[jira] [Updated] (HIVE-13887) LazySimpleSerDe should parse "NULL" dates faster

[Gopal V (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-13887?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gopal V updated HIVE-13887:
---
Attachment: HIVE-13887.1.patch

> LazySimpleSerDe should parse "NULL" dates faster
> 
>
> Key: HIVE-13887
> URL: https://issues.apache.org/jira/browse/HIVE-13887
> Project: Hive
>  Issue Type: Bug
>  Components: Serializers/Deserializers, Vectorization
>Affects Versions: 2.1.0
>Reporter: Gopal V
>Assignee: Gopal V
>  Labels: Performance
> Attachments: HIVE-13887.1.patch
>
>
> Date string which contain "NULL" or "(null)" are being parsed through a very 
> slow codepath involving exception handling as a normal codepath.
> These are currently ~4x slower than parsing an actual date field.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-13887) LazySimpleSerDe should parse "NULL" dates faster

[Gopal V (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-13887?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gopal V updated HIVE-13887:
---
Labels: Performance  (was: )

> LazySimpleSerDe should parse "NULL" dates faster
> 
>
> Key: HIVE-13887
> URL: https://issues.apache.org/jira/browse/HIVE-13887
> Project: Hive
>  Issue Type: Bug
>  Components: Serializers/Deserializers, Vectorization
>Affects Versions: 2.1.0
>Reporter: Gopal V
>Assignee: Gopal V
>  Labels: Performance
> Attachments: HIVE-13887.1.patch
>
>
> Date string which contain "NULL" or "(null)" are being parsed through a very 
> slow codepath involving exception handling as a normal codepath.
> These are currently ~4x slower than parsing an actual date field.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-13887) LazySimpleSerDe should parse "NULL" dates faster

[Gopal V (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-13887?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gopal V updated HIVE-13887:
---
Status: Patch Available  (was: Open)

> LazySimpleSerDe should parse "NULL" dates faster
> 
>
> Key: HIVE-13887
> URL: https://issues.apache.org/jira/browse/HIVE-13887
> Project: Hive
>  Issue Type: Bug
>  Components: Serializers/Deserializers, Vectorization
>Affects Versions: 2.1.0
>Reporter: Gopal V
>Assignee: Gopal V
> Attachments: HIVE-13887.1.patch
>
>
> Date string which contain "NULL" or "(null)" are being parsed through a very 
> slow codepath involving exception handling as a normal codepath.
> These are currently ~4x slower than parsing an actual date field.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-13808) Use constant expressions to backtrack when we create ReduceSink

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-13808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15305749#comment-15305749
 ] 

Hive QA commented on HIVE-13808:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12806611/HIVE-13808.01.patch

{color:red}ERROR:{color} -1 due to no test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 160 failed/errored test(s), 9949 tests 
executed
*Failed tests:*
{noformat}
TestHWISessionManager - did not produce a TEST-*.xml file
TestJdbcWithMiniHA - did not produce a TEST-*.xml file
TestJdbcWithMiniMr - did not produce a TEST-*.xml file
TestMiniTezCliDriver-auto_join30.q-vector_decimal_10_0.q-acid_globallimit.q-and-12-more
 - did not produce a TEST-*.xml file
TestMiniTezCliDriver-auto_sortmerge_join_7.q-orc_merge9.q-tez_union_dynamic_partition.q-and-12-more
 - did not produce a TEST-*.xml file
TestMiniTezCliDriver-delete_where_non_partitioned.q-auto_sortmerge_join_16.q-skewjoin.q-and-12-more
 - did not produce a TEST-*.xml file
TestMiniTezCliDriver-groupby2.q-tez_dynpart_hashjoin_1.q-custom_input_output_format.q-and-12-more
 - did not produce a TEST-*.xml file
TestMiniTezCliDriver-insert_values_non_partitioned.q-schema_evol_orc_nonvec_mapwork_part.q-union5.q-and-12-more
 - did not produce a TEST-*.xml file
TestMiniTezCliDriver-load_dyn_part2.q-selectDistinctStar.q-vector_decimal_5.q-and-12-more
 - did not produce a TEST-*.xml file
TestMiniTezCliDriver-order_null.q-vector_acid3.q-orc_merge10.q-and-12-more - 
did not produce a TEST-*.xml file
TestMiniTezCliDriver-tez_union_group_by.q-vector_auto_smb_mapjoin_14.q-union_fast_stats.q-and-12-more
 - did not produce a TEST-*.xml file
TestMiniTezCliDriver-update_orig_table.q-explainuser_4.q-update_after_multiple_inserts.q-and-12-more
 - did not produce a TEST-*.xml file
TestMiniTezCliDriver-vector_coalesce.q-cbo_windowing.q-tez_join.q-and-12-more - 
did not produce a TEST-*.xml file
TestMiniTezCliDriver-vector_distinct_2.q-tez_joins_explain.q-cte_mat_1.q-and-12-more
 - did not produce a TEST-*.xml file
TestMiniTezCliDriver-vector_interval_2.q-constprog_dpp.q-dynamic_partition_pruning.q-and-12-more
 - did not produce a TEST-*.xml file
TestMinimrCliDriver-index_bitmap3.q-bucketizedhiveinputformat.q-orc_mr_pathalias.q-and-1-more
 - did not produce a TEST-*.xml file
TestMinimrCliDriver-insert_dir_distcp.q-skewjoin_onesideskew.q-bucket_many.q - 
did not produce a TEST-*.xml file
TestOperationLoggingAPIWithTez - did not produce a TEST-*.xml file
TestSQLStdHiveAccessControllerCLI - did not produce a TEST-*.xml file
TestSparkCliDriver-script_pipe.q-stats12.q-auto_join24.q-and-12-more - did not 
produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cp_sel
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ivyDownload
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_12
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_13
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_reducesink_dedup
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_smb_mapjoin_18
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_list_bucket
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_constprog_partitioner
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join12
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join13
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join18_multi_distinct
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join19
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join20
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join22
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join23
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join27
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join5
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join6
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_smb_mapjoin_14
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_sortmerge_join_10
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_sortmerge_join_7
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_sortmerge_join_9
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_bucket4
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_bucket_map_join_spark1
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_bucketmapjoin13
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_bucketmapjoin9
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_bucketmapjoin_negative3
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_bucketsortoptimize_insert_2

[jira] [Updated] (HIVE-13876) Vectorization: Port HIVE-11544 to LazySimpleDeserializeRead

[Gopal V (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-13876?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gopal V updated HIVE-13876:
---
Status: Patch Available  (was: Open)

> Vectorization: Port HIVE-11544 to LazySimpleDeserializeRead
> ---
>
> Key: HIVE-13876
> URL: https://issues.apache.org/jira/browse/HIVE-13876
> Project: Hive
>  Issue Type: Bug
>  Components: File Formats, Vectorization
>Affects Versions: 2.1.0
>Reporter: Gopal V
>Assignee: Gopal V
>Priority: Critical
> Attachments: HIVE-13876.1.patch
>
>
> High CPU usage due to exception handling code.
> {code}
>  TezTaskRunner [RUNNABLE] [DAEMON]
> java.lang.Throwable.fillInStackTrace(int) Throwable.java (native)
> java.lang.Throwable.fillInStackTrace() Throwable.java:783
> java.lang.Throwable.(String) Throwable.java:265
> java.lang.Exception.(String) Exception.java:66
> java.lang.RuntimeException.(String) RuntimeException.java:62
> java.lang.IllegalArgumentException.(String) 
> IllegalArgumentException.java:52
> java.lang.NumberFormatException.(String) NumberFormatException.java:55
> sun.misc.FloatingDecimal.readJavaFormatString(String) 
> FloatingDecimal.java:1842
> sun.misc.FloatingDecimal.parseFloat(String) FloatingDecimal.java:122
> java.lang.Float.parseFloat(String) Float.java:451
> org.apache.hadoop.hive.serde2.lazy.fast.LazySimpleDeserializeRead.readCheckNull()
>  LazySimpleDeserializeRead.java:309
> org.apache.hadoop.hive.ql.exec.vector.VectorDeserializeRow.deserializeRowColumn(VectorizedRowBatch,
>  int, int) VectorDeserializeRow.java:346
> org.apache.hadoop.hive.ql.exec.vector.VectorDeserializeRow.deserialize(VectorizedRowBatch,
>  int) VectorDeserializeRow.java:659
> org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.process(Writable) 
> VectorMapOperator.java:814
> org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.processRow(Object) 
> MapRecordSource.java:86
> org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.pushRecord() 
> MapRecordSource.java:70
> org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.run() 
> MapRecordProcessor.java:361
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(Map,
>  Map) TezProcessor.java:172
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(Map, Map) 
> TezProcessor.java:160
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run() 
> LogicalIOProcessorRuntimeTask.java:370
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run() 
> TaskRunner2Callable.java:73
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run() 
> TaskRunner2Callable.java:61
> java.security.AccessController.doPrivileged(PrivilegedExceptionAction, 
> AccessControlContext) AccessController.java (native)
> javax.security.auth.Subject.doAs(Subject, PrivilegedExceptionAction) 
> Subject.java:422
> org.apache.hadoop.security.UserGroupInformation.doAs(PrivilegedExceptionAction)
>  UserGroupInformation.java:1657
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal() 
> TaskRunner2Callable.java:61
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal() 
> TaskRunner2Callable.java:37
> org.apache.tez.common.CallableWithNdc.call() CallableWithNdc.java:36
> java.util.concurrent.FutureTask.run() FutureTask.java:266
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor$Worker) 
> ThreadPoolExecutor.java:1142
> java.util.concurrent.ThreadPoolExecutor$Worker.run() 
> ThreadPoolExecutor.java:617
> java.lang.Thread.run() Thread.java:745
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-13877) Hplsql UDF doesn't work in Hive Cli

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-13877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15305722#comment-15305722
 ] 

Hive QA commented on HIVE-13877:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12806617/HIVE-13877.2.patch

{color:red}ERROR:{color} -1 due to no test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 138 failed/errored test(s), 9747 tests 
executed
*Failed tests:*
{noformat}
TestAccumuloPredicateHandler - did not produce a TEST-*.xml file
TestHWISessionManager - did not produce a TEST-*.xml file
TestJdbcWithMiniHA - did not produce a TEST-*.xml file
TestJdbcWithMiniMr - did not produce a TEST-*.xml file
TestLongComparison - did not produce a TEST-*.xml file
TestMiniTezCliDriver-auto_join30.q-vector_decimal_10_0.q-acid_globallimit.q-and-12-more
 - did not produce a TEST-*.xml file
TestMiniTezCliDriver-cte_4.q-vector_non_string_partition.q-empty_join.q-and-12-more
 - did not produce a TEST-*.xml file
TestMiniTezCliDriver-delete_where_non_partitioned.q-auto_sortmerge_join_16.q-skewjoin.q-and-12-more
 - did not produce a TEST-*.xml file
TestMiniTezCliDriver-groupby2.q-tez_dynpart_hashjoin_1.q-custom_input_output_format.q-and-12-more
 - did not produce a TEST-*.xml file
TestMiniTezCliDriver-load_dyn_part2.q-selectDistinctStar.q-vector_decimal_5.q-and-12-more
 - did not produce a TEST-*.xml file
TestMiniTezCliDriver-order_null.q-vector_acid3.q-orc_merge10.q-and-12-more - 
did not produce a TEST-*.xml file
TestMiniTezCliDriver-schema_evol_text_nonvec_mapwork_part_all_primitive.q-bucket3.q-tez_fsstat.q-and-12-more
 - did not produce a TEST-*.xml file
TestMiniTezCliDriver-script_pipe.q-vectorized_parquet.q-vector_decimal_aggregate.q-and-12-more
 - did not produce a TEST-*.xml file
TestMiniTezCliDriver-tez_union_group_by.q-vector_auto_smb_mapjoin_14.q-union_fast_stats.q-and-12-more
 - did not produce a TEST-*.xml file
TestMiniTezCliDriver-union2.q-bucket4.q-union8.q-and-12-more - did not produce 
a TEST-*.xml file
TestMiniTezCliDriver-update_orig_table.q-explainuser_4.q-update_after_multiple_inserts.q-and-12-more
 - did not produce a TEST-*.xml file
TestMiniTezCliDriver-vector_coalesce.q-cbo_windowing.q-tez_join.q-and-12-more - 
did not produce a TEST-*.xml file
TestMiniTezCliDriver-vector_distinct_2.q-tez_joins_explain.q-cte_mat_1.q-and-12-more
 - did not produce a TEST-*.xml file
TestMiniTezCliDriver-vector_interval_2.q-constprog_dpp.q-dynamic_partition_pruning.q-and-12-more
 - did not produce a TEST-*.xml file
TestMiniTezCliDriver-vector_varchar_4.q-smb_cache.q-transform_ppr2.q-and-7-more 
- did not produce a TEST-*.xml file
TestMiniTezCliDriver-vectorization_16.q-vector_decimal_round.q-orc_merge6.q-and-12-more
 - did not produce a TEST-*.xml file
TestMinimrCliDriver-groupby2.q-infer_bucket_sort_map_operators.q-load_hdfs_file_with_space_in_the_name.q-and-1-more
 - did not produce a TEST-*.xml file
TestMinimrCliDriver-infer_bucket_sort_num_buckets.q-infer_bucket_sort_reducers_power_two.q-parallel_orderby.q-and-1-more
 - did not produce a TEST-*.xml file
TestMinimrCliDriver-join1.q-infer_bucket_sort_bucketed_table.q-root_dir_external_table.q-and-1-more
 - did not produce a TEST-*.xml file
TestNegativeMinimrCliDriver-udf_local_resource.q-mapreduce_stack_trace_turnoff_hadoop20.q-mapreduce_stack_trace.q-and-6-more
 - did not produce a TEST-*.xml file
TestOperationLoggingAPIWithTez - did not produce a TEST-*.xml file
TestSparkCliDriver-skewjoin_noskew.q-sample2.q-skewjoinopt10.q-and-12-more - 
did not produce a TEST-*.xml file
TestVectorGroupByOperator - did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ivyDownload
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_12
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_list_bucket
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_constprog_partitioner
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_complex_all
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_alter_merge_stats_orc
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join10
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join21
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join28
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join29
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join31
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join_stats2
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_sortmerge_join_1
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_sortmerge_join_9
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_avro_compression_enabled_native

[jira] [Updated] (HIVE-13798) Fix the unit test failure org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ivyDownload

[Aihua Xu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-13798?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aihua Xu updated HIVE-13798:

Status: In Progress  (was: Patch Available)

> Fix the unit test failure 
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ivyDownload
> 
>
> Key: HIVE-13798
> URL: https://issues.apache.org/jira/browse/HIVE-13798
> Project: Hive
>  Issue Type: Sub-task
>Reporter: Aihua Xu
>Assignee: Aihua Xu
> Attachments: HIVE-13798.2.patch, HIVE-13798.3.patch, 
> HIVE-13798.4.patch, HIVE-13798.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-13798) Fix the unit test failure org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ivyDownload

[Aihua Xu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-13798?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aihua Xu updated HIVE-13798:

Attachment: HIVE-13798.4.patch

> Fix the unit test failure 
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ivyDownload
> 
>
> Key: HIVE-13798
> URL: https://issues.apache.org/jira/browse/HIVE-13798
> Project: Hive
>  Issue Type: Sub-task
>Reporter: Aihua Xu
>Assignee: Aihua Xu
> Attachments: HIVE-13798.2.patch, HIVE-13798.3.patch, 
> HIVE-13798.4.patch, HIVE-13798.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-13798) Fix the unit test failure org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ivyDownload

[Aihua Xu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-13798?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aihua Xu updated HIVE-13798:

Status: Patch Available  (was: In Progress)

> Fix the unit test failure 
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ivyDownload
> 
>
> Key: HIVE-13798
> URL: https://issues.apache.org/jira/browse/HIVE-13798
> Project: Hive
>  Issue Type: Sub-task
>Reporter: Aihua Xu
>Assignee: Aihua Xu
> Attachments: HIVE-13798.2.patch, HIVE-13798.3.patch, 
> HIVE-13798.4.patch, HIVE-13798.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-13798) Fix the unit test failure org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ivyDownload

[Aihua Xu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-13798?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aihua Xu updated HIVE-13798:

Attachment: (was: HIVE-13798.4.patch)

> Fix the unit test failure 
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ivyDownload
> 
>
> Key: HIVE-13798
> URL: https://issues.apache.org/jira/browse/HIVE-13798
> Project: Hive
>  Issue Type: Sub-task
>Reporter: Aihua Xu
>Assignee: Aihua Xu
> Attachments: HIVE-13798.2.patch, HIVE-13798.3.patch, 
> HIVE-13798.4.patch, HIVE-13798.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-13798) Fix the unit test failure org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ivyDownload

[Aihua Xu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-13798?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aihua Xu updated HIVE-13798:

Attachment: (was: HIVE-13798.4.patch)

> Fix the unit test failure 
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ivyDownload
> 
>
> Key: HIVE-13798
> URL: https://issues.apache.org/jira/browse/HIVE-13798
> Project: Hive
>  Issue Type: Sub-task
>Reporter: Aihua Xu
>Assignee: Aihua Xu
> Attachments: HIVE-13798.2.patch, HIVE-13798.3.patch, 
> HIVE-13798.4.patch, HIVE-13798.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-13798) Fix the unit test failure org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ivyDownload

[Aihua Xu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-13798?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aihua Xu updated HIVE-13798:

Attachment: HIVE-13798.4.patch

> Fix the unit test failure 
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ivyDownload
> 
>
> Key: HIVE-13798
> URL: https://issues.apache.org/jira/browse/HIVE-13798
> Project: Hive
>  Issue Type: Sub-task
>Reporter: Aihua Xu
>Assignee: Aihua Xu
> Attachments: HIVE-13798.2.patch, HIVE-13798.3.patch, 
> HIVE-13798.4.patch, HIVE-13798.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-13792) Show create table should not show stats info in the table properties

[Aihua Xu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-13792?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aihua Xu updated HIVE-13792:

   Resolution: Fixed
Fix Version/s: 2.2.0
   Status: Resolved  (was: Patch Available)

Pushed to master. Thanks Chaoyu for reviewing.

> Show create table should not show stats info in the table properties
> 
>
> Key: HIVE-13792
> URL: https://issues.apache.org/jira/browse/HIVE-13792
> Project: Hive
>  Issue Type: Sub-task
>  Components: Query Planning
>Affects Versions: 2.1.0
>Reporter: Aihua Xu
>Assignee: Aihua Xu
> Fix For: 2.2.0
>
> Attachments: HIVE-13792.1.patch, HIVE-13792.2.patch, 
> HIVE-13792.3.patch
>
>
> From the test 
> org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_queries 
> failure, we are printing table stats in show create table parameters. This 
> info should be skipped since it would be incorrect when you just copy them to 
> create a table. 
> {noformat}
> PREHOOK: query: SHOW CREATE TABLE hbase_table_1_like
> PREHOOK: type: SHOW_CREATETABLE
> PREHOOK: Input: default@hbase_table_1_like
> POSTHOOK: query: SHOW CREATE TABLE hbase_table_1_like
> POSTHOOK: type: SHOW_CREATETABLE
> POSTHOOK: Input: default@hbase_table_1_like
> CREATE EXTERNAL TABLE `hbase_table_1_like`(
>   `key` int COMMENT 'It is a column key',
>   `value` string COMMENT 'It is the column string value')
> ROW FORMAT SERDE
>   'org.apache.hadoop.hive.hbase.HBaseSerDe'
> STORED BY
>   'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
> WITH SERDEPROPERTIES (
>   'hbase.columns.mapping'='cf:string',
>   'serialization.format'='1')
> TBLPROPERTIES (
>   'COLUMN_STATS_ACCURATE'='{\"BASIC_STATS\":\"true\"}',
>   'hbase.table.name'='hbase_table_0',
>   'numFiles'='0',
>   'numRows'='0',
>   'rawDataSize'='0',
>   'totalSize'='0',
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-13453) Support ORDER BY and windowing clause in partitioning clause with distinct function

[Aihua Xu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-13453?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aihua Xu updated HIVE-13453:

   Resolution: Fixed
Fix Version/s: 2.2.0
   Status: Resolved  (was: Patch Available)

Committed to master. Thanks Yongzhi for reviewing.

> Support ORDER BY and windowing clause in partitioning clause with distinct 
> function
> ---
>
> Key: HIVE-13453
> URL: https://issues.apache.org/jira/browse/HIVE-13453
> Project: Hive
>  Issue Type: Sub-task
>  Components: PTF-Windowing
>Reporter: Aihua Xu
>Assignee: Aihua Xu
> Fix For: 2.2.0
>
> Attachments: HIVE-13453.1.patch, HIVE-13453.2.patch, 
> HIVE-13453.3.patch, HIVE-13453.4.patch
>
>
> Current distinct function on partitioning doesn't support order by and 
> windowing clause due to performance reason. Explore an efficient way to 
> support that.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-13886) HiveSplitGenerator does not read dfs.blocksize correctly

[shanyu zhao (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-13886?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

shanyu zhao updated HIVE-13886:
---
Status: Patch Available  (was: Open)

> HiveSplitGenerator does not read dfs.blocksize correctly
> 
>
> Key: HIVE-13886
> URL: https://issues.apache.org/jira/browse/HIVE-13886
> Project: Hive
>  Issue Type: Bug
>  Components: Query Planning
>Affects Versions: 1.2.1
>Reporter: shanyu zhao
> Attachments: HIVE-13886.patch
>
>
> In src/java/org/apache/hadoop/hive/ql/exec/tez/HiveSplitGenerator.java, it 
> uses conf.getLong() to read "dfs.blocksize". This results in Exception if one 
> configures it to something like "256m". It should use conf.getLongBytes() to 
> read it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-13886) HiveSplitGenerator does not read dfs.blocksize correctly

[shanyu zhao (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-13886?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

shanyu zhao updated HIVE-13886:
---
Attachment: HIVE-13886.patch

patch attached.

> HiveSplitGenerator does not read dfs.blocksize correctly
> 
>
> Key: HIVE-13886
> URL: https://issues.apache.org/jira/browse/HIVE-13886
> Project: Hive
>  Issue Type: Bug
>  Components: Query Planning
>Affects Versions: 1.2.1
>Reporter: shanyu zhao
> Attachments: HIVE-13886.patch
>
>
> In src/java/org/apache/hadoop/hive/ql/exec/tez/HiveSplitGenerator.java, it 
> uses conf.getLong() to read "dfs.blocksize". This results in Exception if one 
> configures it to something like "256m". It should use conf.getLongBytes() to 
> read it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-13882) When hive.server2.async.exec.async.compile is turned on, from JDBC we will get "The query did not generate a result set"

[Aihua Xu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-13882?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aihua Xu updated HIVE-13882:

Attachment: HIVE-13882.2.patch

Patch-2: address comments.

> When hive.server2.async.exec.async.compile is turned on, from JDBC we will 
> get "The query did not generate a result set" 
> -
>
> Key: HIVE-13882
> URL: https://issues.apache.org/jira/browse/HIVE-13882
> Project: Hive
>  Issue Type: Bug
>  Components: JDBC
>Affects Versions: 2.2.0
>Reporter: Aihua Xu
>Assignee: Aihua Xu
> Attachments: HIVE-13882.1.patch, HIVE-13882.2.patch
>
>
>  The following would fail with  "The query did not generate a result set"
> stmt.execute("SET hive.driver.parallel.compilation=true");
> stmt.execute("SET hive.server2.async.exec.async.compile=true");
> ResultSet res =  stmt.executeQuery("SELECT * FROM " + tableName);
> res.next();



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-13837) current_timestamp() output format is different in some cases

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-13837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15305645#comment-15305645
 ] 

Hive QA commented on HIVE-13837:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12806574/HIVE-13837.02.patch

{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 99 failed/errored test(s), 9129 tests 
executed
*Failed tests:*
{noformat}
TestHWISessionManager - did not produce a TEST-*.xml file
TestJdbcWithMiniHA - did not produce a TEST-*.xml file
TestJdbcWithMiniMr - did not produce a TEST-*.xml file
TestMiniTezCliDriver-auto_join30.q-vector_decimal_10_0.q-acid_globallimit.q-and-12-more
 - did not produce a TEST-*.xml file
TestMiniTezCliDriver-auto_sortmerge_join_7.q-orc_merge9.q-tez_union_dynamic_partition.q-and-12-more
 - did not produce a TEST-*.xml file
TestMiniTezCliDriver-enforce_order.q-vector_partition_diff_num_cols.q-unionDistinct_1.q-and-12-more
 - did not produce a TEST-*.xml file
TestMiniTezCliDriver-groupby2.q-tez_dynpart_hashjoin_1.q-custom_input_output_format.q-and-12-more
 - did not produce a TEST-*.xml file
TestMiniTezCliDriver-insert_values_non_partitioned.q-schema_evol_orc_nonvec_mapwork_part.q-union5.q-and-12-more
 - did not produce a TEST-*.xml file
TestMiniTezCliDriver-join1.q-mapjoin_decimal.q-vectorized_distinct_gby.q-and-12-more
 - did not produce a TEST-*.xml file
TestMiniTezCliDriver-load_dyn_part2.q-selectDistinctStar.q-vector_decimal_5.q-and-12-more
 - did not produce a TEST-*.xml file
TestMiniTezCliDriver-mapjoin_mapjoin.q-cte_5.q-insert_into1.q-and-12-more - did 
not produce a TEST-*.xml file
TestMiniTezCliDriver-order_null.q-vector_acid3.q-orc_merge10.q-and-12-more - 
did not produce a TEST-*.xml file
TestMiniTezCliDriver-schema_evol_text_nonvec_mapwork_part_all_primitive.q-bucket3.q-tez_fsstat.q-and-12-more
 - did not produce a TEST-*.xml file
TestMiniTezCliDriver-schema_evol_text_nonvec_mapwork_table.q-vector_decimal_trailing.q-subquery_in.q-and-12-more
 - did not produce a TEST-*.xml file
TestMiniTezCliDriver-script_pipe.q-vectorized_parquet.q-vector_decimal_aggregate.q-and-12-more
 - did not produce a TEST-*.xml file
TestMiniTezCliDriver-union2.q-bucket4.q-union8.q-and-12-more - did not produce 
a TEST-*.xml file
TestMiniTezCliDriver-update_orig_table.q-explainuser_4.q-update_after_multiple_inserts.q-and-12-more
 - did not produce a TEST-*.xml file
TestMiniTezCliDriver-vector_coalesce.q-cbo_windowing.q-tez_join.q-and-12-more - 
did not produce a TEST-*.xml file
TestMiniTezCliDriver-vector_distinct_2.q-tez_joins_explain.q-cte_mat_1.q-and-12-more
 - did not produce a TEST-*.xml file
TestMiniTezCliDriver-vector_interval_2.q-constprog_dpp.q-dynamic_partition_pruning.q-and-12-more
 - did not produce a TEST-*.xml file
TestMiniTezCliDriver-vector_varchar_4.q-smb_cache.q-transform_ppr2.q-and-7-more 
- did not produce a TEST-*.xml file
TestMiniTezCliDriver-vectorization_13.q-auto_sortmerge_join_13.q-tez_bmj_schema_evolution.q-and-12-more
 - did not produce a TEST-*.xml file
TestMinimrCliDriver-external_table_with_space_in_location_path.q-schemeAuthority2.q-auto_sortmerge_join_16.q-and-1-more
 - did not produce a TEST-*.xml file
TestMinimrCliDriver-index_bitmap3.q-bucketizedhiveinputformat.q-orc_mr_pathalias.q-and-1-more
 - did not produce a TEST-*.xml file
TestNegativeCliDriver-udf_invalid.q-nopart_insert.q-insert_into_with_schema.q-and-734-more
 - did not produce a TEST-*.xml file
TestOperationLoggingAPIWithTez - did not produce a TEST-*.xml file
TestSparkCliDriver-timestamp_lazy.q-union29.q-join23.q-and-12-more - did not 
produce a TEST-*.xml file
TestSparkCliDriver-union_remove_23.q-transform_ppr2.q-join20.q-and-12-more - 
did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ivyDownload
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_constprog_partitioner
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_annotate_stats_join
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join17
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join29
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_sortmerge_join_16
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_sortmerge_join_4
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_bucket2
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_bucketsortoptimize_insert_4
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_cbo_semijoin
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_cbo_stats
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_cbo_subq_in
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_cbo_subq_not_in

[jira] [Updated] (HIVE-13511) Run clidriver tests from within the qtest dir for the precommit tests

[Siddharth Seth (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-13511?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Siddharth Seth updated HIVE-13511:
--
Attachment: sparkTestMemAllocateErrorLogs_run432.tar.gz
miniTezKilledLogs_run432.tar.gz

Log files from build432 indicating errors.

> Run clidriver tests from within the qtest dir for the precommit tests
> -
>
> Key: HIVE-13511
> URL: https://issues.apache.org/jira/browse/HIVE-13511
> Project: Hive
>  Issue Type: Sub-task
>Reporter: Siddharth Seth
>Assignee: Siddharth Seth
> Fix For: 2.1.0
>
> Attachments: HIVE-13511.01.patch, HIVE-13511.02.patch, 
> HIVE-13511.03.addendum.patch, HIVE-13511.03.patch, example_maven-test.txt, 
> example_testExecution.txt, failedScriptPostPatch.txt, 
> miniTezKilledLogs_run432.tar.gz, sparkTestMemAllocateErrorLogs_run432.tar.gz
>
>
> The tests are currently run from the itests directory - which means there's 
> additional overhead of having to at least check whether files have changed. 
> Will attach a sample output - this adds up to 40+ seconds per batch. Getting 
> rid of this should be a reasonable saving overall.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-13443) LLAP: signing for the second state of submit (the event)

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-13443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15305564#comment-15305564
 ] 

Hive QA commented on HIVE-13443:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12806545/HIVE-13443.01.patch

{color:red}ERROR:{color} -1 due to build exiting with an error

Test results: 
http://ec2-54-177-240-2.us-west-1.compute.amazonaws.com/job/PreCommit-HIVE-MASTER-Build/436/testReport
Console output: 
http://ec2-54-177-240-2.us-west-1.compute.amazonaws.com/job/PreCommit-HIVE-MASTER-Build/436/console
Test logs: 
http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-436/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hive-ptest/working/scratch/source-prep.sh' failed with exit 
status 1 and output '+ [[ -n /usr/java/jdk1.8.0_25 ]]
+ export JAVA_HOME=/usr/java/jdk1.8.0_25
+ JAVA_HOME=/usr/java/jdk1.8.0_25
+ export 
PATH=/usr/java/jdk1.8.0_25/bin/:/usr/lib64/qt-3.3/bin:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.7.0_45-cloudera/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin
+ 
PATH=/usr/java/jdk1.8.0_25/bin/:/usr/lib64/qt-3.3/bin:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.7.0_45-cloudera/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'M2_OPTS=-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost 
-Dhttp.proxyPort=3128'
+ M2_OPTS='-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost 
-Dhttp.proxyPort=3128'
+ cd /data/hive-ptest/working/
+ tee /data/hive-ptest/logs/PreCommit-HIVE-MASTER-Build-436/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at e1e68b2 HIVE-13511. Run clidriver tests from within the qtest 
dir for the precommit tests. (Siddharth Seth, reviewed by Ashutosh Chauhan)
+ git clean -f -d
Removing ql/src/test/queries/clientpositive/vectorized_mapjoin2.q
Removing ql/src/test/results/clientpositive/tez/vectorized_mapjoin2.q.out
Removing ql/src/test/results/clientpositive/vectorized_mapjoin2.q.out
+ git checkout master
Already on 'master'
+ git reset --hard origin/master
HEAD is now at e1e68b2 HIVE-13511. Run clidriver tests from within the qtest 
dir for the precommit tests. (Siddharth Seth, reviewed by Ashutosh Chauhan)
+ git merge --ff-only origin/master
Already up-to-date.
+ git gc
+ patchCommandPath=/data/hive-ptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hive-ptest/working/scratch/build.patch
+ [[ -f /data/hive-ptest/working/scratch/build.patch ]]
+ chmod +x /data/hive-ptest/working/scratch/smart-apply-patch.sh
+ /data/hive-ptest/working/scratch/smart-apply-patch.sh 
/data/hive-ptest/working/scratch/build.patch
The patch does not appear to apply with p0, p1, or p2
+ exit 1
'
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12806545 - PreCommit-HIVE-MASTER-Build

> LLAP: signing for the second state of submit (the event)
> 
>
> Key: HIVE-13443
> URL: https://issues.apache.org/jira/browse/HIVE-13443
> Project: Hive
>  Issue Type: Sub-task
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-13443.01.patch, HIVE-13443.WIP.nogen.patch, 
> HIVE-13443.patch, HIVE-13443.wo.13444.13675.nogen.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-13818) Fast Vector MapJoin Long hashtable has to handle all integral types

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-13818?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15305562#comment-15305562
 ] 

Hive QA commented on HIVE-13818:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12806542/HIVE-13818.1.patch

{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 124 failed/errored test(s), 9096 tests 
executed
*Failed tests:*
{noformat}
TestDbTxnManager - did not produce a TEST-*.xml file
TestHWISessionManager - did not produce a TEST-*.xml file
TestJdbcWithMiniHA - did not produce a TEST-*.xml file
TestJdbcWithMiniMr - did not produce a TEST-*.xml file
TestMiniTezCliDriver-auto_join1.q-schema_evol_text_vec_mapwork_part_all_complex.q-vector_complex_join.q-and-12-more
 - did not produce a TEST-*.xml file
TestMiniTezCliDriver-auto_sortmerge_join_7.q-orc_merge9.q-tez_union_dynamic_partition.q-and-12-more
 - did not produce a TEST-*.xml file
TestMiniTezCliDriver-cte_4.q-vector_non_string_partition.q-empty_join.q-and-12-more
 - did not produce a TEST-*.xml file
TestMiniTezCliDriver-insert_values_non_partitioned.q-schema_evol_orc_nonvec_mapwork_part.q-union5.q-and-12-more
 - did not produce a TEST-*.xml file
TestMiniTezCliDriver-join1.q-mapjoin_decimal.q-vectorized_distinct_gby.q-and-12-more
 - did not produce a TEST-*.xml file
TestMiniTezCliDriver-load_dyn_part2.q-selectDistinctStar.q-vector_decimal_5.q-and-12-more
 - did not produce a TEST-*.xml file
TestMiniTezCliDriver-mapjoin_mapjoin.q-cte_5.q-insert_into1.q-and-12-more - did 
not produce a TEST-*.xml file
TestMiniTezCliDriver-schema_evol_text_nonvec_mapwork_part_all_primitive.q-bucket3.q-tez_fsstat.q-and-12-more
 - did not produce a TEST-*.xml file
TestMiniTezCliDriver-schema_evol_text_nonvec_mapwork_table.q-vector_decimal_trailing.q-subquery_in.q-and-12-more
 - did not produce a TEST-*.xml file
TestMiniTezCliDriver-update_orig_table.q-explainuser_4.q-update_after_multiple_inserts.q-and-12-more
 - did not produce a TEST-*.xml file
TestMiniTezCliDriver-vector_coalesce.q-cbo_windowing.q-tez_join.q-and-12-more - 
did not produce a TEST-*.xml file
TestMiniTezCliDriver-vector_distinct_2.q-tez_joins_explain.q-cte_mat_1.q-and-12-more
 - did not produce a TEST-*.xml file
TestMiniTezCliDriver-vectorization_13.q-auto_sortmerge_join_13.q-tez_bmj_schema_evolution.q-and-12-more
 - did not produce a TEST-*.xml file
TestMiniTezCliDriver-vectorization_16.q-vector_decimal_round.q-orc_merge6.q-and-12-more
 - did not produce a TEST-*.xml file
TestMinimrCliDriver-insert_dir_distcp.q-skewjoin_onesideskew.q-bucket_many.q - 
did not produce a TEST-*.xml file
TestMinimrCliDriver-orc_merge_diff_fs.q-bucket4.q-file_with_header_footer.q-and-1-more
 - did not produce a TEST-*.xml file
TestNegativeCliDriver-udf_invalid.q-nopart_insert.q-insert_into_with_schema.q-and-734-more
 - did not produce a TEST-*.xml file
TestOperationLoggingAPIWithTez - did not produce a TEST-*.xml file
TestSchedulerQueue - did not produce a TEST-*.xml file
TestSparkCliDriver-groupby6_map.q-join13.q-join_reorder3.q-and-12-more - did 
not produce a TEST-*.xml file
TestSparkCliDriver-groupby_complex_types.q-groupby_map_ppr_multi_distinct.q-vectorization_16.q-and-12-more
 - did not produce a TEST-*.xml file
TestSparkCliDriver-join_cond_pushdown_3.q-groupby7.q-auto_join17.q-and-12-more 
- did not produce a TEST-*.xml file
TestSparkCliDriver-load_dyn_part5.q-load_dyn_part2.q-skewjoinopt16.q-and-12-more
 - did not produce a TEST-*.xml file
TestSparkCliDriver-mapreduce2.q-groupby7_noskew.q-vectorization_5.q-and-12-more 
- did not produce a TEST-*.xml file
TestSparkCliDriver-ptf_general_queries.q-groupby4.q-tez_joins_explain.q-and-12-more
 - did not produce a TEST-*.xml file
TestSparkCliDriver-smb_mapjoin_4.q-groupby8_map.q-groupby4_map.q-and-12-more - 
did not produce a TEST-*.xml file
TestVectorMapJoinFastLongHashMap - did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ivyDownload
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_12
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_list_bucket
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_constprog_partitioner
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.org.apache.hadoop.hive.cli.TestMiniTezCliDriver
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_create_merge_compressed
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_cte_1
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_delete_where_partitioned
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_explainuser_2
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_join0
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_stats

[jira] [Comment Edited] (HIVE-13511) Run clidriver tests from within the qtest dir for the precommit tests

[Siddharth Seth (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-13511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1530#comment-1530
 ] 

Siddharth Seth edited comment on HIVE-13511 at 5/28/16 7:30 PM:


Thanks [~spena]. I think the problem introduced by this patch is now resolved.

However, the new runs are seeing a lot more failures. There's some interesting 
stuff in the logs
1. It looks like test runs are getting killed randomly (I believe this problem 
existed earlier as well ?)
2. Some tests fail  with not enough memory available for the VM to continue
{code}
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 105381888 bytes for committing 
reserved memory.
# An error report file with more information is saved as:
# 
/home/hiveptest/54.177.86.160-hiveptest-1/apache-github-source-source/itests/qtest-spark/hs_err_pid9151.log
{code}

I wonder if the random kills are related as well. i.e. at some point the box 
gets a little overloaded and the OS decides to kill processes ? syslog output 
on the systems would be interesting to look at.

>From looking at various process sizes
maven seems to be 2GB heap per test (potentially 2 processes per testt?)
TestMiniTezCliDriver - runs AMs and Containers with a heap of 819M
Spark tests run AMs and executors with a heap of 1G each

I can look at reducing the size occupied by Tez AMs and containers. Given the 
small amount of data processed in tests - this change seems reasonable. Can the 
same be done for Spark ?

Alternately, should we try reducing the number of drones per box to 2 instead 
of 3, and see how that affects the system ?



was (Author: sseth):
Thanks [~spena]. I think the problem introduced by this patch is now resolved.

However, the new runs are seeing a lot more failures. There's some interesting 
stuff in the logs
1. It looks like test runs are getting killed randomly (I believe this problem 
existed earlier as well ?)
2. Some tests fail to even start with not enough memory available for the VM to 
continue
{code}
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 105381888 bytes for committing 
reserved memory.
# An error report file with more information is saved as:
# 
/home/hiveptest/54.177.86.160-hiveptest-1/apache-github-source-source/itests/qtest-spark/hs_err_pid9151.log
{code}

I wonder if the random kills are related as well. i.e. at some point the box 
gets a little overloaded and the OS decides to kill processes ? syslog output 
on the systems would be interesting to look at.

>From looking at various process sizes
maven seems to be 2GB heap per test (potentially 2 processes per testt?)
TestMiniTezCliDriver - runs AMs and Containers with a heap of 819M
Spark tests run AMs and executors with a heap of 1G each

I can look at reducing the size occupied by Tez AMs and containers. Given the 
small amount of data processed in tests - this change seems reasonable. Can the 
same be done for Spark ?

Alternately, should we try reducing the number of drones per box to 2 instead 
of 3, and see how that affects the system ?


> Run clidriver tests from within the qtest dir for the precommit tests
> -
>
> Key: HIVE-13511
> URL: https://issues.apache.org/jira/browse/HIVE-13511
> Project: Hive
>  Issue Type: Sub-task
>Reporter: Siddharth Seth
>Assignee: Siddharth Seth
> Fix For: 2.1.0
>
> Attachments: HIVE-13511.01.patch, HIVE-13511.02.patch, 
> HIVE-13511.03.addendum.patch, HIVE-13511.03.patch, example_maven-test.txt, 
> example_testExecution.txt, failedScriptPostPatch.txt
>
>
> The tests are currently run from the itests directory - which means there's 
> additional overhead of having to at least check whether files have changed. 
> Will attach a sample output - this adds up to 40+ seconds per batch. Getting 
> rid of this should be a reasonable saving overall.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-13511) Run clidriver tests from within the qtest dir for the precommit tests

[Siddharth Seth (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-13511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1530#comment-1530
 ] 

Siddharth Seth commented on HIVE-13511:
---

Thanks [~spena]. I think the problem introduced by this patch is now resolved.

However, the new runs are seeing a lot more failures. There's some interesting 
stuff in the logs
1. It looks like test runs are getting killed randomly (I believe this problem 
existed earlier as well ?)
2. Some tests fail to even start with not enough memory available for the VM to 
continue
{code}
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 105381888 bytes for committing 
reserved memory.
# An error report file with more information is saved as:
# 
/home/hiveptest/54.177.86.160-hiveptest-1/apache-github-source-source/itests/qtest-spark/hs_err_pid9151.log
{code}

I wonder if the random kills are related as well. i.e. at some point the box 
gets a little overloaded and the OS decides to kill processes ? syslog output 
on the systems would be interesting to look at.

>From looking at various process sizes
maven seems to be 2GB heap per test (potentially 2 processes per testt?)
TestMiniTezCliDriver - runs AMs and Containers with a heap of 819M
Spark tests run AMs and executors with a heap of 1G each

I can look at reducing the size occupied by Tez AMs and containers. Given the 
small amount of data processed in tests - this change seems reasonable. Can the 
same be done for Spark ?

Alternately, should we try reducing the number of drones per box to 2 instead 
of 3, and see how that affects the system ?


> Run clidriver tests from within the qtest dir for the precommit tests
> -
>
> Key: HIVE-13511
> URL: https://issues.apache.org/jira/browse/HIVE-13511
> Project: Hive
>  Issue Type: Sub-task
>Reporter: Siddharth Seth
>Assignee: Siddharth Seth
> Fix For: 2.1.0
>
> Attachments: HIVE-13511.01.patch, HIVE-13511.02.patch, 
> HIVE-13511.03.addendum.patch, HIVE-13511.03.patch, example_maven-test.txt, 
> example_testExecution.txt, failedScriptPostPatch.txt
>
>
> The tests are currently run from the itests directory - which means there's 
> additional overhead of having to at least check whether files have changed. 
> Will attach a sample output - this adds up to 40+ seconds per batch. Getting 
> rid of this should be a reasonable saving overall.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-13713) We miss vectorization in a case of count(*) when aggregation mode is COMPLETE

[Matt McCline (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-13713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt McCline updated HIVE-13713:

Attachment: (was: HIVE-13713.02.patch)

> We miss vectorization in a case of count(*) when aggregation mode is COMPLETE
> -
>
> Key: HIVE-13713
> URL: https://issues.apache.org/jira/browse/HIVE-13713
> Project: Hive
>  Issue Type: Bug
>  Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Attachments: HIVE-13713.01.patch, HIVE-13713.02.patch
>
>
> E.g. vectorization_limit.q doesn't vectorize Reducer 2 for the query:
> {code}
> "select ctinyint, count(distinct(cdouble)) from alltypesorc group by ctinyint 
> order by ctinyint limit 20"
> {code}
> It was producing a stack trace with this error... when trying to vectorize 
> the COMPLETE mode GROUP BY operator.
> {code}
> Vector aggregate not implemented: "count" for type: "NONE 
> (reduce-merge-partial = true)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-13713) We miss vectorization in a case of count(*) when aggregation mode is COMPLETE

[Matt McCline (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-13713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt McCline updated HIVE-13713:

Attachment: HIVE-13713.02.patch

> We miss vectorization in a case of count(*) when aggregation mode is COMPLETE
> -
>
> Key: HIVE-13713
> URL: https://issues.apache.org/jira/browse/HIVE-13713
> Project: Hive
>  Issue Type: Bug
>  Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Attachments: HIVE-13713.01.patch, HIVE-13713.02.patch
>
>
> E.g. vectorization_limit.q doesn't vectorize Reducer 2 for the query:
> {code}
> "select ctinyint, count(distinct(cdouble)) from alltypesorc group by ctinyint 
> order by ctinyint limit 20"
> {code}
> It was producing a stack trace with this error... when trying to vectorize 
> the COMPLETE mode GROUP BY operator.
> {code}
> Vector aggregate not implemented: "count" for type: "NONE 
> (reduce-merge-partial = true)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-13870) Decimal vector is not resized correctly

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-13870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15305519#comment-15305519
 ] 

Hive QA commented on HIVE-13870:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12806543/HIVE-13870.patch

{color:red}ERROR:{color} -1 due to no test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 130 failed/errored test(s), 9069 tests 
executed
*Failed tests:*
{noformat}
TestAccumuloConnectionParameters - did not produce a TEST-*.xml file
TestHWISessionManager - did not produce a TEST-*.xml file
TestJdbcWithMiniHA - did not produce a TEST-*.xml file
TestJdbcWithMiniMr - did not produce a TEST-*.xml file
TestMiniTezCliDriver-auto_join1.q-schema_evol_text_vec_mapwork_part_all_complex.q-vector_complex_join.q-and-12-more
 - did not produce a TEST-*.xml file
TestMiniTezCliDriver-auto_sortmerge_join_7.q-orc_merge9.q-tez_union_dynamic_partition.q-and-12-more
 - did not produce a TEST-*.xml file
TestMiniTezCliDriver-cte_4.q-vector_non_string_partition.q-empty_join.q-and-12-more
 - did not produce a TEST-*.xml file
TestMiniTezCliDriver-delete_where_non_partitioned.q-auto_sortmerge_join_16.q-skewjoin.q-and-12-more
 - did not produce a TEST-*.xml file
TestMiniTezCliDriver-dynpart_sort_optimization2.q-tez_dynpart_hashjoin_3.q-orc_vectorization_ppd.q-and-12-more
 - did not produce a TEST-*.xml file
TestMiniTezCliDriver-enforce_order.q-vector_partition_diff_num_cols.q-unionDistinct_1.q-and-12-more
 - did not produce a TEST-*.xml file
TestMiniTezCliDriver-groupby2.q-tez_dynpart_hashjoin_1.q-custom_input_output_format.q-and-12-more
 - did not produce a TEST-*.xml file
TestMiniTezCliDriver-insert_values_non_partitioned.q-schema_evol_orc_nonvec_mapwork_part.q-union5.q-and-12-more
 - did not produce a TEST-*.xml file
TestMiniTezCliDriver-join1.q-mapjoin_decimal.q-vectorized_distinct_gby.q-and-12-more
 - did not produce a TEST-*.xml file
TestMiniTezCliDriver-mapjoin_mapjoin.q-cte_5.q-insert_into1.q-and-12-more - did 
not produce a TEST-*.xml file
TestMiniTezCliDriver-order_null.q-vector_acid3.q-orc_merge10.q-and-12-more - 
did not produce a TEST-*.xml file
TestMiniTezCliDriver-schema_evol_text_nonvec_mapwork_part_all_primitive.q-bucket3.q-tez_fsstat.q-and-12-more
 - did not produce a TEST-*.xml file
TestMiniTezCliDriver-schema_evol_text_nonvec_mapwork_table.q-vector_decimal_trailing.q-subquery_in.q-and-12-more
 - did not produce a TEST-*.xml file
TestMiniTezCliDriver-update_orig_table.q-explainuser_4.q-update_after_multiple_inserts.q-and-12-more
 - did not produce a TEST-*.xml file
TestMiniTezCliDriver-vector_coalesce.q-cbo_windowing.q-tez_join.q-and-12-more - 
did not produce a TEST-*.xml file
TestMiniTezCliDriver-vector_distinct_2.q-tez_joins_explain.q-cte_mat_1.q-and-12-more
 - did not produce a TEST-*.xml file
TestMiniTezCliDriver-vector_interval_2.q-constprog_dpp.q-dynamic_partition_pruning.q-and-12-more
 - did not produce a TEST-*.xml file
TestMiniTezCliDriver-vector_varchar_4.q-smb_cache.q-transform_ppr2.q-and-7-more 
- did not produce a TEST-*.xml file
TestMiniTezCliDriver-vectorization_13.q-auto_sortmerge_join_13.q-tez_bmj_schema_evolution.q-and-12-more
 - did not produce a TEST-*.xml file
TestMiniTezCliDriver-vectorization_16.q-vector_decimal_round.q-orc_merge6.q-and-12-more
 - did not produce a TEST-*.xml file
TestMinimrCliDriver-external_table_with_space_in_location_path.q-schemeAuthority2.q-auto_sortmerge_join_16.q-and-1-more
 - did not produce a TEST-*.xml file
TestMinimrCliDriver-join1.q-infer_bucket_sort_bucketed_table.q-root_dir_external_table.q-and-1-more
 - did not produce a TEST-*.xml file
TestNegativeCliDriver-udf_invalid.q-nopart_insert.q-insert_into_with_schema.q-and-734-more
 - did not produce a TEST-*.xml file
TestNegativeMinimrCliDriver-udf_local_resource.q-mapreduce_stack_trace_turnoff_hadoop20.q-mapreduce_stack_trace.q-and-6-more
 - did not produce a TEST-*.xml file
TestOperationLoggingAPIWithTez - did not produce a TEST-*.xml file
TestSparkCliDriver-avro_joins.q-join36.q-join1.q-and-12-more - did not produce 
a TEST-*.xml file
TestSparkCliDriver-bucketmapjoin10.q-join_rc.q-skewjoinopt13.q-and-12-more - 
did not produce a TEST-*.xml file
TestSparkCliDriver-ppd_transform.q-union_remove_7.q-date_udf.q-and-12-more - 
did not produce a TEST-*.xml file
TestSparkCliDriver-skewjoin_union_remove_2.q-timestamp_null.q-union32.q-and-12-more
 - did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ivyDownload
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_13
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_constprog_partitioner
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_alter_merge_stats_orc

[jira] [Commented] (HIVE-13248) Change date_add/date_sub/to_date functions to return Date type rather than String

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-13248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15305450#comment-15305450
 ] 

Hive QA commented on HIVE-13248:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12806234/HIVE-13248.3.patch

{color:green}SUCCESS:{color} +1 due to 6 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 136 failed/errored test(s), 9023 tests 
executed
*Failed tests:*
{noformat}
TestColumnColumnFilterVectorExpressionEvaluation - did not produce a TEST-*.xml 
file
TestHWISessionManager - did not produce a TEST-*.xml file
TestJdbcWithMiniHA - did not produce a TEST-*.xml file
TestJdbcWithMiniMr - did not produce a TEST-*.xml file
TestMiniTezCliDriver-auto_join1.q-schema_evol_text_vec_mapwork_part_all_complex.q-vector_complex_join.q-and-12-more
 - did not produce a TEST-*.xml file
TestMiniTezCliDriver-auto_sortmerge_join_7.q-orc_merge9.q-tez_union_dynamic_partition.q-and-12-more
 - did not produce a TEST-*.xml file
TestMiniTezCliDriver-delete_where_non_partitioned.q-auto_sortmerge_join_16.q-skewjoin.q-and-12-more
 - did not produce a TEST-*.xml file
TestMiniTezCliDriver-enforce_order.q-vector_partition_diff_num_cols.q-unionDistinct_1.q-and-12-more
 - did not produce a TEST-*.xml file
TestMiniTezCliDriver-join1.q-mapjoin_decimal.q-vectorized_distinct_gby.q-and-12-more
 - did not produce a TEST-*.xml file
TestMiniTezCliDriver-load_dyn_part2.q-selectDistinctStar.q-vector_decimal_5.q-and-12-more
 - did not produce a TEST-*.xml file
TestMiniTezCliDriver-mapjoin_mapjoin.q-cte_5.q-insert_into1.q-and-12-more - did 
not produce a TEST-*.xml file
TestMiniTezCliDriver-order_null.q-vector_acid3.q-orc_merge10.q-and-12-more - 
did not produce a TEST-*.xml file
TestMiniTezCliDriver-schema_evol_text_nonvec_mapwork_part_all_primitive.q-bucket3.q-tez_fsstat.q-and-12-more
 - did not produce a TEST-*.xml file
TestMiniTezCliDriver-schema_evol_text_nonvec_mapwork_table.q-vector_decimal_trailing.q-subquery_in.q-and-12-more
 - did not produce a TEST-*.xml file
TestMiniTezCliDriver-tez_union_group_by.q-vector_auto_smb_mapjoin_14.q-union_fast_stats.q-and-12-more
 - did not produce a TEST-*.xml file
TestMiniTezCliDriver-update_orig_table.q-explainuser_4.q-update_after_multiple_inserts.q-and-12-more
 - did not produce a TEST-*.xml file
TestMiniTezCliDriver-vector_coalesce.q-cbo_windowing.q-tez_join.q-and-12-more - 
did not produce a TEST-*.xml file
TestMiniTezCliDriver-vector_distinct_2.q-tez_joins_explain.q-cte_mat_1.q-and-12-more
 - did not produce a TEST-*.xml file
TestMiniTezCliDriver-vector_grouping_sets.q-update_all_partitioned.q-schema_evol_stats.q-and-12-more
 - did not produce a TEST-*.xml file
TestMiniTezCliDriver-vector_interval_2.q-constprog_dpp.q-dynamic_partition_pruning.q-and-12-more
 - did not produce a TEST-*.xml file
TestMiniTezCliDriver-vector_varchar_4.q-smb_cache.q-transform_ppr2.q-and-7-more 
- did not produce a TEST-*.xml file
TestMiniTezCliDriver-vectorization_13.q-auto_sortmerge_join_13.q-tez_bmj_schema_evolution.q-and-12-more
 - did not produce a TEST-*.xml file
TestNegativeCliDriver-udf_invalid.q-nopart_insert.q-insert_into_with_schema.q-and-734-more
 - did not produce a TEST-*.xml file
TestNegativeMinimrCliDriver-udf_local_resource.q-mapreduce_stack_trace_turnoff_hadoop20.q-mapreduce_stack_trace.q-and-6-more
 - did not produce a TEST-*.xml file
TestOperationLoggingAPIWithTez - did not produce a TEST-*.xml file
TestSparkCliDriver-auto_join_reordering_values.q-ptf_seqfile.q-auto_join18.q-and-12-more
 - did not produce a TEST-*.xml file
TestSparkCliDriver-join9.q-join_casesensitive.q-filter_join_breaktask.q-and-12-more
 - did not produce a TEST-*.xml file
TestVectorSelectOperator - did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ivyDownload
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_text_vec_mapwork_part_all_primitive
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_text_vecrow_mapwork_part_all_primitive
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_list_bucket
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_constprog_partitioner
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_text_vec_mapwork_part_all_primitive
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_add_part_multiple
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join17
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join19
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join22
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join23
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join29
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join_nulls

[jira] [Commented] (HIVE-13567) Auto-gather column stats - phase 2

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-13567?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15305396#comment-15305396
 ] 

Hive QA commented on HIVE-13567:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12806232/HIVE-13567.01.patch

{color:red}ERROR:{color} -1 due to build exiting with an error

Test results: 
http://ec2-54-177-240-2.us-west-1.compute.amazonaws.com/job/PreCommit-HIVE-MASTER-Build/418/testReport
Console output: 
http://ec2-54-177-240-2.us-west-1.compute.amazonaws.com/job/PreCommit-HIVE-MASTER-Build/418/console
Test logs: 
http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-418/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: InterruptedException: null
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12806232 - PreCommit-HIVE-MASTER-Build

> Auto-gather column stats - phase 2
> --
>
> Key: HIVE-13567
> URL: https://issues.apache.org/jira/browse/HIVE-13567
> Project: Hive
>  Issue Type: Sub-task
>Reporter: Pengcheng Xiong
>Assignee: Pengcheng Xiong
> Attachments: HIVE-13567.01.patch
>
>
> in phase 2, we are going to set auto-gather column on as default. This needs 
> to update golden files.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-13511) Run clidriver tests from within the qtest dir for the precommit tests

[JIRA]

[ 
https://issues.apache.org/jira/browse/HIVE-13511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15305348#comment-15305348
 ] 

Sergio Peña commented on HIVE-13511:


Thanks [~sseth] for letting me know.
I applied the patch (revert the old one, and re-apply both patches).
I'll restart the server.

> Run clidriver tests from within the qtest dir for the precommit tests
> -
>
> Key: HIVE-13511
> URL: https://issues.apache.org/jira/browse/HIVE-13511
> Project: Hive
>  Issue Type: Sub-task
>Reporter: Siddharth Seth
>Assignee: Siddharth Seth
> Fix For: 2.1.0
>
> Attachments: HIVE-13511.01.patch, HIVE-13511.02.patch, 
> HIVE-13511.03.addendum.patch, HIVE-13511.03.patch, example_maven-test.txt, 
> example_testExecution.txt, failedScriptPostPatch.txt
>
>
> The tests are currently run from the itests directory - which means there's 
> additional overhead of having to at least check whether files have changed. 
> Will attach a sample output - this adds up to 40+ seconds per batch. Getting 
> rid of this should be a reasonable saving overall.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-13838) Set basic stats as inaccurate for all ACID tables

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-13838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15305347#comment-15305347
 ] 

Hive QA commented on HIVE-13838:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12806226/HIVE-13838.01.patch

{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 250 failed/errored test(s), 6207 tests 
executed
*Failed tests:*
{noformat}
TestCliDriver-alter_file_format.q-bucket_map_join_tez1.q-fetch_aggregation.q-and-12-more
 - did not produce a TEST-*.xml file
TestCliDriver-alter_rename_table.q-udf_aes_decrypt.q-udf_add.q-and-12-more - 
did not produce a TEST-*.xml file
TestCliDriver-alter_table_not_sorted.q-cbo_udf_max.q-exim_11_managed_external.q-and-12-more
 - did not produce a TEST-*.xml file
TestCliDriver-ambiguitycheck.q-udf_bitwise_and.q-insert_overwrite_directory.q-and-12-more
 - did not produce a TEST-*.xml file
TestCliDriver-authorization_1_sql_std.q-disallow_incompatible_type_change_off.q-schema_evol_text_nonvec_mapwork_table.q-and-12-more
 - did not produce a TEST-*.xml file
TestCliDriver-authorization_cli_nonsql.q-drop_index.q-vector_date_1.q-and-12-more
 - did not produce a TEST-*.xml file
TestCliDriver-authorization_cli_stdconfigauth.q-str_to_map.q-touch.q-and-12-more
 - did not produce a TEST-*.xml file
TestCliDriver-authorization_create_table_owner_privs.q-cte_7.q-groupby_sort_7.q-and-12-more
 - did not produce a TEST-*.xml file
TestCliDriver-authorization_create_temp_table.q-update_all_partitioned.q-lock4.q-and-12-more
 - did not produce a TEST-*.xml file
TestCliDriver-authorization_parts.q-authorization_view_1.q-parquet_map_of_maps.q-and-12-more
 - did not produce a TEST-*.xml file
TestCliDriver-authorization_role_grant2.q-union_remove_7.q-alter_view_as_select.q-and-12-more
 - did not produce a TEST-*.xml file
TestCliDriver-auto_join18.q-dynamic_partition_pruning.q-date_comparison.q-and-12-more
 - did not produce a TEST-*.xml file
TestCliDriver-auto_join30.q-udf_bitmap_empty.q-input17.q-and-12-more - did not 
produce a TEST-*.xml file
TestCliDriver-auto_join_reordering_values.q-subquery_in.q-partition_timestamp2.q-and-12-more
 - did not produce a TEST-*.xml file
TestCliDriver-auto_sortmerge_join_7.q-exim_04_evolved_parts.q-query_with_semi.q-and-12-more
 - did not produce a TEST-*.xml file
TestCliDriver-autogen_colalias.q-skewjoinopt3.q-rcfile_merge1.q-and-12-more - 
did not produce a TEST-*.xml file
TestCliDriver-avro_add_column.q-bucketmapjoin_negative.q-bucket_map_join_spark2.q-and-12-more
 - did not produce a TEST-*.xml file
TestCliDriver-ba_table1.q-udf_notequal.q-input_part1.q-and-12-more - did not 
produce a TEST-*.xml file
TestCliDriver-bucketcontext_4.q-orc_ends_with_nulls.q-correlationoptimizer9.q-and-12-more
 - did not produce a TEST-*.xml file
TestCliDriver-bucketsortoptimize_insert_7.q-list_bucket_query_multiskew_1.q-skewjoin_noskew.q-and-12-more
 - did not produce a TEST-*.xml file
TestCliDriver-cast_to_int.q-literal_double.q-skewjoinopt20.q-and-12-more - did 
not produce a TEST-*.xml file
TestCliDriver-cbo_rp_gby_empty.q-load_nonpart_authsuccess.q-truncate_table.q-and-12-more
 - did not produce a TEST-*.xml file
TestCliDriver-cbo_rp_join1.q-union_top_level.q-join18.q-and-12-more - did not 
produce a TEST-*.xml file
TestCliDriver-cbo_rp_limit.q-load_exist_part_authsuccess.q-schema_evol_text_nonvec_mapwork_part_all_primitive.q-and-12-more
 - did not produce a TEST-*.xml file
TestCliDriver-cbo_rp_lineage2.q-authorization_update.q-udf_pmod.q-and-12-more - 
did not produce a TEST-*.xml file
TestCliDriver-cbo_rp_semijoin.q-custom_input_output_format.q-udf_reflect2.q-and-12-more
 - did not produce a TEST-*.xml file
TestCliDriver-cbo_rp_udf_udaf.q-date_udf.q-limit0.q-and-12-more - did not 
produce a TEST-*.xml file
TestCliDriver-cbo_rp_views.q-orc_split_elimination.q-authorization_view_disable_cbo_4.q-and-12-more
 - did not produce a TEST-*.xml file
TestCliDriver-columnStatsUpdateForStatsOptimizer_2.q-cp_mj_rc.q-show_columns.q-and-12-more
 - did not produce a TEST-*.xml file
TestCliDriver-columnstats_partlvl_dp.q-input31.q-udaf_percentile.q-and-12-more 
- did not produce a TEST-*.xml file
TestCliDriver-compute_stats_string.q-noalias_subq1.q-vector_auto_smb_mapjoin_14.q-and-12-more
 - did not produce a TEST-*.xml file
TestCliDriver-create_func1.q-bucketmapjoin3.q-enforce_order.q-and-12-more - did 
not produce a TEST-*.xml file
TestCliDriver-create_genericudf.q-dynamic_partition_insert.q-skewjoinopt16.q-and-12-more
 - did not produce a TEST-*.xml file
TestCliDriver-create_or_replace_view.q-dbtxnmgr_query5.q-compute_stats_double.q-and-12-more
 - did not produce a TEST-*.xml file
TestCliDriver-cte_mat_1.q-groupby_sort_6.q-update_after_multiple_inserts_special_characters.q-and-12-more
 - did not produce a TEST-*.xml file

[jira] [Comment Edited] (HIVE-13859) mask() UDF not retaining day and month field values

[Madhan Neethiraj (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-13859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15305279#comment-15305279
 ] 

Madhan Neethiraj edited comment on HIVE-13859 at 5/28/16 10:47 AM:
---

Fixed mask() UDF implementation to retain day and month field values when '-1' 
is specified as masked value.

CC: [~ashutoshc], [~hagleitn]


was (Author: madhan.neethiraj):
Fixed mask() UDF implementation to retain day and month field values when '-1' 
is specified as masked value.

> mask() UDF not retaining day and month field values
> ---
>
> Key: HIVE-13859
> URL: https://issues.apache.org/jira/browse/HIVE-13859
> Project: Hive
>  Issue Type: Bug
>  Components: UDF
>Affects Versions: 2.1.0
>Reporter: Madhan Neethiraj
> Fix For: 2.1.0
>
> Attachments: HIVE-13859.patch
>
>
> For date type parameters, mask() UDF replaces year/month/day field values 
> with the values given in arguments to the UDF. Argument value -1 is treated 
> as special, to specify that mask() should retain the value in the parameter. 
> This allows to selectively mask only year/month/day fields.
> Specifying "-1" does not retain the values for day/month fields; however the 
> year value is retained, as shown below.
> {code}
> 0: jdbc:hive2://localhost:1> select id, join_date from employee where id 
> < 4;
> +-+-+--+
> | id  |  join_date  |
> +-+-+--+
> | 1   | 2012-01-01  |
> | 2   | 2014-02-01  |
> | 3   | 2013-03-01  |
> +-+-+--+
> 3 rows selected (0.435 seconds)
> 0: jdbc:hive2://localhost:1> select id, mask(join_date, -1, -1, -1, 
> -1,-1, -1,-1,-1) join_date from employee where id < 4;
> +-+-+--+
> | id  |  join_date  |
> +-+-+--+
> | 1   | 2012-01-01  |
> | 2   | 2014-01-01  |
> | 3   | 2013-01-01  |
> +-+-+--+
> 3 rows selected (0.344 seconds)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-13885) Hive session close is not resetting thread name

[Rajat Khandelwal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-13885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rajat Khandelwal updated HIVE-13885:

Status: Patch Available  (was: Open)

> Hive session close is not resetting thread name
> ---
>
> Key: HIVE-13885
> URL: https://issues.apache.org/jira/browse/HIVE-13885
> Project: Hive
>  Issue Type: Bug
>Reporter: Rajat Khandelwal
>Assignee: Rajat Khandelwal
> Attachments: HIVE-13885.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-13885) Hive session close is not resetting thread name

[Rajat Khandelwal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-13885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rajat Khandelwal updated HIVE-13885:

Attachment: HIVE-13885.patch

[~jcamachorodriguez] This is s pretty small fix. Would like to get this too in 
2.1 release. 

> Hive session close is not resetting thread name
> ---
>
> Key: HIVE-13885
> URL: https://issues.apache.org/jira/browse/HIVE-13885
> Project: Hive
>  Issue Type: Bug
>Reporter: Rajat Khandelwal
>Assignee: Rajat Khandelwal
> Attachments: HIVE-13885.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-13518) Hive on Tez: Shuffle joins do not choose the right 'big' table.

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-13518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15305289#comment-15305289
 ] 

Hive QA commented on HIVE-13518:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12806300/HIVE-13518.3.patch

{color:red}ERROR:{color} -1 due to no test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 252 failed/errored test(s), 6197 tests 
executed
*Failed tests:*
{noformat}
TestCliDriver-alter_file_format.q-bucket_map_join_tez1.q-fetch_aggregation.q-and-12-more
 - did not produce a TEST-*.xml file
TestCliDriver-alter_rename_table.q-udf_aes_decrypt.q-udf_add.q-and-12-more - 
did not produce a TEST-*.xml file
TestCliDriver-alter_table_not_sorted.q-cbo_udf_max.q-exim_11_managed_external.q-and-12-more
 - did not produce a TEST-*.xml file
TestCliDriver-ambiguitycheck.q-udf_bitwise_and.q-insert_overwrite_directory.q-and-12-more
 - did not produce a TEST-*.xml file
TestCliDriver-authorization_1_sql_std.q-disallow_incompatible_type_change_off.q-schema_evol_text_nonvec_mapwork_table.q-and-12-more
 - did not produce a TEST-*.xml file
TestCliDriver-authorization_cli_nonsql.q-orc_createas1.q-skewjoinopt19.q-and-12-more
 - did not produce a TEST-*.xml file
TestCliDriver-authorization_cli_stdconfigauth.q-str_to_map.q-touch.q-and-12-more
 - did not produce a TEST-*.xml file
TestCliDriver-authorization_create_table_owner_privs.q-cte_7.q-groupby_sort_7.q-and-12-more
 - did not produce a TEST-*.xml file
TestCliDriver-authorization_create_temp_table.q-update_all_partitioned.q-lock4.q-and-12-more
 - did not produce a TEST-*.xml file
TestCliDriver-authorization_parts.q-authorization_view_1.q-parquet_map_of_maps.q-and-12-more
 - did not produce a TEST-*.xml file
TestCliDriver-authorization_role_grant2.q-union_remove_7.q-alter_view_as_select.q-and-12-more
 - did not produce a TEST-*.xml file
TestCliDriver-auto_join18.q-dynamic_partition_pruning.q-date_comparison.q-and-12-more
 - did not produce a TEST-*.xml file
TestCliDriver-auto_join30.q-udf_bitmap_empty.q-input17.q-and-12-more - did not 
produce a TEST-*.xml file
TestCliDriver-auto_join_reordering_values.q-subquery_in.q-partition_timestamp2.q-and-12-more
 - did not produce a TEST-*.xml file
TestCliDriver-auto_sortmerge_join_7.q-exim_04_evolved_parts.q-query_with_semi.q-and-12-more
 - did not produce a TEST-*.xml file
TestCliDriver-autogen_colalias.q-skewjoinopt3.q-rcfile_merge1.q-and-12-more - 
did not produce a TEST-*.xml file
TestCliDriver-avro_add_column.q-bucketmapjoin_negative.q-bucket_map_join_spark2.q-and-12-more
 - did not produce a TEST-*.xml file
TestCliDriver-ba_table1.q-udf_notequal.q-input_part1.q-and-12-more - did not 
produce a TEST-*.xml file
TestCliDriver-bucketcontext_4.q-orc_ends_with_nulls.q-correlationoptimizer9.q-and-12-more
 - did not produce a TEST-*.xml file
TestCliDriver-bucketsortoptimize_insert_7.q-list_bucket_query_multiskew_1.q-skewjoin_noskew.q-and-12-more
 - did not produce a TEST-*.xml file
TestCliDriver-cast_to_int.q-literal_double.q-skewjoinopt20.q-and-12-more - did 
not produce a TEST-*.xml file
TestCliDriver-cbo_rp_gby_empty.q-load_nonpart_authsuccess.q-truncate_table.q-and-12-more
 - did not produce a TEST-*.xml file
TestCliDriver-cbo_rp_join1.q-union_top_level.q-join18.q-and-12-more - did not 
produce a TEST-*.xml file
TestCliDriver-cbo_rp_limit.q-load_exist_part_authsuccess.q-schema_evol_text_nonvec_mapwork_part_all_primitive.q-and-12-more
 - did not produce a TEST-*.xml file
TestCliDriver-cbo_rp_lineage2.q-authorization_update.q-udf_pmod.q-and-12-more - 
did not produce a TEST-*.xml file
TestCliDriver-cbo_rp_semijoin.q-custom_input_output_format.q-udf_reflect2.q-and-12-more
 - did not produce a TEST-*.xml file
TestCliDriver-cbo_rp_udf_udaf.q-date_udf.q-limit0.q-and-12-more - did not 
produce a TEST-*.xml file
TestCliDriver-cbo_rp_views.q-orc_split_elimination.q-authorization_view_disable_cbo_4.q-and-12-more
 - did not produce a TEST-*.xml file
TestCliDriver-columnStatsUpdateForStatsOptimizer_2.q-cp_mj_rc.q-show_columns.q-and-12-more
 - did not produce a TEST-*.xml file
TestCliDriver-columnstats_partlvl_dp.q-input31.q-udaf_percentile.q-and-12-more 
- did not produce a TEST-*.xml file
TestCliDriver-compute_stats_string.q-noalias_subq1.q-vector_auto_smb_mapjoin_14.q-and-12-more
 - did not produce a TEST-*.xml file
TestCliDriver-create_func1.q-bucketmapjoin3.q-enforce_order.q-and-12-more - did 
not produce a TEST-*.xml file
TestCliDriver-create_genericudf.q-dynamic_partition_insert.q-skewjoinopt16.q-and-12-more
 - did not produce a TEST-*.xml file
TestCliDriver-create_or_replace_view.q-compute_stats_double.q-reduce_deduplicate_exclude_gby.q-and-12-more
 - did not produce a TEST-*.xml file
TestCliDriver-cte_mat_1.q-groupby_sort_6.q-update_after_multiple_inserts_special_characters.q-and-12-more
 - did not produce a TEST-*.xml file

[jira] [Updated] (HIVE-13859) mask() UDF not retaining day and month field values

[Madhan Neethiraj (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-13859?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Madhan Neethiraj updated HIVE-13859:

Status: Patch Available  (was: Open)

> mask() UDF not retaining day and month field values
> ---
>
> Key: HIVE-13859
> URL: https://issues.apache.org/jira/browse/HIVE-13859
> Project: Hive
>  Issue Type: Bug
>  Components: UDF
>Affects Versions: 2.1.0
>Reporter: Madhan Neethiraj
> Fix For: 2.1.0
>
> Attachments: HIVE-13859.patch
>
>
> For date type parameters, mask() UDF replaces year/month/day field values 
> with the values given in arguments to the UDF. Argument value -1 is treated 
> as special, to specify that mask() should retain the value in the parameter. 
> This allows to selectively mask only year/month/day fields.
> Specifying "-1" does not retain the values for day/month fields; however the 
> year value is retained, as shown below.
> {code}
> 0: jdbc:hive2://localhost:1> select id, join_date from employee where id 
> < 4;
> +-+-+--+
> | id  |  join_date  |
> +-+-+--+
> | 1   | 2012-01-01  |
> | 2   | 2014-02-01  |
> | 3   | 2013-03-01  |
> +-+-+--+
> 3 rows selected (0.435 seconds)
> 0: jdbc:hive2://localhost:1> select id, mask(join_date, -1, -1, -1, 
> -1,-1, -1,-1,-1) join_date from employee where id < 4;
> +-+-+--+
> | id  |  join_date  |
> +-+-+--+
> | 1   | 2012-01-01  |
> | 2   | 2014-01-01  |
> | 3   | 2013-01-01  |
> +-+-+--+
> 3 rows selected (0.344 seconds)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-13859) mask() UDF not retaining day and month field values

[Madhan Neethiraj (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-13859?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Madhan Neethiraj updated HIVE-13859:

Attachment: HIVE-13859.patch

Fixed mask() UDF implementation to retain day and month field values when '-1' 
is specified as masked value.

> mask() UDF not retaining day and month field values
> ---
>
> Key: HIVE-13859
> URL: https://issues.apache.org/jira/browse/HIVE-13859
> Project: Hive
>  Issue Type: Bug
>  Components: UDF
>Affects Versions: 2.1.0
>Reporter: Madhan Neethiraj
> Fix For: 2.1.0
>
> Attachments: HIVE-13859.patch
>
>
> For date type parameters, mask() UDF replaces year/month/day field values 
> with the values given in arguments to the UDF. Argument value -1 is treated 
> as special, to specify that mask() should retain the value in the parameter. 
> This allows to selectively mask only year/month/day fields.
> Specifying "-1" does not retain the values for day/month fields; however the 
> year value is retained, as shown below.
> {code}
> 0: jdbc:hive2://localhost:1> select id, join_date from employee where id 
> < 4;
> +-+-+--+
> | id  |  join_date  |
> +-+-+--+
> | 1   | 2012-01-01  |
> | 2   | 2014-02-01  |
> | 3   | 2013-03-01  |
> +-+-+--+
> 3 rows selected (0.435 seconds)
> 0: jdbc:hive2://localhost:1> select id, mask(join_date, -1, -1, -1, 
> -1,-1, -1,-1,-1) join_date from employee where id < 4;
> +-+-+--+
> | id  |  join_date  |
> +-+-+--+
> | 1   | 2012-01-01  |
> | 2   | 2014-01-01  |
> | 3   | 2013-01-01  |
> +-+-+--+
> 3 rows selected (0.344 seconds)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-13693) Multi-insert query drops Filter before file output when there is a.val <> b.val

[Jesus Camacho Rodriguez (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-13693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jesus Camacho Rodriguez updated HIVE-13693:
---
Target Version/s: 2.1.0

> Multi-insert query drops Filter before file output when there is a.val <> 
> b.val
> ---
>
> Key: HIVE-13693
> URL: https://issues.apache.org/jira/browse/HIVE-13693
> Project: Hive
>  Issue Type: Bug
>  Components: Logical Optimizer
>Affects Versions: 1.3.0, 2.0.0, 2.1.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
> Attachments: HIVE-13693.01.patch, HIVE-13693.01.patch, 
> HIVE-13693.patch
>
>
> To reproduce:
> {noformat}
> CREATE TABLE T_A ( id STRING, val STRING ); 
> CREATE TABLE T_B ( id STRING, val STRING ); 
> CREATE TABLE join_result_1 ( ida STRING, vala STRING, idb STRING, valb STRING 
> ); 
> CREATE TABLE join_result_3 ( ida STRING, vala STRING, idb STRING, valb STRING 
> ); 
> INSERT INTO TABLE T_A 
> VALUES ('Id_1', 'val_101'), ('Id_2', 'val_102'), ('Id_3', 'val_103'); 
> INSERT INTO TABLE T_B 
> VALUES ('Id_1', 'val_103'), ('Id_2', 'val_104'); 
> explain
> FROM T_A a LEFT JOIN T_B b ON a.id = b.id
> INSERT OVERWRITE TABLE join_result_1
> SELECT a.*, b.*
> WHERE b.id = 'Id_1' AND b.val = 'val_103'
> INSERT OVERWRITE TABLE join_result_3
> SELECT a.*, b.*
> WHERE b.val = 'val_104' AND b.id = 'Id_2' AND a.val <> b.val;
> {noformat}
> The (wrong) plan is the following:
> {noformat}
> STAGE DEPENDENCIES:
>   Stage-2 is a root stage
>   Stage-3 depends on stages: Stage-2
>   Stage-0 depends on stages: Stage-3
>   Stage-4 depends on stages: Stage-0
>   Stage-1 depends on stages: Stage-3
>   Stage-5 depends on stages: Stage-1
> STAGE PLANS:
>   Stage: Stage-2
> Tez
>   DagId: haha_20160504140944_174465c9-5d1a-42f9-9665-fae02eeb2767:2
>   Edges:
> Reducer 2 <- Map 1 (SIMPLE_EDGE), Map 3 (SIMPLE_EDGE)
>   DagName: 
>   Vertices:
> Map 1 
> Map Operator Tree:
> TableScan
>   alias: a
>   Statistics: Num rows: 3 Data size: 36 Basic stats: COMPLETE 
> Column stats: NONE
>   Reduce Output Operator
> key expressions: id (type: string)
> sort order: +
> Map-reduce partition columns: id (type: string)
> Statistics: Num rows: 3 Data size: 36 Basic stats: 
> COMPLETE Column stats: NONE
> value expressions: val (type: string)
> Map 3 
> Map Operator Tree:
> TableScan
>   alias: b
>   Statistics: Num rows: 2 Data size: 24 Basic stats: COMPLETE 
> Column stats: NONE
>   Reduce Output Operator
> key expressions: id (type: string)
> sort order: +
> Map-reduce partition columns: id (type: string)
> Statistics: Num rows: 2 Data size: 24 Basic stats: 
> COMPLETE Column stats: NONE
> value expressions: val (type: string)
> Reducer 2 
> Reduce Operator Tree:
>   Merge Join Operator
> condition map:
>  Left Outer Join0 to 1
> keys:
>   0 id (type: string)
>   1 id (type: string)
> outputColumnNames: _col0, _col1, _col6
> Statistics: Num rows: 3 Data size: 39 Basic stats: COMPLETE 
> Column stats: NONE
> Select Operator
>   expressions: _col0 (type: string), _col1 (type: string), 
> 'Id_1' (type: string), 'val_103' (type: string)
>   outputColumnNames: _col0, _col1, _col2, _col3
>   Statistics: Num rows: 3 Data size: 39 Basic stats: COMPLETE 
> Column stats: NONE
>   File Output Operator
> compressed: false
> Statistics: Num rows: 3 Data size: 39 Basic stats: 
> COMPLETE Column stats: NONE
> table:
> input format: org.apache.hadoop.mapred.TextInputFormat
> output format: 
> org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
> serde: 
> org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
> name: bugtest2.join_result_1
> Filter Operator
>   predicate: (_col1 <> _col6) (type: boolean)
>   Statistics: Num rows: 3 Data size: 39 Basic stats: COMPLETE 
> Column stats: NONE
>   Select Operator
> expressions: _col0 (type: string), _col1 (type: string), 
> 'Id_2' (type: string), 'val_104' (type: string)
> outputColumnNames: 

[jira] [Updated] (HIVE-13831) Error pushing predicates to HBase storage handler

[Jesus Camacho Rodriguez (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-13831?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jesus Camacho Rodriguez updated HIVE-13831:
---
Target Version/s: 2.1.0

> Error pushing predicates to HBase storage handler
> -
>
> Key: HIVE-13831
> URL: https://issues.apache.org/jira/browse/HIVE-13831
> Project: Hive
>  Issue Type: Bug
>  Components: HBase Handler
>Affects Versions: 2.1.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
> Attachments: HIVE-13831.01.patch, HIVE-13831.02.patch, 
> HIVE-13831.patch
>
>
> Discovered while working on HIVE-13693.
> There is an error on the predicates that we can push to HBaseStorageHandler. 
> In particular, range predicates of the shape {{(bounded, open)}} and {{(open, 
> bounded)}} over long or int columns get pushed and return wrong results.
> The problem has to do with the storage order for keys in HBase. Keys are 
> sorted lexicographically. Since the byte representation of negative values 
> comes after the positive values, open range predicates need special handling 
> that we do not have right now.
> Thus, for instance, when we push the predicate {{key > 2}}, we return all 
> records with column _key_ greater than 2, plus the records with negative 
> values for the column _key_. This problem does not get exposed if a filter is 
> kept in the Hive operator tree, but we should not assume the latest.
> This fix avoids pushing this kind of predicates to the storage handler, 
> returning them in the _residual_ part of the predicate that cannot be pushed. 
> In the future, special handling might be added to support this kind of 
> predicates.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-13254) GBY cardinality estimation is wrong partition columns is involved

[Jesus Camacho Rodriguez (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-13254?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jesus Camacho Rodriguez updated HIVE-13254:
---
Target Version/s:   (was: 2.1.0)

> GBY cardinality estimation is wrong partition columns is involved
> -
>
> Key: HIVE-13254
> URL: https://issues.apache.org/jira/browse/HIVE-13254
> Project: Hive
>  Issue Type: Bug
>  Components: Hive
>Affects Versions: 1.3.0, 2.0.0, 2.1.0
>Reporter: Prasanth Jayachandran
>Assignee: Jesus Camacho Rodriguez
> Attachments: q3.svg, q3_ef_transpose_aggr.svg
>
>
> When running the following query on TPCDS-1000 scale, setting 
> hive.transpose.aggr.join=true is expected to generate optimal plan but it was 
> not generating. 
> {code:title=Query}
> SELECT `date_dim`.`d_day_name` AS `d_day_name`, 
>`item`.`i_category` AS `i_category` 
> FROM   `store_sales` `store_sales` 
>INNER JOIN `item` `item` 
>ON ( `store_sales`.`ss_item_sk` = `item`.`i_item_sk` ) 
>INNER JOIN `date_dim` `date_dim` 
>ON ( `store_sales`.`ss_sold_date_sk` = `date_dim`.`d_date_sk` 
> ) 
> GROUP  BY `d_day_name`, 
>   `i_category`;
> {code}
> The reason for that is stats annotation rule for GROUP BY is not considering 
> partition column into account. For the above query, the generated plan is 
> attached. As we can see from the plan, GBY is pushed to fact table 
> (store_sales) but that output of GBY shuffled to perform join instead of 
> MapJoin conversion. This is because of wrong estimation of cardinality/data 
> size of GBY on store_sales (Map 1). 
> What's happening internally is, GBY computes estimated cardinality which in 
> this case is NDV(ss_item_sk) * NDV(ss_sold_date_sk) = 338901 * 1823 ~= 617M. 
> This estimate is wrong as ss_sold_date_sk is partition column and estimator 
> assumes its non-partition column. In this case, not every tasks reads data 
> from all partitions. We need to take estimated task parallelism into account. 
> For example: If task parallelism is determined to be 100 the estimate from 
> GBY should be ~6M which should convert this vertex into map join vertex. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-13849) Wrong plan for hive.optimize.sort.dynamic.partition=true

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-13849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15305247#comment-15305247
 ] 

Hive QA commented on HIVE-13849:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12806164/HIVE-13849.patch

{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 251 failed/errored test(s), 6205 tests 
executed
*Failed tests:*
{noformat}
TestCliDriver-alter_rename_table.q-udf_aes_decrypt.q-udf_add.q-and-12-more - 
did not produce a TEST-*.xml file
TestCliDriver-alter_table_not_sorted.q-cbo_udf_max.q-exim_11_managed_external.q-and-12-more
 - did not produce a TEST-*.xml file
TestCliDriver-ambiguitycheck.q-udf_bitwise_and.q-insert_overwrite_directory.q-and-12-more
 - did not produce a TEST-*.xml file
TestCliDriver-authorization_1_sql_std.q-disallow_incompatible_type_change_off.q-schema_evol_text_nonvec_mapwork_table.q-and-12-more
 - did not produce a TEST-*.xml file
TestCliDriver-authorization_cli_nonsql.q-drop_index.q-vector_date_1.q-and-12-more
 - did not produce a TEST-*.xml file
TestCliDriver-authorization_cli_stdconfigauth.q-str_to_map.q-touch.q-and-12-more
 - did not produce a TEST-*.xml file
TestCliDriver-authorization_create_table_owner_privs.q-cte_7.q-groupby_sort_7.q-and-12-more
 - did not produce a TEST-*.xml file
TestCliDriver-authorization_create_temp_table.q-update_all_partitioned.q-lock4.q-and-12-more
 - did not produce a TEST-*.xml file
TestCliDriver-authorization_parts.q-authorization_view_1.q-parquet_map_of_maps.q-and-12-more
 - did not produce a TEST-*.xml file
TestCliDriver-authorization_role_grant2.q-union_remove_7.q-alter_char2.q-and-12-more
 - did not produce a TEST-*.xml file
TestCliDriver-auto_join18.q-dynamic_partition_pruning.q-date_comparison.q-and-12-more
 - did not produce a TEST-*.xml file
TestCliDriver-auto_join30.q-udf_bitmap_empty.q-input17.q-and-12-more - did not 
produce a TEST-*.xml file
TestCliDriver-auto_join_reordering_values.q-subquery_in.q-partition_timestamp2.q-and-12-more
 - did not produce a TEST-*.xml file
TestCliDriver-autogen_colalias.q-skewjoinopt3.q-rcfile_merge1.q-and-12-more - 
did not produce a TEST-*.xml file
TestCliDriver-avro_add_column.q-bucketmapjoin_negative.q-bucket_map_join_spark2.q-and-12-more
 - did not produce a TEST-*.xml file
TestCliDriver-ba_table1.q-udf_notequal.q-input_part1.q-and-12-more - did not 
produce a TEST-*.xml file
TestCliDriver-bucket_map_join_tez1.q-fetch_aggregation.q-drop_database_removes_partition_dirs.q-and-12-more
 - did not produce a TEST-*.xml file
TestCliDriver-bucketcontext_4.q-orc_ends_with_nulls.q-correlationoptimizer9.q-and-12-more
 - did not produce a TEST-*.xml file
TestCliDriver-bucketsortoptimize_insert_7.q-list_bucket_query_multiskew_1.q-skewjoin_noskew.q-and-12-more
 - did not produce a TEST-*.xml file
TestCliDriver-cast_to_int.q-literal_double.q-skewjoinopt20.q-and-12-more - did 
not produce a TEST-*.xml file
TestCliDriver-cbo_rp_gby_empty.q-bucketsortoptimize_insert_4.q-truncate_table.q-and-12-more
 - did not produce a TEST-*.xml file
TestCliDriver-cbo_rp_join1.q-union_top_level.q-join18.q-and-12-more - did not 
produce a TEST-*.xml file
TestCliDriver-cbo_rp_limit.q-load_exist_part_authsuccess.q-schema_evol_text_nonvec_mapwork_part_all_primitive.q-and-12-more
 - did not produce a TEST-*.xml file
TestCliDriver-cbo_rp_lineage2.q-authorization_update.q-udf_pmod.q-and-12-more - 
did not produce a TEST-*.xml file
TestCliDriver-cbo_rp_semijoin.q-custom_input_output_format.q-udf_reflect2.q-and-12-more
 - did not produce a TEST-*.xml file
TestCliDriver-cbo_rp_udf_udaf.q-orc_merge9.q-ba_table3.q-and-12-more - did not 
produce a TEST-*.xml file
TestCliDriver-cbo_rp_views.q-orc_split_elimination.q-authorization_view_disable_cbo_4.q-and-12-more
 - did not produce a TEST-*.xml file
TestCliDriver-columnStatsUpdateForStatsOptimizer_2.q-cp_mj_rc.q-udf_logic_java_boolean.q-and-12-more
 - did not produce a TEST-*.xml file
TestCliDriver-columnstats_partlvl_dp.q-input31.q-udaf_percentile.q-and-12-more 
- did not produce a TEST-*.xml file
TestCliDriver-compute_stats_string.q-show_columns.q-noalias_subq1.q-and-12-more 
- did not produce a TEST-*.xml file
TestCliDriver-constprog_dpp.q-udaf_number_format.q-llap_uncompressed.q-and-12-more
 - did not produce a TEST-*.xml file
TestCliDriver-create_func1.q-bucketmapjoin3.q-enforce_order.q-and-12-more - did 
not produce a TEST-*.xml file
TestCliDriver-create_genericudf.q-dynamic_partition_insert.q-skewjoinopt16.q-and-12-more
 - did not produce a TEST-*.xml file
TestCliDriver-create_or_replace_view.q-dbtxnmgr_query5.q-compute_stats_double.q-and-12-more
 - did not produce a TEST-*.xml file
TestCliDriver-cross_join_merge.q-load_nonpart_authsuccess.q-skewjoinopt13.q-and-12-more
 - did not produce a TEST-*.xml file

[jira] [Updated] (HIVE-5999) Allow other characters for LINES TERMINATED BY

[Nemon Lou (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-5999?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nemon Lou updated HIVE-5999:

Status: Patch Available  (was: Open)

> Allow other characters for LINES TERMINATED BY 
> ---
>
> Key: HIVE-5999
> URL: https://issues.apache.org/jira/browse/HIVE-5999
> Project: Hive
>  Issue Type: Improvement
>  Components: Beeline, Database/Schema, Hive
>Affects Versions: 0.12.0
>Reporter: Mariano Dominguez
>Assignee: Nemon Lou
>Priority: Critical
>  Labels: Delimiter, Hive, Row, SerDe
> Attachments: HIVE-5999.patch
>
>
> LINES TERMINATED BY only supports newline '\n' right now.
> It would be nice to loosen this constraint and allow other characters.
> This limitation seems to be hardcoded here:
> https://github.com/apache/hive/blob/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/BaseSemanticAnalyzer.java#L171
> The DDL Definition on the Hive Language manual shows this as a configurable 
> property whereas it is not. This may lead to mileading assement of being able 
> to choose a choice of field delimiter.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-5999) Allow other characters for LINES TERMINATED BY

[Nemon Lou (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-5999?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nemon Lou updated HIVE-5999:

Attachment: HIVE-5999.patch

Limitations:
1,"lines terminated by" only works with text file.
2,"Multiple table with the same data path but different delimiter" is not 
supported in the same query, due to path binding with tableDesc(See  
Map pathToPartitionInfo).
3,Line delimiter "10" will  be treated as string "10" instead of "\n" .As we 
support line delimiter with string .

> Allow other characters for LINES TERMINATED BY 
> ---
>
> Key: HIVE-5999
> URL: https://issues.apache.org/jira/browse/HIVE-5999
> Project: Hive
>  Issue Type: Improvement
>  Components: Beeline, Database/Schema, Hive
>Affects Versions: 0.12.0
>Reporter: Mariano Dominguez
>Assignee: Nemon Lou
>Priority: Critical
>  Labels: Delimiter, Hive, Row, SerDe
> Attachments: HIVE-5999.patch
>
>
> LINES TERMINATED BY only supports newline '\n' right now.
> It would be nice to loosen this constraint and allow other characters.
> This limitation seems to be hardcoded here:
> https://github.com/apache/hive/blob/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/BaseSemanticAnalyzer.java#L171
> The DDL Definition on the Hive Language manual shows this as a configurable 
> property whereas it is not. This may lead to mileading assement of being able 
> to choose a choice of field delimiter.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-12316) Improved integration test for Hive

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-12316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15305223#comment-15305223
 ] 

Hive QA commented on HIVE-12316:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12776619/HIVE-12316.5.patch

{color:red}ERROR:{color} -1 due to build exiting with an error

Test results: 
http://ec2-54-177-240-2.us-west-1.compute.amazonaws.com/job/PreCommit-HIVE-MASTER-Build/412/testReport
Console output: 
http://ec2-54-177-240-2.us-west-1.compute.amazonaws.com/job/PreCommit-HIVE-MASTER-Build/412/console
Test logs: 
http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-412/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hive-ptest/working/scratch/source-prep.sh' failed with exit 
status 1 and output '+ [[ -n /usr/java/jdk1.8.0_25 ]]
+ export JAVA_HOME=/usr/java/jdk1.8.0_25
+ JAVA_HOME=/usr/java/jdk1.8.0_25
+ export 
PATH=/usr/java/jdk1.8.0_25/bin/:/usr/java/jdk1.7.0_45-cloudera/bin:/usr/local/apache-maven-3.0.5/bin:/usr/lib64/qt-3.3/bin:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.7.0_45-cloudera/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin
+ 
PATH=/usr/java/jdk1.8.0_25/bin/:/usr/java/jdk1.7.0_45-cloudera/bin:/usr/local/apache-maven-3.0.5/bin:/usr/lib64/qt-3.3/bin:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.7.0_45-cloudera/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'M2_OPTS=-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost 
-Dhttp.proxyPort=3128'
+ M2_OPTS='-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost 
-Dhttp.proxyPort=3128'
+ cd /data/hive-ptest/working/
+ tee /data/hive-ptest/logs/PreCommit-HIVE-MASTER-Build-412/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at 0a24c88 HIVE-13084: Vectorization add support for PROJECTION 
Multi-AND/OR (Matt McCline, reviewed by Sergey Shelukhin)
+ git clean -f -d
+ git checkout master
Already on 'master'
+ git reset --hard origin/master
HEAD is now at 0a24c88 HIVE-13084: Vectorization add support for PROJECTION 
Multi-AND/OR (Matt McCline, reviewed by Sergey Shelukhin)
+ git merge --ff-only origin/master
Already up-to-date.
+ git gc
+ patchCommandPath=/data/hive-ptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hive-ptest/working/scratch/build.patch
+ [[ -f /data/hive-ptest/working/scratch/build.patch ]]
+ chmod +x /data/hive-ptest/working/scratch/smart-apply-patch.sh
+ /data/hive-ptest/working/scratch/smart-apply-patch.sh 
/data/hive-ptest/working/scratch/build.patch
The patch does not appear to apply with p0, p1, or p2
+ exit 1
'
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12776619 - PreCommit-HIVE-MASTER-Build

> Improved integration test for Hive
> --
>
> Key: HIVE-12316
> URL: https://issues.apache.org/jira/browse/HIVE-12316
> Project: Hive
>  Issue Type: New Feature
>  Components: Testing Infrastructure
>Affects Versions: 2.0.0
>Reporter: Alan Gates
>Assignee: Alan Gates
> Attachments: HIVE-12316.2.patch, HIVE-12316.5.patch, HIVE-12316.patch
>
>
> In working with Hive testing I have found there are several issues that are 
> causing problems for developers, testers, and users:
> * Because Hive has many tunable knobs (file format, security, etc.) we end up 
> with tests that cover the same functionality with different permutations of 
> these features.
> * The Hive integration tests (ie qfiles) cannot be run on a cluster.  This 
> means we cannot run any of those tests at scale.  The HBase community by 
> contrast uses the same test suite locally and on a cluster, and has found 
> that this helps them greatly in testing.
> * Golden files are a grievous evil.  Test writers are forced to eyeball 
> results the first time they run a test and decide whether they look 
> reasonable, which is error prone and makes testing at scale impossible.  And 
> changes to one part of Hive often end up changing the plan (and the output of 
> explain) thus breaking many tests that are not related.  This is particularly 
> an issue for people working on the optimizer.  
> * The lack of ability to run on a cluster means that when people test Hive 

[jira] [Commented] (HIVE-13847) Avoid file open call in RecordReaderUtils as the stream is already available

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-13847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15305221#comment-15305221
 ] 

Hive QA commented on HIVE-13847:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12806111/HIVE-13847.1.patch

{color:red}ERROR:{color} -1 due to build exiting with an error

Test results: 
http://ec2-54-177-240-2.us-west-1.compute.amazonaws.com/job/PreCommit-HIVE-MASTER-Build/411/testReport
Console output: 
http://ec2-54-177-240-2.us-west-1.compute.amazonaws.com/job/PreCommit-HIVE-MASTER-Build/411/console
Test logs: 
http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-411/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hive-ptest/working/scratch/source-prep.sh' failed with exit 
status 1 and output '+ [[ -n /usr/java/jdk1.8.0_25 ]]
+ export JAVA_HOME=/usr/java/jdk1.8.0_25
+ JAVA_HOME=/usr/java/jdk1.8.0_25
+ export 
PATH=/usr/java/jdk1.8.0_25/bin/:/usr/java/jdk1.7.0_45-cloudera/bin:/usr/local/apache-maven-3.0.5/bin:/usr/lib64/qt-3.3/bin:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.7.0_45-cloudera/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin
+ 
PATH=/usr/java/jdk1.8.0_25/bin/:/usr/java/jdk1.7.0_45-cloudera/bin:/usr/local/apache-maven-3.0.5/bin:/usr/lib64/qt-3.3/bin:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.7.0_45-cloudera/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'M2_OPTS=-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost 
-Dhttp.proxyPort=3128'
+ M2_OPTS='-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost 
-Dhttp.proxyPort=3128'
+ cd /data/hive-ptest/working/
+ tee /data/hive-ptest/logs/PreCommit-HIVE-MASTER-Build-411/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at 0a24c88 HIVE-13084: Vectorization add support for PROJECTION 
Multi-AND/OR (Matt McCline, reviewed by Sergey Shelukhin)
+ git clean -f -d
Removing hbase-handler/src/java/org/apache/hadoop/hive/hbase/phoenix/
Removing 
serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyObjectBaseWrapper.java
Removing serde/src/java/org/apache/hadoop/hive/serde2/lazydio/LazyDioDate.java
Removing 
serde/src/java/org/apache/hadoop/hive/serde2/lazydio/LazyDioTimestamp.java
+ git checkout master
Already on 'master'
+ git reset --hard origin/master
HEAD is now at 0a24c88 HIVE-13084: Vectorization add support for PROJECTION 
Multi-AND/OR (Matt McCline, reviewed by Sergey Shelukhin)
+ git merge --ff-only origin/master
Already up-to-date.
+ git gc
+ patchCommandPath=/data/hive-ptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hive-ptest/working/scratch/build.patch
+ [[ -f /data/hive-ptest/working/scratch/build.patch ]]
+ chmod +x /data/hive-ptest/working/scratch/smart-apply-patch.sh
+ /data/hive-ptest/working/scratch/smart-apply-patch.sh 
/data/hive-ptest/working/scratch/build.patch
The patch does not appear to apply with p0, p1, or p2
+ exit 1
'
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12806111 - PreCommit-HIVE-MASTER-Build

> Avoid file open call in RecordReaderUtils as the stream is already available
> 
>
> Key: HIVE-13847
> URL: https://issues.apache.org/jira/browse/HIVE-13847
> Project: Hive
>  Issue Type: Improvement
>  Components: ORC
>Reporter: Rajesh Balamohan
>Assignee: Rajesh Balamohan
>Priority: Minor
> Attachments: HIVE-13847.1.patch
>
>
> File open call in RecordReaderUtils::readRowIndex can be avoided.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-11233) Include Apache Phoenix support in HBaseStorageHandler

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-11233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15305220#comment-15305220
 ] 

Hive QA commented on HIVE-11233:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12806127/HIVE-11233.4.patch

{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 254 failed/errored test(s), 6210 tests 
executed
*Failed tests:*
{noformat}
TestCliDriver-alter_file_format.q-bucket_map_join_tez1.q-fetch_aggregation.q-and-12-more
 - did not produce a TEST-*.xml file
TestCliDriver-alter_rename_table.q-udf_aes_decrypt.q-udf_add.q-and-12-more - 
did not produce a TEST-*.xml file
TestCliDriver-alter_table_not_sorted.q-cbo_udf_max.q-exim_11_managed_external.q-and-12-more
 - did not produce a TEST-*.xml file
TestCliDriver-ambiguitycheck.q-udf_bitwise_and.q-insert_overwrite_directory.q-and-12-more
 - did not produce a TEST-*.xml file
TestCliDriver-authorization_1_sql_std.q-disallow_incompatible_type_change_off.q-schema_evol_text_nonvec_mapwork_table.q-and-12-more
 - did not produce a TEST-*.xml file
TestCliDriver-authorization_cli_nonsql.q-orc_createas1.q-skewjoinopt19.q-and-12-more
 - did not produce a TEST-*.xml file
TestCliDriver-authorization_cli_stdconfigauth.q-str_to_map.q-touch.q-and-12-more
 - did not produce a TEST-*.xml file
TestCliDriver-authorization_create_table_owner_privs.q-cte_7.q-groupby_sort_7.q-and-12-more
 - did not produce a TEST-*.xml file
TestCliDriver-authorization_create_temp_table.q-update_all_partitioned.q-lock4.q-and-12-more
 - did not produce a TEST-*.xml file
TestCliDriver-authorization_parts.q-authorization_view_1.q-parquet_map_of_maps.q-and-12-more
 - did not produce a TEST-*.xml file
TestCliDriver-authorization_role_grant2.q-union_remove_7.q-alter_view_as_select.q-and-12-more
 - did not produce a TEST-*.xml file
TestCliDriver-auto_join18.q-dynamic_partition_pruning.q-date_comparison.q-and-12-more
 - did not produce a TEST-*.xml file
TestCliDriver-auto_join30.q-udf_bitmap_empty.q-input17.q-and-12-more - did not 
produce a TEST-*.xml file
TestCliDriver-auto_join_reordering_values.q-subquery_in.q-partition_timestamp2.q-and-12-more
 - did not produce a TEST-*.xml file
TestCliDriver-auto_sortmerge_join_7.q-exim_04_evolved_parts.q-query_with_semi.q-and-12-more
 - did not produce a TEST-*.xml file
TestCliDriver-autogen_colalias.q-skewjoinopt3.q-rcfile_merge1.q-and-12-more - 
did not produce a TEST-*.xml file
TestCliDriver-avro_add_column.q-bucketmapjoin_negative.q-bucket_map_join_spark2.q-and-12-more
 - did not produce a TEST-*.xml file
TestCliDriver-ba_table1.q-udf_notequal.q-input_part1.q-and-12-more - did not 
produce a TEST-*.xml file
TestCliDriver-bucketcontext_4.q-orc_ends_with_nulls.q-correlationoptimizer9.q-and-12-more
 - did not produce a TEST-*.xml file
TestCliDriver-bucketsortoptimize_insert_7.q-list_bucket_query_multiskew_1.q-skewjoin_noskew.q-and-12-more
 - did not produce a TEST-*.xml file
TestCliDriver-cast_to_int.q-literal_double.q-skewjoinopt20.q-and-12-more - did 
not produce a TEST-*.xml file
TestCliDriver-cbo_rp_gby_empty.q-load_nonpart_authsuccess.q-truncate_table.q-and-12-more
 - did not produce a TEST-*.xml file
TestCliDriver-cbo_rp_join1.q-union_top_level.q-join18.q-and-12-more - did not 
produce a TEST-*.xml file
TestCliDriver-cbo_rp_limit.q-load_exist_part_authsuccess.q-schema_evol_text_nonvec_mapwork_part_all_primitive.q-and-12-more
 - did not produce a TEST-*.xml file
TestCliDriver-cbo_rp_lineage2.q-authorization_update.q-udf_pmod.q-and-12-more - 
did not produce a TEST-*.xml file
TestCliDriver-cbo_rp_semijoin.q-custom_input_output_format.q-udf_reflect2.q-and-12-more
 - did not produce a TEST-*.xml file
TestCliDriver-cbo_rp_udf_udaf.q-date_udf.q-limit0.q-and-12-more - did not 
produce a TEST-*.xml file
TestCliDriver-cbo_rp_views.q-orc_split_elimination.q-authorization_view_disable_cbo_4.q-and-12-more
 - did not produce a TEST-*.xml file
TestCliDriver-columnStatsUpdateForStatsOptimizer_2.q-cp_mj_rc.q-show_columns.q-and-12-more
 - did not produce a TEST-*.xml file
TestCliDriver-columnstats_partlvl_dp.q-input31.q-udaf_percentile.q-and-12-more 
- did not produce a TEST-*.xml file
TestCliDriver-compute_stats_string.q-noalias_subq1.q-vector_auto_smb_mapjoin_14.q-and-12-more
 - did not produce a TEST-*.xml file
TestCliDriver-create_func1.q-bucketmapjoin3.q-enforce_order.q-and-12-more - did 
not produce a TEST-*.xml file
TestCliDriver-create_genericudf.q-dynamic_partition_insert.q-skewjoinopt16.q-and-12-more
 - did not produce a TEST-*.xml file
TestCliDriver-create_or_replace_view.q-compute_stats_double.q-reduce_deduplicate_exclude_gby.q-and-12-more
 - did not produce a TEST-*.xml file
TestCliDriver-cte_mat_1.q-groupby_sort_6.q-update_after_multiple_inserts_special_characters.q-and-12-more
 - did not produce a TEST-*.xml file

[jira] [Comment Edited] (HIVE-12983) Provide a builtin function to get Hive version

[Lenni Kuff (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-12983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15305199#comment-15305199
 ] 

Lenni Kuff edited comment on HIVE-12983 at 5/28/16 6:10 AM:


Updated based on [~szehon]'s feedback. Now returns a shorted build info 
(version + githash). I do think including the githash is valuable. One of the 
use cases for this is to understand exactly what bits are deployed someplace, 
so it's useful to have the githash. 


was (Author: lskuff):
Updated based on [~szehon]'s feedback. Now returns a shorted build info 
(version + githash). 

> Provide a builtin function to get Hive version
> --
>
> Key: HIVE-12983
> URL: https://issues.apache.org/jira/browse/HIVE-12983
> Project: Hive
>  Issue Type: Improvement
>  Components: UDF
>Affects Versions: 2.0.0
>Reporter: Lenni Kuff
>Assignee: Lenni Kuff
> Attachments: HIVE-12983.1.patch, HIVE-12983.2.patch
>
>
> It would be nice to have a builtin function that would return the Hive 
> version.  This would make it easier for a users and tests to programmatically 
> check the Hive version in a SQL script. It's also useful so a client can 
> check the Hive version on a remote cluster.
> For example:
> {code}
> beeline> SELECT version();
> 2.1.0-SNAPSHOT r208ab352311a6cbbcd1f7fcd40964da2dbc6703d
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-12983) Provide a builtin function to get Hive version

[Lenni Kuff (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-12983?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lenni Kuff updated HIVE-12983:
--
Description: 
It would be nice to have a builtin function that would return the Hive version. 
 This would make it easier for a users and tests to programmatically check the 
Hive version in a SQL script. It's also useful so a client can check the Hive 
version on a remote cluster.

For example:
{code}
beeline> SELECT version();

2.1.0-SNAPSHOT r208ab352311a6cbbcd1f7fcd40964da2dbc6703d
{code}

  was:
It would be nice to have a builtin function that would return the Hive version. 
 This would make it easier for a users and tests to programmatically check the 
Hive version in a SQL script. It's also useful so a client can check the Hive 
version on a remote cluster.

For example:
{code}
beeline> SELECT version();

2.1.0-SNAPSHOT from 208ab352311a6cbbcd1f7fcd40964da2dbc6703d by lskuff source 
checksum 8e971cda755f6b3fb528c233c40eb50a
{code}


> Provide a builtin function to get Hive version
> --
>
> Key: HIVE-12983
> URL: https://issues.apache.org/jira/browse/HIVE-12983
> Project: Hive
>  Issue Type: Improvement
>  Components: UDF
>Affects Versions: 2.0.0
>Reporter: Lenni Kuff
>Assignee: Lenni Kuff
> Attachments: HIVE-12983.1.patch, HIVE-12983.2.patch
>
>
> It would be nice to have a builtin function that would return the Hive 
> version.  This would make it easier for a users and tests to programmatically 
> check the Hive version in a SQL script. It's also useful so a client can 
> check the Hive version on a remote cluster.
> For example:
> {code}
> beeline> SELECT version();
> 2.1.0-SNAPSHOT r208ab352311a6cbbcd1f7fcd40964da2dbc6703d
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-12983) Provide a builtin function to get Hive version

[Lenni Kuff (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-12983?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lenni Kuff updated HIVE-12983:
--
Attachment: HIVE-12983.2.patch

Updated based on [~szehon]'s feedback. Now returns a shorted build info 
(version + githash). 

> Provide a builtin function to get Hive version
> --
>
> Key: HIVE-12983
> URL: https://issues.apache.org/jira/browse/HIVE-12983
> Project: Hive
>  Issue Type: Improvement
>  Components: UDF
>Affects Versions: 2.0.0
>Reporter: Lenni Kuff
>Assignee: Lenni Kuff
> Attachments: HIVE-12983.1.patch, HIVE-12983.2.patch
>
>
> It would be nice to have a builtin function that would return the Hive 
> version.  This would make it easier for a users and tests to programmatically 
> check the Hive version in a SQL script. It's also useful so a client can 
> check the Hive version on a remote cluster.
> For example:
> {code}
> beeline> SELECT version();
> 2.1.0-SNAPSHOT from 208ab352311a6cbbcd1f7fcd40964da2dbc6703d by lskuff source 
> checksum 8e971cda755f6b3fb528c233c40eb50a
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-13584) HBaseStorageHandler should support table pre-split

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-13584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15305196#comment-15305196
 ] 

Hive QA commented on HIVE-13584:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12806129/HIVE-13584.3.patch

{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 251 failed/errored test(s), 6207 tests 
executed
*Failed tests:*
{noformat}
TestCliDriver-alter_file_format.q-bucket_map_join_tez1.q-fetch_aggregation.q-and-12-more
 - did not produce a TEST-*.xml file
TestCliDriver-alter_rename_table.q-udf_aes_decrypt.q-udf_add.q-and-12-more - 
did not produce a TEST-*.xml file
TestCliDriver-alter_table_not_sorted.q-cbo_udf_max.q-exim_11_managed_external.q-and-12-more
 - did not produce a TEST-*.xml file
TestCliDriver-ambiguitycheck.q-udf_bitwise_and.q-insert_overwrite_directory.q-and-12-more
 - did not produce a TEST-*.xml file
TestCliDriver-authorization_1_sql_std.q-disallow_incompatible_type_change_off.q-schema_evol_text_nonvec_mapwork_table.q-and-12-more
 - did not produce a TEST-*.xml file
TestCliDriver-authorization_cli_nonsql.q-orc_createas1.q-skewjoinopt19.q-and-12-more
 - did not produce a TEST-*.xml file
TestCliDriver-authorization_cli_stdconfigauth.q-str_to_map.q-touch.q-and-12-more
 - did not produce a TEST-*.xml file
TestCliDriver-authorization_create_table_owner_privs.q-cte_7.q-groupby_sort_7.q-and-12-more
 - did not produce a TEST-*.xml file
TestCliDriver-authorization_create_temp_table.q-update_all_partitioned.q-lock4.q-and-12-more
 - did not produce a TEST-*.xml file
TestCliDriver-authorization_parts.q-authorization_view_1.q-parquet_map_of_maps.q-and-12-more
 - did not produce a TEST-*.xml file
TestCliDriver-authorization_role_grant2.q-union_remove_7.q-alter_view_as_select.q-and-12-more
 - did not produce a TEST-*.xml file
TestCliDriver-auto_join18.q-dynamic_partition_pruning.q-date_comparison.q-and-12-more
 - did not produce a TEST-*.xml file
TestCliDriver-auto_join30.q-udf_bitmap_empty.q-input17.q-and-12-more - did not 
produce a TEST-*.xml file
TestCliDriver-auto_join_reordering_values.q-subquery_in.q-partition_timestamp2.q-and-12-more
 - did not produce a TEST-*.xml file
TestCliDriver-auto_sortmerge_join_7.q-exim_04_evolved_parts.q-query_with_semi.q-and-12-more
 - did not produce a TEST-*.xml file
TestCliDriver-autogen_colalias.q-skewjoinopt3.q-rcfile_merge1.q-and-12-more - 
did not produce a TEST-*.xml file
TestCliDriver-avro_add_column.q-bucketmapjoin_negative.q-bucket_map_join_spark2.q-and-12-more
 - did not produce a TEST-*.xml file
TestCliDriver-ba_table1.q-udf_notequal.q-input_part1.q-and-12-more - did not 
produce a TEST-*.xml file
TestCliDriver-bucketcontext_4.q-orc_ends_with_nulls.q-correlationoptimizer9.q-and-12-more
 - did not produce a TEST-*.xml file
TestCliDriver-bucketsortoptimize_insert_7.q-list_bucket_query_multiskew_1.q-skewjoin_noskew.q-and-12-more
 - did not produce a TEST-*.xml file
TestCliDriver-cast_to_int.q-literal_double.q-skewjoinopt20.q-and-12-more - did 
not produce a TEST-*.xml file
TestCliDriver-cbo_rp_gby_empty.q-load_nonpart_authsuccess.q-truncate_table.q-and-12-more
 - did not produce a TEST-*.xml file
TestCliDriver-cbo_rp_join1.q-union_top_level.q-join18.q-and-12-more - did not 
produce a TEST-*.xml file
TestCliDriver-cbo_rp_limit.q-load_exist_part_authsuccess.q-schema_evol_text_nonvec_mapwork_part_all_primitive.q-and-12-more
 - did not produce a TEST-*.xml file
TestCliDriver-cbo_rp_lineage2.q-authorization_update.q-udf_pmod.q-and-12-more - 
did not produce a TEST-*.xml file
TestCliDriver-cbo_rp_semijoin.q-custom_input_output_format.q-udf_reflect2.q-and-12-more
 - did not produce a TEST-*.xml file
TestCliDriver-cbo_rp_udf_udaf.q-date_udf.q-limit0.q-and-12-more - did not 
produce a TEST-*.xml file
TestCliDriver-cbo_rp_views.q-orc_split_elimination.q-authorization_view_disable_cbo_4.q-and-12-more
 - did not produce a TEST-*.xml file
TestCliDriver-columnStatsUpdateForStatsOptimizer_2.q-cp_mj_rc.q-show_columns.q-and-12-more
 - did not produce a TEST-*.xml file
TestCliDriver-columnstats_partlvl_dp.q-input31.q-udaf_percentile.q-and-12-more 
- did not produce a TEST-*.xml file
TestCliDriver-compute_stats_string.q-noalias_subq1.q-vector_auto_smb_mapjoin_14.q-and-12-more
 - did not produce a TEST-*.xml file
TestCliDriver-create_func1.q-bucketmapjoin3.q-enforce_order.q-and-12-more - did 
not produce a TEST-*.xml file
TestCliDriver-create_genericudf.q-dynamic_partition_insert.q-skewjoinopt16.q-and-12-more
 - did not produce a TEST-*.xml file
TestCliDriver-create_or_replace_view.q-compute_stats_double.q-reduce_deduplicate_exclude_gby.q-and-12-more
 - did not produce a TEST-*.xml file
TestCliDriver-cte_mat_1.q-groupby_sort_6.q-update_after_multiple_inserts_special_characters.q-and-12-more
 - did not produce a TEST-*.xml file