[jira] [Commented] (HIVE-13068) Disable Hive ConstantPropagate optimizer when CBO has optimized the plan II
[
https://issues.apache.org/jira/browse/HIVE-13068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15265164#comment-15265164
]
Hive QA commented on HIVE-13068:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12801199/HIVE-13068.02.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 171 failed/errored test(s), 9973 tests
executed
*Failed tests:*
{noformat}
TestHWISessionManager - did not produce a TEST-*.xml file
TestMiniTezCliDriver-vector_grouping_sets.q-mapjoin_mapjoin.q-update_all_partitioned.q-and-12-more
- did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_authorization_explain
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join33
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join_filters
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join_nulls
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_sortmerge_join_12
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cast1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cbo_rp_outer_join_ppr
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cbo_rp_subq_not_in
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cbo_rp_udaf_percentile_approx_23
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cbo_subq_not_in
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_colstats_all_nulls
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_constantPropagateForSubQuery
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_constprog3
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_constprog_semijoin
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_create_genericudf
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_create_view
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cross_join
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cross_join_merge
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cross_product_check_1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cross_product_check_2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_decimal_stats
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_explain_logical
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby1_limit
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_input25
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_input8
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join38
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join42
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_alt_syntax
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_cond_pushdown_1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_cond_pushdown_3
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_cond_pushdown_unqual1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_cond_pushdown_unqual3
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_filters
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_nulls
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_reorder
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_lineage2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_lineage3
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_query_oneskew_1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_masking_2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_mergejoin
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_nomore_ambiguous_table_col
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_nonblock_op_deduplicate
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_orc_llap
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd_constant_expr
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd_join5
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd_repeated_alias
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd_udf_case
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_recursive_dir
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_regexp_extract
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_semijoin4
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_skewjoin
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_notin
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_notin_having
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_unqualcolumnrefs
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_views
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udf_unix_timestamp
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_fast_stats
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_offcbo
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_12
[jira] [Work started] (HIVE-13660) Vectorizing IN expression with list of columns throws java.lang.ClassCastException ExprNodeColumnDesc cannot be cast to ExprNodeConstantDesc
[ https://issues.apache.org/jira/browse/HIVE-13660?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-13660 started by Matt McCline. --- > Vectorizing IN expression with list of columns throws > java.lang.ClassCastException ExprNodeColumnDesc cannot be cast to > ExprNodeConstantDesc > > > Key: HIVE-13660 > URL: https://issues.apache.org/jira/browse/HIVE-13660 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-13660.01.patch > > > Example: > SELECT * FROM alltypesorc WHERE cint in (ctinyint, cbigint); -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13660) Vectorizing IN expression with list of columns throws java.lang.ClassCastException ExprNodeColumnDesc cannot be cast to ExprNodeConstantDesc
[ https://issues.apache.org/jira/browse/HIVE-13660?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-13660: Status: Patch Available (was: In Progress) > Vectorizing IN expression with list of columns throws > java.lang.ClassCastException ExprNodeColumnDesc cannot be cast to > ExprNodeConstantDesc > > > Key: HIVE-13660 > URL: https://issues.apache.org/jira/browse/HIVE-13660 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-13660.01.patch > > > Example: > SELECT * FROM alltypesorc WHERE cint in (ctinyint, cbigint); -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13660) Vectorizing IN expression with list of columns throws java.lang.ClassCastException ExprNodeColumnDesc cannot be cast to ExprNodeConstantDesc
[ https://issues.apache.org/jira/browse/HIVE-13660?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-13660: Attachment: HIVE-13660.01.patch > Vectorizing IN expression with list of columns throws > java.lang.ClassCastException ExprNodeColumnDesc cannot be cast to > ExprNodeConstantDesc > > > Key: HIVE-13660 > URL: https://issues.apache.org/jira/browse/HIVE-13660 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-13660.01.patch > > > Example: > SELECT * FROM alltypesorc WHERE cint in (ctinyint, cbigint); -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13421) Propagate job progress in operation status
[ https://issues.apache.org/jira/browse/HIVE-13421?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amareshwari Sriramadasu updated HIVE-13421: --- Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Committed. Thanks [~prongs] > Propagate job progress in operation status > -- > > Key: HIVE-13421 > URL: https://issues.apache.org/jira/browse/HIVE-13421 > Project: Hive > Issue Type: Improvement >Reporter: Rajat Khandelwal >Assignee: Rajat Khandelwal > Fix For: 2.1.0 > > Attachments: HIVE-13421.01.patch, HIVE-13421.02.patch, > HIVE-13421.03.patch, HIVE-13421.04.patch, HIVE-13421.05.patch, > HIVE-13421.06.patch, HIVE-13421.07.patch, HIVE-13421.08.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13421) Propagate job progress in operation status
[ https://issues.apache.org/jira/browse/HIVE-13421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15265097#comment-15265097 ] Amareshwari Sriramadasu commented on HIVE-13421: I see there is no difference between HIVE-13421.04.patch and HIVE-13421.08.patch - except that it is updated to master. Checked all the failed tests - most of them have the more age than 1 and the failed ones with age 1 are not related to the patch - they look intermittent failures. Going ahead and committing patch HIVE-13421.08.patch > Propagate job progress in operation status > -- > > Key: HIVE-13421 > URL: https://issues.apache.org/jira/browse/HIVE-13421 > Project: Hive > Issue Type: Improvement >Reporter: Rajat Khandelwal >Assignee: Rajat Khandelwal > Fix For: 2.1.0 > > Attachments: HIVE-13421.01.patch, HIVE-13421.02.patch, > HIVE-13421.03.patch, HIVE-13421.04.patch, HIVE-13421.05.patch, > HIVE-13421.06.patch, HIVE-13421.07.patch, HIVE-13421.08.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HIVE-13658) An empty where condition leads to vectorization exceptions instead of throwing a compile time error
[
https://issues.apache.org/jira/browse/HIVE-13658?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashutosh Chauhan resolved HIVE-13658.
-
Resolution: Duplicate
Dupe of HIVE-13569
> An empty where condition leads to vectorization exceptions instead of
> throwing a compile time error
> ---
>
> Key: HIVE-13658
> URL: https://issues.apache.org/jira/browse/HIVE-13658
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
>
> A partial query
> select count (distinct field) from table where field;
> Note the missing 'field=value'
> resulted in the following error in task logs, instead of failing early during
> compile
> {code}
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:173)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:139)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:344)
> at
> org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:181)
> at
> org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:172)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
> at
> org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:172)
> at
> org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:168)
> at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.RuntimeException:
> org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while
> processing row
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.processRow(MapRecordSource.java:91)
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.pushRecord(MapRecordSource.java:68)
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.run(MapRecordProcessor.java:326)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:150)
> ... 14 more
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime
> Error while processing row
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.process(VectorMapOperator.java:52)
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.processRow(MapRecordSource.java:83)
> ... 17 more
> Caused by: java.lang.ClassCastException:
> org.apache.hadoop.hive.ql.exec.vector.BytesColumnVector cannot be cast to
> org.apache.hadoop.hive.ql.exec.vector.LongColumnVector
> at
> org.apache.hadoop.hive.ql.exec.vector.expressions.SelectColumnIsTrue.evaluate(SelectColumnIsTrue.java:46)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorFilterOperator.process(VectorFilterOperator.java:106)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:838)
> at
> org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:97)
> at
> org.apache.hadoop.hive.ql.exec.MapOperator$MapOpCtx.forward(MapOperator.java:164)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.process(VectorMapOperator.java:45)
> ... 18 more
> {code}
> Comment by Gunther:
> think this works by implicitly converting the expr to boolean (if a cast is
> necessary). This query looks legal to me and probably needs to be handle in
> Vectorization.
> Comment by Ashutosh:
> Oracle, postgres & sql server throws error for this if type of field is not
> boolean. However, MySQL & Hive (with vectorization off) executes the query by
> implicitly adding a cast to boolean. Hive shall be consistent in its behavior
> regardless whether vectorization is on or off.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-11339) org.apache.hadoop.hive.serde2.io.TimestampWritable.write(DataOutput out) makes incorrect cast
[
https://issues.apache.org/jira/browse/HIVE-11339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15265086#comment-15265086
]
Hive QA commented on HIVE-11339:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12801031/HIVE-11339.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 23 failed/errored test(s), 10003 tests
executed
*Failed tests:*
{noformat}
TestHWISessionManager - did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby1_limit
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_nomore_ambiguous_table_col
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_regexp_extract
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_clustern3
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_clustern4
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_nonkey_groupby
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_selectDistinctStarNeg_2
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_subquery_shared_alias
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_udtf_not_supported1
org.apache.hadoop.hive.metastore.TestAuthzApiEmbedAuthorizerInRemote.org.apache.hadoop.hive.metastore.TestAuthzApiEmbedAuthorizerInRemote
org.apache.hadoop.hive.metastore.TestFilterHooks.org.apache.hadoop.hive.metastore.TestFilterHooks
org.apache.hadoop.hive.metastore.TestHiveMetaStoreGetMetaConf.testGetMetaConfDefault
org.apache.hadoop.hive.metastore.TestMetaStoreAuthorization.testMetaStoreAuthorization
org.apache.hadoop.hive.metastore.TestRetryingHMSHandler.testRetryingHMSHandler
org.apache.hadoop.hive.ql.security.TestClientSideAuthorizationProvider.testSimplePrivileges
org.apache.hadoop.hive.ql.security.TestExtendedAcls.org.apache.hadoop.hive.ql.security.TestExtendedAcls
org.apache.hadoop.hive.thrift.TestHadoopAuthBridge23.testDelegationTokenSharedStore
org.apache.hadoop.hive.thrift.TestHadoopAuthBridge23.testMetastoreProxyUser
org.apache.hadoop.hive.thrift.TestHadoopAuthBridge23.testSaslWithHiveMetaStore
org.apache.hive.hcatalog.listener.TestDbNotificationListener.dropDatabase
org.apache.hive.minikdc.TestJdbcWithDBTokenStore.testNegativeTokenAuth
{noformat}
Test results:
http://ec2-54-177-240-2.us-west-1.compute.amazonaws.com/job/PreCommit-HIVE-MASTER-Build/122/testReport
Console output:
http://ec2-54-177-240-2.us-west-1.compute.amazonaws.com/job/PreCommit-HIVE-MASTER-Build/122/console
Test logs:
http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-122/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 23 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12801031 - PreCommit-HIVE-MASTER-Build
> org.apache.hadoop.hive.serde2.io.TimestampWritable.write(DataOutput out)
> makes incorrect cast
> -
>
> Key: HIVE-11339
> URL: https://issues.apache.org/jira/browse/HIVE-11339
> Project: Hive
> Issue Type: Bug
> Components: Serializers/Deserializers
>Affects Versions: 0.14.0
>Reporter: Arnaud Linz
>Assignee: Zoltan Haindrich
> Labels: easyfix, newbie
> Attachments: HIVE-11339.patch
>
>
> Hi, it's my first Jira and I don't know how to make patches, so I'll explain
> the issue in the description as it is rather simple.
> I have a problem serializing "DefaultHCatRecord" using Apache Flink when
> those records include Timestamps because of an incorrect class cast in
> org.apache.hadoop.hive.serde2.io.TimestampWritable.write(DataOutput out). It
> is implemented using a cast to Outputstream :
> public void write(DataOutput out) throws IOException {
> write((OutputStream) out);
> }
> but nothing says that a DataOutput object is an OutputStream, (and it's not
> the case in Flink) it should rather be implmented using the same code as
> write(OutputStream) :
> {
> checkBytes();
> out.write(currentBytes, offset, getTotalLength());
> }
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13659) An empty where condition leads to vectorization exceptions instead of throwing a compile time error
[
https://issues.apache.org/jira/browse/HIVE-13659?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt McCline updated HIVE-13659:
Status: Patch Available (was: Open)
> An empty where condition leads to vectorization exceptions instead of
> throwing a compile time error
> ---
>
> Key: HIVE-13659
> URL: https://issues.apache.org/jira/browse/HIVE-13659
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Fix For: 2.1.0
>
> Attachments: HIVE-13659.01.patch
>
>
> A partial query
> select count (distinct field) from table where field;
> Note the missing 'field=value'
> resulted in the following error in task logs, instead of failing early during
> compile
> {code}
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:173)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:139)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:344)
> at
> org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:181)
> at
> org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:172)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
> at
> org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:172)
> at
> org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:168)
> at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.RuntimeException:
> org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while
> processing row
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.processRow(MapRecordSource.java:91)
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.pushRecord(MapRecordSource.java:68)
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.run(MapRecordProcessor.java:326)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:150)
> ... 14 more
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime
> Error while processing row
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.process(VectorMapOperator.java:52)
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.processRow(MapRecordSource.java:83)
> ... 17 more
> Caused by: java.lang.ClassCastException:
> org.apache.hadoop.hive.ql.exec.vector.BytesColumnVector cannot be cast to
> org.apache.hadoop.hive.ql.exec.vector.LongColumnVector
> at
> org.apache.hadoop.hive.ql.exec.vector.expressions.SelectColumnIsTrue.evaluate(SelectColumnIsTrue.java:46)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorFilterOperator.process(VectorFilterOperator.java:106)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:838)
> at
> org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:97)
> at
> org.apache.hadoop.hive.ql.exec.MapOperator$MapOpCtx.forward(MapOperator.java:164)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.process(VectorMapOperator.java:45)
> ... 18 more
> {code}
> Comment by Gunther:
> I think this works by implicitly converting the expr to boolean (if a cast is
> necessary). This query looks legal to me and probably needs to be handle in
> Vectorization
> Comment by Ashutosh:
> Oracle, postgres & sql server throws error for this if type of field is not
> boolean. However, MySQL & Hive (with vectorization off) executes the query by
> implicitly adding a cast to boolean. Hive shall be consistent in its behavior
> regardless whether vectorization is on or off.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13659) An empty where condition leads to vectorization exceptions instead of throwing a compile time error
[
https://issues.apache.org/jira/browse/HIVE-13659?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt McCline updated HIVE-13659:
Attachment: HIVE-13659.01.patch
> An empty where condition leads to vectorization exceptions instead of
> throwing a compile time error
> ---
>
> Key: HIVE-13659
> URL: https://issues.apache.org/jira/browse/HIVE-13659
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Fix For: 2.1.0
>
> Attachments: HIVE-13659.01.patch
>
>
> A partial query
> select count (distinct field) from table where field;
> Note the missing 'field=value'
> resulted in the following error in task logs, instead of failing early during
> compile
> {code}
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:173)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:139)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:344)
> at
> org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:181)
> at
> org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:172)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
> at
> org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:172)
> at
> org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:168)
> at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.RuntimeException:
> org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while
> processing row
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.processRow(MapRecordSource.java:91)
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.pushRecord(MapRecordSource.java:68)
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.run(MapRecordProcessor.java:326)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:150)
> ... 14 more
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime
> Error while processing row
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.process(VectorMapOperator.java:52)
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.processRow(MapRecordSource.java:83)
> ... 17 more
> Caused by: java.lang.ClassCastException:
> org.apache.hadoop.hive.ql.exec.vector.BytesColumnVector cannot be cast to
> org.apache.hadoop.hive.ql.exec.vector.LongColumnVector
> at
> org.apache.hadoop.hive.ql.exec.vector.expressions.SelectColumnIsTrue.evaluate(SelectColumnIsTrue.java:46)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorFilterOperator.process(VectorFilterOperator.java:106)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:838)
> at
> org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:97)
> at
> org.apache.hadoop.hive.ql.exec.MapOperator$MapOpCtx.forward(MapOperator.java:164)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.process(VectorMapOperator.java:45)
> ... 18 more
> {code}
> Comment by Gunther:
> I think this works by implicitly converting the expr to boolean (if a cast is
> necessary). This query looks legal to me and probably needs to be handle in
> Vectorization
> Comment by Ashutosh:
> Oracle, postgres & sql server throws error for this if type of field is not
> boolean. However, MySQL & Hive (with vectorization off) executes the query by
> implicitly adding a cast to boolean. Hive shall be consistent in its behavior
> regardless whether vectorization is on or off.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-12963) LIMIT statement with SORT BY creates additional MR job with hardcoded only one reducer
[ https://issues.apache.org/jira/browse/HIVE-12963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15265038#comment-15265038 ] Lefty Leverenz commented on HIVE-12963: --- Doc note: This adds *hive.groupby.limit.extrastep* to HiveConf.java, so it needs to be documented in the wiki for release 2.1.0. * [Configuration Properties -- Query and DDL Execution | https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-QueryandDDLExecution] > LIMIT statement with SORT BY creates additional MR job with hardcoded only > one reducer > -- > > Key: HIVE-12963 > URL: https://issues.apache.org/jira/browse/HIVE-12963 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 1.0.0, 1.2.1, 0.13 >Reporter: Alina Abramova >Assignee: Alina Abramova > Labels: TODOC2.1 > Fix For: 2.1.0 > > Attachments: HIVE-12963.1.patch, HIVE-12963.2.patch, > HIVE-12963.3.patch, HIVE-12963.4.patch, HIVE-12963.6.patch > > > I execute query: > hive> select age from test1 sort by age.age limit 10; > Total jobs = 2 > Launching Job 1 out of 2 > Number of reduce tasks not specified. Estimated from input data size: 1 > Launching Job 2 out of 2 > Number of reduce tasks determined at compile time: 1 > When I have a large number of rows then the last stage of the job takes a > long time. I think we could allow to user choose number of reducers of last > job or refuse extra MR job. > The same behavior I observed with querie: > hive> create table new_test as select age from test1 group by age.age limit > 10; -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12963) LIMIT statement with SORT BY creates additional MR job with hardcoded only one reducer
[ https://issues.apache.org/jira/browse/HIVE-12963?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lefty Leverenz updated HIVE-12963: -- Labels: TODOC2.1 (was: ) > LIMIT statement with SORT BY creates additional MR job with hardcoded only > one reducer > -- > > Key: HIVE-12963 > URL: https://issues.apache.org/jira/browse/HIVE-12963 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 1.0.0, 1.2.1, 0.13 >Reporter: Alina Abramova >Assignee: Alina Abramova > Labels: TODOC2.1 > Fix For: 2.1.0 > > Attachments: HIVE-12963.1.patch, HIVE-12963.2.patch, > HIVE-12963.3.patch, HIVE-12963.4.patch, HIVE-12963.6.patch > > > I execute query: > hive> select age from test1 sort by age.age limit 10; > Total jobs = 2 > Launching Job 1 out of 2 > Number of reduce tasks not specified. Estimated from input data size: 1 > Launching Job 2 out of 2 > Number of reduce tasks determined at compile time: 1 > When I have a large number of rows then the last stage of the job takes a > long time. I think we could allow to user choose number of reducers of last > job or refuse extra MR job. > The same behavior I observed with querie: > hive> create table new_test as select age from test1 group by age.age limit > 10; -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13380) Decimal should have lower precedence than double in type hierachy
[ https://issues.apache.org/jira/browse/HIVE-13380?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lefty Leverenz updated HIVE-13380: -- Labels: TODOC2.1 (was: ) > Decimal should have lower precedence than double in type hierachy > - > > Key: HIVE-13380 > URL: https://issues.apache.org/jira/browse/HIVE-13380 > Project: Hive > Issue Type: Bug > Components: Types >Reporter: Ashutosh Chauhan >Assignee: Ashutosh Chauhan > Labels: TODOC2.1 > Fix For: 2.1.0 > > Attachments: HIVE-13380.2.patch, HIVE-13380.4.patch, > HIVE-13380.5.patch, HIVE-13380.patch > > > Currently its other way round. Also, decimal should be lower than float. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12827) Vectorization: VectorCopyRow/VectorAssignRow/VectorDeserializeRow assign needs explicit isNull[offset] modification
[
https://issues.apache.org/jira/browse/HIVE-12827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15265020#comment-15265020
]
Lefty Leverenz commented on HIVE-12827:
---
[~mmccline], the branch-1 commit doesn't include the JIRA number. (Déjà vu:
see comment about previous commit.)
Please add this to the errata.txt file that was created by HIVE-11704.
Commit: 3734d5b674b4e8de9c0cc751650aee3194bfb93a.
> Vectorization: VectorCopyRow/VectorAssignRow/VectorDeserializeRow assign
> needs explicit isNull[offset] modification
> ---
>
> Key: HIVE-12827
> URL: https://issues.apache.org/jira/browse/HIVE-12827
> Project: Hive
> Issue Type: Bug
>Reporter: Gopal V
>Assignee: Gopal V
> Fix For: 1.3.0, 2.0.0, 2.1.0
>
> Attachments: HIVE-12827.2.patch
>
>
> Some scenarios do set Double.NaN instead of isNull=true, but all types aren't
> consistent.
> Examples of un-set isNull for the valid values are
> {code}
> private class FloatReader extends AbstractDoubleReader {
> FloatReader(int columnIndex) {
> super(columnIndex);
> }
> @Override
> void apply(VectorizedRowBatch batch, int batchIndex) throws IOException {
> DoubleColumnVector colVector = (DoubleColumnVector)
> batch.cols[columnIndex];
> if (deserializeRead.readCheckNull()) {
> VectorizedBatchUtil.setNullColIsNullValue(colVector, batchIndex);
> } else {
> float value = deserializeRead.readFloat();
> colVector.vector[batchIndex] = (double) value;
> }
> }
> }
> {code}
> {code}
> private class DoubleCopyRow extends CopyRow {
> DoubleCopyRow(int inColumnIndex, int outColumnIndex) {
> super(inColumnIndex, outColumnIndex);
> }
> @Override
> void copy(VectorizedRowBatch inBatch, int inBatchIndex,
> VectorizedRowBatch outBatch, int outBatchIndex) {
> DoubleColumnVector inColVector = (DoubleColumnVector)
> inBatch.cols[inColumnIndex];
> DoubleColumnVector outColVector = (DoubleColumnVector)
> outBatch.cols[outColumnIndex];
> if (inColVector.isRepeating) {
> if (inColVector.noNulls || !inColVector.isNull[0]) {
> outColVector.vector[outBatchIndex] = inColVector.vector[0];
> } else {
> VectorizedBatchUtil.setNullColIsNullValue(outColVector,
> outBatchIndex);
> }
> } else {
> if (inColVector.noNulls || !inColVector.isNull[inBatchIndex]) {
> outColVector.vector[outBatchIndex] =
> inColVector.vector[inBatchIndex];
> } else {
> VectorizedBatchUtil.setNullColIsNullValue(outColVector,
> outBatchIndex);
> }
> }
> }
> }
> {code}
> {code}
> private static abstract class VectorDoubleColumnAssign
> extends VectorColumnAssignVectorBase {
> protected void assignDouble(double value, int destIndex) {
> outCol.vector[destIndex] = value;
> }
> }
> {code}
> The pattern to imitate would be the earlier code from VectorBatchUtil
> {code}
> case DOUBLE: {
> DoubleColumnVector dcv = (DoubleColumnVector) batch.cols[offset +
> colIndex];
> if (writableCol != null) {
> dcv.vector[rowIndex] = ((DoubleWritable) writableCol).get();
> dcv.isNull[rowIndex] = false;
> } else {
> dcv.vector[rowIndex] = Double.NaN;
> setNullColIsNullValue(dcv, rowIndex);
> }
> }
> break;
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13587) Set Hive pom to use Hadoop 2.6.1
[
https://issues.apache.org/jira/browse/HIVE-13587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15265015#comment-15265015
]
Hive QA commented on HIVE-13587:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12801527/HIVE-13587-java8.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 225 failed/errored test(s), 9944 tests
executed
*Failed tests:*
{noformat}
TestHWISessionManager - did not produce a TEST-*.xml file
TestJdbcWithMiniHA - did not produce a TEST-*.xml file
TestJdbcWithMiniMr - did not produce a TEST-*.xml file
TestOperationLoggingAPIWithTez - did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_authorization_explain
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_binary_output_format
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucket1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucket2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucket3
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cbo_rp_outer_join_ppr
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_char_udf1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_columnStatsUpdateForStatsOptimizer_1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_columnStatsUpdateForStatsOptimizer_2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_constantPropagateForSubQuery
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ctas
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_describe_table
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_extrapolate_part_stats_full
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_extrapolate_part_stats_partial
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_extrapolate_part_stats_partial_ndv
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_fouter_join_ppr
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_map_ppr
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_map_ppr_multi_distinct
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_ppr
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_ppr_multi_distinct
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_input23
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_input42
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_input_part1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_input_part2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_input_part7
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_input_part9
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join17
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join26
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join32
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join32_lessSize
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join33
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join34
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join35
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join9
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_map_ppr
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_json_serde1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_11
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_12
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_13
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_14
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_3
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_4
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_5
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_6
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_7
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_8
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_9
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_load_dyn_part8
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_louter_join_ppr
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_mapjoin_mapjoin
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_nomore_ambiguous_table_col
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_offset_limit_global_optimizer
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_optimize_nullscan
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_outer_join_ppr
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_partition_coltype_literals
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_pcr
[jira] [Commented] (HIVE-12827) Vectorization: VectorCopyRow/VectorAssignRow/VectorDeserializeRow assign needs explicit isNull[offset] modification
[
https://issues.apache.org/jira/browse/HIVE-12827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15264949#comment-15264949
]
Matt McCline commented on HIVE-12827:
-
Also committed to branch-1
> Vectorization: VectorCopyRow/VectorAssignRow/VectorDeserializeRow assign
> needs explicit isNull[offset] modification
> ---
>
> Key: HIVE-12827
> URL: https://issues.apache.org/jira/browse/HIVE-12827
> Project: Hive
> Issue Type: Bug
>Reporter: Gopal V
>Assignee: Gopal V
> Fix For: 1.3.0, 2.0.0, 2.1.0
>
> Attachments: HIVE-12827.2.patch
>
>
> Some scenarios do set Double.NaN instead of isNull=true, but all types aren't
> consistent.
> Examples of un-set isNull for the valid values are
> {code}
> private class FloatReader extends AbstractDoubleReader {
> FloatReader(int columnIndex) {
> super(columnIndex);
> }
> @Override
> void apply(VectorizedRowBatch batch, int batchIndex) throws IOException {
> DoubleColumnVector colVector = (DoubleColumnVector)
> batch.cols[columnIndex];
> if (deserializeRead.readCheckNull()) {
> VectorizedBatchUtil.setNullColIsNullValue(colVector, batchIndex);
> } else {
> float value = deserializeRead.readFloat();
> colVector.vector[batchIndex] = (double) value;
> }
> }
> }
> {code}
> {code}
> private class DoubleCopyRow extends CopyRow {
> DoubleCopyRow(int inColumnIndex, int outColumnIndex) {
> super(inColumnIndex, outColumnIndex);
> }
> @Override
> void copy(VectorizedRowBatch inBatch, int inBatchIndex,
> VectorizedRowBatch outBatch, int outBatchIndex) {
> DoubleColumnVector inColVector = (DoubleColumnVector)
> inBatch.cols[inColumnIndex];
> DoubleColumnVector outColVector = (DoubleColumnVector)
> outBatch.cols[outColumnIndex];
> if (inColVector.isRepeating) {
> if (inColVector.noNulls || !inColVector.isNull[0]) {
> outColVector.vector[outBatchIndex] = inColVector.vector[0];
> } else {
> VectorizedBatchUtil.setNullColIsNullValue(outColVector,
> outBatchIndex);
> }
> } else {
> if (inColVector.noNulls || !inColVector.isNull[inBatchIndex]) {
> outColVector.vector[outBatchIndex] =
> inColVector.vector[inBatchIndex];
> } else {
> VectorizedBatchUtil.setNullColIsNullValue(outColVector,
> outBatchIndex);
> }
> }
> }
> }
> {code}
> {code}
> private static abstract class VectorDoubleColumnAssign
> extends VectorColumnAssignVectorBase {
> protected void assignDouble(double value, int destIndex) {
> outCol.vector[destIndex] = value;
> }
> }
> {code}
> The pattern to imitate would be the earlier code from VectorBatchUtil
> {code}
> case DOUBLE: {
> DoubleColumnVector dcv = (DoubleColumnVector) batch.cols[offset +
> colIndex];
> if (writableCol != null) {
> dcv.vector[rowIndex] = ((DoubleWritable) writableCol).get();
> dcv.isNull[rowIndex] = false;
> } else {
> dcv.vector[rowIndex] = Double.NaN;
> setNullColIsNullValue(dcv, rowIndex);
> }
> }
> break;
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-12827) Vectorization: VectorCopyRow/VectorAssignRow/VectorDeserializeRow assign needs explicit isNull[offset] modification
[
https://issues.apache.org/jira/browse/HIVE-12827?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt McCline updated HIVE-12827:
Fix Version/s: 1.3.0
> Vectorization: VectorCopyRow/VectorAssignRow/VectorDeserializeRow assign
> needs explicit isNull[offset] modification
> ---
>
> Key: HIVE-12827
> URL: https://issues.apache.org/jira/browse/HIVE-12827
> Project: Hive
> Issue Type: Bug
>Reporter: Gopal V
>Assignee: Gopal V
> Fix For: 1.3.0, 2.0.0, 2.1.0
>
> Attachments: HIVE-12827.2.patch
>
>
> Some scenarios do set Double.NaN instead of isNull=true, but all types aren't
> consistent.
> Examples of un-set isNull for the valid values are
> {code}
> private class FloatReader extends AbstractDoubleReader {
> FloatReader(int columnIndex) {
> super(columnIndex);
> }
> @Override
> void apply(VectorizedRowBatch batch, int batchIndex) throws IOException {
> DoubleColumnVector colVector = (DoubleColumnVector)
> batch.cols[columnIndex];
> if (deserializeRead.readCheckNull()) {
> VectorizedBatchUtil.setNullColIsNullValue(colVector, batchIndex);
> } else {
> float value = deserializeRead.readFloat();
> colVector.vector[batchIndex] = (double) value;
> }
> }
> }
> {code}
> {code}
> private class DoubleCopyRow extends CopyRow {
> DoubleCopyRow(int inColumnIndex, int outColumnIndex) {
> super(inColumnIndex, outColumnIndex);
> }
> @Override
> void copy(VectorizedRowBatch inBatch, int inBatchIndex,
> VectorizedRowBatch outBatch, int outBatchIndex) {
> DoubleColumnVector inColVector = (DoubleColumnVector)
> inBatch.cols[inColumnIndex];
> DoubleColumnVector outColVector = (DoubleColumnVector)
> outBatch.cols[outColumnIndex];
> if (inColVector.isRepeating) {
> if (inColVector.noNulls || !inColVector.isNull[0]) {
> outColVector.vector[outBatchIndex] = inColVector.vector[0];
> } else {
> VectorizedBatchUtil.setNullColIsNullValue(outColVector,
> outBatchIndex);
> }
> } else {
> if (inColVector.noNulls || !inColVector.isNull[inBatchIndex]) {
> outColVector.vector[outBatchIndex] =
> inColVector.vector[inBatchIndex];
> } else {
> VectorizedBatchUtil.setNullColIsNullValue(outColVector,
> outBatchIndex);
> }
> }
> }
> }
> {code}
> {code}
> private static abstract class VectorDoubleColumnAssign
> extends VectorColumnAssignVectorBase {
> protected void assignDouble(double value, int destIndex) {
> outCol.vector[destIndex] = value;
> }
> }
> {code}
> The pattern to imitate would be the earlier code from VectorBatchUtil
> {code}
> case DOUBLE: {
> DoubleColumnVector dcv = (DoubleColumnVector) batch.cols[offset +
> colIndex];
> if (writableCol != null) {
> dcv.vector[rowIndex] = ((DoubleWritable) writableCol).get();
> dcv.isNull[rowIndex] = false;
> } else {
> dcv.vector[rowIndex] = Double.NaN;
> setNullColIsNullValue(dcv, rowIndex);
> }
> }
> break;
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13390) HiveServer2: Add more test to ZK service discovery using MiniHS2
[ https://issues.apache.org/jira/browse/HIVE-13390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15264928#comment-15264928 ] Sushanth Sowmyan commented on HIVE-13390: - +1 > HiveServer2: Add more test to ZK service discovery using MiniHS2 > > > Key: HIVE-13390 > URL: https://issues.apache.org/jira/browse/HIVE-13390 > Project: Hive > Issue Type: Bug > Components: HiveServer2, JDBC >Reporter: Vaibhav Gumashta >Assignee: Vaibhav Gumashta > Attachments: HIVE-13390.1.patch, HIVE-13390.1.patch, > HIVE-13390.2.patch, HIVE-13390.3.patch, keystore.jks, > keystore_exampledotcom.jks, truststore.jks > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12878) Support Vectorization for TEXTFILE and other formats
[ https://issues.apache.org/jira/browse/HIVE-12878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15264926#comment-15264926 ] Matt McCline commented on HIVE-12878: - Around #146 or #147. > Support Vectorization for TEXTFILE and other formats > > > Key: HIVE-12878 > URL: https://issues.apache.org/jira/browse/HIVE-12878 > Project: Hive > Issue Type: New Feature > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-12878.01.patch, HIVE-12878.02.patch, > HIVE-12878.03.patch, HIVE-12878.04.patch, HIVE-12878.05.patch, > HIVE-12878.06.patch, HIVE-12878.07.patch, HIVE-12878.08.patch, > HIVE-12878.09.patch, HIVE-12878.091.patch, HIVE-12878.092.patch, > HIVE-12878.093.patch > > > Support vectorizing when the input format is TEXTFILE and other formats for > better Map Vertex performance. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12878) Support Vectorization for TEXTFILE and other formats
[ https://issues.apache.org/jira/browse/HIVE-12878?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-12878: Attachment: HIVE-12878.093.patch > Support Vectorization for TEXTFILE and other formats > > > Key: HIVE-12878 > URL: https://issues.apache.org/jira/browse/HIVE-12878 > Project: Hive > Issue Type: New Feature > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-12878.01.patch, HIVE-12878.02.patch, > HIVE-12878.03.patch, HIVE-12878.04.patch, HIVE-12878.05.patch, > HIVE-12878.06.patch, HIVE-12878.07.patch, HIVE-12878.08.patch, > HIVE-12878.09.patch, HIVE-12878.091.patch, HIVE-12878.092.patch, > HIVE-12878.093.patch > > > Support vectorizing when the input format is TEXTFILE and other formats for > better Map Vertex performance. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12878) Support Vectorization for TEXTFILE and other formats
[ https://issues.apache.org/jira/browse/HIVE-12878?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-12878: Status: Patch Available (was: In Progress) > Support Vectorization for TEXTFILE and other formats > > > Key: HIVE-12878 > URL: https://issues.apache.org/jira/browse/HIVE-12878 > Project: Hive > Issue Type: New Feature > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-12878.01.patch, HIVE-12878.02.patch, > HIVE-12878.03.patch, HIVE-12878.04.patch, HIVE-12878.05.patch, > HIVE-12878.06.patch, HIVE-12878.07.patch, HIVE-12878.08.patch, > HIVE-12878.09.patch, HIVE-12878.091.patch, HIVE-12878.092.patch, > HIVE-12878.093.patch > > > Support vectorizing when the input format is TEXTFILE and other formats for > better Map Vertex performance. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12878) Support Vectorization for TEXTFILE and other formats
[ https://issues.apache.org/jira/browse/HIVE-12878?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-12878: Status: In Progress (was: Patch Available) > Support Vectorization for TEXTFILE and other formats > > > Key: HIVE-12878 > URL: https://issues.apache.org/jira/browse/HIVE-12878 > Project: Hive > Issue Type: New Feature > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-12878.01.patch, HIVE-12878.02.patch, > HIVE-12878.03.patch, HIVE-12878.04.patch, HIVE-12878.05.patch, > HIVE-12878.06.patch, HIVE-12878.07.patch, HIVE-12878.08.patch, > HIVE-12878.09.patch, HIVE-12878.091.patch, HIVE-12878.092.patch > > > Support vectorizing when the input format is TEXTFILE and other formats for > better Map Vertex performance. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-4924) JDBC: Support query timeout for jdbc
[
https://issues.apache.org/jira/browse/HIVE-4924?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vaibhav Gumashta updated HIVE-4924:
---
Attachment: HIVE-4924.6.patch
> JDBC: Support query timeout for jdbc
>
>
> Key: HIVE-4924
> URL: https://issues.apache.org/jira/browse/HIVE-4924
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2, JDBC
>Affects Versions: 2.0.0
>Reporter: Navis
>Assignee: Vaibhav Gumashta
> Attachments: HIVE-4924.3.patch.txt, HIVE-4924.4.patch,
> HIVE-4924.5.patch, HIVE-4924.6.patch, HIVE-4924.D11787.1.patch,
> HIVE-4924.D11787.2.patch
>
>
> support
> {code}
> Statement.setQueryTimeout(int timeout)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13615) nomore_ambiguous_table_col.q is failing on master
[ https://issues.apache.org/jira/browse/HIVE-13615?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Sankar Sivarama Subramaniyan updated HIVE-13615: - Status: Open (was: Patch Available) > nomore_ambiguous_table_col.q is failing on master > - > > Key: HIVE-13615 > URL: https://issues.apache.org/jira/browse/HIVE-13615 > Project: Hive > Issue Type: Test > Components: Parser >Affects Versions: 2.1.0 >Reporter: Ashutosh Chauhan >Assignee: Hari Sankar Sivarama Subramaniyan > Attachments: HIVE-13615.1.patch, HIVE-13615.2.patch > > > Fails with: > FAILED: ParseException line 3:9 cannot recognize input near 'src' 'key' > 'INSERT' in from source 0 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13615) nomore_ambiguous_table_col.q is failing on master
[ https://issues.apache.org/jira/browse/HIVE-13615?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Sankar Sivarama Subramaniyan updated HIVE-13615: - Attachment: HIVE-13615.2.patch > nomore_ambiguous_table_col.q is failing on master > - > > Key: HIVE-13615 > URL: https://issues.apache.org/jira/browse/HIVE-13615 > Project: Hive > Issue Type: Test > Components: Parser >Affects Versions: 2.1.0 >Reporter: Ashutosh Chauhan >Assignee: Hari Sankar Sivarama Subramaniyan > Attachments: HIVE-13615.1.patch, HIVE-13615.2.patch > > > Fails with: > FAILED: ParseException line 3:9 cannot recognize input near 'src' 'key' > 'INSERT' in from source 0 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13615) nomore_ambiguous_table_col.q is failing on master
[ https://issues.apache.org/jira/browse/HIVE-13615?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Sankar Sivarama Subramaniyan updated HIVE-13615: - Status: Patch Available (was: Open) > nomore_ambiguous_table_col.q is failing on master > - > > Key: HIVE-13615 > URL: https://issues.apache.org/jira/browse/HIVE-13615 > Project: Hive > Issue Type: Test > Components: Parser >Affects Versions: 2.1.0 >Reporter: Ashutosh Chauhan >Assignee: Hari Sankar Sivarama Subramaniyan > Attachments: HIVE-13615.1.patch, HIVE-13615.2.patch > > > Fails with: > FAILED: ParseException line 3:9 cannot recognize input near 'src' 'key' > 'INSERT' in from source 0 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13615) nomore_ambiguous_table_col.q is failing on master
[
https://issues.apache.org/jira/browse/HIVE-13615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15264872#comment-15264872
]
Hari Sankar Sivarama Subramaniyan commented on HIVE-13615:
--
[~ashutoshc]
Reason for the following:
{code}
In addition to that error message has changed for -ve test cases of
=nonkey_groupby.q,subquery_shared_alias.q,clustern3.q,clustern4.q,udtf_not_supported1.q,selectDistinctStarNeg_2.q
We are losing line number and char position in error message.
{code}
Its because of the below line :
{code}
nonReserved -> Identifier[$nonReserved.text]
{code}
The above line generates the following code via ANTLR.
{code}
// 617:19: -> Identifier[$nonReserved.text]
{
adaptor.addChild(root_0,
(CommonTree)adaptor.create(Identifier,
(nonReserved277!=null?input.toString(nonReserved277.start,nonReserved277.stop):null))
);
}
{code}
The problem here is that it creates a new Identifier object at the expense of
losing the nonReserved277 token information such as start /end index, line
number, etc.
The fix would be to change the line to below:
{code}
nonReserved -> Identifier[$nonReserved.start]
{code}
This would prevent losing any valuable token information when the keyword input
is transformed to an identifier token.
> nomore_ambiguous_table_col.q is failing on master
> -
>
> Key: HIVE-13615
> URL: https://issues.apache.org/jira/browse/HIVE-13615
> Project: Hive
> Issue Type: Test
> Components: Parser
>Affects Versions: 2.1.0
>Reporter: Ashutosh Chauhan
>Assignee: Hari Sankar Sivarama Subramaniyan
> Attachments: HIVE-13615.1.patch
>
>
> Fails with:
> FAILED: ParseException line 3:9 cannot recognize input near 'src' 'key'
> 'INSERT' in from source 0
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (HIVE-13656) need to set direct memory limit higher in LlapServiceDriver for certain edge case configurations
[ https://issues.apache.org/jira/browse/HIVE-13656?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin reassigned HIVE-13656: --- Assignee: Sergey Shelukhin > need to set direct memory limit higher in LlapServiceDriver for certain edge > case configurations > > > Key: HIVE-13656 > URL: https://issues.apache.org/jira/browse/HIVE-13656 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-13656.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13656) need to set direct memory limit higher in LlapServiceDriver for certain edge case configurations
[ https://issues.apache.org/jira/browse/HIVE-13656?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-13656: Status: Patch Available (was: Open) > need to set direct memory limit higher in LlapServiceDriver for certain edge > case configurations > > > Key: HIVE-13656 > URL: https://issues.apache.org/jira/browse/HIVE-13656 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-13656.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13656) need to set direct memory limit higher in LlapServiceDriver for certain edge case configurations
[ https://issues.apache.org/jira/browse/HIVE-13656?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-13656: Attachment: HIVE-13656.patch [~gopalv] [~sseth] can you take a look? [~sumitmohanty] fyi > need to set direct memory limit higher in LlapServiceDriver for certain edge > case configurations > > > Key: HIVE-13656 > URL: https://issues.apache.org/jira/browse/HIVE-13656 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin > Attachments: HIVE-13656.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-2573) Create per-session function registry
[ https://issues.apache.org/jira/browse/HIVE-2573?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thejas M Nair updated HIVE-2573: Release Note: Marked as incompatible because of a bug - This patch broke automatic loading of externally added permanent functions in HiveServer2. This only affects multi-HS2 setup, where only one HS2 adds the function; or functions added from Hive CLI bypassing HS2. In such cases, HS2 will not be able to run a query with the new function. Additionally, RELOAD FUNCTION call has been broken by this patch. Workaround: The HS2s that don't have the function need to be restarted. > Create per-session function registry > - > > Key: HIVE-2573 > URL: https://issues.apache.org/jira/browse/HIVE-2573 > Project: Hive > Issue Type: Improvement > Components: Server Infrastructure >Reporter: Navis >Assignee: Navis >Priority: Minor > Labels: TODOC1.2 > Fix For: 1.2.0 > > Attachments: ASF.LICENSE.NOT.GRANTED--HIVE-2573.D3231.1.patch, > HIVE-2573.1.patch.txt, HIVE-2573.10.patch.txt, HIVE-2573.11.patch.txt, > HIVE-2573.12.patch.txt, HIVE-2573.13.patch.txt, HIVE-2573.14.patch.txt, > HIVE-2573.15.patch.txt, HIVE-2573.2.patch.txt, HIVE-2573.3.patch.txt, > HIVE-2573.4.patch.txt, HIVE-2573.5.patch, HIVE-2573.6.patch, > HIVE-2573.7.patch, HIVE-2573.8.patch.txt, HIVE-2573.9.patch.txt > > > Currently the function registry is shared resource and could be overrided by > other users when using HiveServer. If per-session function registry is > provided, this situation could be prevented. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13421) Propagate job progress in operation status
[
https://issues.apache.org/jira/browse/HIVE-13421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15264774#comment-15264774
]
Hive QA commented on HIVE-13421:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12801443/HIVE-13421.08.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 57 failed/errored test(s), 9925 tests
executed
*Failed tests:*
{noformat}
TestHWISessionManager - did not produce a TEST-*.xml file
TestMiniTezCliDriver-auto_join1.q-vector_complex_join.q-vectorization_limit.q-and-12-more
- did not produce a TEST-*.xml file
TestMiniTezCliDriver-cbo_windowing.q-tez_join.q-bucket_map_join_tez1.q-and-12-more
- did not produce a TEST-*.xml file
TestMiniTezCliDriver-groupby2.q-tez_dynpart_hashjoin_1.q-custom_input_output_format.q-and-12-more
- did not produce a TEST-*.xml file
TestMiniTezCliDriver-vector_varchar_4.q-smb_cache.q-tez_join_hash.q-and-8-more
- did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_nomore_ambiguous_table_col
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_regexp_extract
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_clustern3
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_clustern4
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_nonkey_groupby
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_selectDistinctStarNeg_2
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_subquery_shared_alias
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_udtf_not_supported1
org.apache.hadoop.hive.llap.tezplugins.TestLlapTaskCommunicator.testFinishableStateUpdateFailure
org.apache.hadoop.hive.llap.tezplugins.TestLlapTaskSchedulerService.testDelayedLocalityNodeCommErrorImmediateAllocation
org.apache.hadoop.hive.metastore.TestAuthzApiEmbedAuthorizerInRemote.org.apache.hadoop.hive.metastore.TestAuthzApiEmbedAuthorizerInRemote
org.apache.hadoop.hive.metastore.TestFilterHooks.org.apache.hadoop.hive.metastore.TestFilterHooks
org.apache.hadoop.hive.metastore.TestHiveMetaStoreGetMetaConf.testGetMetaConfDefault
org.apache.hadoop.hive.metastore.TestHiveMetaStoreGetMetaConf.testGetMetaConfDefaultEmptyString
org.apache.hadoop.hive.metastore.TestHiveMetaStoreGetMetaConf.testGetMetaConfOverridden
org.apache.hadoop.hive.metastore.TestHiveMetaStoreGetMetaConf.testGetMetaConfUnknownPreperty
org.apache.hadoop.hive.metastore.TestHiveMetaStorePartitionSpecs.testAddPartitions
org.apache.hadoop.hive.metastore.TestHiveMetaStorePartitionSpecs.testFetchingPartitionsWithDifferentSchemas
org.apache.hadoop.hive.metastore.TestHiveMetaStorePartitionSpecs.testGetPartitionSpecs_WithAndWithoutPartitionGrouping
org.apache.hadoop.hive.metastore.TestMetaStoreEndFunctionListener.testEndFunctionListener
org.apache.hadoop.hive.metastore.TestMetaStoreEventListenerOnlyOnCommit.testEventStatus
org.apache.hadoop.hive.metastore.TestMetaStoreInitListener.testMetaStoreInitListener
org.apache.hadoop.hive.metastore.TestMetaStoreMetrics.org.apache.hadoop.hive.metastore.TestMetaStoreMetrics
org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAddPartitionWithCommas
org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAddPartitionWithUnicode
org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAddPartitionWithValidPartVal
org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAppendPartitionWithCommas
org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAppendPartitionWithUnicode
org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAppendPartitionWithValidCharacters
org.apache.hadoop.hive.metastore.TestRetryingHMSHandler.testRetryingHMSHandler
org.apache.hadoop.hive.ql.TestTxnCommands2.testBucketizedInputFormat
org.apache.hadoop.hive.ql.TestTxnCommands2.testInitiatorWithMultipleFailedCompactions
org.apache.hadoop.hive.ql.TestTxnCommands2.testOrcNoPPD
org.apache.hadoop.hive.ql.TestTxnCommands2.testUpdateMixedCase
org.apache.hadoop.hive.ql.security.TestExtendedAcls.org.apache.hadoop.hive.ql.security.TestExtendedAcls
org.apache.hadoop.hive.ql.security.TestFolderPermissions.org.apache.hadoop.hive.ql.security.TestFolderPermissions
org.apache.hadoop.hive.ql.security.TestMetastoreAuthorizationProvider.testSimplePrivileges
org.apache.hadoop.hive.ql.security.TestMultiAuthorizationPreEventListener.org.apache.hadoop.hive.ql.security.TestMultiAuthorizationPreEventListener
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationDrops.testDropPartition
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationProvider.testSimplePrivileges
[jira] [Updated] (HIVE-13587) Set Hive pom to use Hadoop 2.6.1
[ https://issues.apache.org/jira/browse/HIVE-13587?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mohit Sabharwal updated HIVE-13587: --- Attachment: HIVE-13587-java8.patch > Set Hive pom to use Hadoop 2.6.1 > > > Key: HIVE-13587 > URL: https://issues.apache.org/jira/browse/HIVE-13587 > Project: Hive > Issue Type: Sub-task >Reporter: Mohit Sabharwal >Assignee: Mohit Sabharwal > Attachments: HIVE-13587-java8.patch > > > A number of the JDK8 unit test failures are due to HADOOP-10786, fixed in > Hadoop 2.6.1. > * TestMiniHiveKdc.testLogin > * TestHiveAuthFactory.testStartTokenManagerForDBTokenStore > * TestHiveAuthFactory.testStartTokenManagerForMemoryTokenStore > i.e. Hive under Kerberos is broken in Java8 unless we move dependency to > hadoop 2.6.1 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13587) Set Hive pom to use Hadoop 2.6.1
[ https://issues.apache.org/jira/browse/HIVE-13587?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mohit Sabharwal updated HIVE-13587: --- Status: Patch Available (was: Open) > Set Hive pom to use Hadoop 2.6.1 > > > Key: HIVE-13587 > URL: https://issues.apache.org/jira/browse/HIVE-13587 > Project: Hive > Issue Type: Sub-task >Reporter: Mohit Sabharwal >Assignee: Mohit Sabharwal > Attachments: HIVE-13587-java8.patch > > > A number of the JDK8 unit test failures are due to HADOOP-10786, fixed in > Hadoop 2.6.1. > * TestMiniHiveKdc.testLogin > * TestHiveAuthFactory.testStartTokenManagerForDBTokenStore > * TestHiveAuthFactory.testStartTokenManagerForMemoryTokenStore > i.e. Hive under Kerberos is broken in Java8 unless we move dependency to > hadoop 2.6.1 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HIVE-13654) Add JAVA8_URL to jenkins-submit-build.sh
[ https://issues.apache.org/jira/browse/HIVE-13654?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergio Peña resolved HIVE-13654. Resolution: Fixed Fix Version/s: 2.1.0 > Add JAVA8_URL to jenkins-submit-build.sh > > > Key: HIVE-13654 > URL: https://issues.apache.org/jira/browse/HIVE-13654 > Project: Hive > Issue Type: Sub-task >Reporter: Sergio Peña >Assignee: Sergio Peña > Fix For: 2.1.0 > > Attachments: HIVE-13654.1.patch > > > NO PRECOMMIT TESTS -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13654) Add JAVA8_URL to jenkins-submit-build.sh
[ https://issues.apache.org/jira/browse/HIVE-13654?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergio Peña updated HIVE-13654: --- Attachment: HIVE-13654.1.patch > Add JAVA8_URL to jenkins-submit-build.sh > > > Key: HIVE-13654 > URL: https://issues.apache.org/jira/browse/HIVE-13654 > Project: Hive > Issue Type: Sub-task >Reporter: Sergio Peña >Assignee: Sergio Peña > Attachments: HIVE-13654.1.patch > > > NO PRECOMMIT TESTS -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13654) Add JAVA8_URL to jenkins-submit-build.sh
[ https://issues.apache.org/jira/browse/HIVE-13654?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergio Peña updated HIVE-13654: --- Description: NO PRECOMMIT TESTS > Add JAVA8_URL to jenkins-submit-build.sh > > > Key: HIVE-13654 > URL: https://issues.apache.org/jira/browse/HIVE-13654 > Project: Hive > Issue Type: Sub-task >Reporter: Sergio Peña >Assignee: Sergio Peña > > NO PRECOMMIT TESTS -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13561) HiveServer2 is leaking ClassLoaders when add jar / temporary functions are used
[ https://issues.apache.org/jira/browse/HIVE-13561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15264703#comment-15264703 ] Vaibhav Gumashta commented on HIVE-13561: - [~tleftwich] Thanks for the patch. Look forward to the one for master. I'll review this one shortly. > HiveServer2 is leaking ClassLoaders when add jar / temporary functions are > used > --- > > Key: HIVE-13561 > URL: https://issues.apache.org/jira/browse/HIVE-13561 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Affects Versions: 1.2.0, 1.2.1, 2.0.0 >Reporter: Trystan Leftwich >Assignee: Trystan Leftwich > Attachments: HIVE-13561-branch-1.2.3.patch, HIVE-13561.2.patch > > > I can repo this on branch-1.2 and branch-2.0. > It looks to be the same issues as: HIVE-11408 > The patch from HIVE-11408 looks to fix the issue as well. > I've updated the patch from HIVE-11408 to be aligned with branch-1.2 and > master -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13561) HiveServer2 is leaking ClassLoaders when add jar / temporary functions are used
[ https://issues.apache.org/jira/browse/HIVE-13561?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Trystan Leftwich updated HIVE-13561: Status: Open (was: Patch Available) > HiveServer2 is leaking ClassLoaders when add jar / temporary functions are > used > --- > > Key: HIVE-13561 > URL: https://issues.apache.org/jira/browse/HIVE-13561 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Affects Versions: 2.0.0, 1.2.1, 1.2.0 >Reporter: Trystan Leftwich >Assignee: Trystan Leftwich > Attachments: HIVE-13561-branch-1.2.3.patch, HIVE-13561.2.patch > > > I can repo this on branch-1.2 and branch-2.0. > It looks to be the same issues as: HIVE-11408 > The patch from HIVE-11408 looks to fix the issue as well. > I've updated the patch from HIVE-11408 to be aligned with branch-1.2 and > master -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13561) HiveServer2 is leaking ClassLoaders when add jar / temporary functions are used
[ https://issues.apache.org/jira/browse/HIVE-13561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15264699#comment-15264699 ] Trystan Leftwich commented on HIVE-13561: - Apologies for the delay. Added a new patch that will count the size of the constructor cache entries before and after an add jar. Only for branch-1 but will add a patch for master. > HiveServer2 is leaking ClassLoaders when add jar / temporary functions are > used > --- > > Key: HIVE-13561 > URL: https://issues.apache.org/jira/browse/HIVE-13561 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Affects Versions: 1.2.0, 1.2.1, 2.0.0 >Reporter: Trystan Leftwich >Assignee: Trystan Leftwich > Attachments: HIVE-13561-branch-1.2.3.patch, HIVE-13561.2.patch > > > I can repo this on branch-1.2 and branch-2.0. > It looks to be the same issues as: HIVE-11408 > The patch from HIVE-11408 looks to fix the issue as well. > I've updated the patch from HIVE-11408 to be aligned with branch-1.2 and > master -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13561) HiveServer2 is leaking ClassLoaders when add jar / temporary functions are used
[ https://issues.apache.org/jira/browse/HIVE-13561?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Trystan Leftwich updated HIVE-13561: Status: Patch Available (was: Open) > HiveServer2 is leaking ClassLoaders when add jar / temporary functions are > used > --- > > Key: HIVE-13561 > URL: https://issues.apache.org/jira/browse/HIVE-13561 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Affects Versions: 2.0.0, 1.2.1, 1.2.0 >Reporter: Trystan Leftwich >Assignee: Trystan Leftwich > Attachments: HIVE-13561-branch-1.2.3.patch, HIVE-13561.2.patch > > > I can repo this on branch-1.2 and branch-2.0. > It looks to be the same issues as: HIVE-11408 > The patch from HIVE-11408 looks to fix the issue as well. > I've updated the patch from HIVE-11408 to be aligned with branch-1.2 and > master -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-2573) Create per-session function registry
[ https://issues.apache.org/jira/browse/HIVE-2573?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thejas M Nair updated HIVE-2573: Hadoop Flags: Incompatible change > Create per-session function registry > - > > Key: HIVE-2573 > URL: https://issues.apache.org/jira/browse/HIVE-2573 > Project: Hive > Issue Type: Improvement > Components: Server Infrastructure >Reporter: Navis >Assignee: Navis >Priority: Minor > Labels: TODOC1.2 > Fix For: 1.2.0 > > Attachments: ASF.LICENSE.NOT.GRANTED--HIVE-2573.D3231.1.patch, > HIVE-2573.1.patch.txt, HIVE-2573.10.patch.txt, HIVE-2573.11.patch.txt, > HIVE-2573.12.patch.txt, HIVE-2573.13.patch.txt, HIVE-2573.14.patch.txt, > HIVE-2573.15.patch.txt, HIVE-2573.2.patch.txt, HIVE-2573.3.patch.txt, > HIVE-2573.4.patch.txt, HIVE-2573.5.patch, HIVE-2573.6.patch, > HIVE-2573.7.patch, HIVE-2573.8.patch.txt, HIVE-2573.9.patch.txt > > > Currently the function registry is shared resource and could be overrided by > other users when using HiveServer. If per-session function registry is > provided, this situation could be prevented. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13561) HiveServer2 is leaking ClassLoaders when add jar / temporary functions are used
[ https://issues.apache.org/jira/browse/HIVE-13561?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Trystan Leftwich updated HIVE-13561: Attachment: HIVE-13561-branch-1.2.3.patch > HiveServer2 is leaking ClassLoaders when add jar / temporary functions are > used > --- > > Key: HIVE-13561 > URL: https://issues.apache.org/jira/browse/HIVE-13561 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Affects Versions: 1.2.0, 1.2.1, 2.0.0 >Reporter: Trystan Leftwich >Assignee: Trystan Leftwich > Attachments: HIVE-13561-branch-1.2.3.patch, HIVE-13561.2.patch > > > I can repo this on branch-1.2 and branch-2.0. > It looks to be the same issues as: HIVE-11408 > The patch from HIVE-11408 looks to fix the issue as well. > I've updated the patch from HIVE-11408 to be aligned with branch-1.2 and > master -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13653) improve config error messages for LLAP cache size/etc
[ https://issues.apache.org/jira/browse/HIVE-13653?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-13653: Attachment: HIVE-13653.patch [~sseth] [~prasanth_j] can you please take a look > improve config error messages for LLAP cache size/etc > - > > Key: HIVE-13653 > URL: https://issues.apache.org/jira/browse/HIVE-13653 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-13653.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9422) LLAP: row-level vectorized SARGs
[ https://issues.apache.org/jira/browse/HIVE-9422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15264683#comment-15264683 ] Sergey Shelukhin commented on HIVE-9422: [~prasanth_j] do you want to review this? I took a look a little bit but I'm not very familiar with SARG stuff. Also perhaps you can recommend if this can be done earlier in decoding pathway, or with less objects > LLAP: row-level vectorized SARGs > > > Key: HIVE-9422 > URL: https://issues.apache.org/jira/browse/HIVE-9422 > Project: Hive > Issue Type: Sub-task > Components: llap >Reporter: Sergey Shelukhin >Assignee: Yohei Abe > Attachments: HIVE-9422.2.patch, HIVE-9422.WIP1.patch > > > When VRBs are built from encoded data, sargs can be applied on low level to > reduce the number of rows to process. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-4924) JDBC: Support query timeout for jdbc
[
https://issues.apache.org/jira/browse/HIVE-4924?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vaibhav Gumashta updated HIVE-4924:
---
Attachment: HIVE-4924.5.patch
> JDBC: Support query timeout for jdbc
>
>
> Key: HIVE-4924
> URL: https://issues.apache.org/jira/browse/HIVE-4924
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2, JDBC
>Affects Versions: 2.0.0
>Reporter: Navis
>Assignee: Vaibhav Gumashta
> Attachments: HIVE-4924.3.patch.txt, HIVE-4924.4.patch,
> HIVE-4924.5.patch, HIVE-4924.D11787.1.patch, HIVE-4924.D11787.2.patch
>
>
> support
> {code}
> Statement.setQueryTimeout(int timeout)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13380) Decimal should have lower precedence than double in type hierachy
[ https://issues.apache.org/jira/browse/HIVE-13380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15264642#comment-15264642 ] Thejas M Nair commented on HIVE-13380: -- [~leftylev] Yes, I think this needs documentation. [~ashutoshc] Can you also please update the release note to include an example ? Otherwise, the implications are not clear for the end user. > Decimal should have lower precedence than double in type hierachy > - > > Key: HIVE-13380 > URL: https://issues.apache.org/jira/browse/HIVE-13380 > Project: Hive > Issue Type: Bug > Components: Types >Reporter: Ashutosh Chauhan >Assignee: Ashutosh Chauhan > Fix For: 2.1.0 > > Attachments: HIVE-13380.2.patch, HIVE-13380.4.patch, > HIVE-13380.5.patch, HIVE-13380.patch > > > Currently its other way round. Also, decimal should be lower than float. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HIVE-10438) HiveServer2: Enable ResultSet compression for HS2/*DBC driver
[ https://issues.apache.org/jira/browse/HIVE-10438?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kevin Liew reassigned HIVE-10438: - Assignee: Kevin Liew (was: Rohit Dholakia) > HiveServer2: Enable ResultSet compression for HS2/*DBC driver > - > > Key: HIVE-10438 > URL: https://issues.apache.org/jira/browse/HIVE-10438 > Project: Hive > Issue Type: Sub-task > Components: Hive, Thrift API >Affects Versions: 1.2.0 >Reporter: Rohit Dholakia >Assignee: Kevin Liew > Labels: patch > Attachments: HIVE-10438-1.patch, HIVE-10438.patch, > Proposal-rscompressor.pdf, README.txt, > Results_Snappy_protobuf_TBinary_TCompact.pdf, hs2ResultSetCompressor.zip, > hs2driver-master.zip > > > This JIRA proposes an architecture for enabling ResultSet compression which > uses an external plugin. > The patch has three aspects to it: > 0. An architecture for enabling ResultSet compression with external plugins > 1. An example plugin to demonstrate end-to-end functionality > 2. A container to allow everyone to write and test ResultSet compressors with > a query submitter (https://github.com/xiaom/hs2driver) > Also attaching a design document explaining the changes, experimental results > document, and a pdf explaining how to setup the docker container to observe > end-to-end functionality of ResultSet compression. > https://reviews.apache.org/r/35792/ Review board link. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12963) LIMIT statement with SORT BY creates additional MR job with hardcoded only one reducer
[ https://issues.apache.org/jira/browse/HIVE-12963?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-12963: Resolution: Fixed Fix Version/s: 2.1.0 Status: Resolved (was: Patch Available) Committed to master, sorry for the delay. Test result changes with the setting enabled are in the explain plan only... > LIMIT statement with SORT BY creates additional MR job with hardcoded only > one reducer > -- > > Key: HIVE-12963 > URL: https://issues.apache.org/jira/browse/HIVE-12963 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 1.0.0, 1.2.1, 0.13 >Reporter: Alina Abramova >Assignee: Alina Abramova > Fix For: 2.1.0 > > Attachments: HIVE-12963.1.patch, HIVE-12963.2.patch, > HIVE-12963.3.patch, HIVE-12963.4.patch, HIVE-12963.6.patch > > > I execute query: > hive> select age from test1 sort by age.age limit 10; > Total jobs = 2 > Launching Job 1 out of 2 > Number of reduce tasks not specified. Estimated from input data size: 1 > Launching Job 2 out of 2 > Number of reduce tasks determined at compile time: 1 > When I have a large number of rows then the last stage of the job takes a > long time. I think we could allow to user choose number of reducers of last > job or refuse extra MR job. > The same behavior I observed with querie: > hive> create table new_test as select age from test1 group by age.age limit > 10; -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13448) LLAP: check ZK acls for ZKSM and fail if they are too permissive
[ https://issues.apache.org/jira/browse/HIVE-13448?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-13448: Status: Patch Available (was: Open) > LLAP: check ZK acls for ZKSM and fail if they are too permissive > > > Key: HIVE-13448 > URL: https://issues.apache.org/jira/browse/HIVE-13448 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-13448.patch > > > Only the current user should have any access. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13448) LLAP: check ZK acls for ZKSM and fail if they are too permissive
[ https://issues.apache.org/jira/browse/HIVE-13448?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-13448: Attachment: HIVE-13448.patch The patch. [~prasanth_j] can you take a look > LLAP: check ZK acls for ZKSM and fail if they are too permissive > > > Key: HIVE-13448 > URL: https://issues.apache.org/jira/browse/HIVE-13448 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-13448.patch > > > Only the current user should have any access. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13643) Various enhancements / fixes to llap cli tools
[ https://issues.apache.org/jira/browse/HIVE-13643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15264525#comment-15264525 ] Sergey Shelukhin commented on HIVE-13643: - What I mean is, if I try to start a cluster w/o a name on command line, with hardcoded registry, the cluster name may become something like "mynode1.foo.bar,mynode2.foo.bar,mynode3.foo.bar" etc. 1) Will this even work? 2) Is this what we want? > Various enhancements / fixes to llap cli tools > -- > > Key: HIVE-13643 > URL: https://issues.apache.org/jira/browse/HIVE-13643 > Project: Hive > Issue Type: Improvement >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Attachments: HIVE-13643.01.patch, HIVE-13643.02.patch > > > - BUG - llapstatus does not always produce output (PrintWriter.close) > - llapstatus should parse hiveconf > - package llap-cli-log4j2.template > - Log at the start and end of the script to include parameters > - Generate logs to a file different from hive.log > - hive --service llap --name does not propagate the name to the daemons -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HIVE-13643) Various enhancements / fixes to llap cli tools
[ https://issues.apache.org/jira/browse/HIVE-13643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15264525#comment-15264525 ] Sergey Shelukhin edited comment on HIVE-13643 at 4/29/16 6:47 PM: -- What I mean is, if I try to start a cluster w/o a name on command line, with hardcoded registry, the cluster name may become something like "mynode1.foo.bar,mynode2.foo.bar,mynode3.foo.bar" etc. 1) Will this even work in Slider and other places? 2) Is this what we want? was (Author: sershe): What I mean is, if I try to start a cluster w/o a name on command line, with hardcoded registry, the cluster name may become something like "mynode1.foo.bar,mynode2.foo.bar,mynode3.foo.bar" etc. 1) Will this even work? 2) Is this what we want? > Various enhancements / fixes to llap cli tools > -- > > Key: HIVE-13643 > URL: https://issues.apache.org/jira/browse/HIVE-13643 > Project: Hive > Issue Type: Improvement >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Attachments: HIVE-13643.01.patch, HIVE-13643.02.patch > > > - BUG - llapstatus does not always produce output (PrintWriter.close) > - llapstatus should parse hiveconf > - package llap-cli-log4j2.template > - Log at the start and end of the script to include parameters > - Generate logs to a file different from hive.log > - hive --service llap --name does not propagate the name to the daemons -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13643) Various enhancements / fixes to llap cli tools
[ https://issues.apache.org/jira/browse/HIVE-13643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15264517#comment-15264517 ] Siddharth Seth commented on HIVE-13643: --- bq. slider apps can still use hardcoded registry, can't they? the patch doesn't change anything related to this. > Various enhancements / fixes to llap cli tools > -- > > Key: HIVE-13643 > URL: https://issues.apache.org/jira/browse/HIVE-13643 > Project: Hive > Issue Type: Improvement >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Attachments: HIVE-13643.01.patch, HIVE-13643.02.patch > > > - BUG - llapstatus does not always produce output (PrintWriter.close) > - llapstatus should parse hiveconf > - package llap-cli-log4j2.template > - Log at the start and end of the script to include parameters > - Generate logs to a file different from hive.log > - hive --service llap --name does not propagate the name to the daemons -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13512) Make initializing dag ids in TezWork thread safe for parallel compilation
[
https://issues.apache.org/jira/browse/HIVE-13512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15264476#comment-15264476
]
Peter Slawski commented on HIVE-13512:
--
[~gopalv], could you please confirm my above statement regarding the test
failures. I would like to know the next steps I need to take for getting this
patch in. Thank you!
> Make initializing dag ids in TezWork thread safe for parallel compilation
> -
>
> Key: HIVE-13512
> URL: https://issues.apache.org/jira/browse/HIVE-13512
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2, Query Planning
>Affects Versions: 2.0.0
>Reporter: Peter Slawski
>Assignee: Peter Slawski
>Priority: Minor
> Attachments: HIVE-13512.1.patch, HIVE-13512.1.patch
>
>
> When parallel query compilation is enabled, it is possible for concurrent
> running threads to create TezWork objects that have the same dag id. This is
> because the counter used to obtain the next dag id is not thread safe. The
> counter should be an AtomicInteger rather than an int.
> {code:java}
> private static int counter;
> ...
> public TezWork(String queryId, Configuration conf) {
> this.dagId = queryId + ":" + (++counter);
> ...
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-12878) Support Vectorization for TEXTFILE and other formats
[
https://issues.apache.org/jira/browse/HIVE-12878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15264470#comment-15264470
]
Hive QA commented on HIVE-12878:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12801019/HIVE-12878.092.patch
{color:green}SUCCESS:{color} +1 due to 24 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 64 failed/errored test(s), 1 tests
executed
*Failed tests:*
{noformat}
TestHWISessionManager - did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_nomore_ambiguous_table_col
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_regexp_extract
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_bucket4
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_bucket5
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_bucket6
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_disable_merge_for_bucketing
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_infer_bucket_sort_map_operators
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_infer_bucket_sort_num_buckets
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_infer_bucket_sort_reducers_power_two
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_list_bucket_dml_10
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_orc_merge1
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_orc_merge2
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_orc_merge9
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_orc_merge_diff_fs
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_reduce_deduplicate
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_vector_outer_join1
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_vector_outer_join2
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_vector_outer_join3
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_vector_outer_join4
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_vector_outer_join5
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_clustern3
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_clustern4
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_nonkey_groupby
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_selectDistinctStarNeg_2
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_subquery_shared_alias
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_udtf_not_supported1
org.apache.hadoop.hive.metastore.TestAuthzApiEmbedAuthorizerInRemote.org.apache.hadoop.hive.metastore.TestAuthzApiEmbedAuthorizerInRemote
org.apache.hadoop.hive.metastore.TestFilterHooks.org.apache.hadoop.hive.metastore.TestFilterHooks
org.apache.hadoop.hive.metastore.TestHiveMetaStoreGetMetaConf.testGetMetaConfDefault
org.apache.hadoop.hive.metastore.TestMetaStoreEndFunctionListener.testEndFunctionListener
org.apache.hadoop.hive.metastore.TestMetaStoreEventListenerOnlyOnCommit.testEventStatus
org.apache.hadoop.hive.metastore.TestMetaStoreInitListener.testMetaStoreInitListener
org.apache.hadoop.hive.metastore.TestMetaStoreMetrics.org.apache.hadoop.hive.metastore.TestMetaStoreMetrics
org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAddPartitionWithCommas
org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAddPartitionWithUnicode
org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAddPartitionWithValidPartVal
org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAppendPartitionWithCommas
org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAppendPartitionWithUnicode
org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAppendPartitionWithValidCharacters
org.apache.hadoop.hive.metastore.TestRemoteUGIHiveMetaStoreIpAddress.testIpAddress
org.apache.hadoop.hive.metastore.TestRetryingHMSHandler.testRetryingHMSHandler
org.apache.hadoop.hive.ql.io.orc.TestInputOutputFormat.testVectorization
org.apache.hadoop.hive.ql.io.orc.TestInputOutputFormat.testVectorizationWithAcid
org.apache.hadoop.hive.ql.io.orc.TestInputOutputFormat.testVectorizationWithBuckets
org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager2.insertOverwriteCreate
org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager2.testDummyTxnManagerOnAcidTable
org.apache.hadoop.hive.ql.security.TestClientSideAuthorizationProvider.testSimplePrivileges
[jira] [Updated] (HIVE-13596) HS2 should be able to get UDFs on demand from metastore
[ https://issues.apache.org/jira/browse/HIVE-13596?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-13596: Resolution: Fixed Fix Version/s: 2.1.0 Status: Resolved (was: Patch Available) Committed to master - thanks for the review! > HS2 should be able to get UDFs on demand from metastore > --- > > Key: HIVE-13596 > URL: https://issues.apache.org/jira/browse/HIVE-13596 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Fix For: 2.1.0 > > Attachments: HIVE-13596.01.patch, HIVE-13596.02.patch, > HIVE-13596.03.patch, HIVE-13596.patch > > > When multiple HS2s are run, creating a permanent fn is only executed on one > of them, and the other HS2s don't get the new function. Unlike say with > tables, where we always get stuff from db on demand, fns are registered at > certain points in the code and if the new one is not registered, it will not > be available. > We should restore the pre-HIVE-2573 behavior of being able to refresh the > UDFs on demand. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13343) Need to disable hybrid grace hash join in llap mode except for dynamically partitioned hash join
[ https://issues.apache.org/jira/browse/HIVE-13343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15264447#comment-15264447 ] Wei Zheng commented on HIVE-13343: -- [~vikram.dixit] In patch 4 I didn't see LLAP_ENABLE_GRACE_JOIN_IN_LLAP being used anywhere. > Need to disable hybrid grace hash join in llap mode except for dynamically > partitioned hash join > > > Key: HIVE-13343 > URL: https://issues.apache.org/jira/browse/HIVE-13343 > Project: Hive > Issue Type: Bug > Components: llap >Affects Versions: 2.1.0 >Reporter: Vikram Dixit K >Assignee: Vikram Dixit K > Attachments: HIVE-13343.1.patch, HIVE-13343.2.patch, > HIVE-13343.3.patch, HIVE-13343.4.patch > > > Due to performance reasons, we should disable use of hybrid grace hash join > in llap when dynamic partition hash join is not used. With dynamic partition > hash join, we need hybrid grace hash join due to the possibility of skews. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13447) LLAP: check ZK acls for registry and fail if they are too permissive
[
https://issues.apache.org/jira/browse/HIVE-13447?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Shelukhin updated HIVE-13447:
Resolution: Fixed
Fix Version/s: 2.1.0
Status: Resolved (was: Patch Available)
Committed to master. Thanks for the review!
> LLAP: check ZK acls for registry and fail if they are too permissive
>
>
> Key: HIVE-13447
> URL: https://issues.apache.org/jira/browse/HIVE-13447
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Fix For: 2.1.0
>
> Attachments: HIVE-13447.01.patch, HIVE-13447.patch
>
>
> Only the current ("hive") user can have write access.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13596) HS2 should be able to get UDFs on demand from metastore
[ https://issues.apache.org/jira/browse/HIVE-13596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15264438#comment-15264438 ] Sergey Shelukhin commented on HIVE-13596: - Test failures are the same as in other recent JIRAs... metastore problems on the server, or broken by other jiras > HS2 should be able to get UDFs on demand from metastore > --- > > Key: HIVE-13596 > URL: https://issues.apache.org/jira/browse/HIVE-13596 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-13596.01.patch, HIVE-13596.02.patch, > HIVE-13596.03.patch, HIVE-13596.patch > > > When multiple HS2s are run, creating a permanent fn is only executed on one > of them, and the other HS2s don't get the new function. Unlike say with > tables, where we always get stuff from db on demand, fns are registered at > certain points in the code and if the new one is not registered, it will not > be available. > We should restore the pre-HIVE-2573 behavior of being able to refresh the > UDFs on demand. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9660) store end offset of compressed data for RG in RowIndex in ORC
[ https://issues.apache.org/jira/browse/HIVE-9660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15264416#comment-15264416 ] Sergey Shelukhin commented on HIVE-9660: [~prasanth_j] this is now ready for +1 :) > store end offset of compressed data for RG in RowIndex in ORC > - > > Key: HIVE-9660 > URL: https://issues.apache.org/jira/browse/HIVE-9660 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-9660.01.patch, HIVE-9660.02.patch, > HIVE-9660.03.patch, HIVE-9660.04.patch, HIVE-9660.05.patch, > HIVE-9660.06.patch, HIVE-9660.07.patch, HIVE-9660.07.patch, > HIVE-9660.08.patch, HIVE-9660.09.patch, HIVE-9660.10.patch, > HIVE-9660.10.patch, HIVE-9660.11.patch, HIVE-9660.patch, HIVE-9660.patch > > > Right now the end offset is estimated, which in some cases results in tons of > extra data being read. > We can add a separate array to RowIndex (positions_v2?) that stores number of > compressed buffers for each RG, or end offset, or something, to remove this > estimation magic -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9660) store end offset of compressed data for RG in RowIndex in ORC
[ https://issues.apache.org/jira/browse/HIVE-9660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15264415#comment-15264415 ] Sergey Shelukhin commented on HIVE-9660: Some test failures are caused by metastore issues, and some are broken by other jiras it appears > store end offset of compressed data for RG in RowIndex in ORC > - > > Key: HIVE-9660 > URL: https://issues.apache.org/jira/browse/HIVE-9660 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-9660.01.patch, HIVE-9660.02.patch, > HIVE-9660.03.patch, HIVE-9660.04.patch, HIVE-9660.05.patch, > HIVE-9660.06.patch, HIVE-9660.07.patch, HIVE-9660.07.patch, > HIVE-9660.08.patch, HIVE-9660.09.patch, HIVE-9660.10.patch, > HIVE-9660.10.patch, HIVE-9660.11.patch, HIVE-9660.patch, HIVE-9660.patch > > > Right now the end offset is estimated, which in some cases results in tons of > extra data being read. > We can add a separate array to RowIndex (positions_v2?) that stores number of > compressed buffers for each RG, or end offset, or something, to remove this > estimation magic -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13643) Various enhancements / fixes to llap cli tools
[ https://issues.apache.org/jira/browse/HIVE-13643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15264375#comment-15264375 ] Sergey Shelukhin commented on HIVE-13643: - slider apps can still use hardcoded registry, can't they? Otherwise looks good > Various enhancements / fixes to llap cli tools > -- > > Key: HIVE-13643 > URL: https://issues.apache.org/jira/browse/HIVE-13643 > Project: Hive > Issue Type: Improvement >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Attachments: HIVE-13643.01.patch, HIVE-13643.02.patch > > > - BUG - llapstatus does not always produce output (PrintWriter.close) > - llapstatus should parse hiveconf > - package llap-cli-log4j2.template > - Log at the start and end of the script to include parameters > - Generate logs to a file different from hive.log > - hive --service llap --name does not propagate the name to the daemons -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13343) Need to disable hybrid grace hash join in llap mode except for dynamically partitioned hash join
[
https://issues.apache.org/jira/browse/HIVE-13343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15264267#comment-15264267
]
Hive QA commented on HIVE-13343:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12801089/HIVE-13343.4.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 68 failed/errored test(s), 9882 tests
executed
*Failed tests:*
{noformat}
TestHWISessionManager - did not produce a TEST-*.xml file
TestMiniTezCliDriver-auto_join30.q-script_pipe.q-vector_decimal_10_0.q-and-12-more
- did not produce a TEST-*.xml file
TestMiniTezCliDriver-cbo_windowing.q-tez_join.q-bucket_map_join_tez1.q-and-12-more
- did not produce a TEST-*.xml file
TestMiniTezCliDriver-insert_values_non_partitioned.q-schema_evol_orc_nonvec_mapwork_part.q-union5.q-and-12-more
- did not produce a TEST-*.xml file
TestMiniTezCliDriver-vector_grouping_sets.q-mapjoin_mapjoin.q-update_all_partitioned.q-and-12-more
- did not produce a TEST-*.xml file
TestMiniTezCliDriver-vector_interval_2.q-vectorization_10.q-orc_merge5.q-and-12-more
- did not produce a TEST-*.xml file
TestMiniTezCliDriver-vector_non_string_partition.q-delete_where_non_partitioned.q-auto_sortmerge_join_16.q-and-12-more
- did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_nomore_ambiguous_table_col
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_regexp_extract
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_hybridgrace_hashjoin_1
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_bucket4
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_bucket5
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_bucket6
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_disable_merge_for_bucketing
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_infer_bucket_sort_map_operators
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_infer_bucket_sort_num_buckets
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_infer_bucket_sort_reducers_power_two
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_list_bucket_dml_10
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_orc_merge1
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_orc_merge2
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_orc_merge9
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_orc_merge_diff_fs
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_reduce_deduplicate
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_vector_outer_join1
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_vector_outer_join2
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_vector_outer_join3
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_vector_outer_join4
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_vector_outer_join5
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.org.apache.hadoop.hive.cli.TestMiniTezCliDriver
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_dynamic_partition_pruning
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_hybridgrace_hashjoin_1
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_cast_constant
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_clustern3
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_clustern4
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_nonkey_groupby
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_selectDistinctStarNeg_2
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_subquery_shared_alias
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_udtf_not_supported1
org.apache.hadoop.hive.metastore.TestAuthzApiEmbedAuthorizerInRemote.org.apache.hadoop.hive.metastore.TestAuthzApiEmbedAuthorizerInRemote
org.apache.hadoop.hive.metastore.TestFilterHooks.org.apache.hadoop.hive.metastore.TestFilterHooks
org.apache.hadoop.hive.metastore.TestHiveMetaStoreGetMetaConf.testGetMetaConfDefault
org.apache.hadoop.hive.metastore.TestHiveMetaStoreGetMetaConf.testGetMetaConfDefaultEmptyString
org.apache.hadoop.hive.metastore.TestHiveMetaStoreGetMetaConf.testGetMetaConfOverridden
org.apache.hadoop.hive.metastore.TestHiveMetaStoreGetMetaConf.testGetMetaConfUnknownPreperty
org.apache.hadoop.hive.metastore.TestMetaStoreEndFunctionListener.testEndFunctionListener
org.apache.hadoop.hive.metastore.TestMetaStoreEventListenerOnlyOnCommit.testEventStatus
[jira] [Commented] (HIVE-13509) HCatalog getSplits should ignore the partition with invalid path
[
https://issues.apache.org/jira/browse/HIVE-13509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15264026#comment-15264026
]
Chaoyu Tang commented on HIVE-13509:
Thanks, [~leftylev]
> HCatalog getSplits should ignore the partition with invalid path
>
>
> Key: HIVE-13509
> URL: https://issues.apache.org/jira/browse/HIVE-13509
> Project: Hive
> Issue Type: Improvement
> Components: HCatalog
>Reporter: Chaoyu Tang
>Assignee: Chaoyu Tang
> Fix For: 2.1.0
>
> Attachments: HIVE-13509.1.patch, HIVE-13509.2.patch, HIVE-13509.patch
>
>
> It is quite common that there is the discrepancy between partition directory
> and its HMS metadata, simply because the directory could be added/deleted
> externally using hdfs shell command. Technically it should be fixed by MSCK
> and alter table .. add/drop command etc, but sometimes it might not be
> practical especially in a multi-tenant env. This discrepancy does not cause
> any problem to Hive, Hive returns no rows for a partition with an invalid
> (e.g. non-existing) path, but it fails the Pig load with HCatLoader, because
> the HCatBaseInputFormat getSplits throws an error when getting a split for a
> non-existing path. The error message might looks like:
> {code}
> Caused by: org.apache.hadoop.mapred.InvalidInputException: Input path does
> not exist:
> hdfs://xyz.com:8020/user/hive/warehouse/xyz/date=2016-01-01/country=BR
> at
> org.apache.hadoop.mapred.FileInputFormat.singleThreadedListStatus(FileInputFormat.java:287)
> at
> org.apache.hadoop.mapred.FileInputFormat.listStatus(FileInputFormat.java:229)
> at
> org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:315)
> at
> org.apache.hive.hcatalog.mapreduce.HCatBaseInputFormat.getSplits(HCatBaseInputFormat.java:162)
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.getSplits(PigInputFormat.java:274)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13621) compute stats in certain cases fails with NPE
[
https://issues.apache.org/jira/browse/HIVE-13621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15264025#comment-15264025
]
Hive QA commented on HIVE-13621:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12800922/HIVE-13621.1.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 17 failed/errored test(s), 10018 tests
executed
*Failed tests:*
{noformat}
TestHWISessionManager - did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_deleteAnalyze
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_nomore_ambiguous_table_col
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_regexp_extract
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_clustern3
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_clustern4
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_nonkey_groupby
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_selectDistinctStarNeg_2
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_subquery_shared_alias
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_udtf_not_supported1
org.apache.hadoop.hive.metastore.TestAuthzApiEmbedAuthorizerInRemote.org.apache.hadoop.hive.metastore.TestAuthzApiEmbedAuthorizerInRemote
org.apache.hadoop.hive.metastore.TestFilterHooks.org.apache.hadoop.hive.metastore.TestFilterHooks
org.apache.hadoop.hive.metastore.TestHiveMetaStoreGetMetaConf.testGetMetaConfDefault
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationProviderWithACL.testSimplePrivileges
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationReads.testReadTableFailure
org.apache.hive.hcatalog.listener.TestDbNotificationListener.dropDatabase
{noformat}
Test results:
http://ec2-54-177-240-2.us-west-1.compute.amazonaws.com/job/PreCommit-HIVE-MASTER-Build/118/testReport
Console output:
http://ec2-54-177-240-2.us-west-1.compute.amazonaws.com/job/PreCommit-HIVE-MASTER-Build/118/console
Test logs:
http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-118/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 17 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12800922 - PreCommit-HIVE-MASTER-Build
> compute stats in certain cases fails with NPE
> -
>
> Key: HIVE-13621
> URL: https://issues.apache.org/jira/browse/HIVE-13621
> Project: Hive
> Issue Type: Bug
> Components: HBase Metastore, Metastore
>Affects Versions: 2.1.0, 2.0.1
>Reporter: Vikram Dixit K
>Assignee: Vikram Dixit K
> Attachments: HIVE-13621.1.patch
>
>
> {code}
> FAILED: NullPointerException null
> java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.stats.StatsUtils.getColStatistics(StatsUtils.java:693)
> at
> org.apache.hadoop.hive.ql.stats.StatsUtils.convertColStats(StatsUtils.java:739)
> at
> org.apache.hadoop.hive.ql.stats.StatsUtils.getTableColumnStats(StatsUtils.java:728)
> at
> org.apache.hadoop.hive.ql.stats.StatsUtils.collectStatistics(StatsUtils.java:183)
> at
> org.apache.hadoop.hive.ql.stats.StatsUtils.collectStatistics(StatsUtils.java:136)
> at
> org.apache.hadoop.hive.ql.stats.StatsUtils.collectStatistics(StatsUtils.java:124){code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-10502) Cannot specify log4j.properties file location in Beeline
[
https://issues.apache.org/jira/browse/HIVE-10502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15264009#comment-15264009
]
Matyas Orhidi commented on HIVE-10502:
--
Here is an option that worked for me:
1) Created a log4j config in a folder. (should be beeline-log4j.properties
exactly)
/root/mypreferredclasses/beeline-log4j.properties
--
log4j.rootLogger=TRACE, console
console appender
log4j.appender.console=org.apache.log4j.ConsoleAppender
log4j.appender.console.target=System.err
log4j.appender.console.layout=org.apache.log4j.PatternLayout
log4j.appender.console.layout.ConversionPattern=### magic here ### %d{yy/MM/dd
HH:mm:ss} [%t]: %p %c{2}: %m%n
log4j.appender.console.encoding=UTF-8
2) Called the beeline command by adding this folder into the beginning of the
beeline classpath:
export HADOOP_USER_CLASSPATH_FIRST=true; export
HADOOP_CLASSPATH=/root/mypreferredclasses; beeline -u
jdbc:hive2://nosuchhost:1
scan complete in 2ms
Connecting to jdbc:hive2://nosuchhost:1
### magic here ### 16/04/29 05:44:42 [main]: INFO jdbc.Utils: Supplied
authorities: nosuchhost:1
### magic here ### 16/04/29 05:44:42 [main]: INFO jdbc.Utils: Resolved
authority: nosuchhost:1
### magic here ### 16/04/29 05:44:42 [main]: INFO jdbc.HiveConnection: Will try
to open client transport with JDBC Uri: jdbc:hive2://nosuchhost:1
### magic here ### 16/04/29 05:44:42 [main]: DEBUG transport.TSaslTransport:
opening transport org.apache.thrift.transport.TSaslClientTransport@2977cbf7
### magic here ### 16/04/29 05:44:42 [main]: INFO jdbc.HiveConnection: Could
not open client transport with JDBC Uri: jdbc:hive2://nosuchhost:1
Error: Could not open client transport with JDBC Uri:
jdbc:hive2://nosuchhost:1: java.net.UnknownHostException: nosuchhost
(state=08S01,code=0)
Beeline version 1.1.0-cdh5.4.5 by Apache Hive
0: jdbc:hive2://nosuchhost:1 (closed)>
> Cannot specify log4j.properties file location in Beeline
>
>
> Key: HIVE-10502
> URL: https://issues.apache.org/jira/browse/HIVE-10502
> Project: Hive
> Issue Type: Bug
> Components: Beeline
>Affects Versions: 1.1.0
>Reporter: Szehon Ho
>Assignee: Chaoyu Tang
>
> In HiveCLI, HiveServer2, HMS, etc, the following is called early in the
> startup to initialize log4j logging: LogUtils.initHiveLog4j().
> However, seems like this is not the case in Beeline, which also needs log4j
> like as follows:
> {noformat}
> at org.apache.log4j.LogManager.(LogManager.java:127)
> at org.slf4j.impl.Log4jLoggerFactory.getLogger(Log4jLoggerFactory.java:66)
> at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:270)
> at
> org.apache.commons.logging.impl.SLF4JLogFactory.getInstance(SLF4JLogFactory.java:156)
> at
> org.apache.commons.logging.impl.SLF4JLogFactory.getInstance(SLF4JLogFactory.java:132)
> at org.apache.commons.logging.LogFactory.getLog(LogFactory.java:657)
> at org.apache.hadoop.util.VersionInfo.(VersionInfo.java:37)
> {noformat}
> It would be good to specify it, so it doesn't pick the first one in the
> classpath.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13652) Import table change order of dynamic partitions
[
https://issues.apache.org/jira/browse/HIVE-13652?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lukas Waldmann updated HIVE-13652:
--
Description:
Table with multiple dynamic partitions like year,month, day exported using
"export table" command is imported (using "import table") such a way that order
of partitions is changed to day, month, year.
Export DB: Hive 0.14
Import DB: Hive 1.2.1000.2.4.0.0-169
Tables created as:
create table T1
( ... ) PARTITIONED BY (period_year string, period_month string, period_day
string) STORED AS ORC TBLPROPERTIES ("orc.compress"="SNAPPY");
export command:
export table t1 to 'path'
import command:
import table t1 from 'path'
HDFS file structure on both original table location and export path keeps the
original partition order ../year/month/day
HDFS file structure after import is .../day/month/year
was:
Table with multiple dynamic partitions like year,month, day exported using
"export table" command is imported (using "import table") such a way that order
of partitions is changed to day, month, year.
Exported table structure on hdfs keeps the original partition order
../year/month/day while imported table has hdfs structure .../day/month/year
Export DB: Hive 0.14
Import DB: Hive 1.2.1000.2.4.0.0-169
Tables created as:
create table T1
( ... ) PARTITIONED BY (period_year string, period_month string, period_day
string) STORED AS ORC TBLPROPERTIES ("orc.compress"="SNAPPY");
export command:
export table t1 to 'path'
import command:
import table t1 from 'path'
> Import table change order of dynamic partitions
> ---
>
> Key: HIVE-13652
> URL: https://issues.apache.org/jira/browse/HIVE-13652
> Project: Hive
> Issue Type: Bug
>Affects Versions: 1.2.0, 1.2.1
>Reporter: Lukas Waldmann
>
> Table with multiple dynamic partitions like year,month, day exported using
> "export table" command is imported (using "import table") such a way that
> order of partitions is changed to day, month, year.
> Export DB: Hive 0.14
> Import DB: Hive 1.2.1000.2.4.0.0-169
> Tables created as:
> create table T1
> ( ... ) PARTITIONED BY (period_year string, period_month string, period_day
> string) STORED AS ORC TBLPROPERTIES ("orc.compress"="SNAPPY");
> export command:
> export table t1 to 'path'
> import command:
> import table t1 from 'path'
> HDFS file structure on both original table location and export path keeps the
> original partition order ../year/month/day
> HDFS file structure after import is .../day/month/year
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13652) Import table change order of dynamic partitions
[
https://issues.apache.org/jira/browse/HIVE-13652?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lukas Waldmann updated HIVE-13652:
--
Description:
Table with multiple dynamic partitions like year,month, day exported using
"export table" command is imported (using "import table") such a way that order
of partitions is changed to day, month, year.
Exported table structure on hdfs keeps the original partition order
../year/month/day while imported table has hdfs structure .../day/month/year
Export DB: Hive 0.14
Import DB: Hive 1.2.1000.2.4.0.0-169
Tables created as:
create table T1
( ... ) PARTITIONED BY (period_year string, period_month string, period_day
string) STORED AS ORC TBLPROPERTIES ("orc.compress"="SNAPPY");
export command:
export table t1 to 'path'
import command:
import table t1 from 'path'
was:
Table with multiple dynamic partitions like year,month, day exported using
"export table" command is imported (using "import table") such a way that order
of partitions is changed to day, month, year.
Exported table structure on hdfs keeps the original partition order
../year/month/day while imported table has hdfs structure .../day/month/year
Export DB: Hive 0.14
Import DB: Hive 1.2.1000.2.4.0.0-169
Tables created as:
create table T1
( ... ) PARTITIONED BY (period_year string, period_month string, period_day
string) STORED AS ORC TBLPROPERTIES ("orc.compress"="SNAPPY");
export command:
export table t1
> Import table change order of dynamic partitions
> ---
>
> Key: HIVE-13652
> URL: https://issues.apache.org/jira/browse/HIVE-13652
> Project: Hive
> Issue Type: Bug
>Affects Versions: 1.2.0, 1.2.1
>Reporter: Lukas Waldmann
>
> Table with multiple dynamic partitions like year,month, day exported using
> "export table" command is imported (using "import table") such a way that
> order of partitions is changed to day, month, year.
> Exported table structure on hdfs keeps the original partition order
> ../year/month/day while imported table has hdfs structure .../day/month/year
> Export DB: Hive 0.14
> Import DB: Hive 1.2.1000.2.4.0.0-169
> Tables created as:
> create table T1
> ( ... ) PARTITIONED BY (period_year string, period_month string, period_day
> string) STORED AS ORC TBLPROPERTIES ("orc.compress"="SNAPPY");
> export command:
> export table t1 to 'path'
> import command:
> import table t1 from 'path'
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13652) Import table change order of dynamic partitions
[
https://issues.apache.org/jira/browse/HIVE-13652?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lukas Waldmann updated HIVE-13652:
--
Description:
Table with multiple dynamic partitions like year,month, day exported using
"export table" command is imported (using "import table") such a way that order
of partitions is changed to day, month, year.
Exported table structure on hdfs keeps the original partition order
../year/month/day while imported table has hdfs structure .../day/month/year
Export DB: Hive 0.14
Import DB: Hive 1.2.1000.2.4.0.0-169
Tables created as:
create table T1
( ... ) PARTITIONED BY (period_year string, period_month string, period_day
string) STORED AS ORC TBLPROPERTIES ("orc.compress"="SNAPPY");
export command:
export table t1
was:
Table with multiple dynamic partitions like year,month, day exported using
"export table" command is imported (using "import table") such a way that order
of partitions is changed to day, month, year.
Exported table structure on hdfs keeps the original partition order
../year/month/day while imported table has hdfs structure .../day/month/year
> Import table change order of dynamic partitions
> ---
>
> Key: HIVE-13652
> URL: https://issues.apache.org/jira/browse/HIVE-13652
> Project: Hive
> Issue Type: Bug
>Affects Versions: 1.2.0, 1.2.1
>Reporter: Lukas Waldmann
>
> Table with multiple dynamic partitions like year,month, day exported using
> "export table" command is imported (using "import table") such a way that
> order of partitions is changed to day, month, year.
> Exported table structure on hdfs keeps the original partition order
> ../year/month/day while imported table has hdfs structure .../day/month/year
> Export DB: Hive 0.14
> Import DB: Hive 1.2.1000.2.4.0.0-169
> Tables created as:
> create table T1
> ( ... ) PARTITIONED BY (period_year string, period_month string, period_day
> string) STORED AS ORC TBLPROPERTIES ("orc.compress"="SNAPPY");
> export command:
> export table t1
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13525) HoS hangs when job is empty
[
https://issues.apache.org/jira/browse/HIVE-13525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rui Li updated HIVE-13525:
--
Attachment: HIVE-13525.3.patch
Update patch to solve the metrics lost problem.
Due to SPARK-14958, it's still possible the job can hang if task fails. What we
can do on our side to mitigate it is:
# Avoid task failure.
# Avoid deserialization error that triggers SPARK-14958.
For 1, the v3 patch makes HoS able to handle empty small table for map join.
For 2, the error that triggers SPARK-14958 is {{NoClassDefFoundError:
org/antlr/runtime/tree/CommonTree}}. So the v3 patch adds {{antlr-runtime}} to
driver's class path.
With the patch all the disabled tests can run (but may fail). I'll fix and
re-enable them in a follow on JIRA.
> HoS hangs when job is empty
> ---
>
> Key: HIVE-13525
> URL: https://issues.apache.org/jira/browse/HIVE-13525
> Project: Hive
> Issue Type: Bug
>Reporter: Rui Li
>Assignee: Rui Li
> Attachments: HIVE-13525.1.patch, HIVE-13525.2.patch,
> HIVE-13525.3.patch
>
>
> Observed in local tests. This should be the cause of HIVE-13402.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13421) Propagate job progress in operation status
[ https://issues.apache.org/jira/browse/HIVE-13421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15263935#comment-15263935 ] Rajat Khandelwal commented on HIVE-13421: - Taking patch from reviewboard and attaching > Propagate job progress in operation status > -- > > Key: HIVE-13421 > URL: https://issues.apache.org/jira/browse/HIVE-13421 > Project: Hive > Issue Type: Improvement >Reporter: Rajat Khandelwal >Assignee: Rajat Khandelwal > Fix For: 2.1.0 > > Attachments: HIVE-13421.01.patch, HIVE-13421.02.patch, > HIVE-13421.03.patch, HIVE-13421.04.patch, HIVE-13421.05.patch, > HIVE-13421.06.patch, HIVE-13421.07.patch, HIVE-13421.08.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13421) Propagate job progress in operation status
[ https://issues.apache.org/jira/browse/HIVE-13421?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajat Khandelwal updated HIVE-13421: Attachment: HIVE-13421.08.patch > Propagate job progress in operation status > -- > > Key: HIVE-13421 > URL: https://issues.apache.org/jira/browse/HIVE-13421 > Project: Hive > Issue Type: Improvement >Reporter: Rajat Khandelwal >Assignee: Rajat Khandelwal > Fix For: 2.1.0 > > Attachments: HIVE-13421.01.patch, HIVE-13421.02.patch, > HIVE-13421.03.patch, HIVE-13421.04.patch, HIVE-13421.05.patch, > HIVE-13421.06.patch, HIVE-13421.07.patch, HIVE-13421.08.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9660) store end offset of compressed data for RG in RowIndex in ORC
[
https://issues.apache.org/jira/browse/HIVE-9660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15263918#comment-15263918
]
Hive QA commented on HIVE-9660:
---
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12800915/HIVE-9660.11.patch
{color:green}SUCCESS:{color} +1 due to 12 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 53 failed/errored test(s), 9894 tests
executed
*Failed tests:*
{noformat}
TestHBaseAggrStatsCacheIntegration - did not produce a TEST-*.xml file
TestHWISessionManager - did not produce a TEST-*.xml file
TestMiniTezCliDriver-auto_join30.q-script_pipe.q-vector_decimal_10_0.q-and-12-more
- did not produce a TEST-*.xml file
TestMiniTezCliDriver-explainuser_4.q-update_after_multiple_inserts.q-mapreduce2.q-and-12-more
- did not produce a TEST-*.xml file
TestMiniTezCliDriver-order_null.q-vector_acid3.q-orc_merge10.q-and-12-more -
did not produce a TEST-*.xml file
TestMiniTezCliDriver-vector_distinct_2.q-tez_joins_explain.q-cte_mat_1.q-and-12-more
- did not produce a TEST-*.xml file
TestMiniTezCliDriver-vector_varchar_4.q-smb_cache.q-tez_join_hash.q-and-8-more
- did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_extrapolate_part_stats_partial_ndv
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_nomore_ambiguous_table_col
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_regexp_extract
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_bucket4
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_bucket6
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_disable_merge_for_bucketing
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_infer_bucket_sort_num_buckets
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_orc_merge9
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_vector_outer_join1
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_vector_outer_join2
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_vector_outer_join3
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_vector_outer_join4
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_vector_outer_join5
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_clustern3
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_clustern4
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_nonkey_groupby
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_selectDistinctStarNeg_2
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_subquery_shared_alias
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_udtf_not_supported1
org.apache.hadoop.hive.metastore.TestAuthzApiEmbedAuthorizerInRemote.org.apache.hadoop.hive.metastore.TestAuthzApiEmbedAuthorizerInRemote
org.apache.hadoop.hive.metastore.TestFilterHooks.org.apache.hadoop.hive.metastore.TestFilterHooks
org.apache.hadoop.hive.metastore.TestHiveMetaStorePartitionSpecs.testAddPartitions
org.apache.hadoop.hive.metastore.TestHiveMetaStorePartitionSpecs.testFetchingPartitionsWithDifferentSchemas
org.apache.hadoop.hive.metastore.TestHiveMetaStorePartitionSpecs.testGetPartitionSpecs_WithAndWithoutPartitionGrouping
org.apache.hadoop.hive.metastore.TestMetaStoreEndFunctionListener.testEndFunctionListener
org.apache.hadoop.hive.metastore.TestMetaStoreEventListenerOnlyOnCommit.testEventStatus
org.apache.hadoop.hive.metastore.TestMetaStoreInitListener.testMetaStoreInitListener
org.apache.hadoop.hive.metastore.TestMetaStoreMetrics.org.apache.hadoop.hive.metastore.TestMetaStoreMetrics
org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAppendPartitionWithCommas
org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAppendPartitionWithValidCharacters
org.apache.hadoop.hive.metastore.TestRetryingHMSHandler.testRetryingHMSHandler
org.apache.hadoop.hive.ql.security.TestClientSideAuthorizationProvider.testSimplePrivileges
org.apache.hadoop.hive.ql.security.TestExtendedAcls.org.apache.hadoop.hive.ql.security.TestExtendedAcls
org.apache.hadoop.hive.ql.security.TestFolderPermissions.org.apache.hadoop.hive.ql.security.TestFolderPermissions
org.apache.hadoop.hive.ql.security.TestMultiAuthorizationPreEventListener.org.apache.hadoop.hive.ql.security.TestMultiAuthorizationPreEventListener
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationDrops.testDropDatabase
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationDrops.testDropPartition
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationProvider.testSimplePrivileges
[jira] [Updated] (HIVE-13510) Dynamic partitioning doesn’t work when remote metastore is used
[
https://issues.apache.org/jira/browse/HIVE-13510?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashutosh Chauhan updated HIVE-13510:
Resolution: Fixed
Fix Version/s: 2.1.0
Status: Resolved (was: Patch Available)
Pushed to master. Thanks, [~yalovyyi]
> Dynamic partitioning doesn’t work when remote metastore is used
> ---
>
> Key: HIVE-13510
> URL: https://issues.apache.org/jira/browse/HIVE-13510
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 2.1.0
> Environment: Hadoop 2.7.1
>Reporter: Illya Yalovyy
>Assignee: Illya Yalovyy
>Priority: Critical
> Fix For: 2.1.0
>
> Attachments: HIVE-13510.1.patch, HIVE-13510.2.patch
>
>
> *Steps to reproduce:*
> # Configure remote metastore (hive.metastore.uris)
> # Create table t1 (a string);
> # Create table t2 (a string) partitioned by (b string);
> # set hive.exec.dynamic.partition.mode=nonstrict;
> # Insert overwrite table t2 partition (b) select a,a from t1;
> *Result:*
> {noformat}
> FAILED: SemanticException org.apache.hadoop.hive.ql.metadata.HiveException:
> org.apache.thrift.TApplicationException: getMetaConf failed: unknown result
> 16/04/13 15:04:51 [c679e424-2501-4347-8146-cf1b1cae217c main]: ERROR
> ql.Driver: FAILED: SemanticException
> org.apache.hadoop.hive.ql.metadata.HiveException:
> org.apache.thrift.TApplicationException: getMetaConf failed: unknown result
> org.apache.hadoop.hive.ql.parse.SemanticException:
> org.apache.hadoop.hive.ql.metadata.HiveException:
> org.apache.thrift.TApplicationException: getMetaConf failed: unknown result
> at
> org.apache.hadoop.hive.ql.plan.DynamicPartitionCtx.(DynamicPartitionCtx.java:84)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genFileSinkPlan(SemanticAnalyzer.java:6550)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPostGroupByBodyPlan(SemanticAnalyzer.java:9315)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genBodyPlan(SemanticAnalyzer.java:9204)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:10071)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:9949)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genOPTree(SemanticAnalyzer.java:10607)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.genOPTree(CalcitePlanner.java:358)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:10618)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:233)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:245)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:476)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:318)
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1192)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1287)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1118)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1106)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:236)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:187)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:403)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:339)
> at
> org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:748)
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:721)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:648)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
> org.apache.thrift.TApplicationException: getMetaConf failed: unknown result
> at org.apache.hadoop.hive.ql.metadata.Hive.getMetaConf(Hive.java:3493)
> at
> org.apache.hadoop.hive.ql.plan.DynamicPartitionCtx.(DynamicPartitionCtx.java:82)
> ... 29 more
> Caused by: org.apache.thrift.TApplicationException: getMetaConf failed:
> unknown result
> at
>
[jira] [Commented] (HIVE-13637) Fold CASE into NVL when CBO optimized the plan
[ https://issues.apache.org/jira/browse/HIVE-13637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15263859#comment-15263859 ] Jesus Camacho Rodriguez commented on HIVE-13637: I was also doubting... Problem is that Calcite does not support NVL natively, so we might miss folding optimization opportunities if we convert CASE into NVL. I could also create an additional rule to transform it at the end of query planning, but it seems an overkill. I have just uploaded a new patch that integrates the conversion in the parser instead of the ASTConverter, and thus integrates better with the work done in HIVE-13068 (it will need that patch to go in first though). > Fold CASE into NVL when CBO optimized the plan > -- > > Key: HIVE-13637 > URL: https://issues.apache.org/jira/browse/HIVE-13637 > Project: Hive > Issue Type: Sub-task > Components: CBO >Affects Versions: 2.1.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > Attachments: HIVE-13637.patch, HIVE-13637.patch > > > After HIVE-13068 goes in, folding CASE into NVL got disabled when CBO has > optimized the plan, as it was done by ConstantPropagate in Hive. We need to > enable it back. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13637) Fold CASE into NVL when CBO optimized the plan
[ https://issues.apache.org/jira/browse/HIVE-13637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-13637: --- Attachment: HIVE-13637.patch > Fold CASE into NVL when CBO optimized the plan > -- > > Key: HIVE-13637 > URL: https://issues.apache.org/jira/browse/HIVE-13637 > Project: Hive > Issue Type: Sub-task > Components: CBO >Affects Versions: 2.1.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > Attachments: HIVE-13637.patch, HIVE-13637.patch > > > After HIVE-13068 goes in, folding CASE into NVL got disabled when CBO has > optimized the plan, as it was done by ConstantPropagate in Hive. We need to > enable it back. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13469) LLAP: Support delayed scheduling for locality
[ https://issues.apache.org/jira/browse/HIVE-13469?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated HIVE-13469: -- Resolution: Fixed Fix Version/s: 2.1.0 Status: Resolved (was: Patch Available) > LLAP: Support delayed scheduling for locality > - > > Key: HIVE-13469 > URL: https://issues.apache.org/jira/browse/HIVE-13469 > Project: Hive > Issue Type: Improvement >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Fix For: 2.1.0 > > Attachments: HIVE-13469.01.patch, HIVE-13469.02.patch, > HIVE-13469.02.patch, HIVE-13469.03.patch > > > LLAP currently supports forcing locality. Change this to support a time based > delay for locality as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13469) LLAP: Support delayed scheduling for locality
[ https://issues.apache.org/jira/browse/HIVE-13469?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated HIVE-13469: -- Attachment: HIVE-13469.03.patch Rebased patch to trunk. The test failures from the previous run are not related. Committing. > LLAP: Support delayed scheduling for locality > - > > Key: HIVE-13469 > URL: https://issues.apache.org/jira/browse/HIVE-13469 > Project: Hive > Issue Type: Improvement >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Attachments: HIVE-13469.01.patch, HIVE-13469.02.patch, > HIVE-13469.02.patch, HIVE-13469.03.patch > > > LLAP currently supports forcing locality. Change this to support a time based > delay for locality as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13516) Adding BTEQ .IF, .QUIT, ERRORCODE to HPL/SQL
[ https://issues.apache.org/jira/browse/HIVE-13516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dmitry Tolpeko updated HIVE-13516: -- Status: Patch Available (was: Open) > Adding BTEQ .IF, .QUIT, ERRORCODE to HPL/SQL > > > Key: HIVE-13516 > URL: https://issues.apache.org/jira/browse/HIVE-13516 > Project: Hive > Issue Type: Improvement > Components: hpl/sql >Affects Versions: 2.1.0 >Reporter: Dmitry Tolpeko >Assignee: Dmitry Tolpeko > Attachments: HIVE-13516.1.patch > > > Adding Teradata BTEQ features to HPL/SQL such as .IF, .QUIT, ERRORCODE. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13516) Adding BTEQ .IF, .QUIT, ERRORCODE to HPL/SQL
[ https://issues.apache.org/jira/browse/HIVE-13516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dmitry Tolpeko updated HIVE-13516: -- Status: Open (was: Patch Available) > Adding BTEQ .IF, .QUIT, ERRORCODE to HPL/SQL > > > Key: HIVE-13516 > URL: https://issues.apache.org/jira/browse/HIVE-13516 > Project: Hive > Issue Type: Improvement > Components: hpl/sql >Affects Versions: 2.1.0 >Reporter: Dmitry Tolpeko >Assignee: Dmitry Tolpeko > Attachments: HIVE-13516.1.patch > > > Adding Teradata BTEQ features to HPL/SQL such as .IF, .QUIT, ERRORCODE. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13647) named_struct changes the column name to lowercase
[ https://issues.apache.org/jira/browse/HIVE-13647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sivashankar updated HIVE-13647: --- Priority: Minor (was: Trivial) Description: the named_struct is changing the column name "dropOffLocation" and "pickUpLocation" to lower case as dropofflocation and pickuplocation respectively named_struct( 'notes',notes, 'dropOffLocation',dropOffLocation, 'pickUpLocation',pickUpLocation, 'email',email, 'phone',phone, 'username',username, 'lastname',lastname, 'firstname',firstname, 'loc',loc )as chng_hist_d from( change_history.d.notes as notes, change_history.d.dropOffLocation as dropOffLocation, change_history.d.pickUpLocation as pickUpLocation, change_history.d.email as email, change_history.d.phone as phone, change_history.d.username as username, change_history.d.lastName as lastname, change_history.d.firstName as firstname, change_history.d.loc as loc from --TABLENAME-- LATERAL VIEW explode(changehistory) exploded_table as change)a)b; result came as 2.906Zi","notes":null,"dropofflocation":null,"pickuplocation":null,"email":null,"phone":null,"username":null, Summary: named_struct changes the column name to lowercase (was: named_struct changes the column to lowercase) > named_struct changes the column name to lowercase > - > > Key: HIVE-13647 > URL: https://issues.apache.org/jira/browse/HIVE-13647 > Project: Hive > Issue Type: Bug >Affects Versions: 0.14.0 >Reporter: Sivashankar >Assignee: Sivashankar >Priority: Minor > > the named_struct is changing the column name "dropOffLocation" and > "pickUpLocation" to lower case as dropofflocation and pickuplocation > respectively > named_struct( > 'notes',notes, > 'dropOffLocation',dropOffLocation, > 'pickUpLocation',pickUpLocation, > 'email',email, > 'phone',phone, > 'username',username, > 'lastname',lastname, > 'firstname',firstname, > 'loc',loc > )as chng_hist_d > from( > change_history.d.notes as notes, > change_history.d.dropOffLocation as dropOffLocation, > change_history.d.pickUpLocation as pickUpLocation, > change_history.d.email as email, > change_history.d.phone as phone, > change_history.d.username as username, > change_history.d.lastName as lastname, > change_history.d.firstName as firstname, > change_history.d.loc as loc > from --TABLENAME-- LATERAL VIEW explode(changehistory) exploded_table as > change)a)b; > result came as > 2.906Zi","notes":null,"dropofflocation":null,"pickuplocation":null,"email":null,"phone":null,"username":null, -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13643) Various enhancements / fixes to llap cli tools
[ https://issues.apache.org/jira/browse/HIVE-13643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated HIVE-13643: -- Attachment: HIVE-13643.02.patch > Various enhancements / fixes to llap cli tools > -- > > Key: HIVE-13643 > URL: https://issues.apache.org/jira/browse/HIVE-13643 > Project: Hive > Issue Type: Improvement >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Attachments: HIVE-13643.01.patch, HIVE-13643.02.patch > > > - BUG - llapstatus does not always produce output (PrintWriter.close) > - llapstatus should parse hiveconf > - package llap-cli-log4j2.template > - Log at the start and end of the script to include parameters > - Generate logs to a file different from hive.log > - hive --service llap --name does not propagate the name to the daemons -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13643) Various enhancements / fixes to llap cli tools
[ https://issues.apache.org/jira/browse/HIVE-13643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated HIVE-13643: -- Attachment: (was: HIVE-13643.02.patch) > Various enhancements / fixes to llap cli tools > -- > > Key: HIVE-13643 > URL: https://issues.apache.org/jira/browse/HIVE-13643 > Project: Hive > Issue Type: Improvement >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Attachments: HIVE-13643.01.patch > > > - BUG - llapstatus does not always produce output (PrintWriter.close) > - llapstatus should parse hiveconf > - package llap-cli-log4j2.template > - Log at the start and end of the script to include parameters > - Generate logs to a file different from hive.log > - hive --service llap --name does not propagate the name to the daemons -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13643) Various enhancements / fixes to llap cli tools
[ https://issues.apache.org/jira/browse/HIVE-13643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated HIVE-13643: -- Attachment: HIVE-13643.02.patch Updated patch. - Capitalized LLAP in log lines. - This only work with slider apps. Min length of service hosts from config has to be 2. Moved the checks a bit to make this simpler, and improved logging. This patch contains 2 more changes. 1. adds a configurable appTimeout to llap status - when waiting for the yarn app to show up 2. Changes llap service driver to pick up the name correctly from the config if it is not passed in on the command line. (The earlier behaviour would result in the app launching wiht a name llap0 (default) - and daemons registering with the name in hive-site) [~prasanth_j] - another change I'd like to make is to set the log level via hiveconf. Do you know what is required for this ? Once the process starts up - it'll pick up the log level from the config file. Assuming something specific needs to be done to make this work. I can make that change here, or a separate jira depending on when jenkins ends up running for this. > Various enhancements / fixes to llap cli tools > -- > > Key: HIVE-13643 > URL: https://issues.apache.org/jira/browse/HIVE-13643 > Project: Hive > Issue Type: Improvement >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Attachments: HIVE-13643.01.patch, HIVE-13643.02.patch > > > - BUG - llapstatus does not always produce output (PrintWriter.close) > - llapstatus should parse hiveconf > - package llap-cli-log4j2.template > - Log at the start and end of the script to include parameters > - Generate logs to a file different from hive.log > - hive --service llap --name does not propagate the name to the daemons -- This message was sent by Atlassian JIRA (v6.3.4#6332)
