[jira] [Updated] (HIVE-11927) Implement/Enable constant related optimization rules in Calcite: enable HiveReduceExpressionsRule to fold constants
[ https://issues.apache.org/jira/browse/HIVE-11927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-11927: --- Attachment: HIVE-11927.15.patch > Implement/Enable constant related optimization rules in Calcite: enable > HiveReduceExpressionsRule to fold constants > --- > > Key: HIVE-11927 > URL: https://issues.apache.org/jira/browse/HIVE-11927 > Project: Hive > Issue Type: Sub-task >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Attachments: HIVE-11927.01.patch, HIVE-11927.02.patch, > HIVE-11927.03.patch, HIVE-11927.04.patch, HIVE-11927.05.patch, > HIVE-11927.06.patch, HIVE-11927.07.patch, HIVE-11927.08.patch, > HIVE-11927.09.patch, HIVE-11927.10.patch, HIVE-11927.11.patch, > HIVE-11927.12.patch, HIVE-11927.13.patch, HIVE-11927.14.patch, > HIVE-11927.15.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12429) Switch default Hive authorization to SQLStandardAuth in 2.0
[ https://issues.apache.org/jira/browse/HIVE-12429?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Dai updated HIVE-12429: -- Attachment: HIVE-12429.3.patch > Switch default Hive authorization to SQLStandardAuth in 2.0 > --- > > Key: HIVE-12429 > URL: https://issues.apache.org/jira/browse/HIVE-12429 > Project: Hive > Issue Type: Task > Components: Authorization, Security >Affects Versions: 2.0.0 >Reporter: Alan Gates >Assignee: Daniel Dai > Attachments: HIVE-12429.1.patch, HIVE-12429.2.patch, > HIVE-12429.3.patch > > > Hive's default authorization is not real security, as it does not secure a > number of features and anyone can grant access to any object to any user. We > should switch the default to SQLStandardAuth, which provides real > authentication. > As this is a backwards incompatible change this was hard to do previously, > but 2.0 gives us a place to do this type of change. > By default authorization will still be off, as there are a few other things > to set when turning on authorization (such as the list of admin users). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12619) Switching the field order within an array of structs causes the query to fail
[ https://issues.apache.org/jira/browse/HIVE-12619?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mohammad Kamrul Islam updated HIVE-12619: - Attachment: HIVE-12619.2.patch > Switching the field order within an array of structs causes the query to fail > - > > Key: HIVE-12619 > URL: https://issues.apache.org/jira/browse/HIVE-12619 > Project: Hive > Issue Type: Bug >Affects Versions: 1.1.0 >Reporter: Ang Zhang >Assignee: Mohammad Kamrul Islam >Priority: Minor > Attachments: HIVE-12619.2.patch > > > Switching the field order within an array of structs causes the query to fail > or return the wrong data for the fields, but switching the field order within > just a struct works. > How to reproduce: > Case1 if the two fields have the same type, query will return wrong data for > the fields > drop table if exists schema_test; > create table schema_test (msg array) stored > as parquet; > insert into table schema_test select stack(2, array(named_struct('f1', 'abc', > 'f2', 'abc2')), array(named_struct('f1', 'efg', 'f2', 'efg2'))) from one > limit 2; > select * from schema_test; > --returns > --[{"f1":"efg","f2":"efg2"}] > --[{"f1":"abc","f2":"abc2"}] > alter table schema_test change msg msg array ; > select * from schema_test; > --returns > --[{"f2":"efg","f1":"efg2"}] > --[{"f2":"abc","f1":"abc2"}] > Case2: if the two fields have different type, the query will fail > drop table if exists schema_test; > create table schema_test (msg array ) stored as > parquet; > insert into table schema_test select stack(2, array(named_struct('f1', 'abc', > 'f2', 1)), array(named_struct('f1', 'efg', 'f2', 2))) from one limit 2; > select * from schema_test; > --returns > --[{"f1":"efg","f2":2}] > --[{"f1":"abc","f2":1}] > alter table schema_test change msg msg array ; > select * from schema_test; > Failed with exception > java.io.IOException:org.apache.hadoop.hive.ql.metadata.HiveException: > java.lang.ClassCastException: org.apache.hadoop.io.Text cannot be cast to > org.apache.hadoop.io.IntWritable -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12619) Switching the field order within an array of structs causes the query to fail
[ https://issues.apache.org/jira/browse/HIVE-12619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15063788#comment-15063788 ] Mohammad Kamrul Islam commented on HIVE-12619: -- RB: https://reviews.apache.org/r/41541/ [~spena] please review it. > Switching the field order within an array of structs causes the query to fail > - > > Key: HIVE-12619 > URL: https://issues.apache.org/jira/browse/HIVE-12619 > Project: Hive > Issue Type: Bug >Affects Versions: 1.1.0 >Reporter: Ang Zhang >Assignee: Mohammad Kamrul Islam >Priority: Minor > Attachments: HIVE-12619.2.patch > > > Switching the field order within an array of structs causes the query to fail > or return the wrong data for the fields, but switching the field order within > just a struct works. > How to reproduce: > Case1 if the two fields have the same type, query will return wrong data for > the fields > drop table if exists schema_test; > create table schema_test (msg array) stored > as parquet; > insert into table schema_test select stack(2, array(named_struct('f1', 'abc', > 'f2', 'abc2')), array(named_struct('f1', 'efg', 'f2', 'efg2'))) from one > limit 2; > select * from schema_test; > --returns > --[{"f1":"efg","f2":"efg2"}] > --[{"f1":"abc","f2":"abc2"}] > alter table schema_test change msg msg array ; > select * from schema_test; > --returns > --[{"f2":"efg","f1":"efg2"}] > --[{"f2":"abc","f1":"abc2"}] > Case2: if the two fields have different type, the query will fail > drop table if exists schema_test; > create table schema_test (msg array ) stored as > parquet; > insert into table schema_test select stack(2, array(named_struct('f1', 'abc', > 'f2', 1)), array(named_struct('f1', 'efg', 'f2', 2))) from one limit 2; > select * from schema_test; > --returns > --[{"f1":"efg","f2":2}] > --[{"f1":"abc","f2":1}] > alter table schema_test change msg msg array ; > select * from schema_test; > Failed with exception > java.io.IOException:org.apache.hadoop.hive.ql.metadata.HiveException: > java.lang.ClassCastException: org.apache.hadoop.io.Text cannot be cast to > org.apache.hadoop.io.IntWritable -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12619) Switching the field order within an array of structs causes the query to fail
[ https://issues.apache.org/jira/browse/HIVE-12619?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mohammad Kamrul Islam updated HIVE-12619: - Attachment: (was: HIVE-12619.1.patch) > Switching the field order within an array of structs causes the query to fail > - > > Key: HIVE-12619 > URL: https://issues.apache.org/jira/browse/HIVE-12619 > Project: Hive > Issue Type: Bug >Affects Versions: 1.1.0 >Reporter: Ang Zhang >Assignee: Mohammad Kamrul Islam >Priority: Minor > > Switching the field order within an array of structs causes the query to fail > or return the wrong data for the fields, but switching the field order within > just a struct works. > How to reproduce: > Case1 if the two fields have the same type, query will return wrong data for > the fields > drop table if exists schema_test; > create table schema_test (msg array) stored > as parquet; > insert into table schema_test select stack(2, array(named_struct('f1', 'abc', > 'f2', 'abc2')), array(named_struct('f1', 'efg', 'f2', 'efg2'))) from one > limit 2; > select * from schema_test; > --returns > --[{"f1":"efg","f2":"efg2"}] > --[{"f1":"abc","f2":"abc2"}] > alter table schema_test change msg msg array ; > select * from schema_test; > --returns > --[{"f2":"efg","f1":"efg2"}] > --[{"f2":"abc","f1":"abc2"}] > Case2: if the two fields have different type, the query will fail > drop table if exists schema_test; > create table schema_test (msg array ) stored as > parquet; > insert into table schema_test select stack(2, array(named_struct('f1', 'abc', > 'f2', 1)), array(named_struct('f1', 'efg', 'f2', 2))) from one limit 2; > select * from schema_test; > --returns > --[{"f1":"efg","f2":2}] > --[{"f1":"abc","f2":1}] > alter table schema_test change msg msg array ; > select * from schema_test; > Failed with exception > java.io.IOException:org.apache.hadoop.hive.ql.metadata.HiveException: > java.lang.ClassCastException: org.apache.hadoop.io.Text cannot be cast to > org.apache.hadoop.io.IntWritable -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12658) Task rejection by an llap daemon spams the log with RejectedExecutionExceptions
[
https://issues.apache.org/jira/browse/HIVE-12658?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15063802#comment-15063802
]
Hive QA commented on HIVE-12658:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12778327/HIVE-12658.3.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 18 failed/errored test(s), 9950 tests
executed
*Failed tests:*
{noformat}
TestHWISessionManager - did not produce a TEST-*.xml file
TestMiniTezCliDriver-vector_coalesce.q-auto_sortmerge_join_7.q-dynamic_partition_pruning.q-and-12-more
- did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_order2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union9
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver_encryption_insert_partition_dynamic
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_import
org.apache.hadoop.hive.ql.exec.spark.session.TestSparkSessionManagerImpl.testMultiSessionMultipleUse
org.apache.hadoop.hive.ql.exec.spark.session.TestSparkSessionManagerImpl.testSingleSessionMultipleUse
org.apache.hive.jdbc.TestJdbcWithMiniHS2.testAddJarDataNucleusUnCaching
org.apache.hive.jdbc.TestSSL.testSSLVersion
org.apache.hive.spark.client.TestSparkClient.testAddJarsAndFiles
org.apache.hive.spark.client.TestSparkClient.testCounters
org.apache.hive.spark.client.TestSparkClient.testErrorJob
org.apache.hive.spark.client.TestSparkClient.testJobSubmission
org.apache.hive.spark.client.TestSparkClient.testMetricsCollection
org.apache.hive.spark.client.TestSparkClient.testRemoteClient
org.apache.hive.spark.client.TestSparkClient.testSimpleSparkJob
org.apache.hive.spark.client.TestSparkClient.testSyncRpc
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6393/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6393/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-6393/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 18 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12778327 - PreCommit-HIVE-TRUNK-Build
> Task rejection by an llap daemon spams the log with
> RejectedExecutionExceptions
> ---
>
> Key: HIVE-12658
> URL: https://issues.apache.org/jira/browse/HIVE-12658
> Project: Hive
> Issue Type: Task
>Reporter: Siddharth Seth
>Assignee: Prasanth Jayachandran
> Attachments: HIVE-12658.1.patch, HIVE-12658.2.patch,
> HIVE-12658.3.patch, HIVE-12658.3.patch
>
>
> The execution queue throws a RejectedExecutionException - which is logged by
> the hadoop IPC layer.
> Instead of relying on an Exception in the protocol - move to sending back an
> explicit response to indicate a rejected fragment.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-11355) Hive on tez: memory manager for sort buffers (input/output) and operators
[ https://issues.apache.org/jira/browse/HIVE-11355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vikram Dixit K updated HIVE-11355: -- Attachment: HIVE-11355.9.patch Fix for failing tests. > Hive on tez: memory manager for sort buffers (input/output) and operators > - > > Key: HIVE-11355 > URL: https://issues.apache.org/jira/browse/HIVE-11355 > Project: Hive > Issue Type: Improvement > Components: Tez >Affects Versions: 2.0.0 >Reporter: Vikram Dixit K >Assignee: Vikram Dixit K > Attachments: HIVE-11355.1.patch, HIVE-11355.2.patch, > HIVE-11355.3.patch, HIVE-11355.4.patch, HIVE-11355.5.patch, > HIVE-11355.6.patch, HIVE-11355.7.patch, HIVE-11355.8.patch, HIVE-11355.9.patch > > > We need to better manage the sort buffer allocations to ensure better > performance. Also, we need to provide configurations to certain operators to > stay within memory limits. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12682) Reducers in dynamic partitioning job spend a lot of time running hadoop.conf.Configuration.getOverlay
[
https://issues.apache.org/jira/browse/HIVE-12682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15063949#comment-15063949
]
Hive QA commented on HIVE-12682:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12778337/HIVE-12682.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 22 failed/errored test(s), 9921 tests
executed
*Failed tests:*
{noformat}
TestHWISessionManager - did not produce a TEST-*.xml file
TestMiniTezCliDriver-tez_bmj_schema_evolution.q-orc_merge5.q-vectorization_limit.q-and-12-more
- did not produce a TEST-*.xml file
TestMiniTezCliDriver-tez_smb_empty.q-transform_ppr2.q-vector_outer_join5.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-timestamp_lazy.q-bucketsortoptimize_insert_4.q-date_udf.q-and-12-more
- did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_order2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union9
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver_encryption_insert_partition_dynamic
org.apache.hadoop.hive.cli.TestMinimrCliDriver.org.apache.hadoop.hive.cli.TestMinimrCliDriver
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_bucket_many
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_bucket_num_reducers
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_import
org.apache.hadoop.hive.ql.exec.spark.session.TestSparkSessionManagerImpl.testMultiSessionMultipleUse
org.apache.hadoop.hive.ql.exec.spark.session.TestSparkSessionManagerImpl.testSingleSessionMultipleUse
org.apache.hive.jdbc.TestSSL.testSSLVersion
org.apache.hive.spark.client.TestSparkClient.testAddJarsAndFiles
org.apache.hive.spark.client.TestSparkClient.testCounters
org.apache.hive.spark.client.TestSparkClient.testErrorJob
org.apache.hive.spark.client.TestSparkClient.testJobSubmission
org.apache.hive.spark.client.TestSparkClient.testMetricsCollection
org.apache.hive.spark.client.TestSparkClient.testRemoteClient
org.apache.hive.spark.client.TestSparkClient.testSimpleSparkJob
org.apache.hive.spark.client.TestSparkClient.testSyncRpc
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6394/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6394/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-6394/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 22 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12778337 - PreCommit-HIVE-TRUNK-Build
> Reducers in dynamic partitioning job spend a lot of time running
> hadoop.conf.Configuration.getOverlay
> -
>
> Key: HIVE-12682
> URL: https://issues.apache.org/jira/browse/HIVE-12682
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 1.2.1
>Reporter: Carter Shanklin
>Assignee: Prasanth Jayachandran
> Attachments: HIVE-12682.1.patch, reducer.png
>
>
> I tested this on Hive 1.2.1 but looks like it's still applicable to 2.0.
> I ran this query:
> {code}
> create table flights (
> …
> )
> PARTITIONED BY (Year int)
> CLUSTERED BY (Month)
> SORTED BY (DayofMonth) into 12 buckets
> STORED AS ORC
> TBLPROPERTIES("orc.bloom.filter.columns"="*")
> ;
> {code}
> (Taken from here:
> https://github.com/t3rmin4t0r/all-airlines-data/blob/master/ddl/orc.sql)
> I profiled just the reduce phase and noticed something odd, the attached
> graph shows where time was spent during the reducer phase.
> !reducer.png!
> Problem seems to relate to
> https://github.com/apache/hive/blob/branch-2.0/ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java#L903
> /cc [~gopalv]
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-12698) Remove exposure to internal privilege and principal classes in HiveAuthorizer
[
https://issues.apache.org/jira/browse/HIVE-12698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15064191#comment-15064191
]
Hive QA commented on HIVE-12698:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12778425/HIVE-12698.4.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 16 failed/errored test(s), 9965 tests
executed
*Failed tests:*
{noformat}
TestHWISessionManager - did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_order2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union9
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver_encryption_insert_partition_dynamic
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_import
org.apache.hadoop.hive.ql.exec.spark.session.TestSparkSessionManagerImpl.testMultiSessionMultipleUse
org.apache.hadoop.hive.ql.exec.spark.session.TestSparkSessionManagerImpl.testSingleSessionMultipleUse
org.apache.hive.jdbc.TestSSL.testSSLVersion
org.apache.hive.spark.client.TestSparkClient.testAddJarsAndFiles
org.apache.hive.spark.client.TestSparkClient.testCounters
org.apache.hive.spark.client.TestSparkClient.testErrorJob
org.apache.hive.spark.client.TestSparkClient.testJobSubmission
org.apache.hive.spark.client.TestSparkClient.testMetricsCollection
org.apache.hive.spark.client.TestSparkClient.testRemoteClient
org.apache.hive.spark.client.TestSparkClient.testSimpleSparkJob
org.apache.hive.spark.client.TestSparkClient.testSyncRpc
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6396/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6396/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-6396/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 16 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12778425 - PreCommit-HIVE-TRUNK-Build
> Remove exposure to internal privilege and principal classes in HiveAuthorizer
> -
>
> Key: HIVE-12698
> URL: https://issues.apache.org/jira/browse/HIVE-12698
> Project: Hive
> Issue Type: Bug
> Components: Authorization
>Affects Versions: 1.3.0, 2.0.0
>Reporter: Thejas M Nair
>Assignee: Thejas M Nair
> Fix For: 1.3.0, 2.0.0
>
> Attachments: HIVE-12698.1.patch, HIVE-12698.2.patch,
> HIVE-12698.3.patch, HIVE-12698.4.patch
>
>
> The changes in HIVE-11179 expose several internal classes to
> HiveAuthorization implementations. These include PrivilegeObjectDesc,
> PrivilegeDesc, PrincipalDesc and AuthorizationUtils.
> We should avoid exposing that to all Authorization implementations, but also
> make the ability to customize the mapping of internal classes to the public
> api classes possible for Apache Sentry (incubating).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-11355) Hive on tez: memory manager for sort buffers (input/output) and operators
[ https://issues.apache.org/jira/browse/HIVE-11355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vikram Dixit K updated HIVE-11355: -- Attachment: HIVE-11355.10.patch > Hive on tez: memory manager for sort buffers (input/output) and operators > - > > Key: HIVE-11355 > URL: https://issues.apache.org/jira/browse/HIVE-11355 > Project: Hive > Issue Type: Improvement > Components: Tez >Affects Versions: 2.0.0 >Reporter: Vikram Dixit K >Assignee: Vikram Dixit K > Attachments: HIVE-11355.1.patch, HIVE-11355.10.patch, > HIVE-11355.2.patch, HIVE-11355.3.patch, HIVE-11355.4.patch, > HIVE-11355.5.patch, HIVE-11355.6.patch, HIVE-11355.7.patch, > HIVE-11355.8.patch, HIVE-11355.9.patch > > > We need to better manage the sort buffer allocations to ensure better > performance. Also, we need to provide configurations to certain operators to > stay within memory limits. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11775) Implement limit push down through union all in CBO
[ https://issues.apache.org/jira/browse/HIVE-11775?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-11775: --- Attachment: HIVE-11775.13.patch address offset==null issue > Implement limit push down through union all in CBO > -- > > Key: HIVE-11775 > URL: https://issues.apache.org/jira/browse/HIVE-11775 > Project: Hive > Issue Type: New Feature >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Attachments: HIVE-11775.01.patch, HIVE-11775.02.patch, > HIVE-11775.03.patch, HIVE-11775.04.patch, HIVE-11775.05.patch, > HIVE-11775.06.patch, HIVE-11775.07.patch, HIVE-11775.08.patch, > HIVE-11775.09.patch, HIVE-11775.10.patch, HIVE-11775.11.patch, > HIVE-11775.12.patch, HIVE-11775.13.patch > > > Enlightened by HIVE-11684 (Kudos to [~jcamachorodriguez]), we can actually > push limit down through union all, which reduces the intermediate number of > rows in union branches. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11775) Implement limit push down through union all in CBO
[ https://issues.apache.org/jira/browse/HIVE-11775?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-11775: --- Attachment: (was: HIVE-11775.13.patch) > Implement limit push down through union all in CBO > -- > > Key: HIVE-11775 > URL: https://issues.apache.org/jira/browse/HIVE-11775 > Project: Hive > Issue Type: New Feature >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Attachments: HIVE-11775.01.patch, HIVE-11775.02.patch, > HIVE-11775.03.patch, HIVE-11775.04.patch, HIVE-11775.05.patch, > HIVE-11775.06.patch, HIVE-11775.07.patch, HIVE-11775.08.patch, > HIVE-11775.09.patch, HIVE-11775.10.patch, HIVE-11775.11.patch, > HIVE-11775.12.patch > > > Enlightened by HIVE-11684 (Kudos to [~jcamachorodriguez]), we can actually > push limit down through union all, which reduces the intermediate number of > rows in union branches. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11865) Disable Hive PPD optimizer when CBO has optimized the plan
[ https://issues.apache.org/jira/browse/HIVE-11865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-11865: --- Attachment: HIVE-11865.05.patch After checking q files changes, I needed to extend the simpler version of PPD. In particular, synthetic join predicates (for dynamic partition pruning) are introduced just below join, so those particular predicates need to be pushed. Further, if we have a Filter operator that is an ancestor of a PTF operator, and between the Filter and the PTF there are only Select operators, we also need to push its predicates on top of the PTF; the reason is that otherwise LimitPushdownOptimizer will not kick in for the Rank function. [~jpullokkaran], QA run should come clean. Could you review the code? Thanks > Disable Hive PPD optimizer when CBO has optimized the plan > -- > > Key: HIVE-11865 > URL: https://issues.apache.org/jira/browse/HIVE-11865 > Project: Hive > Issue Type: Bug > Components: CBO, Logical Optimizer >Affects Versions: 2.0.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > Attachments: HIVE-11865.01.patch, HIVE-11865.02.patch, > HIVE-11865.02.patch, HIVE-11865.03.patch, HIVE-11865.04.patch, > HIVE-11865.05.patch, HIVE-11865.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12661) StatsSetupConst.COLUMN_STATS_ACCURATE is not used correctly
[
https://issues.apache.org/jira/browse/HIVE-12661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15064429#comment-15064429
]
Hive QA commented on HIVE-12661:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12778386/HIVE-12661.05.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 27 failed/errored test(s), 9966 tests
executed
*Failed tests:*
{noformat}
TestHWISessionManager - did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join_stats2
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver_encryption_insert_partition_dynamic
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver_encryption_join_with_different_encryption_keys
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_bucket5
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_infer_bucket_sort_bucketed_table
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_infer_bucket_sort_map_operators
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_infer_bucket_sort_merge
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_infer_bucket_sort_num_buckets
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_infer_bucket_sort_reducers_power_two
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_list_bucket_dml_10
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_parallel_orderby
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_reduce_deduplicate
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_import
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_stats_partialscan_autogether
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_unset_table_property
org.apache.hadoop.hive.ql.exec.spark.session.TestSparkSessionManagerImpl.testMultiSessionMultipleUse
org.apache.hadoop.hive.ql.exec.spark.session.TestSparkSessionManagerImpl.testSingleSessionMultipleUse
org.apache.hive.jdbc.TestSSL.testSSLVersion
org.apache.hive.spark.client.TestSparkClient.testAddJarsAndFiles
org.apache.hive.spark.client.TestSparkClient.testCounters
org.apache.hive.spark.client.TestSparkClient.testErrorJob
org.apache.hive.spark.client.TestSparkClient.testJobSubmission
org.apache.hive.spark.client.TestSparkClient.testMetricsCollection
org.apache.hive.spark.client.TestSparkClient.testRemoteClient
org.apache.hive.spark.client.TestSparkClient.testSimpleSparkJob
org.apache.hive.spark.client.TestSparkClient.testSyncRpc
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6397/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6397/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-6397/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 27 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12778386 - PreCommit-HIVE-TRUNK-Build
> StatsSetupConst.COLUMN_STATS_ACCURATE is not used correctly
> ---
>
> Key: HIVE-12661
> URL: https://issues.apache.org/jira/browse/HIVE-12661
> Project: Hive
> Issue Type: Bug
>Reporter: Pengcheng Xiong
>Assignee: Pengcheng Xiong
> Attachments: HIVE-12661.01.patch, HIVE-12661.02.patch,
> HIVE-12661.03.patch, HIVE-12661.04.patch, HIVE-12661.05.patch
>
>
> PROBLEM:
> Hive stats are autogathered properly till an 'analyze table [tablename]
> compute statistics for columns' is run. Then it does not auto-update the
> stats till the command is run again. repo:
> {code}
> set hive.stats.autogather=true;
> set hive.stats.atomic=false ;
> set hive.stats.collect.rawdatasize=true ;
> set hive.stats.collect.scancols=false ;
> set hive.stats.collect.tablekeys=false ;
> set hive.stats.fetch.column.stats=true;
> set hive.stats.fetch.partition.stats=true ;
> set hive.stats.reliable=false ;
> set hive.compute.query.using.stats=true;
> CREATE TABLE `default`.`calendar` (`year` int) ROW FORMAT SERDE
> 'org.apache.hadoop.hive.ql.io.orc.OrcSerde' STORED AS INPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat' OUTPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat' TBLPROPERTIES (
> 'orc.compress'='NONE') ;
> insert into calendar values (2010), (2011), (2012);
> select * from calendar;
[jira] [Updated] (HIVE-11775) Implement limit push down through union all in CBO
[ https://issues.apache.org/jira/browse/HIVE-11775?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-11775: --- Attachment: (was: HIVE-11775.13.patch) > Implement limit push down through union all in CBO > -- > > Key: HIVE-11775 > URL: https://issues.apache.org/jira/browse/HIVE-11775 > Project: Hive > Issue Type: New Feature >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Attachments: HIVE-11775.01.patch, HIVE-11775.02.patch, > HIVE-11775.03.patch, HIVE-11775.04.patch, HIVE-11775.05.patch, > HIVE-11775.06.patch, HIVE-11775.07.patch, HIVE-11775.08.patch, > HIVE-11775.09.patch, HIVE-11775.10.patch, HIVE-11775.11.patch, > HIVE-11775.12.patch > > > Enlightened by HIVE-11684 (Kudos to [~jcamachorodriguez]), we can actually > push limit down through union all, which reduces the intermediate number of > rows in union branches. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11775) Implement limit push down through union all in CBO
[ https://issues.apache.org/jira/browse/HIVE-11775?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-11775: --- Attachment: HIVE-11775.13.patch > Implement limit push down through union all in CBO > -- > > Key: HIVE-11775 > URL: https://issues.apache.org/jira/browse/HIVE-11775 > Project: Hive > Issue Type: New Feature >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Attachments: HIVE-11775.01.patch, HIVE-11775.02.patch, > HIVE-11775.03.patch, HIVE-11775.04.patch, HIVE-11775.05.patch, > HIVE-11775.06.patch, HIVE-11775.07.patch, HIVE-11775.08.patch, > HIVE-11775.09.patch, HIVE-11775.10.patch, HIVE-11775.11.patch, > HIVE-11775.12.patch, HIVE-11775.13.patch > > > Enlightened by HIVE-11684 (Kudos to [~jcamachorodriguez]), we can actually > push limit down through union all, which reduces the intermediate number of > rows in union branches. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12708) Hive on Spark doesn't work with Kerboresed HBase [Spark Branch]
[ https://issues.apache.org/jira/browse/HIVE-12708?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuefu Zhang updated HIVE-12708: --- Attachment: HIVE-12708.1-spark.patch > Hive on Spark doesn't work with Kerboresed HBase [Spark Branch] > --- > > Key: HIVE-12708 > URL: https://issues.apache.org/jira/browse/HIVE-12708 > Project: Hive > Issue Type: Bug > Components: Spark >Affects Versions: 1.2.0, 1.1.0, 2.0.0 >Reporter: Xuefu Zhang >Assignee: Xuefu Zhang > Attachments: HIVE-12708.1-spark.patch > > > Spark application launcher (spark-submit) acquires HBase delegation token on > Hive user's behalf when the application is launched. This mechanism, which > doesn't work for long-running sessions, is not in line with what Hive is > doing. Hive actually acquires the token automatically whenever a job needs > it. The right approach for Spark should be allowing applications to > dynamically add whatever tokens they need to the spark context. While this > needs work on Spark side, we provide a workaround solution in Hive. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11935) Race condition in HiveMetaStoreClient: isCompatibleWith and close
[
https://issues.apache.org/jira/browse/HIVE-11935?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Dai updated HIVE-11935:
--
Summary: Race condition in HiveMetaStoreClient: isCompatibleWith and close
(was: Access HiveMetaStoreClient.currentMetaVars should be synchronized)
> Race condition in HiveMetaStoreClient: isCompatibleWith and close
> --
>
> Key: HIVE-11935
> URL: https://issues.apache.org/jira/browse/HIVE-11935
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 1.2.1
>Reporter: Daniel Dai
>Assignee: Daniel Dai
> Fix For: 1.3.0, 2.1.0
>
> Attachments: HIVE-11935.1.patch, HIVE-11935.2.patch
>
>
> We saw intermittent failure of the following stack:
> {code}
> java.lang.NullPointerException
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.isCompatibleWith(HiveMetaStoreClient.java:287)
> at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:156)
> at com.sun.proxy.$Proxy9.isCompatibleWith(Unknown Source)
> at org.apache.hadoop.hive.ql.metadata.Hive.get(Hive.java:206)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.createHiveDB(BaseSemanticAnalyzer.java:205)
> at
> org.apache.hadoop.hive.ql.parse.DDLSemanticAnalyzer.(DDLSemanticAnalyzer.java:223)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzerFactory.get(SemanticAnalyzerFactory.java:259)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:409)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:308)
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1122)
> at
> org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1116)
> at
> org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:110)
> at
> org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation.java:181)
> at
> org.apache.hive.service.cli.operation.Operation.run(Operation.java:257)
> at
> org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:388)
> at
> org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementAsync(HiveSessionImpl.java:375)
> at sun.reflect.GeneratedMethodAccessor21.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at
> org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:78)
> at
> org.apache.hive.service.cli.session.HiveSessionProxy.access$000(HiveSessionProxy.java:36)
> at
> org.apache.hive.service.cli.session.HiveSessionProxy$1.run(HiveSessionProxy.java:63)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
> at
> org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:59)
> at com.sun.proxy.$Proxy20.executeStatementAsync(Unknown Source)
> at
> org.apache.hive.service.cli.CLIService.executeStatementAsync(CLIService.java:274)
> at
> org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(ThriftCLIService.java:486)
> at
> org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1313)
> at
> org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1298)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at
> org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1298)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> at org.apache.thrift.server.TServlet.doPost(TServlet.java:83)
> at
> org.apache.hive.service.cli.thrift.ThriftHttpServlet.doPost(ThriftHttpServlet.java:171)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
> at
> org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:565)
> at
>
[jira] [Updated] (HIVE-12577) NPE in LlapTaskCommunicator when unregistering containers
[
https://issues.apache.org/jira/browse/HIVE-12577?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Shelukhin updated HIVE-12577:
Attachment: HIVE-12577.03.patch
The same patch for HiveQA
> NPE in LlapTaskCommunicator when unregistering containers
> -
>
> Key: HIVE-12577
> URL: https://issues.apache.org/jira/browse/HIVE-12577
> Project: Hive
> Issue Type: Bug
> Components: llap
>Affects Versions: 2.0.0
>Reporter: Siddharth Seth
>Assignee: Siddharth Seth
>Priority: Critical
> Attachments: HIVE-12577.03.patch, HIVE-12577.1.review.txt,
> HIVE-12577.1.txt, HIVE-12577.1.wip.txt, HIVE-12577.2.review.txt,
> HIVE-12577.2.txt
>
>
> {code}
> 2015-12-02 13:29:00,160 [ERROR] [Dispatcher thread {Central}]
> |common.AsyncDispatcher|: Error in dispatcher thread
> java.lang.NullPointerException
> at
> org.apache.hadoop.hive.llap.tezplugins.LlapTaskCommunicator$EntityTracker.unregisterContainer(LlapTaskCommunicator.java:586)
> at
> org.apache.hadoop.hive.llap.tezplugins.LlapTaskCommunicator.registerContainerEnd(LlapTaskCommunicator.java:188)
> at
> org.apache.tez.dag.app.TaskCommunicatorManager.unregisterRunningContainer(TaskCommunicatorManager.java:389)
> at
> org.apache.tez.dag.app.rm.container.AMContainerImpl.unregisterFromTAListener(AMContainerImpl.java:1121)
> at
> org.apache.tez.dag.app.rm.container.AMContainerImpl$StopRequestAtLaunchingTransition.transition(AMContainerImpl.java:699)
> at
> org.apache.tez.dag.app.rm.container.AMContainerImpl$StopRequestAtIdleTransition.transition(AMContainerImpl.java:805)
> at
> org.apache.tez.dag.app.rm.container.AMContainerImpl$StopRequestAtRunningTransition.transition(AMContainerImpl.java:892)
> at
> org.apache.tez.dag.app.rm.container.AMContainerImpl$StopRequestAtRunningTransition.transition(AMContainerImpl.java:887)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory$SingleInternalArc.doTransition(StateMachineFactory.java:362)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:302)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:46)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:448)
> at
> org.apache.tez.dag.app.rm.container.AMContainerImpl.handle(AMContainerImpl.java:415)
> at
> org.apache.tez.dag.app.rm.container.AMContainerImpl.handle(AMContainerImpl.java:72)
> at
> org.apache.tez.dag.app.rm.container.AMContainerMap.handle(AMContainerMap.java:60)
> at
> org.apache.tez.dag.app.rm.container.AMContainerMap.handle(AMContainerMap.java:36)
> at
> org.apache.tez.common.AsyncDispatcher.dispatch(AsyncDispatcher.java:183)
> at
> org.apache.tez.common.AsyncDispatcher$1.run(AsyncDispatcher.java:114)
> at java.lang.Thread.run(Thread.java:745)
> 2015-12-02 13:29:00,167 [ERROR] [Dispatcher thread {Central}]
> |common.AsyncDispatcher|: Error in dispatcher thread
> java.lang.NullPointerException
> at
> org.apache.tez.dag.app.TaskCommunicatorManager.unregisterRunningContainer(TaskCommunicatorManager.java:386)
> at
> org.apache.tez.dag.app.rm.container.AMContainerImpl.unregisterFromTAListener(AMContainerImpl.java:1121)
> at
> org.apache.tez.dag.app.rm.container.AMContainerImpl$StopRequestAtLaunchingTransition.transition(AMContainerImpl.java:699)
> at
> org.apache.tez.dag.app.rm.container.AMContainerImpl$StopRequestAtIdleTransition.transition(AMContainerImpl.java:805)
> at
> org.apache.tez.dag.app.rm.container.AMContainerImpl$StopRequestAtRunningTransition.transition(AMContainerImpl.java:892)
> at
> org.apache.tez.dag.app.rm.container.AMContainerImpl$StopRequestAtRunningTransition.transition(AMContainerImpl.java:887)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory$SingleInternalArc.doTransition(StateMachineFactory.java:362)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:302)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:46)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:448)
> at
> org.apache.tez.dag.app.rm.container.AMContainerImpl.handle(AMContainerImpl.java:415)
> at
> org.apache.tez.dag.app.rm.container.AMContainerImpl.handle(AMContainerImpl.java:72)
> at
> org.apache.tez.dag.app.rm.container.AMContainerMap.handle(AMContainerMap.java:60)
> at
> org.apache.tez.dag.app.rm.container.AMContainerMap.handle(AMContainerMap.java:36)
>
[jira] [Updated] (HIVE-11775) Implement limit push down through union all in CBO
[ https://issues.apache.org/jira/browse/HIVE-11775?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-11775: --- Attachment: HIVE-11775.13.patch > Implement limit push down through union all in CBO > -- > > Key: HIVE-11775 > URL: https://issues.apache.org/jira/browse/HIVE-11775 > Project: Hive > Issue Type: New Feature >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Attachments: HIVE-11775.01.patch, HIVE-11775.02.patch, > HIVE-11775.03.patch, HIVE-11775.04.patch, HIVE-11775.05.patch, > HIVE-11775.06.patch, HIVE-11775.07.patch, HIVE-11775.08.patch, > HIVE-11775.09.patch, HIVE-11775.10.patch, HIVE-11775.11.patch, > HIVE-11775.12.patch, HIVE-11775.13.patch > > > Enlightened by HIVE-11684 (Kudos to [~jcamachorodriguez]), we can actually > push limit down through union all, which reduces the intermediate number of > rows in union branches. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11775) Implement limit push down through union all in CBO
[ https://issues.apache.org/jira/browse/HIVE-11775?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-11775: --- Attachment: HIVE-11775.13.patch > Implement limit push down through union all in CBO > -- > > Key: HIVE-11775 > URL: https://issues.apache.org/jira/browse/HIVE-11775 > Project: Hive > Issue Type: New Feature >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Attachments: HIVE-11775.01.patch, HIVE-11775.02.patch, > HIVE-11775.03.patch, HIVE-11775.04.patch, HIVE-11775.05.patch, > HIVE-11775.06.patch, HIVE-11775.07.patch, HIVE-11775.08.patch, > HIVE-11775.09.patch, HIVE-11775.10.patch, HIVE-11775.11.patch, > HIVE-11775.12.patch, HIVE-11775.13.patch > > > Enlightened by HIVE-11684 (Kudos to [~jcamachorodriguez]), we can actually > push limit down through union all, which reduces the intermediate number of > rows in union branches. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11775) Implement limit push down through union all in CBO
[ https://issues.apache.org/jira/browse/HIVE-11775?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-11775: --- Attachment: (was: HIVE-11775.13.patch) > Implement limit push down through union all in CBO > -- > > Key: HIVE-11775 > URL: https://issues.apache.org/jira/browse/HIVE-11775 > Project: Hive > Issue Type: New Feature >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Attachments: HIVE-11775.01.patch, HIVE-11775.02.patch, > HIVE-11775.03.patch, HIVE-11775.04.patch, HIVE-11775.05.patch, > HIVE-11775.06.patch, HIVE-11775.07.patch, HIVE-11775.08.patch, > HIVE-11775.09.patch, HIVE-11775.10.patch, HIVE-11775.11.patch, > HIVE-11775.12.patch, HIVE-11775.13.patch > > > Enlightened by HIVE-11684 (Kudos to [~jcamachorodriguez]), we can actually > push limit down through union all, which reduces the intermediate number of > rows in union branches. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12644) Support for offset in HiveSortMergeRule
[ https://issues.apache.org/jira/browse/HIVE-12644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15064485#comment-15064485 ] Jesus Camacho Rodriguez commented on HIVE-12644: [~jpullokkaran], this was on HiveSortMergeRule rule, not HiveSortRemoveRule; thus it should be fine. > Support for offset in HiveSortMergeRule > --- > > Key: HIVE-12644 > URL: https://issues.apache.org/jira/browse/HIVE-12644 > Project: Hive > Issue Type: Bug > Components: CBO >Affects Versions: 2.1.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > Attachments: HIVE-12644.patch > > > After HIVE-11531 goes in, HiveSortMergeRule needs to be extended to support > offset properly when it merges operators that contain Limit. Otherwise, limit > pushdown through outer join optimization (introduced in HIVE-11684) will not > work properly. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11935) Access HiveMetaStoreClient.currentMetaVars should be synchronized
[
https://issues.apache.org/jira/browse/HIVE-11935?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Dai updated HIVE-11935:
--
Fix Version/s: 2.1.0
1.3.0
> Access HiveMetaStoreClient.currentMetaVars should be synchronized
> -
>
> Key: HIVE-11935
> URL: https://issues.apache.org/jira/browse/HIVE-11935
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 1.2.1
>Reporter: Daniel Dai
>Assignee: Daniel Dai
> Fix For: 1.3.0, 2.1.0
>
> Attachments: HIVE-11935.1.patch, HIVE-11935.2.patch
>
>
> We saw intermittent failure of the following stack:
> {code}
> java.lang.NullPointerException
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.isCompatibleWith(HiveMetaStoreClient.java:287)
> at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:156)
> at com.sun.proxy.$Proxy9.isCompatibleWith(Unknown Source)
> at org.apache.hadoop.hive.ql.metadata.Hive.get(Hive.java:206)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.createHiveDB(BaseSemanticAnalyzer.java:205)
> at
> org.apache.hadoop.hive.ql.parse.DDLSemanticAnalyzer.(DDLSemanticAnalyzer.java:223)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzerFactory.get(SemanticAnalyzerFactory.java:259)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:409)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:308)
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1122)
> at
> org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1116)
> at
> org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:110)
> at
> org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation.java:181)
> at
> org.apache.hive.service.cli.operation.Operation.run(Operation.java:257)
> at
> org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:388)
> at
> org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementAsync(HiveSessionImpl.java:375)
> at sun.reflect.GeneratedMethodAccessor21.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at
> org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:78)
> at
> org.apache.hive.service.cli.session.HiveSessionProxy.access$000(HiveSessionProxy.java:36)
> at
> org.apache.hive.service.cli.session.HiveSessionProxy$1.run(HiveSessionProxy.java:63)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
> at
> org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:59)
> at com.sun.proxy.$Proxy20.executeStatementAsync(Unknown Source)
> at
> org.apache.hive.service.cli.CLIService.executeStatementAsync(CLIService.java:274)
> at
> org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(ThriftCLIService.java:486)
> at
> org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1313)
> at
> org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1298)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at
> org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1298)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> at org.apache.thrift.server.TServlet.doPost(TServlet.java:83)
> at
> org.apache.hive.service.cli.thrift.ThriftHttpServlet.doPost(ThriftHttpServlet.java:171)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
> at
> org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:565)
> at
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:479)
> at
>
[jira] [Commented] (HIVE-12548) Hive metastore goes down in Kerberos,sentry enabled CDH5.5 cluster
[
https://issues.apache.org/jira/browse/HIVE-12548?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15064686#comment-15064686
]
Andrew Olson commented on HIVE-12548:
-
After much debugging we were finally able to figure this out. Here are some
code and config snippets that might be helpful for anyone else who runs into
this cryptic exception.
{noformat}
for (Token token : UserGroupInformation.getCurrentUser().getTokens()) {
if (token.getKind().equals(DelegationTokenIdentifier.HIVE_DELEGATION_KIND)) {
conf.set(HIVE_METASTORE_TOKEN_SIGNATURE_PROPERTY,
token.getService().toString());
break;
}
}
{noformat}
{noformat}
HiveConf hiveConf = new HiveConf();
hiveConf.addResource(conf);
{noformat}
{noformat}
if (System.getProperty("oozie.action.conf.xml") != null) {
conf.addResource(new Path("file:///",
System.getProperty("oozie.action.conf.xml")));
}
conf.addResource("hive-site.xml");
{noformat}
{noformat}
if (System.getenv("HADOOP_TOKEN_FILE_LOCATION") != null) {
conf.set("mapreduce.job.credentials.binary",
System.getenv("HADOOP_TOKEN_FILE_LOCATION"));
}
{noformat}
{noformat}
conf.set("yarn.application.classpath", conf.get("yarn.application.classpath") +
",/opt/cloudera/parcels/CDH/lib/hive/lib/*");
{noformat}
{noformat}
${jobTracker}
${nameNode}
${concat(additionsPath, 'hive-site.xml')}
${concat(hadoopPropertyFilesPath, 'global-conf.xml')}
oozie.launcher.mapreduce.job.user.classpath.first
true
hcat.metastore.uri
${hiveMetastoreURI}
hcat.metastore.principal
${hivePrincipal}
{noformat}
> Hive metastore goes down in Kerberos,sentry enabled CDH5.5 cluster
> --
>
> Key: HIVE-12548
> URL: https://issues.apache.org/jira/browse/HIVE-12548
> Project: Hive
> Issue Type: Bug
> Components: Hive, HiveServer2
> Environment: RHEL 6.5 CLOUDERA CDH 5.5
>Reporter: narendra reddy ganesana
>
> [pool-3-thread-10]: Error occurred during processing of message.
> java.lang.RuntimeException: org.apache.thrift.transport.TTransportException:
> Invalid status -128
> at
> org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:219)
> at
> org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory$1.run(HadoopThriftAuthBridge.java:739)
> at
> org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory$1.run(HadoopThriftAuthBridge.java:736)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:356)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1651)
> at
> org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory.getTransport(HadoopThriftAuthBridge.java:736)
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:268)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.thrift.transport.TTransportException: Invalid status
> -128
> at
> org.apache.thrift.transport.TSaslTransport.sendAndThrowMessage(TSaslTransport.java:232)
> at
> org.apache.thrift.transport.TSaslTransport.receiveSaslMessage(TSaslTransport.java:184)
> at
> org.apache.thrift.transport.TSaslServerTransport.handleSaslStartMessage(TSaslServerTransport.java:125)
> at
> org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:271)
> at
> org.apache.thrift.transport.TSaslServerTransport.open(TSaslServerTransport.java:41)
> at
> org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:216)
> ... 10 more
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-11775) Implement limit push down through union all in CBO
[ https://issues.apache.org/jira/browse/HIVE-11775?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15064589#comment-15064589 ] Pengcheng Xiong commented on HIVE-11775: [~jpullokkaran], i have addressed the offset issue in the new patch. Could you take a look? Thanks. > Implement limit push down through union all in CBO > -- > > Key: HIVE-11775 > URL: https://issues.apache.org/jira/browse/HIVE-11775 > Project: Hive > Issue Type: New Feature >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Attachments: HIVE-11775.01.patch, HIVE-11775.02.patch, > HIVE-11775.03.patch, HIVE-11775.04.patch, HIVE-11775.05.patch, > HIVE-11775.06.patch, HIVE-11775.07.patch, HIVE-11775.08.patch, > HIVE-11775.09.patch, HIVE-11775.10.patch, HIVE-11775.11.patch, > HIVE-11775.12.patch, HIVE-11775.13.patch > > > Enlightened by HIVE-11684 (Kudos to [~jcamachorodriguez]), we can actually > push limit down through union all, which reduces the intermediate number of > rows in union branches. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11935) Access HiveMetaStoreClient.currentMetaVars should be synchronized
[
https://issues.apache.org/jira/browse/HIVE-11935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15064653#comment-15064653
]
Daniel Dai commented on HIVE-11935:
---
Yes, this is about thread synchronization. The test we saw is a concurrency
test. The only reason HS2 throw the exception is isCompatibleWith and close are
called by different threads and mixed temporally. The patch take Sergey's route
which make a local copy of currentMetaVars. There is slight chance that close
get called between check and make local copy, but chance is much smaller since
we are safe in the while loop.
> Access HiveMetaStoreClient.currentMetaVars should be synchronized
> -
>
> Key: HIVE-11935
> URL: https://issues.apache.org/jira/browse/HIVE-11935
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 1.2.1
>Reporter: Daniel Dai
>Assignee: Daniel Dai
> Attachments: HIVE-11935.1.patch, HIVE-11935.2.patch
>
>
> We saw intermittent failure of the following stack:
> {code}
> java.lang.NullPointerException
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.isCompatibleWith(HiveMetaStoreClient.java:287)
> at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:156)
> at com.sun.proxy.$Proxy9.isCompatibleWith(Unknown Source)
> at org.apache.hadoop.hive.ql.metadata.Hive.get(Hive.java:206)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.createHiveDB(BaseSemanticAnalyzer.java:205)
> at
> org.apache.hadoop.hive.ql.parse.DDLSemanticAnalyzer.(DDLSemanticAnalyzer.java:223)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzerFactory.get(SemanticAnalyzerFactory.java:259)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:409)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:308)
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1122)
> at
> org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1116)
> at
> org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:110)
> at
> org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation.java:181)
> at
> org.apache.hive.service.cli.operation.Operation.run(Operation.java:257)
> at
> org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:388)
> at
> org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementAsync(HiveSessionImpl.java:375)
> at sun.reflect.GeneratedMethodAccessor21.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at
> org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:78)
> at
> org.apache.hive.service.cli.session.HiveSessionProxy.access$000(HiveSessionProxy.java:36)
> at
> org.apache.hive.service.cli.session.HiveSessionProxy$1.run(HiveSessionProxy.java:63)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
> at
> org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:59)
> at com.sun.proxy.$Proxy20.executeStatementAsync(Unknown Source)
> at
> org.apache.hive.service.cli.CLIService.executeStatementAsync(CLIService.java:274)
> at
> org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(ThriftCLIService.java:486)
> at
> org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1313)
> at
> org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1298)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at
> org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1298)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> at org.apache.thrift.server.TServlet.doPost(TServlet.java:83)
> at
> org.apache.hive.service.cli.thrift.ThriftHttpServlet.doPost(ThriftHttpServlet.java:171)
> at
[jira] [Updated] (HIVE-9642) Hive metastore client retries don't happen consistently for all api calls
[ https://issues.apache.org/jira/browse/HIVE-9642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Dai updated HIVE-9642: - Attachment: HIVE-9642.7.patch Missing test case in the previous patch. > Hive metastore client retries don't happen consistently for all api calls > - > > Key: HIVE-9642 > URL: https://issues.apache.org/jira/browse/HIVE-9642 > Project: Hive > Issue Type: Bug >Affects Versions: 1.0.0 >Reporter: Xiaobing Zhou >Assignee: Daniel Dai > Attachments: HIVE-9642.1.patch, HIVE-9642.2.patch, HIVE-9642.3.patch, > HIVE-9642.4.patch, HIVE-9642.5.patch, HIVE-9642.5.patch, HIVE-9642.6.patch, > HIVE-9642.7.patch > > > When org.apache.thrift.transport.TTransportException is thrown for issues > like socket timeout, the retry via RetryingMetaStoreClient happens only in > certain cases. > Retry happens for the getDatabase call in but not for getAllDatabases(). > The reason is RetryingMetaStoreClient checks for TTransportException being > the cause for InvocationTargetException. But in case of some calls such as > getAllDatabases in HiveMetastoreClient, all exceptions get wrapped in a > MetaException. We should remove this unnecessary wrapping of exceptions for > certain functions in HMC. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11527) bypass HiveServer2 thrift interface for query results
[ https://issues.apache.org/jira/browse/HIVE-11527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15064723#comment-15064723 ] Vaibhav Gumashta commented on HIVE-11527: - [~tasanuma0829] Thanks for the useful work. I have one concern regarding the approach: it seems you're not using webhdfs to read the results as intended in the original proposal. > bypass HiveServer2 thrift interface for query results > - > > Key: HIVE-11527 > URL: https://issues.apache.org/jira/browse/HIVE-11527 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Reporter: Sergey Shelukhin >Assignee: Takanobu Asanuma > Attachments: HIVE-11527.WIP.patch > > > Right now, HS2 reads query results and returns them to the caller via its > thrift API. > There should be an option for HS2 to return some pointer to results (an HDFS > link?) and for the user to read the results directly off HDFS inside the > cluster, or via something like WebHDFS outside the cluster > Review board link: https://reviews.apache.org/r/40867 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12577) NPE in LlapTaskCommunicator when unregistering containers
[
https://issues.apache.org/jira/browse/HIVE-12577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15064731#comment-15064731
]
Sergey Shelukhin commented on HIVE-12577:
-
+1 pending test run
> NPE in LlapTaskCommunicator when unregistering containers
> -
>
> Key: HIVE-12577
> URL: https://issues.apache.org/jira/browse/HIVE-12577
> Project: Hive
> Issue Type: Bug
> Components: llap
>Affects Versions: 2.0.0
>Reporter: Siddharth Seth
>Assignee: Siddharth Seth
>Priority: Critical
> Attachments: HIVE-12577.1.review.txt, HIVE-12577.1.txt,
> HIVE-12577.1.wip.txt, HIVE-12577.2.review.txt, HIVE-12577.2.txt
>
>
> {code}
> 2015-12-02 13:29:00,160 [ERROR] [Dispatcher thread {Central}]
> |common.AsyncDispatcher|: Error in dispatcher thread
> java.lang.NullPointerException
> at
> org.apache.hadoop.hive.llap.tezplugins.LlapTaskCommunicator$EntityTracker.unregisterContainer(LlapTaskCommunicator.java:586)
> at
> org.apache.hadoop.hive.llap.tezplugins.LlapTaskCommunicator.registerContainerEnd(LlapTaskCommunicator.java:188)
> at
> org.apache.tez.dag.app.TaskCommunicatorManager.unregisterRunningContainer(TaskCommunicatorManager.java:389)
> at
> org.apache.tez.dag.app.rm.container.AMContainerImpl.unregisterFromTAListener(AMContainerImpl.java:1121)
> at
> org.apache.tez.dag.app.rm.container.AMContainerImpl$StopRequestAtLaunchingTransition.transition(AMContainerImpl.java:699)
> at
> org.apache.tez.dag.app.rm.container.AMContainerImpl$StopRequestAtIdleTransition.transition(AMContainerImpl.java:805)
> at
> org.apache.tez.dag.app.rm.container.AMContainerImpl$StopRequestAtRunningTransition.transition(AMContainerImpl.java:892)
> at
> org.apache.tez.dag.app.rm.container.AMContainerImpl$StopRequestAtRunningTransition.transition(AMContainerImpl.java:887)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory$SingleInternalArc.doTransition(StateMachineFactory.java:362)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:302)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:46)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:448)
> at
> org.apache.tez.dag.app.rm.container.AMContainerImpl.handle(AMContainerImpl.java:415)
> at
> org.apache.tez.dag.app.rm.container.AMContainerImpl.handle(AMContainerImpl.java:72)
> at
> org.apache.tez.dag.app.rm.container.AMContainerMap.handle(AMContainerMap.java:60)
> at
> org.apache.tez.dag.app.rm.container.AMContainerMap.handle(AMContainerMap.java:36)
> at
> org.apache.tez.common.AsyncDispatcher.dispatch(AsyncDispatcher.java:183)
> at
> org.apache.tez.common.AsyncDispatcher$1.run(AsyncDispatcher.java:114)
> at java.lang.Thread.run(Thread.java:745)
> 2015-12-02 13:29:00,167 [ERROR] [Dispatcher thread {Central}]
> |common.AsyncDispatcher|: Error in dispatcher thread
> java.lang.NullPointerException
> at
> org.apache.tez.dag.app.TaskCommunicatorManager.unregisterRunningContainer(TaskCommunicatorManager.java:386)
> at
> org.apache.tez.dag.app.rm.container.AMContainerImpl.unregisterFromTAListener(AMContainerImpl.java:1121)
> at
> org.apache.tez.dag.app.rm.container.AMContainerImpl$StopRequestAtLaunchingTransition.transition(AMContainerImpl.java:699)
> at
> org.apache.tez.dag.app.rm.container.AMContainerImpl$StopRequestAtIdleTransition.transition(AMContainerImpl.java:805)
> at
> org.apache.tez.dag.app.rm.container.AMContainerImpl$StopRequestAtRunningTransition.transition(AMContainerImpl.java:892)
> at
> org.apache.tez.dag.app.rm.container.AMContainerImpl$StopRequestAtRunningTransition.transition(AMContainerImpl.java:887)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory$SingleInternalArc.doTransition(StateMachineFactory.java:362)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:302)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:46)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:448)
> at
> org.apache.tez.dag.app.rm.container.AMContainerImpl.handle(AMContainerImpl.java:415)
> at
> org.apache.tez.dag.app.rm.container.AMContainerImpl.handle(AMContainerImpl.java:72)
> at
> org.apache.tez.dag.app.rm.container.AMContainerMap.handle(AMContainerMap.java:60)
> at
> org.apache.tez.dag.app.rm.container.AMContainerMap.handle(AMContainerMap.java:36)
> at
>
[jira] [Commented] (HIVE-12711) Document howto disable web ui in config of hive.server2.webui.port
[ https://issues.apache.org/jira/browse/HIVE-12711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15064957#comment-15064957 ] Thejas M Nair commented on HIVE-12711: -- [~jxiang] [~szehon] Can you please review ? > Document howto disable web ui in config of hive.server2.webui.port > -- > > Key: HIVE-12711 > URL: https://issues.apache.org/jira/browse/HIVE-12711 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2 >Reporter: Thejas M Nair >Assignee: Thejas M Nair > Attachments: HIVE-12711.1.patch > > > hive.server2.webui.port config does not say that it can be used to disable > webui as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12711) Document howto disable web ui in config of hive.server2.webui.port
[ https://issues.apache.org/jira/browse/HIVE-12711?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thejas M Nair updated HIVE-12711: - Attachment: HIVE-12711.1.patch > Document howto disable web ui in config of hive.server2.webui.port > -- > > Key: HIVE-12711 > URL: https://issues.apache.org/jira/browse/HIVE-12711 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2 >Reporter: Thejas M Nair >Assignee: Thejas M Nair > Attachments: HIVE-12711.1.patch > > > hive.server2.webui.port config does not say that it can be used to disable > webui as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12331) Remove hive.enforce.bucketing & hive.enforce.sorting configs
[ https://issues.apache.org/jira/browse/HIVE-12331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-12331: Hadoop Flags: Incompatible change [~leftylev] updated wiki > Remove hive.enforce.bucketing & hive.enforce.sorting configs > > > Key: HIVE-12331 > URL: https://issues.apache.org/jira/browse/HIVE-12331 > Project: Hive > Issue Type: Improvement > Components: Configuration >Reporter: Ashutosh Chauhan >Assignee: Ashutosh Chauhan > Labels: TODOC2.0 > Fix For: 2.0.0 > > Attachments: HIVE-12331.1.patch, HIVE-12331.patch > > > If table is created as bucketed and/or sorted and this config is set to > false, you will insert data in wrong buckets and/or sort order and then if > you use these tables subsequently in BMJ or SMBJ you will get wrong results. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12605) Implement JDBC Connection.isValid
[
https://issues.apache.org/jira/browse/HIVE-12605?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gabor Liptak updated HIVE-12605:
Attachment: HIVE-12605.3.patch
> Implement JDBC Connection.isValid
> -
>
> Key: HIVE-12605
> URL: https://issues.apache.org/jira/browse/HIVE-12605
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2, JDBC
>Reporter: Thejas M Nair
> Labels: newbie, trivial
> Attachments: HIVE-12605.1.patch, HIVE-12605.2.patch,
> HIVE-12605.3.patch
>
>
> http://docs.oracle.com/javase/7/docs/api/java/sql/Connection.html#isValid(int)
> implementation in Hive JDBC driver throws "SQLException("Method not
> supported")".
> That is a method often used by connection pooling libraries.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-12712) HiveInputFormat may fail to column names to read in some cases
[
https://issues.apache.org/jira/browse/HIVE-12712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15065064#comment-15065064
]
Prasanth Jayachandran commented on HIVE-12712:
--
[~sershe] Could you please review this patch?
> HiveInputFormat may fail to column names to read in some cases
> --
>
> Key: HIVE-12712
> URL: https://issues.apache.org/jira/browse/HIVE-12712
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.0.0, 2.1.0
>Reporter: Takahiko Saito
>Assignee: Prasanth Jayachandran
> Attachments: HIVE-12712.1.patch
>
>
> The primary issue is when plan is generated pathToAliases map is populated
> with directory paths to table aliases. pathToAliases.put() uses
> path.toString() as map key. During probing, path.toUri().toString() is used.
> This can cause probe misses when path contains spaces in them. path.toUri()
> will escape the spaces in the path whereas path.toString() does not escape
> the spaces. As a result, HiveInputFormat can trigger a different code path
> which can fail to set list of columns to read from the source table. This was
> causing unexpected NPE in OrcInputFormat (after refactoring HIVE-11705) which
> removed null check for column names. The resulting exception is
> {code}
> Caused by: java.lang.RuntimeException: ORC split generation failed with
> exception: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.generateSplitsInfo(OrcInputFormat.java:1288)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getSplits(OrcInputFormat.java:1354)
> at
> org.apache.hadoop.hive.ql.io.HiveInputFormat.addSplitsForGroup(HiveInputFormat.java:367)
> at
> org.apache.hadoop.hive.ql.io.HiveInputFormat.getSplits(HiveInputFormat.java:457)
> at
> org.apache.hadoop.hive.ql.exec.tez.HiveSplitGenerator.initialize(HiveSplitGenerator.java:152)
> at
> org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:246)
> at
> org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:240)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
> at
> org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:240)
> at
> org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:227)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> ... 3 more
> Caused by: java.util.concurrent.ExecutionException:
> java.lang.NullPointerException
> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
> at java.util.concurrent.FutureTask.get(FutureTask.java:192)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.generateSplitsInfo(OrcInputFormat.java:1282)
> ... 15 more
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.extractNeededColNames(OrcInputFormat.java:422)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.extractNeededColNames(OrcInputFormat.java:417)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.access$2000(OrcInputFormat.java:134)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$SplitGenerator.call(OrcInputFormat.java:1072)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$SplitGenerator.call(OrcInputFormat.java:919)
> ... 4 more
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-12442) Refactor/repackage HiveServer2's Thrift code so that it can be used in the tasks
[ https://issues.apache.org/jira/browse/HIVE-12442?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vaibhav Gumashta updated HIVE-12442: Description: For implementing HIVE-12427, the tasks will need to have knowledge of thrift types from HS2's thrift API. This jira will look at the least invasive way to do that. was: For implementing HIVE-12427, the tasks will need to have knowledge of thrift types from HS2's thrift API. This jira will look at the least invasive way to do that. https://reviews.apache.org/r/41379 > Refactor/repackage HiveServer2's Thrift code so that it can be used in the > tasks > > > Key: HIVE-12442 > URL: https://issues.apache.org/jira/browse/HIVE-12442 > Project: Hive > Issue Type: Sub-task >Affects Versions: 1.2.1 >Reporter: Vaibhav Gumashta >Assignee: Rohit Dholakia > Labels: hiveserver, thrift > Attachments: hive-12442.1.patch, hive-12442.2.patch > > > For implementing HIVE-12427, the tasks will need to have knowledge of thrift > types from HS2's thrift API. This jira will look at the least invasive way to > do that. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12442) Refactor/repackage HiveServer2's Thrift code so that it can be used in the tasks
[ https://issues.apache.org/jira/browse/HIVE-12442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15065112#comment-15065112 ] Vaibhav Gumashta commented on HIVE-12442: - [~rohitdholakia] Patch v2 looks good except some minor comments. > Refactor/repackage HiveServer2's Thrift code so that it can be used in the > tasks > > > Key: HIVE-12442 > URL: https://issues.apache.org/jira/browse/HIVE-12442 > Project: Hive > Issue Type: Sub-task >Affects Versions: 1.2.1 >Reporter: Vaibhav Gumashta >Assignee: Rohit Dholakia > Labels: hiveserver, thrift > Attachments: hive-12442.1.patch, hive-12442.2.patch > > > For implementing HIVE-12427, the tasks will need to have knowledge of thrift > types from HS2's thrift API. This jira will look at the least invasive way to > do that. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12429) Switch default Hive authorization to SQLStandardAuth in 2.0
[
https://issues.apache.org/jira/browse/HIVE-12429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15064943#comment-15064943
]
Hive QA commented on HIVE-12429:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12778462/HIVE-12429.3.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6400/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6400/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-6400/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hive-ptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ [[ -n /usr/java/jdk1.7.0_45-cloudera ]]
+ export JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera
+ JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera
+ export
PATH=/usr/java/jdk1.7.0_45-cloudera/bin/:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.7.0_45-cloudera/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin
+
PATH=/usr/java/jdk1.7.0_45-cloudera/bin/:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.7.0_45-cloudera/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'M2_OPTS=-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost
-Dhttp.proxyPort=3128'
+ M2_OPTS='-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost
-Dhttp.proxyPort=3128'
+ cd /data/hive-ptest/working/
+ tee /data/hive-ptest/logs/PreCommit-HIVE-TRUNK-Build-6400/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ cd apache-github-source-source
+ git fetch origin
>From https://github.com/apache/hive
a17c95e..1d5e9c9 branch-2.0 -> origin/branch-2.0
9496409..7df6202 master -> origin/master
9af0b27..a116e96 spark -> origin/spark
+ git reset --hard HEAD
HEAD is now at 9496409 HIVE-11935: Race condition in HiveMetaStoreClient:
isCompatibleWith and close (Daniel Dai, Reviewed by Alan Gates)
+ git clean -f -d
Removing
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/HiveRexExecutorImpl.java
Removing
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveReduceExpressionsRule.java
Removing ql/src/test/queries/clientpositive/cbo_const.q
Removing ql/src/test/queries/clientpositive/constantfolding.q
Removing ql/src/test/results/clientpositive/cbo_const.q.out
Removing ql/src/test/results/clientpositive/constantfolding.q.out
+ git checkout master
Already on 'master'
Your branch is behind 'origin/master' by 2 commits, and can be fast-forwarded.
+ git reset --hard origin/master
HEAD is now at 7df6202 HIVE-12075 : add analyze command to explictly cache file
metadata in HBase metastore (Sergey Shelukhin, reviewed by Alan Gates)
+ git merge --ff-only origin/master
Already up-to-date.
+ git gc
+ patchCommandPath=/data/hive-ptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hive-ptest/working/scratch/build.patch
+ [[ -f /data/hive-ptest/working/scratch/build.patch ]]
+ chmod +x /data/hive-ptest/working/scratch/smart-apply-patch.sh
+ /data/hive-ptest/working/scratch/smart-apply-patch.sh
/data/hive-ptest/working/scratch/build.patch
The patch does not appear to apply with p0, p1, or p2
+ exit 1
'
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12778462 - PreCommit-HIVE-TRUNK-Build
> Switch default Hive authorization to SQLStandardAuth in 2.0
> ---
>
> Key: HIVE-12429
> URL: https://issues.apache.org/jira/browse/HIVE-12429
> Project: Hive
> Issue Type: Task
> Components: Authorization, Security
>Affects Versions: 2.0.0
>Reporter: Alan Gates
>Assignee: Daniel Dai
> Attachments: HIVE-12429.1.patch, HIVE-12429.2.patch,
> HIVE-12429.3.patch
>
>
> Hive's default authorization is not real security, as it does not secure a
> number of features and anyone can grant access to any object to any user. We
> should switch the default to SQLStandardAuth, which provides real
> authentication.
> As this is a backwards incompatible change this was hard to do previously,
> but 2.0 gives us a place to do this type of change.
> By default authorization will still be off, as there are a few other things
> to set
[jira] [Commented] (HIVE-12711) Document howto disable web ui in config of hive.server2.webui.port
[ https://issues.apache.org/jira/browse/HIVE-12711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15064961#comment-15064961 ] Szehon Ho commented on HIVE-12711: -- +1 thanks for adding it > Document howto disable web ui in config of hive.server2.webui.port > -- > > Key: HIVE-12711 > URL: https://issues.apache.org/jira/browse/HIVE-12711 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2 >Reporter: Thejas M Nair >Assignee: Thejas M Nair > Attachments: HIVE-12711.1.patch > > > hive.server2.webui.port config does not say that it can be used to disable > webui as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-1841) datanucleus.fixedDatastore should be true in hive-default.xml
[
https://issues.apache.org/jira/browse/HIVE-1841?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashutosh Chauhan updated HIVE-1841:
---
Hadoop Flags: Incompatible change
[~leftylev] Updated wiki.
> datanucleus.fixedDatastore should be true in hive-default.xml
> --
>
> Key: HIVE-1841
> URL: https://issues.apache.org/jira/browse/HIVE-1841
> Project: Hive
> Issue Type: Improvement
> Components: Configuration, Metastore
>Affects Versions: 0.6.0
>Reporter: Edward Capriolo
>Assignee: Ashutosh Chauhan
>Priority: Minor
> Labels: TODOC2.0
> Fix For: 2.0.0
>
> Attachments: HIVE-1841.1.patch.txt, HIVE-1841.2.patch, HIVE-1841.patch
>
>
> Two datanucleus variables:
> {noformat}
>
> datanucleus.autoCreateSchema
> false
>
>
> datanucleus.fixedDatastore
> true
>
> {noformat}
> are dangerous. We do want the schema to auto-create itself, but we do not
> want the schema to auto update itself.
> Someone might accidentally point a trunk at the wrong meta-store and
> unknowingly update. I believe we should set this to false and possibly trap
> exceptions stemming from hive wanting to do any update. This way someone has
> to actively acknowledge the update, by setting this to true and then starting
> up hive, or leaving it false, removing schema modifies for the user that hive
> usages, and doing all the time and doing the updates by hand.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-12413) Default mode for hive.mapred.mode should be strict
[ https://issues.apache.org/jira/browse/HIVE-12413?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-12413: Hadoop Flags: Incompatible change [~leftylev] Updated wiki > Default mode for hive.mapred.mode should be strict > -- > > Key: HIVE-12413 > URL: https://issues.apache.org/jira/browse/HIVE-12413 > Project: Hive > Issue Type: Task > Components: Configuration >Reporter: Ashutosh Chauhan >Assignee: Ashutosh Chauhan > Labels: TODOC2.0 > Fix For: 2.0.0 > > Attachments: HIVE-12413.1.patch, HIVE-12413.2.patch, > HIVE-12413.3.patch, HIVE-12413.4.patch, HIVE-12413.patch > > > Non-strict mode allows some questionable semantics and questionable > operations. Its better that user makes a conscious choice to enable such a > behavior. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12644) Support for offset in HiveSortMergeRule
[ https://issues.apache.org/jira/browse/HIVE-12644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-12644: --- Attachment: HIVE-12644.01.patch > Support for offset in HiveSortMergeRule > --- > > Key: HIVE-12644 > URL: https://issues.apache.org/jira/browse/HIVE-12644 > Project: Hive > Issue Type: Bug > Components: CBO >Affects Versions: 2.1.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > Attachments: HIVE-12644.01.patch, HIVE-12644.patch > > > After HIVE-11531 goes in, HiveSortMergeRule needs to be extended to support > offset properly when it merges operators that contain Limit. Otherwise, limit > pushdown through outer join optimization (introduced in HIVE-11684) will not > work properly. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12605) Implement JDBC Connection.isValid
[
https://issues.apache.org/jira/browse/HIVE-12605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15064990#comment-15064990
]
Thejas M Nair commented on HIVE-12605:
--
Thanks for adding the test!
Can you also please add a negative test case ?
> Implement JDBC Connection.isValid
> -
>
> Key: HIVE-12605
> URL: https://issues.apache.org/jira/browse/HIVE-12605
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2, JDBC
>Reporter: Thejas M Nair
> Labels: newbie, trivial
> Attachments: HIVE-12605.1.patch, HIVE-12605.2.patch
>
>
> http://docs.oracle.com/javase/7/docs/api/java/sql/Connection.html#isValid(int)
> implementation in Hive JDBC driver throws "SQLException("Method not
> supported")".
> That is a method often used by connection pooling libraries.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-12712) HiveInputFormat may fail to column names to read in some cases
[
https://issues.apache.org/jira/browse/HIVE-12712?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Prasanth Jayachandran updated HIVE-12712:
-
Attachment: HIVE-12712.1.patch
> HiveInputFormat may fail to column names to read in some cases
> --
>
> Key: HIVE-12712
> URL: https://issues.apache.org/jira/browse/HIVE-12712
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.0.0, 2.1.0
>Reporter: Takahiko Saito
>Assignee: Prasanth Jayachandran
> Attachments: HIVE-12712.1.patch
>
>
> The primary issue is when plan is generated pathToAliases map is populated
> with directory paths to table aliases. pathToAliases.put() uses
> path.toString() as map key. During probing, path.toUri().toString() is used.
> This can cause probe misses when path contains spaces in them. path.toUri()
> will escape the spaces in the path whereas path.toString() does not escape
> the spaces. As a result, HiveInputFormat can trigger a different code path
> which can fail to set list of columns to read from the source table. This was
> causing unexpected NPE in OrcInputFormat (after refactoring HIVE-11705) which
> removed null check for column names. The resulting exception is
> {code}
> Caused by: java.lang.RuntimeException: ORC split generation failed with
> exception: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.generateSplitsInfo(OrcInputFormat.java:1288)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getSplits(OrcInputFormat.java:1354)
> at
> org.apache.hadoop.hive.ql.io.HiveInputFormat.addSplitsForGroup(HiveInputFormat.java:367)
> at
> org.apache.hadoop.hive.ql.io.HiveInputFormat.getSplits(HiveInputFormat.java:457)
> at
> org.apache.hadoop.hive.ql.exec.tez.HiveSplitGenerator.initialize(HiveSplitGenerator.java:152)
> at
> org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:246)
> at
> org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:240)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
> at
> org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:240)
> at
> org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:227)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> ... 3 more
> Caused by: java.util.concurrent.ExecutionException:
> java.lang.NullPointerException
> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
> at java.util.concurrent.FutureTask.get(FutureTask.java:192)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.generateSplitsInfo(OrcInputFormat.java:1282)
> ... 15 more
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.extractNeededColNames(OrcInputFormat.java:422)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.extractNeededColNames(OrcInputFormat.java:417)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.access$2000(OrcInputFormat.java:134)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$SplitGenerator.call(OrcInputFormat.java:1072)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$SplitGenerator.call(OrcInputFormat.java:919)
> ... 4 more
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-12338) Add webui to HiveServer2
[ https://issues.apache.org/jira/browse/HIVE-12338?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15065090#comment-15065090 ] Thejas M Nair commented on HIVE-12338: -- I am yet to go through the patch, but does this allow all users to see all activity and all config ? Is there a proposal to introduce any 'admin user' concept ? Regarding configuration, there are some parameters which should not be exposed to all users. See HIVE-9013. > Add webui to HiveServer2 > > > Key: HIVE-12338 > URL: https://issues.apache.org/jira/browse/HIVE-12338 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Reporter: Jimmy Xiang >Assignee: Jimmy Xiang > Attachments: HIVE-12338.1.patch, HIVE-12338.2.patch, > HIVE-12338.3.patch, HIVE-12338.4.patch, hs2-conf.png, hs2-logs.png, > hs2-metrics.png, hs2-webui.png > > > A web ui for HiveServer2 can show some useful information such as: > > 1. Sessions, > 2. Queries that are executing on the HS2, their states, starting time, etc. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12502) to_date UDF cannot accept NULLs of VOID type
[ https://issues.apache.org/jira/browse/HIVE-12502?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Dere updated HIVE-12502: -- Attachment: HIVE-12502.1.patch pre-commit tests didn't appear to run for this Jira, re-uploading patch > to_date UDF cannot accept NULLs of VOID type > > > Key: HIVE-12502 > URL: https://issues.apache.org/jira/browse/HIVE-12502 > Project: Hive > Issue Type: Bug > Components: UDF >Affects Versions: 1.0.0 >Reporter: Aaron Tokhy >Assignee: Aaron Tokhy >Priority: Trivial > Attachments: HIVE-12502-branch-1.patch, HIVE-12502.1.patch, > HIVE-12502.patch > > > The to_date method behaves differently based off the 'data type' of null > passed in. > hive> select to_date(null); > FAILED: SemanticException [Error 10014]: Line 1:7 Wrong arguments 'TOK_NULL': > TO_DATE() only takes STRING/TIMESTAMP/DATEWRITABLE types, got VOID > hive> select to_date(cast(null as timestamp)); > OK > NULL > Time taken: 0.031 seconds, Fetched: 1 row(s) > This appears to be a regression introduced in HIVE-5731. The previous > version of to_date would not check the type: > https://github.com/apache/hive/commit/09b6553214d6db5ec7049b88bbe8ff640a7fef72#diff-204f5588c0767cf372a5ca7e3fb964afL56 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12436) Default hive.metastore.schema.verification to true
[ https://issues.apache.org/jira/browse/HIVE-12436?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-12436: Hadoop Flags: Incompatible change [~leftylev] updated wiki > Default hive.metastore.schema.verification to true > -- > > Key: HIVE-12436 > URL: https://issues.apache.org/jira/browse/HIVE-12436 > Project: Hive > Issue Type: Task > Components: Metastore >Reporter: Ashutosh Chauhan >Assignee: Ashutosh Chauhan > Labels: TODOC2.0 > Fix For: 2.0.0 > > Attachments: HIVE-12436.patch > > > It enforces metastore schema version consistency -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11865) Disable Hive PPD optimizer when CBO has optimized the plan
[ https://issues.apache.org/jira/browse/HIVE-11865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15065000#comment-15065000 ] Laljo John Pullokkaran commented on HIVE-11865: --- May be you could address this in a followup Jira. > Disable Hive PPD optimizer when CBO has optimized the plan > -- > > Key: HIVE-11865 > URL: https://issues.apache.org/jira/browse/HIVE-11865 > Project: Hive > Issue Type: Bug > Components: CBO, Logical Optimizer >Affects Versions: 2.0.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > Attachments: HIVE-11865.01.patch, HIVE-11865.02.patch, > HIVE-11865.02.patch, HIVE-11865.03.patch, HIVE-11865.04.patch, > HIVE-11865.05.patch, HIVE-11865.06.patch, HIVE-11865.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11865) Disable Hive PPD optimizer when CBO has optimized the plan
[ https://issues.apache.org/jira/browse/HIVE-11865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15064999#comment-15064999 ] Laljo John Pullokkaran commented on HIVE-11865: --- [~jcamachorodriguez] SimplePredicatePushDown seems to be brute force and could be refined. SimpleFilterPPD shouldn't it check for DPP also. Now it seems like it would process always. Also SimplePredicatePushDown would re evaluate predicates that might have been evaluated before and deemed unfit for push down. This is not ideal. > Disable Hive PPD optimizer when CBO has optimized the plan > -- > > Key: HIVE-11865 > URL: https://issues.apache.org/jira/browse/HIVE-11865 > Project: Hive > Issue Type: Bug > Components: CBO, Logical Optimizer >Affects Versions: 2.0.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > Attachments: HIVE-11865.01.patch, HIVE-11865.02.patch, > HIVE-11865.02.patch, HIVE-11865.03.patch, HIVE-11865.04.patch, > HIVE-11865.05.patch, HIVE-11865.06.patch, HIVE-11865.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12711) Document howto disable web ui in config of hive.server2.webui.port
[ https://issues.apache.org/jira/browse/HIVE-12711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15065001#comment-15065001 ] Jimmy Xiang commented on HIVE-12711: +1 > Document howto disable web ui in config of hive.server2.webui.port > -- > > Key: HIVE-12711 > URL: https://issues.apache.org/jira/browse/HIVE-12711 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2 >Reporter: Thejas M Nair >Assignee: Thejas M Nair > Attachments: HIVE-12711.1.patch > > > hive.server2.webui.port config does not say that it can be used to disable > webui as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12429) Switch default Hive authorization to SQLStandardAuth in 2.0
[ https://issues.apache.org/jira/browse/HIVE-12429?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Dai updated HIVE-12429: -- Attachment: HIVE-12429.4.patch > Switch default Hive authorization to SQLStandardAuth in 2.0 > --- > > Key: HIVE-12429 > URL: https://issues.apache.org/jira/browse/HIVE-12429 > Project: Hive > Issue Type: Task > Components: Authorization, Security >Affects Versions: 2.0.0 >Reporter: Alan Gates >Assignee: Daniel Dai > Attachments: HIVE-12429.1.patch, HIVE-12429.2.patch, > HIVE-12429.3.patch, HIVE-12429.4.patch > > > Hive's default authorization is not real security, as it does not secure a > number of features and anyone can grant access to any object to any user. We > should switch the default to SQLStandardAuth, which provides real > authentication. > As this is a backwards incompatible change this was hard to do previously, > but 2.0 gives us a place to do this type of change. > By default authorization will still be off, as there are a few other things > to set when turning on authorization (such as the list of admin users). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HIVE-12697) Remove deprecated post option from webhcat test files
[ https://issues.apache.org/jira/browse/HIVE-12697?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman resolved HIVE-12697. --- Resolution: Fixed Fix Version/s: 2.1.0 2.0.0 1.3.0 > Remove deprecated post option from webhcat test files > - > > Key: HIVE-12697 > URL: https://issues.apache.org/jira/browse/HIVE-12697 > Project: Hive > Issue Type: Test > Components: WebHCat >Affects Versions: 2.0.0 >Reporter: Aswathy Chellammal Sreekumar >Assignee: Aswathy Chellammal Sreekumar > Labels: test > Fix For: 1.3.0, 2.0.0, 2.1.0 > > Attachments: HIVE-12697.1.patch, HIVE-12697.2.patch > > > Tests are still having the deprecated post option user.name. Need to remove > them and add the same to query string > Submitting user.name as form parameter in POST method was deprecated in Hive > 0.13 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12623) Add an option to force allocation of fragments on requested nodes
[ https://issues.apache.org/jira/browse/HIVE-12623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15065022#comment-15065022 ] Gunther Hagleitner commented on HIVE-12623: --- Change and test LGTM +1. Submitted the patch to HiveQA. This add a new config: hive.llap.task.scheduler.locality.delay. The value is used once at AM startup, which means it's best to change it before starting HS2. > Add an option to force allocation of fragments on requested nodes > - > > Key: HIVE-12623 > URL: https://issues.apache.org/jira/browse/HIVE-12623 > Project: Hive > Issue Type: Improvement > Components: llap >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Attachments: HIVE-12623.1.txt, HIVE-12623.1.wip.txt > > > Currently, fragments are sent to random nodes if the requested node does not > have capacity. In certain situations there's more to be gained by sending the > fragments to the requested node only. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12685) Remove invalid property in common/src/test/resources/hive-site.xml
[
https://issues.apache.org/jira/browse/HIVE-12685?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15065056#comment-15065056
]
Wei Zheng commented on HIVE-12685:
--
There's a new mismatch from the last run of ptest:
{code}
org.apache.hadoop.hive.metastore.TestHiveMetaStorePartitionSpecs.testAddPartitions
{code}
But the test passes locally on my laptop.
> Remove invalid property in common/src/test/resources/hive-site.xml
> --
>
> Key: HIVE-12685
> URL: https://issues.apache.org/jira/browse/HIVE-12685
> Project: Hive
> Issue Type: Bug
> Components: Configuration
>Affects Versions: 2.0.0, 2.1.0
>Reporter: Wei Zheng
>Assignee: Wei Zheng
> Attachments: HIVE-12685.1.patch, HIVE-12685.2.patch,
> HIVE-12685.3.patch, HIVE-12685.4.patch, HIVE-12685.5.patch
>
>
> Currently there's such a property as below, which is obviously wrong
> {code}
>
> javax.jdo.option.ConnectionDriverName
> hive-site.xml
> Override ConfVar defined in HiveConf
>
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-12674) HS2 Tez sessions should have maximum age
[ https://issues.apache.org/jira/browse/HIVE-12674?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-12674: Attachment: HIVE-12674.01.patch > HS2 Tez sessions should have maximum age > > > Key: HIVE-12674 > URL: https://issues.apache.org/jira/browse/HIVE-12674 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-12674.01.patch, HIVE-12674.patch > > > Certain tokens passed to AM by clients (e.g. an HDFS token) have maximum > lifetime beyond which they cannot be renewed. We should cycle long-lived > session AMs after a configurable period to avoid problems with these. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12685) Remove redundant hive-site.xml under common/src/test/resources/
[
https://issues.apache.org/jira/browse/HIVE-12685?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wei Zheng updated HIVE-12685:
-
Summary: Remove redundant hive-site.xml under common/src/test/resources/
(was: Remove invalid property in common/src/test/resources/hive-site.xml)
> Remove redundant hive-site.xml under common/src/test/resources/
> ---
>
> Key: HIVE-12685
> URL: https://issues.apache.org/jira/browse/HIVE-12685
> Project: Hive
> Issue Type: Bug
> Components: Configuration
>Affects Versions: 2.0.0, 2.1.0
>Reporter: Wei Zheng
>Assignee: Wei Zheng
> Attachments: HIVE-12685.1.patch, HIVE-12685.2.patch,
> HIVE-12685.3.patch, HIVE-12685.4.patch, HIVE-12685.5.patch
>
>
> Currently there's such a property as below, which is obviously wrong
> {code}
>
> javax.jdo.option.ConnectionDriverName
> hive-site.xml
> Override ConfVar defined in HiveConf
>
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-11927) Implement/Enable constant related optimization rules in Calcite: enable HiveReduceExpressionsRule to fold constants
[
https://issues.apache.org/jira/browse/HIVE-11927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15064937#comment-15064937
]
Hive QA commented on HIVE-11927:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12778448/HIVE-11927.15.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 23 failed/errored test(s), 9965 tests
executed
*Failed tests:*
{noformat}
TestHWISessionManager - did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join_stats2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats5
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver_encryption_insert_partition_dynamic
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_bucketizedhiveinputformat
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_explainuser_3
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_unionDistinct_2
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_import
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_stats5
org.apache.hadoop.hive.ql.exec.spark.session.TestSparkSessionManagerImpl.testMultiSessionMultipleUse
org.apache.hadoop.hive.ql.exec.spark.session.TestSparkSessionManagerImpl.testSingleSessionMultipleUse
org.apache.hadoop.hive.ql.security.TestMetastoreAuthorizationProvider.testSimplePrivileges
org.apache.hive.jdbc.TestJdbcDriver2.testShowGrant
org.apache.hive.jdbc.TestJdbcDriver2.testShowRoleGrant
org.apache.hive.jdbc.TestSSL.testSSLVersion
org.apache.hive.spark.client.TestSparkClient.testAddJarsAndFiles
org.apache.hive.spark.client.TestSparkClient.testCounters
org.apache.hive.spark.client.TestSparkClient.testErrorJob
org.apache.hive.spark.client.TestSparkClient.testJobSubmission
org.apache.hive.spark.client.TestSparkClient.testMetricsCollection
org.apache.hive.spark.client.TestSparkClient.testRemoteClient
org.apache.hive.spark.client.TestSparkClient.testSimpleSparkJob
org.apache.hive.spark.client.TestSparkClient.testSyncRpc
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6399/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6399/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-6399/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 23 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12778448 - PreCommit-HIVE-TRUNK-Build
> Implement/Enable constant related optimization rules in Calcite: enable
> HiveReduceExpressionsRule to fold constants
> ---
>
> Key: HIVE-11927
> URL: https://issues.apache.org/jira/browse/HIVE-11927
> Project: Hive
> Issue Type: Sub-task
>Reporter: Pengcheng Xiong
>Assignee: Pengcheng Xiong
> Attachments: HIVE-11927.01.patch, HIVE-11927.02.patch,
> HIVE-11927.03.patch, HIVE-11927.04.patch, HIVE-11927.05.patch,
> HIVE-11927.06.patch, HIVE-11927.07.patch, HIVE-11927.08.patch,
> HIVE-11927.09.patch, HIVE-11927.10.patch, HIVE-11927.11.patch,
> HIVE-11927.12.patch, HIVE-11927.13.patch, HIVE-11927.14.patch,
> HIVE-11927.15.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-12710) add better logging for Tez session creation thread failures
[ https://issues.apache.org/jira/browse/HIVE-12710?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-12710: Attachment: HIVE-12710.patch [~gopalv] [~sseth] [~hagleitn] can you review? A small logging patch. > add better logging for Tez session creation thread failures > --- > > Key: HIVE-12710 > URL: https://issues.apache.org/jira/browse/HIVE-12710 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Minor > Attachments: HIVE-12710.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11527) bypass HiveServer2 thrift interface for query results
[ https://issues.apache.org/jira/browse/HIVE-11527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15064988#comment-15064988 ] Sergey Shelukhin commented on HIVE-11527: - One minor comment from my side, there's a better overload to use in logging. > bypass HiveServer2 thrift interface for query results > - > > Key: HIVE-11527 > URL: https://issues.apache.org/jira/browse/HIVE-11527 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Reporter: Sergey Shelukhin >Assignee: Takanobu Asanuma > Attachments: HIVE-11527.WIP.patch > > > Right now, HS2 reads query results and returns them to the caller via its > thrift API. > There should be an option for HS2 to return some pointer to results (an HDFS > link?) and for the user to read the results directly off HDFS inside the > cluster, or via something like WebHDFS outside the cluster > Review board link: https://reviews.apache.org/r/40867 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12644) Support for offset in HiveSortMergeRule
[
https://issues.apache.org/jira/browse/HIVE-12644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15064983#comment-15064983
]
Jesus Camacho Rodriguez commented on HIVE-12644:
[~jpullokkaran], new patch simplifies the logic in the _onMatch_ method in
{{HiveSortMergeRule}}. Further, I checked back {{HiveSortJoinReduceRule}} and
indeed offset needed to be considered in the _matches_ method. Could you check
the patch back? Thanks
> Support for offset in HiveSortMergeRule
> ---
>
> Key: HIVE-12644
> URL: https://issues.apache.org/jira/browse/HIVE-12644
> Project: Hive
> Issue Type: Bug
> Components: CBO
>Affects Versions: 2.1.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
> Attachments: HIVE-12644.01.patch, HIVE-12644.patch
>
>
> After HIVE-11531 goes in, HiveSortMergeRule needs to be extended to support
> offset properly when it merges operators that contain Limit. Otherwise, limit
> pushdown through outer join optimization (introduced in HIVE-11684) will not
> work properly.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-11865) Disable Hive PPD optimizer when CBO has optimized the plan
[ https://issues.apache.org/jira/browse/HIVE-11865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15065018#comment-15065018 ] Jesus Camacho Rodriguez commented on HIVE-11865: [~jpullokkaran], SimpleFilterPPD only considers synthetic predicates, predicates in Filter on top of TS, and predicates in Filter-Select*-PTF chains. Probably you mean that there is more logic in PPD in general that could be refined in for the SimplePPD? > Disable Hive PPD optimizer when CBO has optimized the plan > -- > > Key: HIVE-11865 > URL: https://issues.apache.org/jira/browse/HIVE-11865 > Project: Hive > Issue Type: Bug > Components: CBO, Logical Optimizer >Affects Versions: 2.0.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > Attachments: HIVE-11865.01.patch, HIVE-11865.02.patch, > HIVE-11865.02.patch, HIVE-11865.03.patch, HIVE-11865.04.patch, > HIVE-11865.05.patch, HIVE-11865.06.patch, HIVE-11865.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12712) HiveInputFormat may fail to column names to read in some cases
[
https://issues.apache.org/jira/browse/HIVE-12712?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Prasanth Jayachandran updated HIVE-12712:
-
Reporter: Takahiko Saito (was: Prasanth Jayachandran)
> HiveInputFormat may fail to column names to read in some cases
> --
>
> Key: HIVE-12712
> URL: https://issues.apache.org/jira/browse/HIVE-12712
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.0.0, 2.1.0
>Reporter: Takahiko Saito
>Assignee: Prasanth Jayachandran
>
> The primary issue is when plan is generated pathToAliases map is populated
> with directory paths to table aliases. pathToAliases.put() uses
> path.toString() as map key. During probing, path.toUri().toString() is used.
> This can cause probe misses when path contains spaces in them. path.toUri()
> will escape the spaces in the path whereas path.toString() does not escape
> the spaces. As a result, HiveInputFormat can trigger a different code path
> which can fail to set list of columns to read from the source table. This was
> causing unexpected NPE in OrcInputFormat (after refactoring HIVE-11705) which
> removed null check for column names. The resulting exception is
> {code}
> Caused by: java.lang.RuntimeException: ORC split generation failed with
> exception: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.generateSplitsInfo(OrcInputFormat.java:1288)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getSplits(OrcInputFormat.java:1354)
> at
> org.apache.hadoop.hive.ql.io.HiveInputFormat.addSplitsForGroup(HiveInputFormat.java:367)
> at
> org.apache.hadoop.hive.ql.io.HiveInputFormat.getSplits(HiveInputFormat.java:457)
> at
> org.apache.hadoop.hive.ql.exec.tez.HiveSplitGenerator.initialize(HiveSplitGenerator.java:152)
> at
> org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:246)
> at
> org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:240)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
> at
> org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:240)
> at
> org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:227)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> ... 3 more
> Caused by: java.util.concurrent.ExecutionException:
> java.lang.NullPointerException
> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
> at java.util.concurrent.FutureTask.get(FutureTask.java:192)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.generateSplitsInfo(OrcInputFormat.java:1282)
> ... 15 more
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.extractNeededColNames(OrcInputFormat.java:422)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.extractNeededColNames(OrcInputFormat.java:417)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.access$2000(OrcInputFormat.java:134)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$SplitGenerator.call(OrcInputFormat.java:1072)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$SplitGenerator.call(OrcInputFormat.java:919)
> ... 4 more
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-12442) Refactor/repackage HiveServer2's Thrift code so that it can be used in the tasks
[ https://issues.apache.org/jira/browse/HIVE-12442?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rohit Dholakia updated HIVE-12442: -- Attachment: hive-12442.2.patch > Refactor/repackage HiveServer2's Thrift code so that it can be used in the > tasks > > > Key: HIVE-12442 > URL: https://issues.apache.org/jira/browse/HIVE-12442 > Project: Hive > Issue Type: Sub-task >Affects Versions: 1.2.1 >Reporter: Vaibhav Gumashta >Assignee: Rohit Dholakia > Labels: hiveserver, thrift > Attachments: hive-12442.1.patch, hive-12442.2.patch > > > For implementing HIVE-12427, the tasks will need to have knowledge of thrift > types from HS2's thrift API. This jira will look at the least invasive way to > do that. > https://reviews.apache.org/r/41379 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12619) Switching the field order within an array of structs causes the query to fail
[
https://issues.apache.org/jira/browse/HIVE-12619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15065097#comment-15065097
]
Hive QA commented on HIVE-12619:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12778465/HIVE-12619.2.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 26 failed/errored test(s), 9965 tests
executed
*Failed tests:*
{noformat}
TestHWISessionManager - did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join_stats2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats5
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_filemetadata
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver_encryption_insert_partition_dynamic
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_stats_filemetadata
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_explainuser_3
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_unionDistinct_2
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_import
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_columnstats_partlvl_multiple_part_clause
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_stats5
org.apache.hadoop.hive.ql.exec.spark.session.TestSparkSessionManagerImpl.testMultiSessionMultipleUse
org.apache.hadoop.hive.ql.exec.spark.session.TestSparkSessionManagerImpl.testSingleSessionMultipleUse
org.apache.hadoop.hive.ql.security.TestMetastoreAuthorizationProvider.testSimplePrivileges
org.apache.hadoop.hive.ql.security.authorization.plugin.TestHiveOperationType.checkHiveOperationTypeMatch
org.apache.hive.jdbc.TestJdbcDriver2.testShowGrant
org.apache.hive.jdbc.TestJdbcDriver2.testShowRoleGrant
org.apache.hive.jdbc.TestSSL.testSSLVersion
org.apache.hive.spark.client.TestSparkClient.testAddJarsAndFiles
org.apache.hive.spark.client.TestSparkClient.testCounters
org.apache.hive.spark.client.TestSparkClient.testErrorJob
org.apache.hive.spark.client.TestSparkClient.testJobSubmission
org.apache.hive.spark.client.TestSparkClient.testMetricsCollection
org.apache.hive.spark.client.TestSparkClient.testRemoteClient
org.apache.hive.spark.client.TestSparkClient.testSimpleSparkJob
org.apache.hive.spark.client.TestSparkClient.testSyncRpc
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6401/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6401/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-6401/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 26 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12778465 - PreCommit-HIVE-TRUNK-Build
> Switching the field order within an array of structs causes the query to fail
> -
>
> Key: HIVE-12619
> URL: https://issues.apache.org/jira/browse/HIVE-12619
> Project: Hive
> Issue Type: Bug
>Affects Versions: 1.1.0
>Reporter: Ang Zhang
>Assignee: Mohammad Kamrul Islam
>Priority: Minor
> Attachments: HIVE-12619.2.patch
>
>
> Switching the field order within an array of structs causes the query to fail
> or return the wrong data for the fields, but switching the field order within
> just a struct works.
> How to reproduce:
> Case1 if the two fields have the same type, query will return wrong data for
> the fields
> drop table if exists schema_test;
> create table schema_test (msg array) stored
> as parquet;
> insert into table schema_test select stack(2, array(named_struct('f1', 'abc',
> 'f2', 'abc2')), array(named_struct('f1', 'efg', 'f2', 'efg2'))) from one
> limit 2;
> select * from schema_test;
> --returns
> --[{"f1":"efg","f2":"efg2"}]
> --[{"f1":"abc","f2":"abc2"}]
> alter table schema_test change msg msg array;
> select * from schema_test;
> --returns
> --[{"f2":"efg","f1":"efg2"}]
> --[{"f2":"abc","f1":"abc2"}]
> Case2: if the two fields have different type, the query will fail
> drop table if exists schema_test;
> create table schema_test (msg array) stored as
> parquet;
> insert into table schema_test select stack(2, array(named_struct('f1', 'abc',
> 'f2', 1)), array(named_struct('f1', 'efg', 'f2', 2))) from one limit 2;
> select * from schema_test;
> --returns
> --[{"f1":"efg","f2":2}]
> --[{"f1":"abc","f2":1}]
> alter
[jira] [Updated] (HIVE-12674) HS2 Tez sessions should have maximum age
[ https://issues.apache.org/jira/browse/HIVE-12674?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-12674: Attachment: HIVE-12674.01.patch > HS2 Tez sessions should have maximum age > > > Key: HIVE-12674 > URL: https://issues.apache.org/jira/browse/HIVE-12674 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-12674.01.patch, HIVE-12674.01.patch, > HIVE-12674.patch > > > Certain tokens passed to AM by clients (e.g. an HDFS token) have maximum > lifetime beyond which they cannot be renewed. We should cycle long-lived > session AMs after a configurable period to avoid problems with these. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12712) HiveInputFormat may fail to column names to read in some cases
[
https://issues.apache.org/jira/browse/HIVE-12712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15065109#comment-15065109
]
Sergey Shelukhin commented on HIVE-12712:
-
Left a couple comments on RB
> HiveInputFormat may fail to column names to read in some cases
> --
>
> Key: HIVE-12712
> URL: https://issues.apache.org/jira/browse/HIVE-12712
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.0.0, 2.1.0
>Reporter: Takahiko Saito
>Assignee: Prasanth Jayachandran
> Attachments: HIVE-12712.1.patch
>
>
> The primary issue is when plan is generated pathToAliases map is populated
> with directory paths to table aliases. pathToAliases.put() uses
> path.toString() as map key. During probing, path.toUri().toString() is used.
> This can cause probe misses when path contains spaces in them. path.toUri()
> will escape the spaces in the path whereas path.toString() does not escape
> the spaces. As a result, HiveInputFormat can trigger a different code path
> which can fail to set list of columns to read from the source table. This was
> causing unexpected NPE in OrcInputFormat (after refactoring HIVE-11705) which
> removed null check for column names. The resulting exception is
> {code}
> Caused by: java.lang.RuntimeException: ORC split generation failed with
> exception: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.generateSplitsInfo(OrcInputFormat.java:1288)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getSplits(OrcInputFormat.java:1354)
> at
> org.apache.hadoop.hive.ql.io.HiveInputFormat.addSplitsForGroup(HiveInputFormat.java:367)
> at
> org.apache.hadoop.hive.ql.io.HiveInputFormat.getSplits(HiveInputFormat.java:457)
> at
> org.apache.hadoop.hive.ql.exec.tez.HiveSplitGenerator.initialize(HiveSplitGenerator.java:152)
> at
> org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:246)
> at
> org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:240)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
> at
> org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:240)
> at
> org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:227)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> ... 3 more
> Caused by: java.util.concurrent.ExecutionException:
> java.lang.NullPointerException
> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
> at java.util.concurrent.FutureTask.get(FutureTask.java:192)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.generateSplitsInfo(OrcInputFormat.java:1282)
> ... 15 more
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.extractNeededColNames(OrcInputFormat.java:422)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.extractNeededColNames(OrcInputFormat.java:417)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.access$2000(OrcInputFormat.java:134)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$SplitGenerator.call(OrcInputFormat.java:1072)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$SplitGenerator.call(OrcInputFormat.java:919)
> ... 4 more
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-12706) Incorrect output from from_utc_timestamp()/to_utc_timestamp when local timezone has DST
[
https://issues.apache.org/jira/browse/HIVE-12706?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15064793#comment-15064793
]
Jason Dere commented on HIVE-12706:
---
Looks like this error is occurring because the function is handling the
timestamps as timestamps in the local timezone, though the function says that
the timestamp is to be treated as having either UTC (from_utc_timestamp()) or
the input timezone (to_utc_timestamp()).
> Incorrect output from from_utc_timestamp()/to_utc_timestamp when local
> timezone has DST
> ---
>
> Key: HIVE-12706
> URL: https://issues.apache.org/jira/browse/HIVE-12706
> Project: Hive
> Issue Type: Bug
> Components: UDF
>Reporter: Jason Dere
>Assignee: Jason Dere
>
> Getting wrong output with the local timezone set to PST (which has DST). I
> don't think this happens when the local timezone does not observe DST.
> {noformat}
> select from_utc_timestamp('2015-03-28 17:00:00', 'Europe/London')
> 2015-03-28 17:00:00
> select from_utc_timestamp('2015-03-28 18:00:00', 'Europe/London')
> 2015-03-28 19:00:00 <= Wrong, should be 2015-03-28 18:00:00
> select from_utc_timestamp('2015-03-28 19:00:00', 'Europe/London')
> 2015-03-28 20:00:00 <= Wrong, should be 2015-03-28 19:00:00
> {noformat}
> Also to_utc_timestamp():
> {noformat}
> select to_utc_timestamp('2015-03-28 17:00:00', 'Europe/London')
> 2015-03-28 17:00:00
> select to_utc_timestamp('2015-03-28 18:00:00', 'Europe/London')
> 2015-03-28 17:00:00 <= Wrong
> select to_utc_timestamp('2015-03-28 19:00:00', 'Europe/London')
> 2015-03-28 18:00:00 <= Wrong
> select to_utc_timestamp('2015-03-28 20:00:00', 'Europe/London')
> 2015-03-28 19:00:00 <= Wrong
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-12395) Turn off CBO for hive.support.special.characters.tablename tests until feature is complete
[ https://issues.apache.org/jira/browse/HIVE-12395?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-12395: --- Fix Version/s: 2.1.0 > Turn off CBO for hive.support.special.characters.tablename tests until > feature is complete > -- > > Key: HIVE-12395 > URL: https://issues.apache.org/jira/browse/HIVE-12395 > Project: Hive > Issue Type: Bug >Affects Versions: 1.2.1 >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Fix For: 2.1.0 > > Attachments: HIVE-12395.01.patch, HIVE-12395.02.patch > > > Due to recent stats issue found in HIVE-12381, we need to turn off CBO for > hive.support.special.characters.tablename tests until feature is complete. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12633) LLAP: package included serde jars
[ https://issues.apache.org/jira/browse/HIVE-12633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15064872#comment-15064872 ] Gopal V commented on HIVE-12633: Changed the default aux classes to be always included and change the continue into a if (!) > LLAP: package included serde jars > - > > Key: HIVE-12633 > URL: https://issues.apache.org/jira/browse/HIVE-12633 > Project: Hive > Issue Type: Bug > Components: llap >Affects Versions: 2.0.0, 2.1.0 >Reporter: Takahiko Saito >Assignee: Sergey Shelukhin > Attachments: HIVE-12633.01.patch, HIVE-12633.02.patch, > HIVE-12633.03.patch, HIVE-12633.patch > > > Some SerDes like JSONSerde are not packaged with LLAP. One cannot localize > jars on the daemon (due to security consideration if nothing else), so we > should package them. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12633) LLAP: package included serde jars
[ https://issues.apache.org/jira/browse/HIVE-12633?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gopal V updated HIVE-12633: --- Component/s: llap > LLAP: package included serde jars > - > > Key: HIVE-12633 > URL: https://issues.apache.org/jira/browse/HIVE-12633 > Project: Hive > Issue Type: Bug > Components: llap >Affects Versions: 2.0.0, 2.1.0 >Reporter: Takahiko Saito >Assignee: Sergey Shelukhin > Attachments: HIVE-12633.01.patch, HIVE-12633.02.patch, > HIVE-12633.03.patch, HIVE-12633.patch > > > Some SerDes like JSONSerde are not packaged with LLAP. One cannot localize > jars on the daemon (due to security consideration if nothing else), so we > should package them. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12706) Incorrect output from from_utc_timestamp()/to_utc_timestamp when local timezone has DST
[
https://issues.apache.org/jira/browse/HIVE-12706?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Dere updated HIVE-12706:
--
Attachment: HIVE-12706.1.patch
Patch to fix issue by parsing timestamp string with DateFormat to use correct
timezone, and also using DateFormat to convert to timestamp string of
destination timezone.
> Incorrect output from from_utc_timestamp()/to_utc_timestamp when local
> timezone has DST
> ---
>
> Key: HIVE-12706
> URL: https://issues.apache.org/jira/browse/HIVE-12706
> Project: Hive
> Issue Type: Bug
> Components: UDF
>Reporter: Jason Dere
>Assignee: Jason Dere
> Attachments: HIVE-12706.1.patch
>
>
> Getting wrong output with the local timezone set to PST (which has DST). I
> don't think this happens when the local timezone does not observe DST.
> {noformat}
> select from_utc_timestamp('2015-03-28 17:00:00', 'Europe/London')
> 2015-03-28 17:00:00
> select from_utc_timestamp('2015-03-28 18:00:00', 'Europe/London')
> 2015-03-28 19:00:00 <= Wrong, should be 2015-03-28 18:00:00
> select from_utc_timestamp('2015-03-28 19:00:00', 'Europe/London')
> 2015-03-28 20:00:00 <= Wrong, should be 2015-03-28 19:00:00
> {noformat}
> Also to_utc_timestamp():
> {noformat}
> select to_utc_timestamp('2015-03-28 17:00:00', 'Europe/London')
> 2015-03-28 17:00:00
> select to_utc_timestamp('2015-03-28 18:00:00', 'Europe/London')
> 2015-03-28 17:00:00 <= Wrong
> select to_utc_timestamp('2015-03-28 19:00:00', 'Europe/London')
> 2015-03-28 18:00:00 <= Wrong
> select to_utc_timestamp('2015-03-28 20:00:00', 'Europe/London')
> 2015-03-28 19:00:00 <= Wrong
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-12632) LLAP: don't use IO elevator for ACID tables
[
https://issues.apache.org/jira/browse/HIVE-12632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15064799#comment-15064799
]
Hive QA commented on HIVE-12632:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12778388/HIVE-12632.02.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 17 failed/errored test(s), 9951 tests
executed
*Failed tests:*
{noformat}
TestHWISessionManager - did not produce a TEST-*.xml file
TestSparkCliDriver-timestamp_lazy.q-bucketsortoptimize_insert_4.q-date_udf.q-and-12-more
- did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join_stats2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_llap_acid
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver_encryption_insert_partition_dynamic
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_import
org.apache.hadoop.hive.ql.exec.spark.session.TestSparkSessionManagerImpl.testMultiSessionMultipleUse
org.apache.hadoop.hive.ql.exec.spark.session.TestSparkSessionManagerImpl.testSingleSessionMultipleUse
org.apache.hive.jdbc.TestSSL.testSSLVersion
org.apache.hive.spark.client.TestSparkClient.testAddJarsAndFiles
org.apache.hive.spark.client.TestSparkClient.testCounters
org.apache.hive.spark.client.TestSparkClient.testErrorJob
org.apache.hive.spark.client.TestSparkClient.testJobSubmission
org.apache.hive.spark.client.TestSparkClient.testMetricsCollection
org.apache.hive.spark.client.TestSparkClient.testRemoteClient
org.apache.hive.spark.client.TestSparkClient.testSimpleSparkJob
org.apache.hive.spark.client.TestSparkClient.testSyncRpc
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6398/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6398/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-6398/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 17 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12778388 - PreCommit-HIVE-TRUNK-Build
> LLAP: don't use IO elevator for ACID tables
>
>
> Key: HIVE-12632
> URL: https://issues.apache.org/jira/browse/HIVE-12632
> Project: Hive
> Issue Type: Bug
> Components: llap
>Reporter: Takahiko Saito
>Assignee: Sergey Shelukhin
>Priority: Blocker
> Attachments: HIVE-12632.01.patch, HIVE-12632.02.patch,
> HIVE-12632.patch
>
>
> Until HIVE-12631 is fixed, we need to avoid ACID tables in IO elevator. Right
> now, a FileNotFound error is thrown.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-12273) Improve user level explain
[ https://issues.apache.org/jira/browse/HIVE-12273?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-12273: --- Issue Type: Sub-task (was: Improvement) Parent: HIVE-9780 > Improve user level explain > -- > > Key: HIVE-12273 > URL: https://issues.apache.org/jira/browse/HIVE-12273 > Project: Hive > Issue Type: Sub-task >Affects Versions: 1.2.0, 1.2.1 >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Fix For: 2.0.0 > > Attachments: HIVE-12273.01.patch, HIVE-12273.02.patch, > HIVE-12273.03.patch > > > add (1) vectorization flags (2) Hybrid hash join flags (join algo.) (3) mode > of execution (4) ACID table flag -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12528) don't start HS2 Tez sessions in a single thread
[ https://issues.apache.org/jira/browse/HIVE-12528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15064857#comment-15064857 ] Sergey Shelukhin commented on HIVE-12528: - [~sseth] ping > don't start HS2 Tez sessions in a single thread > --- > > Key: HIVE-12528 > URL: https://issues.apache.org/jira/browse/HIVE-12528 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-12528.01.patch, HIVE-12528.02.patch, > HIVE-12528.patch > > > Starting sessions in parallel would improve the startup time. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12528) don't start HS2 Tez sessions in a single thread
[ https://issues.apache.org/jira/browse/HIVE-12528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-12528: Attachment: HIVE-12528.02.patch Fixed the tests, added a test case. > don't start HS2 Tez sessions in a single thread > --- > > Key: HIVE-12528 > URL: https://issues.apache.org/jira/browse/HIVE-12528 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-12528.01.patch, HIVE-12528.02.patch, > HIVE-12528.patch > > > Starting sessions in parallel would improve the startup time. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12633) LLAP: package included serde jars
[ https://issues.apache.org/jira/browse/HIVE-12633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15064875#comment-15064875 ] Sergey Shelukhin commented on HIVE-12633: - Makes sense. Is that also a +1? :) > LLAP: package included serde jars > - > > Key: HIVE-12633 > URL: https://issues.apache.org/jira/browse/HIVE-12633 > Project: Hive > Issue Type: Bug > Components: llap >Affects Versions: 2.0.0, 2.1.0 >Reporter: Takahiko Saito >Assignee: Sergey Shelukhin > Attachments: HIVE-12633.01.patch, HIVE-12633.02.patch, > HIVE-12633.03.patch, HIVE-12633.patch > > > Some SerDes like JSONSerde are not packaged with LLAP. One cannot localize > jars on the daemon (due to security consideration if nothing else), so we > should package them. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12633) LLAP: package included serde jars
[ https://issues.apache.org/jira/browse/HIVE-12633?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gopal V updated HIVE-12633: --- Affects Version/s: 2.1.0 2.0.0 > LLAP: package included serde jars > - > > Key: HIVE-12633 > URL: https://issues.apache.org/jira/browse/HIVE-12633 > Project: Hive > Issue Type: Bug > Components: llap >Affects Versions: 2.0.0, 2.1.0 >Reporter: Takahiko Saito >Assignee: Sergey Shelukhin > Attachments: HIVE-12633.01.patch, HIVE-12633.02.patch, > HIVE-12633.03.patch, HIVE-12633.patch > > > Some SerDes like JSONSerde are not packaged with LLAP. One cannot localize > jars on the daemon (due to security consideration if nothing else), so we > should package them. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12644) Support for offset in HiveSortMergeRule
[ https://issues.apache.org/jira/browse/HIVE-12644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15064891#comment-15064891 ] Laljo John Pullokkaran commented on HIVE-12644: --- As we discussed, need to simplify the logic about limit overlaps (Top Limit & Bottom Limit): 1. Fully contained 2. Partially contained 3. Outside > Support for offset in HiveSortMergeRule > --- > > Key: HIVE-12644 > URL: https://issues.apache.org/jira/browse/HIVE-12644 > Project: Hive > Issue Type: Bug > Components: CBO >Affects Versions: 2.1.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > Attachments: HIVE-12644.patch > > > After HIVE-11531 goes in, HiveSortMergeRule needs to be extended to support > offset properly when it merges operators that contain Limit. Otherwise, limit > pushdown through outer join optimization (introduced in HIVE-11684) will not > work properly. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12675) PerfLogger should log performance metrics at debug level
[ https://issues.apache.org/jira/browse/HIVE-12675?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Sankar Sivarama Subramaniyan updated HIVE-12675: - Fix Version/s: 2.0.0 > PerfLogger should log performance metrics at debug level > > > Key: HIVE-12675 > URL: https://issues.apache.org/jira/browse/HIVE-12675 > Project: Hive > Issue Type: Bug >Reporter: Hari Sankar Sivarama Subramaniyan >Assignee: Hari Sankar Sivarama Subramaniyan > Fix For: 2.0.0, 2.1.0 > > Attachments: HIVE-12675.1.patch > > > As more and more subcomponents of Hive (Tez, Optimizer) etc are using > PerfLogger to track the performance metrics, it will be more meaningful to > set the PerfLogger logging level to DEBUG. Otherwise, we will print the > performance metrics unnecessarily for each and every query if the underlying > subcomponent does not control the PerfLogging via a parameter on its own. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12526) PerfLogger for hive compiler and optimizer
[ https://issues.apache.org/jira/browse/HIVE-12526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Sankar Sivarama Subramaniyan updated HIVE-12526: - Fix Version/s: 2.0.0 > PerfLogger for hive compiler and optimizer > -- > > Key: HIVE-12526 > URL: https://issues.apache.org/jira/browse/HIVE-12526 > Project: Hive > Issue Type: Bug >Reporter: Hari Sankar Sivarama Subramaniyan >Assignee: Hari Sankar Sivarama Subramaniyan > Fix For: 2.0.0, 2.1.0 > > Attachments: HIVE-12526.1.patch, HIVE-12526.2.patch, > HIVE-12526.3.patch, HIVE-12526.4.patch > > > This jira is intended to use the perflogger to track compilation times and > optimization times (calcite, tez compiler, physical compiler) etc. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12708) Hive on Spark doesn't work with Kerboresed HBase [Spark Branch]
[ https://issues.apache.org/jira/browse/HIVE-12708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15064798#comment-15064798 ] Xuefu Zhang commented on HIVE-12708: [~jxiang], [~szehon], could you help review the patch? The test result looks good. I tested the patch with Kerberosed HBase and it worked. Thanks. > Hive on Spark doesn't work with Kerboresed HBase [Spark Branch] > --- > > Key: HIVE-12708 > URL: https://issues.apache.org/jira/browse/HIVE-12708 > Project: Hive > Issue Type: Bug > Components: Spark >Affects Versions: 1.2.0, 1.1.0, 2.0.0 >Reporter: Xuefu Zhang >Assignee: Xuefu Zhang > Attachments: HIVE-12708.1-spark.patch > > > Spark application launcher (spark-submit) acquires HBase delegation token on > Hive user's behalf when the application is launched. This mechanism, which > doesn't work for long-running sessions, is not in line with what Hive is > doing. Hive actually acquires the token automatically whenever a job needs > it. The right approach for Spark should be allowing applications to > dynamically add whatever tokens they need to the spark context. While this > needs work on Spark side, we provide a workaround solution in Hive. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12395) Turn off CBO for hive.support.special.characters.tablename tests until feature is complete
[ https://issues.apache.org/jira/browse/HIVE-12395?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-12395: --- Affects Version/s: 1.2.1 > Turn off CBO for hive.support.special.characters.tablename tests until > feature is complete > -- > > Key: HIVE-12395 > URL: https://issues.apache.org/jira/browse/HIVE-12395 > Project: Hive > Issue Type: Bug >Affects Versions: 1.2.1 >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Fix For: 2.1.0 > > Attachments: HIVE-12395.01.patch, HIVE-12395.02.patch > > > Due to recent stats issue found in HIVE-12381, we need to turn off CBO for > hive.support.special.characters.tablename tests until feature is complete. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11865) Disable Hive PPD optimizer when CBO has optimized the plan
[ https://issues.apache.org/jira/browse/HIVE-11865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-11865: --- Attachment: HIVE-11865.06.patch > Disable Hive PPD optimizer when CBO has optimized the plan > -- > > Key: HIVE-11865 > URL: https://issues.apache.org/jira/browse/HIVE-11865 > Project: Hive > Issue Type: Bug > Components: CBO, Logical Optimizer >Affects Versions: 2.0.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > Attachments: HIVE-11865.01.patch, HIVE-11865.02.patch, > HIVE-11865.02.patch, HIVE-11865.03.patch, HIVE-11865.04.patch, > HIVE-11865.05.patch, HIVE-11865.06.patch, HIVE-11865.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12708) Hive on Spark doesn't work with Kerboresed HBase [Spark Branch]
[ https://issues.apache.org/jira/browse/HIVE-12708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15064865#comment-15064865 ] Szehon Ho commented on HIVE-12708: -- +1 lgtm. One question about the log message, I might be not understanding it, but are these properties from hbase configuration file instead of hive configuration? > Hive on Spark doesn't work with Kerboresed HBase [Spark Branch] > --- > > Key: HIVE-12708 > URL: https://issues.apache.org/jira/browse/HIVE-12708 > Project: Hive > Issue Type: Bug > Components: Spark >Affects Versions: 1.2.0, 1.1.0, 2.0.0 >Reporter: Xuefu Zhang >Assignee: Xuefu Zhang > Attachments: HIVE-12708.1-spark.patch > > > Spark application launcher (spark-submit) acquires HBase delegation token on > Hive user's behalf when the application is launched. This mechanism, which > doesn't work for long-running sessions, is not in line with what Hive is > doing. Hive actually acquires the token automatically whenever a job needs > it. The right approach for Spark should be allowing applications to > dynamically add whatever tokens they need to the spark context. While this > needs work on Spark side, we provide a workaround solution in Hive. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12528) don't start HS2 Tez sessions in a single thread
[ https://issues.apache.org/jira/browse/HIVE-12528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15064902#comment-15064902 ] Siddharth Seth commented on HIVE-12528: --- [~sershe] - haven't looked at the new patch yet. Will try getting to it later today. > don't start HS2 Tez sessions in a single thread > --- > > Key: HIVE-12528 > URL: https://issues.apache.org/jira/browse/HIVE-12528 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-12528.01.patch, HIVE-12528.02.patch, > HIVE-12528.patch > > > Starting sessions in parallel would improve the startup time. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11865) Disable Hive PPD optimizer when CBO has optimized the plan
[ https://issues.apache.org/jira/browse/HIVE-11865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15064905#comment-15064905 ] Laljo John Pullokkaran commented on HIVE-11865: --- +1 conditional on clean qa run > Disable Hive PPD optimizer when CBO has optimized the plan > -- > > Key: HIVE-11865 > URL: https://issues.apache.org/jira/browse/HIVE-11865 > Project: Hive > Issue Type: Bug > Components: CBO, Logical Optimizer >Affects Versions: 2.0.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > Attachments: HIVE-11865.01.patch, HIVE-11865.02.patch, > HIVE-11865.02.patch, HIVE-11865.03.patch, HIVE-11865.04.patch, > HIVE-11865.05.patch, HIVE-11865.06.patch, HIVE-11865.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12666) PCRExprProcFactory.GenericFuncExprProcessor.process() aggressively removes dynamic partition pruner generated synthetic join predicates.
[
https://issues.apache.org/jira/browse/HIVE-12666?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hari Sankar Sivarama Subramaniyan updated HIVE-12666:
-
Fix Version/s: 2.0.0
> PCRExprProcFactory.GenericFuncExprProcessor.process() aggressively removes
> dynamic partition pruner generated synthetic join predicates.
>
>
> Key: HIVE-12666
> URL: https://issues.apache.org/jira/browse/HIVE-12666
> Project: Hive
> Issue Type: Bug
>Reporter: Hari Sankar Sivarama Subramaniyan
>Assignee: Hari Sankar Sivarama Subramaniyan
>Priority: Blocker
> Fix For: 2.0.0, 2.1.0
>
> Attachments: HIVE-12666.1.patch, HIVE-12666.2.patch
>
>
> Introduced by HIVE-11634. The original idea in HIVE-11634 was to remove the
> IN partition conditions from the predicate list since the static dynamic
> partitioning would kick in and push these predicates down to metastore.
> However, the check is too aggressive and removes events such as below :
> {code}
> -Select Operator
> - expressions: UDFToDouble(UDFToInteger((hr / 2)))
> (type: double)
> - outputColumnNames: _col0
> - Statistics: Num rows: 1 Data size: 7 Basic stats:
> COMPLETE Column stats: NONE
> - Group By Operator
> -keys: _col0 (type: double)
> -mode: hash
> -outputColumnNames: _col0
> -Statistics: Num rows: 1 Data size: 7 Basic stats:
> COMPLETE Column stats: NONE
> -Dynamic Partitioning Event Operator
> - Target Input: srcpart
> - Partition key expr: UDFToDouble(hr)
> - Statistics: Num rows: 1 Data size: 7 Basic stats:
> COMPLETE Column stats: NONE
> - Target column: hr
> - Target Vertex: Map 1
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-12633) LLAP: package included serde jars
[ https://issues.apache.org/jira/browse/HIVE-12633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15064786#comment-15064786 ] Gopal V commented on HIVE-12633: [~sershe]: making minor edits instead of review, is that ok? > LLAP: package included serde jars > - > > Key: HIVE-12633 > URL: https://issues.apache.org/jira/browse/HIVE-12633 > Project: Hive > Issue Type: Bug >Reporter: Takahiko Saito >Assignee: Sergey Shelukhin > Attachments: HIVE-12633.01.patch, HIVE-12633.02.patch, > HIVE-12633.patch > > > Some SerDes like JSONSerde are not packaged with LLAP. One cannot localize > jars on the daemon (due to security consideration if nothing else), so we > should package them. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12708) Hive on Spark doesn't work with Kerboresed HBase [Spark Branch]
[
https://issues.apache.org/jira/browse/HIVE-12708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15064789#comment-15064789
]
Hive QA commented on HIVE-12708:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12778540/HIVE-12708.1-spark.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 4 failed/errored test(s), 9866 tests executed
*Failed tests:*
{noformat}
TestHWISessionManager - did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_import
org.apache.hadoop.hive.metastore.TestHiveMetaStorePartitionSpecs.testGetPartitionSpecs_WithAndWithoutPartitionGrouping
org.apache.hive.jdbc.TestSSL.testSSLVersion
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/1022/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/1022/console
Test logs:
http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-SPARK-Build-1022/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 4 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12778540 - PreCommit-HIVE-SPARK-Build
> Hive on Spark doesn't work with Kerboresed HBase [Spark Branch]
> ---
>
> Key: HIVE-12708
> URL: https://issues.apache.org/jira/browse/HIVE-12708
> Project: Hive
> Issue Type: Bug
> Components: Spark
>Affects Versions: 1.2.0, 1.1.0, 2.0.0
>Reporter: Xuefu Zhang
>Assignee: Xuefu Zhang
> Attachments: HIVE-12708.1-spark.patch
>
>
> Spark application launcher (spark-submit) acquires HBase delegation token on
> Hive user's behalf when the application is launched. This mechanism, which
> doesn't work for long-running sessions, is not in line with what Hive is
> doing. Hive actually acquires the token automatically whenever a job needs
> it. The right approach for Spark should be allowing applications to
> dynamically add whatever tokens they need to the spark context. While this
> needs work on Spark side, we provide a workaround solution in Hive.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-12632) LLAP: don't use IO elevator for ACID tables
[ https://issues.apache.org/jira/browse/HIVE-12632?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-12632: Attachment: HIVE-12632.03.patch Updated the explain code. > LLAP: don't use IO elevator for ACID tables > > > Key: HIVE-12632 > URL: https://issues.apache.org/jira/browse/HIVE-12632 > Project: Hive > Issue Type: Bug > Components: llap >Reporter: Takahiko Saito >Assignee: Sergey Shelukhin >Priority: Blocker > Attachments: HIVE-12632.01.patch, HIVE-12632.02.patch, > HIVE-12632.03.patch, HIVE-12632.patch > > > Until HIVE-12631 is fixed, we need to avoid ACID tables in IO elevator. Right > now, a FileNotFound error is thrown. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12633) LLAP: package included serde jars
[ https://issues.apache.org/jira/browse/HIVE-12633?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gopal V updated HIVE-12633: --- Attachment: HIVE-12633.03.patch > LLAP: package included serde jars > - > > Key: HIVE-12633 > URL: https://issues.apache.org/jira/browse/HIVE-12633 > Project: Hive > Issue Type: Bug >Reporter: Takahiko Saito >Assignee: Sergey Shelukhin > Attachments: HIVE-12633.01.patch, HIVE-12633.02.patch, > HIVE-12633.03.patch, HIVE-12633.patch > > > Some SerDes like JSONSerde are not packaged with LLAP. One cannot localize > jars on the daemon (due to security consideration if nothing else), so we > should package them. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12633) LLAP: package included serde jars
[ https://issues.apache.org/jira/browse/HIVE-12633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15064877#comment-15064877 ] Gopal V commented on HIVE-12633: Yes, my seal of approval included (+1). > LLAP: package included serde jars > - > > Key: HIVE-12633 > URL: https://issues.apache.org/jira/browse/HIVE-12633 > Project: Hive > Issue Type: Bug > Components: llap >Affects Versions: 2.0.0, 2.1.0 >Reporter: Takahiko Saito >Assignee: Sergey Shelukhin > Attachments: HIVE-12633.01.patch, HIVE-12633.02.patch, > HIVE-12633.03.patch, HIVE-12633.patch > > > Some SerDes like JSONSerde are not packaged with LLAP. One cannot localize > jars on the daemon (due to security consideration if nothing else), so we > should package them. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12708) Hive on Spark doesn't work with Kerboresed HBase [Spark Branch]
[ https://issues.apache.org/jira/browse/HIVE-12708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15064881#comment-15064881 ] Xuefu Zhang commented on HIVE-12708: Good catch. Yes. It's hbase configuration. I'll change the log message. > Hive on Spark doesn't work with Kerboresed HBase [Spark Branch] > --- > > Key: HIVE-12708 > URL: https://issues.apache.org/jira/browse/HIVE-12708 > Project: Hive > Issue Type: Bug > Components: Spark >Affects Versions: 1.2.0, 1.1.0, 2.0.0 >Reporter: Xuefu Zhang >Assignee: Xuefu Zhang > Attachments: HIVE-12708.1-spark.patch > > > Spark application launcher (spark-submit) acquires HBase delegation token on > Hive user's behalf when the application is launched. This mechanism, which > doesn't work for long-running sessions, is not in line with what Hive is > doing. Hive actually acquires the token automatically whenever a job needs > it. The right approach for Spark should be allowing applications to > dynamically add whatever tokens they need to the spark context. While this > needs work on Spark side, we provide a workaround solution in Hive. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12708) Hive on Spark doesn't work with Kerboresed HBase [Spark Branch]
[ https://issues.apache.org/jira/browse/HIVE-12708?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuefu Zhang updated HIVE-12708: --- Attachment: HIVE-12708.1-spark.patch > Hive on Spark doesn't work with Kerboresed HBase [Spark Branch] > --- > > Key: HIVE-12708 > URL: https://issues.apache.org/jira/browse/HIVE-12708 > Project: Hive > Issue Type: Bug > Components: Spark >Affects Versions: 1.2.0, 1.1.0, 2.0.0 >Reporter: Xuefu Zhang >Assignee: Xuefu Zhang > Attachments: HIVE-12708.1-spark.patch, HIVE-12708.1-spark.patch > > > Spark application launcher (spark-submit) acquires HBase delegation token on > Hive user's behalf when the application is launched. This mechanism, which > doesn't work for long-running sessions, is not in line with what Hive is > doing. Hive actually acquires the token automatically whenever a job needs > it. The right approach for Spark should be allowing applications to > dynamically add whatever tokens they need to the spark context. While this > needs work on Spark side, we provide a workaround solution in Hive. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12470) Allow splits to provide custom consistent locations, instead of being tied to data locality
[ https://issues.apache.org/jira/browse/HIVE-12470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15065164#comment-15065164 ] Prasanth Jayachandran commented on HIVE-12470: -- minor comments in RB. lgtm +1 > Allow splits to provide custom consistent locations, instead of being tied to > data locality > --- > > Key: HIVE-12470 > URL: https://issues.apache.org/jira/browse/HIVE-12470 > Project: Hive > Issue Type: Improvement > Components: llap >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Attachments: HIVE-12470.1.txt, HIVE-12470.1.wip.txt > > > LLAP instances may not run on the same nodes as HDFS, or may run on a subset > of the cluster. > Using split locations based on FileSystem locality is not very useful in such > cases - since that guarantees not getting any locality. > Allow a split to map to a specific location - so that there's a chance of > getting cache locality across different queries. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11865) Disable Hive PPD optimizer when CBO has optimized the plan
[
https://issues.apache.org/jira/browse/HIVE-11865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15065179#comment-15065179
]
Hive QA commented on HIVE-11865:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12778605/HIVE-11865.06.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 48 failed/errored test(s), 9968 tests
executed
*Failed tests:*
{noformat}
TestHWISessionManager - did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join_stats2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer14
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_fold_eq_with_case_when
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_mergejoin
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_filemetadata
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver_encryption_insert_partition_dynamic
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_mrr
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_stats_filemetadata
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_join
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_smb_1
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_smb_main
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_infer_bucket_sort_map_operators
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.org.apache.hadoop.hive.cli.TestMiniTezCliDriver
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_cbo_semijoin
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_cbo_subq_exists
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_delete_where_partitioned
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_filter_join_breaktask2
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_mrr
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_orc_acidvec_mapwork_part
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_tez_dml
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_tez_join
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_tez_smb_1
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_tez_smb_main
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_tez_union
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_update_all_non_partitioned
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_update_where_partitioned
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_non_string_partition
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_partitioned_date_time
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_reduce_groupby_decimal
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vectorization_16
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vectorization_8
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vectorization_not
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_import
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_columnstats_partlvl_multiple_part_clause
org.apache.hadoop.hive.metastore.TestHiveMetaStorePartitionSpecs.testGetPartitionSpecs_WithAndWithoutPartitionGrouping
org.apache.hadoop.hive.ql.exec.spark.session.TestSparkSessionManagerImpl.testMultiSessionMultipleUse
org.apache.hadoop.hive.ql.exec.spark.session.TestSparkSessionManagerImpl.testSingleSessionMultipleUse
org.apache.hadoop.hive.ql.security.authorization.plugin.TestHiveOperationType.checkHiveOperationTypeMatch
org.apache.hive.jdbc.TestSSL.testSSLVersion
org.apache.hive.spark.client.TestSparkClient.testAddJarsAndFiles
org.apache.hive.spark.client.TestSparkClient.testCounters
org.apache.hive.spark.client.TestSparkClient.testErrorJob
org.apache.hive.spark.client.TestSparkClient.testJobSubmission
org.apache.hive.spark.client.TestSparkClient.testMetricsCollection
org.apache.hive.spark.client.TestSparkClient.testRemoteClient
org.apache.hive.spark.client.TestSparkClient.testSimpleSparkJob
org.apache.hive.spark.client.TestSparkClient.testSyncRpc
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6402/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6402/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-6402/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 48 tests
[jira] [Updated] (HIVE-12470) Allow splits to provide custom consistent locations, instead of being tied to data locality
[ https://issues.apache.org/jira/browse/HIVE-12470?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated HIVE-12470: -- Attachment: HIVE-12470.2.txt Updated patch with RB comments addressed. > Allow splits to provide custom consistent locations, instead of being tied to > data locality > --- > > Key: HIVE-12470 > URL: https://issues.apache.org/jira/browse/HIVE-12470 > Project: Hive > Issue Type: Improvement > Components: llap >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Attachments: HIVE-12470.1.txt, HIVE-12470.1.wip.txt, HIVE-12470.2.txt > > > LLAP instances may not run on the same nodes as HDFS, or may run on a subset > of the cluster. > Using split locations based on FileSystem locality is not very useful in such > cases - since that guarantees not getting any locality. > Allow a split to map to a specific location - so that there's a chance of > getting cache locality across different queries. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11355) Hive on tez: memory manager for sort buffers (input/output) and operators
[
https://issues.apache.org/jira/browse/HIVE-11355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15065259#comment-15065259
]
Hive QA commented on HIVE-11355:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12778515/HIVE-11355.10.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 26 failed/errored test(s), 9952 tests
executed
*Failed tests:*
{noformat}
TestHWISessionManager - did not produce a TEST-*.xml file
TestSparkCliDriver-timestamp_lazy.q-bucketsortoptimize_insert_4.q-date_udf.q-and-12-more
- did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join_stats2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_filemetadata
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver_encryption_insert_partition_dynamic
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_stats_filemetadata
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_join
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_smb_1
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_smb_main
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_explainuser_3
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_import
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_columnstats_partlvl_multiple_part_clause
org.apache.hadoop.hive.ql.exec.spark.session.TestSparkSessionManagerImpl.testMultiSessionMultipleUse
org.apache.hadoop.hive.ql.exec.spark.session.TestSparkSessionManagerImpl.testSingleSessionMultipleUse
org.apache.hadoop.hive.ql.parse.TestGenTezWork.testCreateMap
org.apache.hadoop.hive.ql.parse.TestGenTezWork.testCreateReduce
org.apache.hadoop.hive.ql.security.authorization.plugin.TestHiveOperationType.checkHiveOperationTypeMatch
org.apache.hive.jdbc.TestSSL.testSSLVersion
org.apache.hive.spark.client.TestSparkClient.testAddJarsAndFiles
org.apache.hive.spark.client.TestSparkClient.testCounters
org.apache.hive.spark.client.TestSparkClient.testErrorJob
org.apache.hive.spark.client.TestSparkClient.testJobSubmission
org.apache.hive.spark.client.TestSparkClient.testMetricsCollection
org.apache.hive.spark.client.TestSparkClient.testRemoteClient
org.apache.hive.spark.client.TestSparkClient.testSimpleSparkJob
org.apache.hive.spark.client.TestSparkClient.testSyncRpc
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6403/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6403/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-6403/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 26 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12778515 - PreCommit-HIVE-TRUNK-Build
> Hive on tez: memory manager for sort buffers (input/output) and operators
> -
>
> Key: HIVE-11355
> URL: https://issues.apache.org/jira/browse/HIVE-11355
> Project: Hive
> Issue Type: Improvement
> Components: Tez
>Affects Versions: 2.0.0
>Reporter: Vikram Dixit K
>Assignee: Vikram Dixit K
> Attachments: HIVE-11355.1.patch, HIVE-11355.10.patch,
> HIVE-11355.2.patch, HIVE-11355.3.patch, HIVE-11355.4.patch,
> HIVE-11355.5.patch, HIVE-11355.6.patch, HIVE-11355.7.patch,
> HIVE-11355.8.patch, HIVE-11355.9.patch
>
>
> We need to better manage the sort buffer allocations to ensure better
> performance. Also, we need to provide configurations to certain operators to
> stay within memory limits.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-11775) Implement limit push down through union all in CBO
[
https://issues.apache.org/jira/browse/HIVE-11775?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15065280#comment-15065280
]
Hive QA commented on HIVE-11775:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12778559/HIVE-11775.13.patch
{color:green}SUCCESS:{color} +1 due to 3 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 22 failed/errored test(s), 9968 tests
executed
*Failed tests:*
{noformat}
TestHWISessionManager - did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join_stats2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_filemetadata
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver_encryption_insert_partition_dynamic
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_stats_filemetadata
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_import
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_columnstats_partlvl_multiple_part_clause
org.apache.hadoop.hive.metastore.TestHiveMetaStorePartitionSpecs.testFetchingPartitionsWithDifferentSchemas
org.apache.hadoop.hive.metastore.TestHiveMetaStorePartitionSpecs.testGetPartitionSpecs_WithAndWithoutPartitionGrouping
org.apache.hadoop.hive.ql.exec.spark.session.TestSparkSessionManagerImpl.testMultiSessionMultipleUse
org.apache.hadoop.hive.ql.exec.spark.session.TestSparkSessionManagerImpl.testSingleSessionMultipleUse
org.apache.hadoop.hive.ql.security.authorization.plugin.TestHiveOperationType.checkHiveOperationTypeMatch
org.apache.hive.jdbc.TestSSL.testSSLVersion
org.apache.hive.service.cli.session.TestSessionManagerMetrics.testThreadPoolMetrics
org.apache.hive.spark.client.TestSparkClient.testAddJarsAndFiles
org.apache.hive.spark.client.TestSparkClient.testCounters
org.apache.hive.spark.client.TestSparkClient.testErrorJob
org.apache.hive.spark.client.TestSparkClient.testJobSubmission
org.apache.hive.spark.client.TestSparkClient.testMetricsCollection
org.apache.hive.spark.client.TestSparkClient.testRemoteClient
org.apache.hive.spark.client.TestSparkClient.testSimpleSparkJob
org.apache.hive.spark.client.TestSparkClient.testSyncRpc
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6404/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6404/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-6404/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 22 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12778559 - PreCommit-HIVE-TRUNK-Build
> Implement limit push down through union all in CBO
> --
>
> Key: HIVE-11775
> URL: https://issues.apache.org/jira/browse/HIVE-11775
> Project: Hive
> Issue Type: New Feature
>Reporter: Pengcheng Xiong
>Assignee: Pengcheng Xiong
> Attachments: HIVE-11775.01.patch, HIVE-11775.02.patch,
> HIVE-11775.03.patch, HIVE-11775.04.patch, HIVE-11775.05.patch,
> HIVE-11775.06.patch, HIVE-11775.07.patch, HIVE-11775.08.patch,
> HIVE-11775.09.patch, HIVE-11775.10.patch, HIVE-11775.11.patch,
> HIVE-11775.12.patch, HIVE-11775.13.patch
>
>
> Enlightened by HIVE-11684 (Kudos to [~jcamachorodriguez]), we can actually
> push limit down through union all, which reduces the intermediate number of
> rows in union branches.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-12644) Support for offset in HiveSortMergeRule
[ https://issues.apache.org/jira/browse/HIVE-12644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15065190#comment-15065190 ] Laljo John Pullokkaran commented on HIVE-12644: --- +1 conditional on clean qa run > Support for offset in HiveSortMergeRule > --- > > Key: HIVE-12644 > URL: https://issues.apache.org/jira/browse/HIVE-12644 > Project: Hive > Issue Type: Bug > Components: CBO >Affects Versions: 2.1.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > Attachments: HIVE-12644.01.patch, HIVE-12644.patch > > > After HIVE-11531 goes in, HiveSortMergeRule needs to be extended to support > offset properly when it merges operators that contain Limit. Otherwise, limit > pushdown through outer join optimization (introduced in HIVE-11684) will not > work properly. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
