[jira] [Updated] (HIVE-13293) Query occurs performance degradation after enabling parallel order by for Hive on Spark
[ https://issues.apache.org/jira/browse/HIVE-13293?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-13293: -- Attachment: HIVE-13293.1.patch Upload same patch to trigger tests again. Our benchmark results show that the patch can improve performance for certain queries in TPC-BB by over 30%. One shortcoming of parallel order by is that we will end up with more files, which can bring some overhead. We should minimize such overhead, e.g. HIVE-13572. Also pinging [~xuefuz] for review. > Query occurs performance degradation after enabling parallel order by for > Hive on Spark > --- > > Key: HIVE-13293 > URL: https://issues.apache.org/jira/browse/HIVE-13293 > Project: Hive > Issue Type: Bug > Components: Spark >Affects Versions: 2.0.0 >Reporter: Lifeng Wang >Assignee: Rui Li > Attachments: HIVE-13293.1.patch, HIVE-13293.1.patch > > > I use TPCx-BB to do some performance test on Hive on Spark engine. And found > query 10 has performance degradation when enabling parallel order by. > It seems that sampling cost much time before running the real query. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13724) Backport HIVE-11591 to branch-1 to use undated annotations
[ https://issues.apache.org/jira/browse/HIVE-13724?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Zheng updated HIVE-13724: - Resolution: Fixed Fix Version/s: 1.3.0 Status: Resolved (was: Patch Available) > Backport HIVE-11591 to branch-1 to use undated annotations > -- > > Key: HIVE-13724 > URL: https://issues.apache.org/jira/browse/HIVE-13724 > Project: Hive > Issue Type: Bug > Components: Thrift API >Affects Versions: 1.2.1 >Reporter: Wei Zheng >Assignee: Wei Zheng > Fix For: 1.3.0 > > Attachments: HIVE-13724.1.patch, HIVE-13724.pom_change_only.patch > > > HIVE-12832 changed branch-1 hive pom file and updated thrift version from > 0.9.2 to 0.9.3. But it didn't update the thrift args part to use undated > annotation from HIVE-11591. > So every time someone is running maven thrift re-gen command, it will still > update a lot of unrelated files, just because of the date change. > Need backport HIVE-11591 to branch-1. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13724) Backport HIVE-11591 to branch-1 to use undated annotations
[ https://issues.apache.org/jira/browse/HIVE-13724?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15277627#comment-15277627 ] Wei Zheng commented on HIVE-13724: -- Thanks Sergey for the review. Committed patch 1 to branch-1. > Backport HIVE-11591 to branch-1 to use undated annotations > -- > > Key: HIVE-13724 > URL: https://issues.apache.org/jira/browse/HIVE-13724 > Project: Hive > Issue Type: Bug > Components: Thrift API >Affects Versions: 1.2.1 >Reporter: Wei Zheng >Assignee: Wei Zheng > Fix For: 1.3.0 > > Attachments: HIVE-13724.1.patch, HIVE-13724.pom_change_only.patch > > > HIVE-12832 changed branch-1 hive pom file and updated thrift version from > 0.9.2 to 0.9.3. But it didn't update the thrift args part to use undated > annotation from HIVE-11591. > So every time someone is running maven thrift re-gen command, it will still > update a lot of unrelated files, just because of the date change. > Need backport HIVE-11591 to branch-1. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13621) compute stats in certain cases fails with NPE
[
https://issues.apache.org/jira/browse/HIVE-13621?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pengcheng Xiong updated HIVE-13621:
---
Status: Open (was: Patch Available)
> compute stats in certain cases fails with NPE
> -
>
> Key: HIVE-13621
> URL: https://issues.apache.org/jira/browse/HIVE-13621
> Project: Hive
> Issue Type: Bug
> Components: HBase Metastore, Metastore
>Affects Versions: 2.1.0, 2.0.1
>Reporter: Vikram Dixit K
>Assignee: Pengcheng Xiong
> Attachments: HIVE-13621.1.patch, HIVE-13621.2.patch
>
>
> {code}
> FAILED: NullPointerException null
> java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.stats.StatsUtils.getColStatistics(StatsUtils.java:693)
> at
> org.apache.hadoop.hive.ql.stats.StatsUtils.convertColStats(StatsUtils.java:739)
> at
> org.apache.hadoop.hive.ql.stats.StatsUtils.getTableColumnStats(StatsUtils.java:728)
> at
> org.apache.hadoop.hive.ql.stats.StatsUtils.collectStatistics(StatsUtils.java:183)
> at
> org.apache.hadoop.hive.ql.stats.StatsUtils.collectStatistics(StatsUtils.java:136)
> at
> org.apache.hadoop.hive.ql.stats.StatsUtils.collectStatistics(StatsUtils.java:124){code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13621) compute stats in certain cases fails with NPE
[
https://issues.apache.org/jira/browse/HIVE-13621?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pengcheng Xiong updated HIVE-13621:
---
Attachment: HIVE-13621.2.patch
> compute stats in certain cases fails with NPE
> -
>
> Key: HIVE-13621
> URL: https://issues.apache.org/jira/browse/HIVE-13621
> Project: Hive
> Issue Type: Bug
> Components: HBase Metastore, Metastore
>Affects Versions: 2.1.0, 2.0.1
>Reporter: Vikram Dixit K
>Assignee: Pengcheng Xiong
> Attachments: HIVE-13621.1.patch, HIVE-13621.2.patch
>
>
> {code}
> FAILED: NullPointerException null
> java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.stats.StatsUtils.getColStatistics(StatsUtils.java:693)
> at
> org.apache.hadoop.hive.ql.stats.StatsUtils.convertColStats(StatsUtils.java:739)
> at
> org.apache.hadoop.hive.ql.stats.StatsUtils.getTableColumnStats(StatsUtils.java:728)
> at
> org.apache.hadoop.hive.ql.stats.StatsUtils.collectStatistics(StatsUtils.java:183)
> at
> org.apache.hadoop.hive.ql.stats.StatsUtils.collectStatistics(StatsUtils.java:136)
> at
> org.apache.hadoop.hive.ql.stats.StatsUtils.collectStatistics(StatsUtils.java:124){code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13621) compute stats in certain cases fails with NPE
[
https://issues.apache.org/jira/browse/HIVE-13621?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pengcheng Xiong updated HIVE-13621:
---
Status: Patch Available (was: Open)
> compute stats in certain cases fails with NPE
> -
>
> Key: HIVE-13621
> URL: https://issues.apache.org/jira/browse/HIVE-13621
> Project: Hive
> Issue Type: Bug
> Components: HBase Metastore, Metastore
>Affects Versions: 2.1.0, 2.0.1
>Reporter: Vikram Dixit K
>Assignee: Pengcheng Xiong
> Attachments: HIVE-13621.1.patch, HIVE-13621.2.patch
>
>
> {code}
> FAILED: NullPointerException null
> java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.stats.StatsUtils.getColStatistics(StatsUtils.java:693)
> at
> org.apache.hadoop.hive.ql.stats.StatsUtils.convertColStats(StatsUtils.java:739)
> at
> org.apache.hadoop.hive.ql.stats.StatsUtils.getTableColumnStats(StatsUtils.java:728)
> at
> org.apache.hadoop.hive.ql.stats.StatsUtils.collectStatistics(StatsUtils.java:183)
> at
> org.apache.hadoop.hive.ql.stats.StatsUtils.collectStatistics(StatsUtils.java:136)
> at
> org.apache.hadoop.hive.ql.stats.StatsUtils.collectStatistics(StatsUtils.java:124){code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (HIVE-13621) compute stats in certain cases fails with NPE
[
https://issues.apache.org/jira/browse/HIVE-13621?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pengcheng Xiong reassigned HIVE-13621:
--
Assignee: Pengcheng Xiong (was: Vikram Dixit K)
> compute stats in certain cases fails with NPE
> -
>
> Key: HIVE-13621
> URL: https://issues.apache.org/jira/browse/HIVE-13621
> Project: Hive
> Issue Type: Bug
> Components: HBase Metastore, Metastore
>Affects Versions: 2.1.0, 2.0.1
>Reporter: Vikram Dixit K
>Assignee: Pengcheng Xiong
> Attachments: HIVE-13621.1.patch
>
>
> {code}
> FAILED: NullPointerException null
> java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.stats.StatsUtils.getColStatistics(StatsUtils.java:693)
> at
> org.apache.hadoop.hive.ql.stats.StatsUtils.convertColStats(StatsUtils.java:739)
> at
> org.apache.hadoop.hive.ql.stats.StatsUtils.getTableColumnStats(StatsUtils.java:728)
> at
> org.apache.hadoop.hive.ql.stats.StatsUtils.collectStatistics(StatsUtils.java:183)
> at
> org.apache.hadoop.hive.ql.stats.StatsUtils.collectStatistics(StatsUtils.java:136)
> at
> org.apache.hadoop.hive.ql.stats.StatsUtils.collectStatistics(StatsUtils.java:124){code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13598) Describe extended table should show the primary keys/foreign keys associated with the table
[ https://issues.apache.org/jira/browse/HIVE-13598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Sankar Sivarama Subramaniyan updated HIVE-13598: - Status: Open (was: Patch Available) > Describe extended table should show the primary keys/foreign keys associated > with the table > --- > > Key: HIVE-13598 > URL: https://issues.apache.org/jira/browse/HIVE-13598 > Project: Hive > Issue Type: Bug >Reporter: Hari Sankar Sivarama Subramaniyan >Assignee: Hari Sankar Sivarama Subramaniyan > Attachments: HIVE-13598.1.patch, HIVE-13598.2.patch, > HIVE-13598.3.patch, HIVE-13598.4.patch, HIVE-13598.5.patch, HIVE-13598.6.patch > > > After HIVE-13290 is committed, we need to show the constraints as part of > table description when extended label is used. Currently, the constraints > would not be shown as part of table description since Constraint is a > separate entity. > The purpose of the jira is to modify Hive.describeTable() to enable the user > to view the constraints associated with the table when the user does a > "describe extended table"; -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13598) Describe extended table should show the primary keys/foreign keys associated with the table
[ https://issues.apache.org/jira/browse/HIVE-13598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Sankar Sivarama Subramaniyan updated HIVE-13598: - Status: Patch Available (was: Open) > Describe extended table should show the primary keys/foreign keys associated > with the table > --- > > Key: HIVE-13598 > URL: https://issues.apache.org/jira/browse/HIVE-13598 > Project: Hive > Issue Type: Bug >Reporter: Hari Sankar Sivarama Subramaniyan >Assignee: Hari Sankar Sivarama Subramaniyan > Attachments: HIVE-13598.1.patch, HIVE-13598.2.patch, > HIVE-13598.3.patch, HIVE-13598.4.patch, HIVE-13598.5.patch, HIVE-13598.6.patch > > > After HIVE-13290 is committed, we need to show the constraints as part of > table description when extended label is used. Currently, the constraints > would not be shown as part of table description since Constraint is a > separate entity. > The purpose of the jira is to modify Hive.describeTable() to enable the user > to view the constraints associated with the table when the user does a > "describe extended table"; -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13598) Describe extended table should show the primary keys/foreign keys associated with the table
[ https://issues.apache.org/jira/browse/HIVE-13598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Sankar Sivarama Subramaniyan updated HIVE-13598: - Attachment: HIVE-13598.6.patch > Describe extended table should show the primary keys/foreign keys associated > with the table > --- > > Key: HIVE-13598 > URL: https://issues.apache.org/jira/browse/HIVE-13598 > Project: Hive > Issue Type: Bug >Reporter: Hari Sankar Sivarama Subramaniyan >Assignee: Hari Sankar Sivarama Subramaniyan > Attachments: HIVE-13598.1.patch, HIVE-13598.2.patch, > HIVE-13598.3.patch, HIVE-13598.4.patch, HIVE-13598.5.patch, HIVE-13598.6.patch > > > After HIVE-13290 is committed, we need to show the constraints as part of > table description when extended label is used. Currently, the constraints > would not be shown as part of table description since Constraint is a > separate entity. > The purpose of the jira is to modify Hive.describeTable() to enable the user > to view the constraints associated with the table when the user does a > "describe extended table"; -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13598) Describe extended table should show the primary keys/foreign keys associated with the table
[ https://issues.apache.org/jira/browse/HIVE-13598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Sankar Sivarama Subramaniyan updated HIVE-13598: - Attachment: (was: HIVE-13598.6.patch) > Describe extended table should show the primary keys/foreign keys associated > with the table > --- > > Key: HIVE-13598 > URL: https://issues.apache.org/jira/browse/HIVE-13598 > Project: Hive > Issue Type: Bug >Reporter: Hari Sankar Sivarama Subramaniyan >Assignee: Hari Sankar Sivarama Subramaniyan > Attachments: HIVE-13598.1.patch, HIVE-13598.2.patch, > HIVE-13598.3.patch, HIVE-13598.4.patch, HIVE-13598.5.patch, HIVE-13598.6.patch > > > After HIVE-13290 is committed, we need to show the constraints as part of > table description when extended label is used. Currently, the constraints > would not be shown as part of table description since Constraint is a > separate entity. > The purpose of the jira is to modify Hive.describeTable() to enable the user > to view the constraints associated with the table when the user does a > "describe extended table"; -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13724) Backport HIVE-11591 to branch-1 to use undated annotations
[ https://issues.apache.org/jira/browse/HIVE-13724?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Zheng updated HIVE-13724: - Status: Patch Available (was: Open) > Backport HIVE-11591 to branch-1 to use undated annotations > -- > > Key: HIVE-13724 > URL: https://issues.apache.org/jira/browse/HIVE-13724 > Project: Hive > Issue Type: Bug > Components: Thrift API >Affects Versions: 1.2.1 >Reporter: Wei Zheng >Assignee: Wei Zheng > Attachments: HIVE-13724.1.patch, HIVE-13724.pom_change_only.patch > > > HIVE-12832 changed branch-1 hive pom file and updated thrift version from > 0.9.2 to 0.9.3. But it didn't update the thrift args part to use undated > annotation from HIVE-11591. > So every time someone is running maven thrift re-gen command, it will still > update a lot of unrelated files, just because of the date change. > Need backport HIVE-11591 to branch-1. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12721) Add UUID built in function
[
https://issues.apache.org/jira/browse/HIVE-12721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15277520#comment-15277520
]
Hive QA commented on HIVE-12721:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12802898/HIVE-12721.2.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 17 failed/errored test(s), 9944 tests
executed
*Failed tests:*
{noformat}
TestCliDriver-gen_udf_example_add10.q-ppd_join4.q-union27.q-and-12-more - did
not produce a TEST-*.xml file
TestCliDriver-partition_timestamp.q-ppd_random.q-vector_outer_join5.q-and-12-more
- did not produce a TEST-*.xml file
TestCliDriver-ptf_general_queries.q-unionDistinct_1.q-groupby1_noskew.q-and-12-more
- did not produce a TEST-*.xml file
TestHWISessionManager - did not produce a TEST-*.xml file
TestMiniLlapCliDriver - did not produce a TEST-*.xml file
TestMiniTezCliDriver-enforce_order.q-vector_partition_diff_num_cols.q-unionDistinct_1.q-and-12-more
- did not produce a TEST-*.xml file
TestMiniTezCliDriver-join1.q-schema_evol_orc_nonvec_mapwork_part.q-mapjoin_decimal.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-bucketsortoptimize_insert_7.q-smb_mapjoin_15.q-mapreduce1.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-skewjoinopt3.q-union27.q-multigroupby_singlemr.q-and-12-more
- did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_external_table_ppd
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_binary_storage_queries
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3
org.apache.hadoop.hive.llap.tez.TestConverters.testFragmentSpecToTaskSpec
org.apache.hadoop.hive.llap.tezplugins.TestLlapTaskCommunicator.testFinishableStateUpdateFailure
org.apache.hadoop.hive.metastore.TestHiveMetaStoreGetMetaConf.testGetMetaConfDefault
org.apache.hadoop.hive.metastore.hbase.TestHBaseSchemaTool.oneMondoTest
org.apache.hive.service.cli.session.TestHiveSessionImpl.testLeakOperationHandle
{noformat}
Test results:
http://ec2-54-177-240-2.us-west-1.compute.amazonaws.com/job/PreCommit-HIVE-MASTER-Build/225/testReport
Console output:
http://ec2-54-177-240-2.us-west-1.compute.amazonaws.com/job/PreCommit-HIVE-MASTER-Build/225/console
Test logs:
http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-225/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 17 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12802898 - PreCommit-HIVE-MASTER-Build
> Add UUID built in function
> --
>
> Key: HIVE-12721
> URL: https://issues.apache.org/jira/browse/HIVE-12721
> Project: Hive
> Issue Type: Improvement
> Components: UDF
>Reporter: Jeremy Beard
>Assignee: Jeremy Beard
> Attachments: HIVE-12721.1.patch, HIVE-12721.2.patch, HIVE-12721.patch
>
>
> A UUID function would be very useful for ETL jobs that need to generate
> surrogate keys.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13598) Describe extended table should show the primary keys/foreign keys associated with the table
[ https://issues.apache.org/jira/browse/HIVE-13598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Sankar Sivarama Subramaniyan updated HIVE-13598: - Status: Open (was: Patch Available) > Describe extended table should show the primary keys/foreign keys associated > with the table > --- > > Key: HIVE-13598 > URL: https://issues.apache.org/jira/browse/HIVE-13598 > Project: Hive > Issue Type: Bug >Reporter: Hari Sankar Sivarama Subramaniyan >Assignee: Hari Sankar Sivarama Subramaniyan > Attachments: HIVE-13598.1.patch, HIVE-13598.2.patch, > HIVE-13598.3.patch, HIVE-13598.4.patch, HIVE-13598.5.patch, HIVE-13598.6.patch > > > After HIVE-13290 is committed, we need to show the constraints as part of > table description when extended label is used. Currently, the constraints > would not be shown as part of table description since Constraint is a > separate entity. > The purpose of the jira is to modify Hive.describeTable() to enable the user > to view the constraints associated with the table when the user does a > "describe extended table"; -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13598) Describe extended table should show the primary keys/foreign keys associated with the table
[ https://issues.apache.org/jira/browse/HIVE-13598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Sankar Sivarama Subramaniyan updated HIVE-13598: - Status: Patch Available (was: Open) > Describe extended table should show the primary keys/foreign keys associated > with the table > --- > > Key: HIVE-13598 > URL: https://issues.apache.org/jira/browse/HIVE-13598 > Project: Hive > Issue Type: Bug >Reporter: Hari Sankar Sivarama Subramaniyan >Assignee: Hari Sankar Sivarama Subramaniyan > Attachments: HIVE-13598.1.patch, HIVE-13598.2.patch, > HIVE-13598.3.patch, HIVE-13598.4.patch, HIVE-13598.5.patch, HIVE-13598.6.patch > > > After HIVE-13290 is committed, we need to show the constraints as part of > table description when extended label is used. Currently, the constraints > would not be shown as part of table description since Constraint is a > separate entity. > The purpose of the jira is to modify Hive.describeTable() to enable the user > to view the constraints associated with the table when the user does a > "describe extended table"; -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13598) Describe extended table should show the primary keys/foreign keys associated with the table
[ https://issues.apache.org/jira/browse/HIVE-13598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Sankar Sivarama Subramaniyan updated HIVE-13598: - Status: Patch Available (was: Open) > Describe extended table should show the primary keys/foreign keys associated > with the table > --- > > Key: HIVE-13598 > URL: https://issues.apache.org/jira/browse/HIVE-13598 > Project: Hive > Issue Type: Bug >Reporter: Hari Sankar Sivarama Subramaniyan >Assignee: Hari Sankar Sivarama Subramaniyan > Attachments: HIVE-13598.1.patch, HIVE-13598.2.patch, > HIVE-13598.3.patch, HIVE-13598.4.patch, HIVE-13598.5.patch, HIVE-13598.6.patch > > > After HIVE-13290 is committed, we need to show the constraints as part of > table description when extended label is used. Currently, the constraints > would not be shown as part of table description since Constraint is a > separate entity. > The purpose of the jira is to modify Hive.describeTable() to enable the user > to view the constraints associated with the table when the user does a > "describe extended table"; -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13598) Describe extended table should show the primary keys/foreign keys associated with the table
[ https://issues.apache.org/jira/browse/HIVE-13598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Sankar Sivarama Subramaniyan updated HIVE-13598: - Attachment: HIVE-13598.6.patch > Describe extended table should show the primary keys/foreign keys associated > with the table > --- > > Key: HIVE-13598 > URL: https://issues.apache.org/jira/browse/HIVE-13598 > Project: Hive > Issue Type: Bug >Reporter: Hari Sankar Sivarama Subramaniyan >Assignee: Hari Sankar Sivarama Subramaniyan > Attachments: HIVE-13598.1.patch, HIVE-13598.2.patch, > HIVE-13598.3.patch, HIVE-13598.4.patch, HIVE-13598.5.patch, HIVE-13598.6.patch > > > After HIVE-13290 is committed, we need to show the constraints as part of > table description when extended label is used. Currently, the constraints > would not be shown as part of table description since Constraint is a > separate entity. > The purpose of the jira is to modify Hive.describeTable() to enable the user > to view the constraints associated with the table when the user does a > "describe extended table"; -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13598) Describe extended table should show the primary keys/foreign keys associated with the table
[ https://issues.apache.org/jira/browse/HIVE-13598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Sankar Sivarama Subramaniyan updated HIVE-13598: - Status: Open (was: Patch Available) > Describe extended table should show the primary keys/foreign keys associated > with the table > --- > > Key: HIVE-13598 > URL: https://issues.apache.org/jira/browse/HIVE-13598 > Project: Hive > Issue Type: Bug >Reporter: Hari Sankar Sivarama Subramaniyan >Assignee: Hari Sankar Sivarama Subramaniyan > Attachments: HIVE-13598.1.patch, HIVE-13598.2.patch, > HIVE-13598.3.patch, HIVE-13598.4.patch, HIVE-13598.5.patch, HIVE-13598.6.patch > > > After HIVE-13290 is committed, we need to show the constraints as part of > table description when extended label is used. Currently, the constraints > would not be shown as part of table description since Constraint is a > separate entity. > The purpose of the jira is to modify Hive.describeTable() to enable the user > to view the constraints associated with the table when the user does a > "describe extended table"; -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13726) Improve dynamic partition loading VI
[ https://issues.apache.org/jira/browse/HIVE-13726?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-13726: Status: Patch Available (was: Open) > Improve dynamic partition loading VI > > > Key: HIVE-13726 > URL: https://issues.apache.org/jira/browse/HIVE-13726 > Project: Hive > Issue Type: Improvement > Components: Query Processor >Affects Versions: 2.0.0, 1.2.0 >Reporter: Ashutosh Chauhan >Assignee: Ashutosh Chauhan > Attachments: HIVE-13726.patch > > > Parallelize deletes and other refactoring. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13726) Improve dynamic partition loading VI
[ https://issues.apache.org/jira/browse/HIVE-13726?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-13726: Attachment: HIVE-13726.patch > Improve dynamic partition loading VI > > > Key: HIVE-13726 > URL: https://issues.apache.org/jira/browse/HIVE-13726 > Project: Hive > Issue Type: Improvement > Components: Query Processor >Affects Versions: 1.2.0, 2.0.0 >Reporter: Ashutosh Chauhan >Assignee: Ashutosh Chauhan > Attachments: HIVE-13726.patch > > > Parallelize deletes and other refactoring. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13726) Improve dynamic partition loading VI
[ https://issues.apache.org/jira/browse/HIVE-13726?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-13726: Status: Open (was: Patch Available) > Improve dynamic partition loading VI > > > Key: HIVE-13726 > URL: https://issues.apache.org/jira/browse/HIVE-13726 > Project: Hive > Issue Type: Improvement > Components: Query Processor >Affects Versions: 2.0.0, 1.2.0 >Reporter: Ashutosh Chauhan >Assignee: Ashutosh Chauhan > > Parallelize deletes and other refactoring. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13726) Improve dynamic partition loading VI
[ https://issues.apache.org/jira/browse/HIVE-13726?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-13726: Attachment: (was: HIVE-13726.patch) > Improve dynamic partition loading VI > > > Key: HIVE-13726 > URL: https://issues.apache.org/jira/browse/HIVE-13726 > Project: Hive > Issue Type: Improvement > Components: Query Processor >Affects Versions: 1.2.0, 2.0.0 >Reporter: Ashutosh Chauhan >Assignee: Ashutosh Chauhan > > Parallelize deletes and other refactoring. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13395) Lost Update problem in ACID
[ https://issues.apache.org/jira/browse/HIVE-13395?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15277442#comment-15277442 ] Prasanth Jayachandran commented on HIVE-13395: -- the typo fix looks good to me, +1 > Lost Update problem in ACID > --- > > Key: HIVE-13395 > URL: https://issues.apache.org/jira/browse/HIVE-13395 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 1.2.0, 2.0.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman >Priority: Blocker > Labels: TODOC1.3, TODOC2.1 > Fix For: 1.3.0, 2.1.0 > > Attachments: HIVE-13395.11.patch, HIVE-13395.12.patch, > HIVE-13395.13.patch, HIVE-13395.14.patch, HIVE-13395.15.patch, > HIVE-13395.16.patch, HIVE-13395.6.patch, HIVE-13395.7.patch, > HIVE-13395.8.patch, HIVE-13395.addendum.patch, HIVE-13395.addendum2.patch > > > ACID users can run into Lost Update problem. > In Hive 1.2, Driver.recordValidTxns() (which records the snapshot to use for > the query) is called in Driver.compile(). > Now suppose to concurrent "update T set x = x + 1" are executed. (for > simplicity assume there is exactly 1 row in T) > What can happen is that both compile at the same time (more precisely before > acquireLocksAndOpenTxn() in runInternal() is called) and thus will lock in > the same snapshot, say the value of x = 7 in this snapshot. > Now 1 will get the lock on the row, the second will block. > Now 1, makes x = 8 and commits. > Now 2 proceeds and makes x = 8 again since in it's snapshot x is still 7. > This specific issue is solved in Hive 1.3/2.0 (HIVE-11077 which is a large > patch that deals with multi-statement txns) by moving recordValidTxns() after > locks are acquired which reduces the likelihood of this but doesn't eliminate > the problem. > > Even in 1.3 version of the code, you could have the same issue. Assume the > same 2 queries: > Both start a txn, say txnid 9 and 10. Say 10 gets the lock first, 9 blocks. > 10 updates the row (so x = 8) and thus ReaderKey.currentTransactionId=10. > 10 commits. > Now 9 can proceed and it will get a snapshot that includes 10, i.e. it will > see x = 8 and it will write x = 9, but it will set > ReaderKey.currentTransactionId = 9. Thus when merge logic runs, it will see > x = 8 is the later version of this row, i.e. lost update. > The problem is that locks alone are insufficient for MVCC architecture. > > At lower level Row ID has (originalTransactionId, rowid, bucket id, > currentTransactionId) and since on update/delete we do a table scan, we could > check that we are about to write a row with currentTransactionId < > (currentTransactionId of row we've read) and fail the query. Currently, > currentTransactionId is not surfaced at higher level where this check can be > made. > This would not work (efficiently) longer term where we want to support fast > update on user defined PK vis streaming ingest. > Also, this would not work with multi statement txns since in that case we'd > lock in the snapshot at the start of the txn, but then 2nd, 3rd etc queries > would use the same snapshot and the locks for these queries would be acquired > after the snapshot is locked in so this would be the same situation as pre > HIVE-11077. > > > A more robust solution (commonly used with MVCC) is to keep track of start > and commit time (logical counter) or each transaction to detect if two txns > overlap. The 2nd part is to keep track of write-set, i.e. which data (rows, > partitions, whatever appropriate level of granularity is) were modified by > any txn and if 2 txns overlap in time and wrote the same element, abort later > one. This is called first-committer-wins rule. This requires a MS DB schema > change > It would be most convenient to use the same sequence for txnId, start and > commit time (in which case txnid=start time). In this case we'd need to add > 1 filed to TXNS table. The complication here is that we'll be using elements > of the sequence faster and they are used as part of file name of delta and > base dir and currently limited to 7 digits which can be exceeded. So this > would require some thought to handling upgrade/migration. > Also, write-set tracking requires either additional metastore table or > keeping info in HIVE_LOCKS around longer with new state. > > In the short term, on SQL side of things we could (in auto commit mode only) > acquire the locks first and then open the txn AND update these locks with txn > id. > This implies another Thrift change to pass in lockId to openTxn. > The same would not work for Streaming API since it opens several txns at once > and then acquires locks for each. > (Not sure if that's is an issue or not since Streaming only does Insert). > Either way this
[jira] [Updated] (HIVE-13726) Improve dynamic partition loading VI
[ https://issues.apache.org/jira/browse/HIVE-13726?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-13726: Status: Patch Available (was: Open) > Improve dynamic partition loading VI > > > Key: HIVE-13726 > URL: https://issues.apache.org/jira/browse/HIVE-13726 > Project: Hive > Issue Type: Improvement > Components: Query Processor >Affects Versions: 2.0.0, 1.2.0 >Reporter: Ashutosh Chauhan >Assignee: Ashutosh Chauhan > Attachments: HIVE-13726.patch > > > Parallelize deletes and other refactoring. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13726) Improve dynamic partition loading VI
[ https://issues.apache.org/jira/browse/HIVE-13726?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-13726: Attachment: HIVE-13726.patch > Improve dynamic partition loading VI > > > Key: HIVE-13726 > URL: https://issues.apache.org/jira/browse/HIVE-13726 > Project: Hive > Issue Type: Improvement > Components: Query Processor >Affects Versions: 1.2.0, 2.0.0 >Reporter: Ashutosh Chauhan >Assignee: Ashutosh Chauhan > Attachments: HIVE-13726.patch > > > Parallelize deletes and other refactoring. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13395) Lost Update problem in ACID
[ https://issues.apache.org/jira/browse/HIVE-13395?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-13395: -- Attachment: HIVE-13395.addendum2.patch > Lost Update problem in ACID > --- > > Key: HIVE-13395 > URL: https://issues.apache.org/jira/browse/HIVE-13395 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 1.2.0, 2.0.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman >Priority: Blocker > Labels: TODOC1.3, TODOC2.1 > Fix For: 1.3.0, 2.1.0 > > Attachments: HIVE-13395.11.patch, HIVE-13395.12.patch, > HIVE-13395.13.patch, HIVE-13395.14.patch, HIVE-13395.15.patch, > HIVE-13395.16.patch, HIVE-13395.6.patch, HIVE-13395.7.patch, > HIVE-13395.8.patch, HIVE-13395.addendum.patch, HIVE-13395.addendum2.patch > > > ACID users can run into Lost Update problem. > In Hive 1.2, Driver.recordValidTxns() (which records the snapshot to use for > the query) is called in Driver.compile(). > Now suppose to concurrent "update T set x = x + 1" are executed. (for > simplicity assume there is exactly 1 row in T) > What can happen is that both compile at the same time (more precisely before > acquireLocksAndOpenTxn() in runInternal() is called) and thus will lock in > the same snapshot, say the value of x = 7 in this snapshot. > Now 1 will get the lock on the row, the second will block. > Now 1, makes x = 8 and commits. > Now 2 proceeds and makes x = 8 again since in it's snapshot x is still 7. > This specific issue is solved in Hive 1.3/2.0 (HIVE-11077 which is a large > patch that deals with multi-statement txns) by moving recordValidTxns() after > locks are acquired which reduces the likelihood of this but doesn't eliminate > the problem. > > Even in 1.3 version of the code, you could have the same issue. Assume the > same 2 queries: > Both start a txn, say txnid 9 and 10. Say 10 gets the lock first, 9 blocks. > 10 updates the row (so x = 8) and thus ReaderKey.currentTransactionId=10. > 10 commits. > Now 9 can proceed and it will get a snapshot that includes 10, i.e. it will > see x = 8 and it will write x = 9, but it will set > ReaderKey.currentTransactionId = 9. Thus when merge logic runs, it will see > x = 8 is the later version of this row, i.e. lost update. > The problem is that locks alone are insufficient for MVCC architecture. > > At lower level Row ID has (originalTransactionId, rowid, bucket id, > currentTransactionId) and since on update/delete we do a table scan, we could > check that we are about to write a row with currentTransactionId < > (currentTransactionId of row we've read) and fail the query. Currently, > currentTransactionId is not surfaced at higher level where this check can be > made. > This would not work (efficiently) longer term where we want to support fast > update on user defined PK vis streaming ingest. > Also, this would not work with multi statement txns since in that case we'd > lock in the snapshot at the start of the txn, but then 2nd, 3rd etc queries > would use the same snapshot and the locks for these queries would be acquired > after the snapshot is locked in so this would be the same situation as pre > HIVE-11077. > > > A more robust solution (commonly used with MVCC) is to keep track of start > and commit time (logical counter) or each transaction to detect if two txns > overlap. The 2nd part is to keep track of write-set, i.e. which data (rows, > partitions, whatever appropriate level of granularity is) were modified by > any txn and if 2 txns overlap in time and wrote the same element, abort later > one. This is called first-committer-wins rule. This requires a MS DB schema > change > It would be most convenient to use the same sequence for txnId, start and > commit time (in which case txnid=start time). In this case we'd need to add > 1 filed to TXNS table. The complication here is that we'll be using elements > of the sequence faster and they are used as part of file name of delta and > base dir and currently limited to 7 digits which can be exceeded. So this > would require some thought to handling upgrade/migration. > Also, write-set tracking requires either additional metastore table or > keeping info in HIVE_LOCKS around longer with new state. > > In the short term, on SQL side of things we could (in auto commit mode only) > acquire the locks first and then open the txn AND update these locks with txn > id. > This implies another Thrift change to pass in lockId to openTxn. > The same would not work for Streaming API since it opens several txns at once > and then acquires locks for each. > (Not sure if that's is an issue or not since Streaming only does Insert). > Either way this feels hacky. > > Here is one simple
[jira] [Updated] (HIVE-13696) Monitor fair-scheduler.xml and automatically update/validate jobs submitted to fair-scheduler
[
https://issues.apache.org/jira/browse/HIVE-13696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Reuben Kuhnert updated HIVE-13696:
--
Attachment: HIVE-13696.02.patch
> Monitor fair-scheduler.xml and automatically update/validate jobs submitted
> to fair-scheduler
> -
>
> Key: HIVE-13696
> URL: https://issues.apache.org/jira/browse/HIVE-13696
> Project: Hive
> Issue Type: Improvement
>Reporter: Reuben Kuhnert
>Assignee: Reuben Kuhnert
> Attachments: HIVE-13696.01.patch, HIVE-13696.02.patch
>
>
> Ensure that jobs are placed into the correct queue according to
> {{fair-scheduler.xml}}. Jobs should be placed into the correct queue, and
> users should not be able to submit jobs to queues they do not have access to.
> This patch builds on the existing functionality in {{FairSchedulerShim}} to
> route jobs to user-specific queue based on {{fair-scheduler.xml}}
> configuration (leveraging the Yarn {{QueuePlacementPolicy}} class). In
> addition to configuring job routing at session connect (current behavior),
> the routing is validated per submission to yarn (when impersonation is off).
> A {{FileSystemWatcher}} class is included to monitor changes in the
> {{fair-scheduler.xml}} file (so updates are automatically reloaded when the
> file pointed to by {{yarn.scheduler.fair.allocation.file}} is changed).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13696) Monitor fair-scheduler.xml and automatically update/validate jobs submitted to fair-scheduler
[
https://issues.apache.org/jira/browse/HIVE-13696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Reuben Kuhnert updated HIVE-13696:
--
Summary: Monitor fair-scheduler.xml and automatically update/validate jobs
submitted to fair-scheduler (was: Validate jobs submitted to fair-scheduler)
> Monitor fair-scheduler.xml and automatically update/validate jobs submitted
> to fair-scheduler
> -
>
> Key: HIVE-13696
> URL: https://issues.apache.org/jira/browse/HIVE-13696
> Project: Hive
> Issue Type: Improvement
>Reporter: Reuben Kuhnert
>Assignee: Reuben Kuhnert
> Attachments: HIVE-13696.01.patch
>
>
> Ensure that jobs are placed into the correct queue according to
> {{fair-scheduler.xml}}. Jobs should be placed into the correct queue, and
> users should not be able to submit jobs to queues they do not have access to.
> This patch builds on the existing functionality in {{FairSchedulerShim}} to
> route jobs to user-specific queue based on {{fair-scheduler.xml}}
> configuration (leveraging the Yarn {{QueuePlacementPolicy}} class). In
> addition to configuring job routing at session connect (current behavior),
> the routing is validated per submission to yarn (when impersonation is off).
> A {{FileSystemWatcher}} class is included to monitor changes in the
> {{fair-scheduler.xml}} file (so updates are automatically reloaded when the
> file pointed to by {{yarn.scheduler.fair.allocation.file}} is changed).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (HIVE-13725) ACID: Streaming API should synchronize calls when multiple threads use the same endpoint
[ https://issues.apache.org/jira/browse/HIVE-13725?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vaibhav Gumashta reassigned HIVE-13725: --- Assignee: Vaibhav Gumashta > ACID: Streaming API should synchronize calls when multiple threads use the > same endpoint > > > Key: HIVE-13725 > URL: https://issues.apache.org/jira/browse/HIVE-13725 > Project: Hive > Issue Type: Bug > Components: HCatalog, Metastore, Transactions >Affects Versions: 1.2.1, 2.0.0 >Reporter: Vaibhav Gumashta >Assignee: Vaibhav Gumashta > Labels: ACID, Streaming > > Currently, the streaming endpoint creates a metastore client which gets used > for RPC. The client itself is not internally thread safe. Therefore, the API > methods should provide the relevant synchronization so that the methods can > be called from different threads. A sample use case is as follows: > 1. Thread 1 creates a streaming endpoint and opens a txn batch. > 2. Thread 2 heartbeats the txn batch. > With the current impl, this can result in an "out of sequence response", > since the response of the calls in thread1 might end up going to thread2 and > vice-versa. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13724) Backport HIVE-11591 to branch-1 to use undated annotations
[ https://issues.apache.org/jira/browse/HIVE-13724?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15277420#comment-15277420 ] Sergey Shelukhin commented on HIVE-13724: - +1 > Backport HIVE-11591 to branch-1 to use undated annotations > -- > > Key: HIVE-13724 > URL: https://issues.apache.org/jira/browse/HIVE-13724 > Project: Hive > Issue Type: Bug > Components: Thrift API >Affects Versions: 1.2.1 >Reporter: Wei Zheng >Assignee: Wei Zheng > Attachments: HIVE-13724.1.patch, HIVE-13724.pom_change_only.patch > > > HIVE-12832 changed branch-1 hive pom file and updated thrift version from > 0.9.2 to 0.9.3. But it didn't update the thrift args part to use undated > annotation from HIVE-11591. > So every time someone is running maven thrift re-gen command, it will still > update a lot of unrelated files, just because of the date change. > Need backport HIVE-11591 to branch-1. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13725) ACID: Streaming API should synchronize calls when multiple threads use the same endpoint
[ https://issues.apache.org/jira/browse/HIVE-13725?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vaibhav Gumashta updated HIVE-13725: Labels: ACID Streaming (was: ) > ACID: Streaming API should synchronize calls when multiple threads use the > same endpoint > > > Key: HIVE-13725 > URL: https://issues.apache.org/jira/browse/HIVE-13725 > Project: Hive > Issue Type: Bug > Components: HCatalog, Metastore, Transactions >Affects Versions: 1.2.1, 2.0.0 >Reporter: Vaibhav Gumashta > Labels: ACID, Streaming > > Currently, the streaming endpoint creates a metastore client which gets used > for RPC. The client itself is not internally thread safe. Therefore, the API > methods should provide the relevant synchronization so that the methods can > be called from different threads. A sample use case is as follows: > 1. Thread 1 creates a streaming endpoint and opens a txn batch. > 2. Thread 2 heartbeats the txn batch. > With the current impl, this can result in an "out of sequence response", > since the response of the calls in thread1 might end up going to thread2 and > vice-versa. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13725) ACID: Streaming API should synchronize calls when multiple threads use the same endpoint
[ https://issues.apache.org/jira/browse/HIVE-13725?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-13725: -- Component/s: Transactions HCatalog > ACID: Streaming API should synchronize calls when multiple threads use the > same endpoint > > > Key: HIVE-13725 > URL: https://issues.apache.org/jira/browse/HIVE-13725 > Project: Hive > Issue Type: Bug > Components: HCatalog, Metastore, Transactions >Affects Versions: 1.2.1, 2.0.0 >Reporter: Vaibhav Gumashta > > Currently, the streaming endpoint creates a metastore client which gets used > for RPC. The client itself is not internally thread safe. Therefore, the API > methods should provide the relevant synchronization so that the methods can > be called from different threads. A sample use case is as follows: > 1. Thread 1 creates a streaming endpoint and opens a txn batch. > 2. Thread 2 heartbeats the txn batch. > With the current impl, this can result in an "out of sequence response", > since the response of the calls in thread1 might end up going to thread2 and > vice-versa. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13618) Trailing spaces in partition column will be treated differently
[ https://issues.apache.org/jira/browse/HIVE-13618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15277419#comment-15277419 ] Lefty Leverenz commented on HIVE-13618: --- Should this be documented in the wiki? > Trailing spaces in partition column will be treated differently > --- > > Key: HIVE-13618 > URL: https://issues.apache.org/jira/browse/HIVE-13618 > Project: Hive > Issue Type: Bug >Affects Versions: 2.0.0 >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Fix For: 2.1.0 > > Attachments: HIVE-13618.01.patch > > > We store the partition spec value in the metastore. In mysql (and derby i > think), the trailing space is ignored. That is, if you have a partition > column "col" (type varchar or string) with value "a " and then select from > the table where col = "a", it will return. However, in postgres and Oracle, > the trailing space is not ignored. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13458) Heartbeater doesn't fail query when heartbeat fails
[ https://issues.apache.org/jira/browse/HIVE-13458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15277413#comment-15277413 ] Eugene Koifman commented on HIVE-13458: --- I think queryId should be set to null in stopHeartbeat() just like heartbeatTask Shouldn't acquireLocksWithHeartbeatDelay() do ctx.setHeartbeater(startHeartbeat()); ? Or perhaps better, this and make acquireLocks(QueryPlan plan, Context ctx, String username) call acquireLocksWithHeartbeatDelay() Why did the 2 .q files change? They don't seem to be related to any of the changes > Heartbeater doesn't fail query when heartbeat fails > --- > > Key: HIVE-13458 > URL: https://issues.apache.org/jira/browse/HIVE-13458 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 2.1.0 >Reporter: Wei Zheng >Assignee: Wei Zheng > Attachments: HIVE-13458.1.patch, HIVE-13458.2.patch, > HIVE-13458.3.patch, HIVE-13458.4.patch, HIVE-13458.5.patch, > HIVE-13458.6.patch, HIVE-13458.7.patch, HIVE-13458.8.patch > > > When a heartbeat fails to locate a lock, it should fail the current query. > That doesn't happen, which is a bug. > Another thing is, we need to make sure stopHeartbeat really stops the > heartbeat, i.e. no additional heartbeat will be sent, since that will break > the assumption and cause the query to fail. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13724) Backport HIVE-11591 to branch-1 to use undated annotations
[ https://issues.apache.org/jira/browse/HIVE-13724?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Zheng updated HIVE-13724: - Attachment: HIVE-13724.pom_change_only.patch HIVE-13724.1.patch [~sershe] Can you review please? > Backport HIVE-11591 to branch-1 to use undated annotations > -- > > Key: HIVE-13724 > URL: https://issues.apache.org/jira/browse/HIVE-13724 > Project: Hive > Issue Type: Bug > Components: Thrift API >Affects Versions: 1.2.1 >Reporter: Wei Zheng >Assignee: Wei Zheng > Attachments: HIVE-13724.1.patch, HIVE-13724.pom_change_only.patch > > > HIVE-12832 changed branch-1 hive pom file and updated thrift version from > 0.9.2 to 0.9.3. But it didn't update the thrift args part to use undated > annotation from HIVE-11591. > So every time someone is running maven thrift re-gen command, it will still > update a lot of unrelated files, just because of the date change. > Need backport HIVE-11591 to branch-1. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11591) upgrade thrift to 0.9.3 and change generation to use undated annotations
[ https://issues.apache.org/jira/browse/HIVE-11591?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15277403#comment-15277403 ] Sergey Shelukhin commented on HIVE-11591: - Sure > upgrade thrift to 0.9.3 and change generation to use undated annotations > > > Key: HIVE-11591 > URL: https://issues.apache.org/jira/browse/HIVE-11591 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Fix For: 2.0.0 > > Attachments: HIVE-11591.WIP.patch, HIVE-11591.nogen.patch, > HIVE-11591.patch > > > Thrift has added class annotations to generated classes; these contain > generation date. Because of this, all the Java thrift files change on every > re-gen, even if you only make a small change that should not affect bazillion > files. We should use undated annotations to avoid this problem. > This depends on upgrading to Thrift 0.9.3, -which doesn't exist yet-. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13657) Spark driver stderr logs should appear in hive client logs
[
https://issues.apache.org/jira/browse/HIVE-13657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15277396#comment-15277396
]
Mohit Sabharwal commented on HIVE-13657:
[~szehon], could you take a look? thanks!
> Spark driver stderr logs should appear in hive client logs
> --
>
> Key: HIVE-13657
> URL: https://issues.apache.org/jira/browse/HIVE-13657
> Project: Hive
> Issue Type: Bug
>Reporter: Mohit Sabharwal
>Assignee: Mohit Sabharwal
> Attachments: HIVE-13657.1.patch, HIVE-13657.patch
>
>
> Currently, spark driver exceptions are not getting logged in beeline.
> Instead, the users sees the not-so-useful:
> {code}
> ERROR : Failed to execute spark task, with exception
> 'org.apache.hadoop.hive.ql.metadata.HiveException(Failed to create spark
> client.)'
>
> {code}
> The user has to look at HS2 logs to discover the root cause:
> {code}
> 2015-04-01 11:33:16,048 INFO org.apache.hive.spark.client.SparkClientImpl:
> 15/04/01 11:33:16 WARN UserGroupInformation: PriviledgedActionException
> as:foo (auth:PROXY) via hive (auth:SIMPLE)
> cause:org.apache.hadoop.security.AccessControlException: Permission denied:
> user=foo, access=WRITE, inode="/user":hdfs:supergroup:drwxr-xr-x
> ...
> {code}
> We should surface these critical errors in hive client.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-12996) Temp tables shouldn't be locked
[
https://issues.apache.org/jira/browse/HIVE-12996?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15277393#comment-15277393
]
Eugene Koifman commented on HIVE-12996:
---
DbTxnManager.isTempTable(Entity entity) should use Table.isTemporary()
otherwise looks good
> Temp tables shouldn't be locked
> ---
>
> Key: HIVE-12996
> URL: https://issues.apache.org/jira/browse/HIVE-12996
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 2.0.0
>Reporter: Wei Zheng
>Assignee: Wei Zheng
> Attachments: HIVE-12996.1.patch, HIVE-12996.2.patch
>
>
> Internally, INSERT INTO ... VALUES statements use temp table to accomplish
> its functionality. But temp tables shouldn't be stored in the metastore
> tables for ACID, because they are by definition only visible inside the
> session that created them, and we don't allow multiple threads inside a
> session. If a temp table is used in a query, it should be ignored by lock
> manager.
> {code}
> mysql> select * from COMPLETED_TXN_COMPONENTS;
> +---+--+---+--+
> | CTC_TXNID | CTC_DATABASE | CTC_TABLE | CTC_PARTITION|
> +---+--+---+--+
> | 1 | acid | t1| NULL |
> | 1 | acid | values__tmp__table__1 | NULL |
> | 2 | acid | t1| NULL |
> | 2 | acid | values__tmp__table__2 | NULL |
> | 3 | acid | values__tmp__table__3 | NULL |
> | 3 | acid | t1| NULL |
> | 4 | acid | values__tmp__table__1 | NULL |
> | 4 | acid | t2p | ds=today |
> | 5 | acid | values__tmp__table__1 | NULL |
> | 5 | acid | t3p | ds=today/hour=12 |
> +---+--+---+--+
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-11591) upgrade thrift to 0.9.3 and change generation to use undated annotations
[ https://issues.apache.org/jira/browse/HIVE-11591?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15277383#comment-15277383 ] Wei Zheng commented on HIVE-11591: -- [~sershe] Should we backport this thrift args change to branch-1 as well? I see a lot of changes due to the date. > upgrade thrift to 0.9.3 and change generation to use undated annotations > > > Key: HIVE-11591 > URL: https://issues.apache.org/jira/browse/HIVE-11591 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Fix For: 2.0.0 > > Attachments: HIVE-11591.WIP.patch, HIVE-11591.nogen.patch, > HIVE-11591.patch > > > Thrift has added class annotations to generated classes; these contain > generation date. Because of this, all the Java thrift files change on every > re-gen, even if you only make a small change that should not affect bazillion > files. We should use undated annotations to avoid this problem. > This depends on upgrading to Thrift 0.9.3, -which doesn't exist yet-. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13723) Executing join query on type Float using Thrift Serde will result in Float cast to Double error
[
https://issues.apache.org/jira/browse/HIVE-13723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15277352#comment-15277352
]
Ziyang Zhao commented on HIVE-13723:
This error is caused by
hive-serde/src/java/org/apache/hadoop/hive/serde2/thrift/ColumnBuffer.java.
When first initialize the ColumnBuffer for type Float, it set the member
variable "type = Type.DOUBLE_TYPE". When addValue(Type, Object) is called, this
member variable is passed in as the first parameter, which is "DOUBLE_TYPE, so
it does "(Double) field" and caused the error.
> Executing join query on type Float using Thrift Serde will result in Float
> cast to Double error
> ---
>
> Key: HIVE-13723
> URL: https://issues.apache.org/jira/browse/HIVE-13723
> Project: Hive
> Issue Type: Bug
> Components: Serializers/Deserializers
>Affects Versions: 2.1.0
>Reporter: Ziyang Zhao
>
> After enable thrift Serde, execute the following queries in beeline,
> >create table test1 (a int);
> >create table test2 (b float);
> >insert into test1 values (1);
> >insert into test2 values (1);
> >select * from test1 join test2 on test1.a=test2.b;
> this will give the error:
> java.lang.Exception: java.lang.RuntimeException:
> org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while
> processing row {"b":1.0}
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:462)
> ~[hadoop-mapreduce-client-common-2.7.1.2.4.0.0-169.jar:?]
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:522)
> [hadoop-mapreduce-client-common-2.7.1.2.4.0.0-169.jar:?]
> Caused by: java.lang.RuntimeException:
> org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while
> processing row {"b":1.0}
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:168)
> ~[hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
> ~[hadoop-mapreduce-client-core-2.7.1.2.4.0.0-169.jar:?]
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
> ~[hadoop-mapreduce-client-core-2.7.1.2.4.0.0-169.jar:?]
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
> ~[hadoop-mapreduce-client-core-2.7.1.2.4.0.0-169.jar:?]
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:243)
> ~[hadoop-mapreduce-client-common-2.7.1.2.4.0.0-169.jar:?]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> ~[?:1.7.0_95]
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> ~[?:1.7.0_95]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> ~[?:1.7.0_95]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> ~[?:1.7.0_95]
> at java.lang.Thread.run(Thread.java:745) ~[?:1.7.0_95]
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime
> Error while processing row {"b":1.0}
> at
> org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:568)
> ~[hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:159)
> ~[hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
> ~[hadoop-mapreduce-client-core-2.7.1.2.4.0.0-169.jar:?]
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
> ~[hadoop-mapreduce-client-core-2.7.1.2.4.0.0-169.jar:?]
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
> ~[hadoop-mapreduce-client-core-2.7.1.2.4.0.0-169.jar:?]
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:243)
> ~[hadoop-mapreduce-client-common-2.7.1.2.4.0.0-169.jar:?]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> ~[?:1.7.0_95]
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> ~[?:1.7.0_95]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> ~[?:1.7.0_95]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> ~[?:1.7.0_95]
> at java.lang.Thread.run(Thread.java:745) ~[?:1.7.0_95]
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Unexpected
> exception from MapJoinOperator :
> org.apache.hadoop.hive.serde2.SerDeException: java.lang.ClassCastException:
> java.lang.Float cannot be cast to java.lang.Double
> at
> org.apache.hadoop.hive.ql.exec.MapJoinOperator.process(MapJoinOperator.java:454)
> ~[hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
>
[jira] [Commented] (HIVE-13670) Improve Beeline connect/reconnect semantics
[ https://issues.apache.org/jira/browse/HIVE-13670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15277340#comment-15277340 ] Thejas M Nair commented on HIVE-13670: -- +1 The latest changes are minor and are isolated to beeline, so I think running the beeline tests for the update would be sufficient. > Improve Beeline connect/reconnect semantics > --- > > Key: HIVE-13670 > URL: https://issues.apache.org/jira/browse/HIVE-13670 > Project: Hive > Issue Type: Improvement >Affects Versions: 2.1.0 >Reporter: Sushanth Sowmyan >Assignee: Sushanth Sowmyan > Attachments: HIVE-13670.2.patch, HIVE-13670.3.patch, > HIVE-13670.4.patch, HIVE-13670.patch > > > For most users of beeline, chances are that they will be using it with a > single HS2 instance most of the time. In this scenario, having them type out > a jdbc uri for HS2 every single time to !connect can get tiresome. Thus, we > should improve semantics so that if a user does a successful !connect, then > we must store the last-connected-to-url, so that if they do a !close, and > then a !reconnect, then !reconnect should attempt to connect to the last > successfully used url. > Also, if they then do a !save, then that last-successfully-used url must be > saved, so that in subsequent sessions, they can simply do !reconnect rather > than specifying a url for !connect. > In addition, it would be useful to introduce a new way of doing !connect that > does involve typing out a jdbc url every time (since this is highly likely to > be error-prone) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13670) Improve Beeline connect/reconnect semantics
[ https://issues.apache.org/jira/browse/HIVE-13670?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sushanth Sowmyan updated HIVE-13670: Attachment: HIVE-13670.4.patch Updated patch per [~thejas]'s comments. > Improve Beeline connect/reconnect semantics > --- > > Key: HIVE-13670 > URL: https://issues.apache.org/jira/browse/HIVE-13670 > Project: Hive > Issue Type: Improvement >Affects Versions: 2.1.0 >Reporter: Sushanth Sowmyan >Assignee: Sushanth Sowmyan > Attachments: HIVE-13670.2.patch, HIVE-13670.3.patch, > HIVE-13670.4.patch, HIVE-13670.patch > > > For most users of beeline, chances are that they will be using it with a > single HS2 instance most of the time. In this scenario, having them type out > a jdbc uri for HS2 every single time to !connect can get tiresome. Thus, we > should improve semantics so that if a user does a successful !connect, then > we must store the last-connected-to-url, so that if they do a !close, and > then a !reconnect, then !reconnect should attempt to connect to the last > successfully used url. > Also, if they then do a !save, then that last-successfully-used url must be > saved, so that in subsequent sessions, they can simply do !reconnect rather > than specifying a url for !connect. > In addition, it would be useful to introduce a new way of doing !connect that > does involve typing out a jdbc url every time (since this is highly likely to > be error-prone) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13700) TestHiveOperationType is failing on master
[
https://issues.apache.org/jira/browse/HIVE-13700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15277332#comment-15277332
]
Hive QA commented on HIVE-13700:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12802844/HIVE-13700.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 17 failed/errored test(s), 9940 tests
executed
*Failed tests:*
{noformat}
TestCliDriver-gen_udf_example_add10.q-ppd_join4.q-union27.q-and-12-more - did
not produce a TEST-*.xml file
TestCliDriver-partition_timestamp.q-ppd_random.q-vector_outer_join5.q-and-12-more
- did not produce a TEST-*.xml file
TestCliDriver-ptf_general_queries.q-unionDistinct_1.q-groupby1_noskew.q-and-12-more
- did not produce a TEST-*.xml file
TestHWISessionManager - did not produce a TEST-*.xml file
TestMiniLlapCliDriver - did not produce a TEST-*.xml file
TestMiniTezCliDriver-enforce_order.q-vector_partition_diff_num_cols.q-unionDistinct_1.q-and-12-more
- did not produce a TEST-*.xml file
TestMiniTezCliDriver-join1.q-schema_evol_orc_nonvec_mapwork_part.q-mapjoin_decimal.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-bucketsortoptimize_insert_7.q-smb_mapjoin_15.q-mapreduce1.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-skewjoinopt3.q-union27.q-multigroupby_singlemr.q-and-12-more
- did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_external_table_ppd
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_binary_storage_queries
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3
org.apache.hadoop.hive.llap.tez.TestConverters.testFragmentSpecToTaskSpec
org.apache.hadoop.hive.llap.tezplugins.TestLlapTaskCommunicator.testFinishableStateUpdateFailure
org.apache.hadoop.hive.metastore.hbase.TestHBaseSchemaTool.oneMondoTest
org.apache.hive.hcatalog.hbase.TestPigHBaseStorageHandler.org.apache.hive.hcatalog.hbase.TestPigHBaseStorageHandler
org.apache.hive.service.cli.session.TestHiveSessionImpl.testLeakOperationHandle
{noformat}
Test results:
http://ec2-54-177-240-2.us-west-1.compute.amazonaws.com/job/PreCommit-HIVE-MASTER-Build/224/testReport
Console output:
http://ec2-54-177-240-2.us-west-1.compute.amazonaws.com/job/PreCommit-HIVE-MASTER-Build/224/console
Test logs:
http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-224/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 17 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12802844 - PreCommit-HIVE-MASTER-Build
> TestHiveOperationType is failing on master
> --
>
> Key: HIVE-13700
> URL: https://issues.apache.org/jira/browse/HIVE-13700
> Project: Hive
> Issue Type: Bug
> Components: Tests
>Affects Versions: 2.1.0
>Reporter: Ashutosh Chauhan
>Assignee: Hari Sankar Sivarama Subramaniyan
> Fix For: 2.1.0
>
> Attachments: HIVE-13700.1.patch
>
>
> Presumably be broken by HIVE-13351
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-11321) Move OrcFile.OrcTableProperties from OrcFile into OrcConf.
[ https://issues.apache.org/jira/browse/HIVE-11321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15277331#comment-15277331 ] Shannon Ladymon commented on HIVE-11321: [~owen.omalley], could you answer a few questions about this patch? I'd like to update the wiki with some more information from it: (1) Why are some of these table properties (ex. *orc.block.size*) not listed in the [HiveQL Syntax | https://cwiki.apache.org/confluence/display/Hive/LanguageManual+ORC#LanguageManualORC-HiveQLSyntax] section? Should they be added? (2) Why are some properties (*hive.orc.splits.include.file.footer, hive.orc.cache.stripe.details.size, hive.orc.compute.splits.num.threads, hive.exec.orc.split.strategy, hive.merge.orcfile.stripe.level*) listed in the [Configuration Properties | https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-ORCFileFormat] section, but not in the HiveQL Syntax section? Should they be added? (3) For the properties that are in both OrcConf.java and HiveConf.java, is there a conflict if they are set to different values? > Move OrcFile.OrcTableProperties from OrcFile into OrcConf. > -- > > Key: HIVE-11321 > URL: https://issues.apache.org/jira/browse/HIVE-11321 > Project: Hive > Issue Type: Sub-task >Reporter: Owen O'Malley >Assignee: Owen O'Malley > Fix For: 2.0.0 > > Attachments: HIVE-11321.patch > > > We should pull all of the configuration/table property knobs into a single > list. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11417) Create shims for the row by row read path that is backed by VectorizedRowBatch
[ https://issues.apache.org/jira/browse/HIVE-11417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15277315#comment-15277315 ] Prasanth Jayachandran commented on HIVE-11417: -- Left 2 minor comments in RB. Also had another comment about possible file descriptor leak in PR. The patch mostly looks like moving around code. The row-by-row shim and removal of next() looks good to me. +1, pending a clean test run > Create shims for the row by row read path that is backed by VectorizedRowBatch > -- > > Key: HIVE-11417 > URL: https://issues.apache.org/jira/browse/HIVE-11417 > Project: Hive > Issue Type: Sub-task >Reporter: Owen O'Malley >Assignee: Owen O'Malley > Fix For: 2.1.0 > > Attachments: HIVE-11417.patch, HIVE-11417.patch, HIVE-11417.patch, > HIVE-11417.patch, HIVE-11417.patch, HIVE-11417.patch > > > I'd like to make the default path for reading and writing ORC files to be > vectorized. To ensure that Hive can still read row by row, we'll need shims > to support the old API. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13700) TestHiveOperationType is failing on master
[ https://issues.apache.org/jira/browse/HIVE-13700?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Sankar Sivarama Subramaniyan updated HIVE-13700: - Resolution: Fixed Fix Version/s: 2.1.0 Status: Resolved (was: Patch Available) > TestHiveOperationType is failing on master > -- > > Key: HIVE-13700 > URL: https://issues.apache.org/jira/browse/HIVE-13700 > Project: Hive > Issue Type: Bug > Components: Tests >Affects Versions: 2.1.0 >Reporter: Ashutosh Chauhan >Assignee: Hari Sankar Sivarama Subramaniyan > Fix For: 2.1.0 > > Attachments: HIVE-13700.1.patch > > > Presumably be broken by HIVE-13351 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13700) TestHiveOperationType is failing on master
[ https://issues.apache.org/jira/browse/HIVE-13700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15277278#comment-15277278 ] Hari Sankar Sivarama Subramaniyan commented on HIVE-13700: -- Tested this locally, this doesnt break anything. I will commit it soon. > TestHiveOperationType is failing on master > -- > > Key: HIVE-13700 > URL: https://issues.apache.org/jira/browse/HIVE-13700 > Project: Hive > Issue Type: Bug > Components: Tests >Affects Versions: 2.1.0 >Reporter: Ashutosh Chauhan >Assignee: Hari Sankar Sivarama Subramaniyan > Attachments: HIVE-13700.1.patch > > > Presumably be broken by HIVE-13351 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13722) Add flag to detect constants to CBO pull up rules
[ https://issues.apache.org/jira/browse/HIVE-13722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15277248#comment-15277248 ] Ashutosh Chauhan commented on HIVE-13722: - +1 > Add flag to detect constants to CBO pull up rules > - > > Key: HIVE-13722 > URL: https://issues.apache.org/jira/browse/HIVE-13722 > Project: Hive > Issue Type: Sub-task > Components: CBO, Logical Optimizer, Physical Optimizer >Affects Versions: 2.1.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > Attachments: HIVE-13722.patch > > > Add flag to avoid firing CBO pull up constants rules indefinitely. This issue > can be reproduced using e.g. union27.q, union_remove_19.q, unionDistinct_1.q. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13565) thrift change
[ https://issues.apache.org/jira/browse/HIVE-13565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-13565: --- Status: Open (was: Patch Available) > thrift change > - > > Key: HIVE-13565 > URL: https://issues.apache.org/jira/browse/HIVE-13565 > Project: Hive > Issue Type: Sub-task >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Attachments: HIVE-13565.01.patch, HIVE-13565.02.patch, > HIVE-13565.03.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13565) thrift change
[ https://issues.apache.org/jira/browse/HIVE-13565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-13565: --- Attachment: HIVE-13565.03.patch > thrift change > - > > Key: HIVE-13565 > URL: https://issues.apache.org/jira/browse/HIVE-13565 > Project: Hive > Issue Type: Sub-task >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Attachments: HIVE-13565.01.patch, HIVE-13565.02.patch, > HIVE-13565.03.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13565) thrift change
[ https://issues.apache.org/jira/browse/HIVE-13565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-13565: --- Status: Patch Available (was: Open) > thrift change > - > > Key: HIVE-13565 > URL: https://issues.apache.org/jira/browse/HIVE-13565 > Project: Hive > Issue Type: Sub-task >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Attachments: HIVE-13565.01.patch, HIVE-13565.02.patch, > HIVE-13565.03.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13565) thrift change
[ https://issues.apache.org/jira/browse/HIVE-13565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-13565: --- Attachment: (was: HIVE-13565.03.patch) > thrift change > - > > Key: HIVE-13565 > URL: https://issues.apache.org/jira/browse/HIVE-13565 > Project: Hive > Issue Type: Sub-task >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Attachments: HIVE-13565.01.patch, HIVE-13565.02.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12634) Add command to kill an ACID transacton
[ https://issues.apache.org/jira/browse/HIVE-12634?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Zheng updated HIVE-12634: - Target Version/s: 1.3.0, 2.1.0 (was: 1.3.0) > Add command to kill an ACID transacton > -- > > Key: HIVE-12634 > URL: https://issues.apache.org/jira/browse/HIVE-12634 > Project: Hive > Issue Type: New Feature > Components: Transactions >Affects Versions: 1.0.0 >Reporter: Eugene Koifman >Assignee: Wei Zheng > Labels: TODOC2.1 > Attachments: HIVE-12634.1.patch, HIVE-12634.2.patch, > HIVE-12634.3.patch, HIVE-12634.4.patch, HIVE-12634.5.patch, > HIVE-12634.6.patch, HIVE-12634.7.patch > > > Should add a CLI command to abort a (runaway) transaction. > This should clean up all state related to this txn. > The initiator of this (if still alive) will get an error trying to > heartbeat/commit, i.e. will become aware that the txn is dead. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-10176) skip.header.line.count causes values to be skipped when performing insert values
[
https://issues.apache.org/jira/browse/HIVE-10176?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashutosh Chauhan updated HIVE-10176:
Status: Patch Available (was: Open)
> skip.header.line.count causes values to be skipped when performing insert
> values
>
>
> Key: HIVE-10176
> URL: https://issues.apache.org/jira/browse/HIVE-10176
> Project: Hive
> Issue Type: Bug
>Affects Versions: 1.2.1, 1.0.0
>Reporter: Wenbo Wang
>Assignee: Vladyslav Pavlenko
> Fix For: 2.0.0
>
> Attachments: HIVE-10176.1.patch, HIVE-10176.10.patch,
> HIVE-10176.11.patch, HIVE-10176.12.patch, HIVE-10176.13.patch,
> HIVE-10176.14.patch, HIVE-10176.15.patch, HIVE-10176.2.patch,
> HIVE-10176.3.patch, HIVE-10176.4.patch, HIVE-10176.5.patch,
> HIVE-10176.6.patch, HIVE-10176.7.patch, HIVE-10176.8.patch,
> HIVE-10176.9.patch, data
>
>
> When inserting values in to tables with TBLPROPERTIES
> ("skip.header.line.count"="1") the first value listed is also skipped.
> create table test (row int, name string) TBLPROPERTIES
> ("skip.header.line.count"="1");
> load data local inpath '/root/data' into table test;
> insert into table test values (1, 'a'), (2, 'b'), (3, 'c');
> (1, 'a') isn't inserted into the table.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-10176) skip.header.line.count causes values to be skipped when performing insert values
[
https://issues.apache.org/jira/browse/HIVE-10176?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashutosh Chauhan updated HIVE-10176:
Status: Open (was: Patch Available)
> skip.header.line.count causes values to be skipped when performing insert
> values
>
>
> Key: HIVE-10176
> URL: https://issues.apache.org/jira/browse/HIVE-10176
> Project: Hive
> Issue Type: Bug
>Affects Versions: 1.2.1, 1.0.0
>Reporter: Wenbo Wang
>Assignee: Vladyslav Pavlenko
> Fix For: 2.0.0
>
> Attachments: HIVE-10176.1.patch, HIVE-10176.10.patch,
> HIVE-10176.11.patch, HIVE-10176.12.patch, HIVE-10176.13.patch,
> HIVE-10176.14.patch, HIVE-10176.15.patch, HIVE-10176.2.patch,
> HIVE-10176.3.patch, HIVE-10176.4.patch, HIVE-10176.5.patch,
> HIVE-10176.6.patch, HIVE-10176.7.patch, HIVE-10176.8.patch,
> HIVE-10176.9.patch, data
>
>
> When inserting values in to tables with TBLPROPERTIES
> ("skip.header.line.count"="1") the first value listed is also skipped.
> create table test (row int, name string) TBLPROPERTIES
> ("skip.header.line.count"="1");
> load data local inpath '/root/data' into table test;
> insert into table test values (1, 'a'), (2, 'b'), (3, 'c');
> (1, 'a') isn't inserted into the table.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-10176) skip.header.line.count causes values to be skipped when performing insert values
[
https://issues.apache.org/jira/browse/HIVE-10176?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashutosh Chauhan updated HIVE-10176:
Attachment: HIVE-10176.15.patch
> skip.header.line.count causes values to be skipped when performing insert
> values
>
>
> Key: HIVE-10176
> URL: https://issues.apache.org/jira/browse/HIVE-10176
> Project: Hive
> Issue Type: Bug
>Affects Versions: 1.0.0, 1.2.1
>Reporter: Wenbo Wang
>Assignee: Vladyslav Pavlenko
> Fix For: 2.0.0
>
> Attachments: HIVE-10176.1.patch, HIVE-10176.10.patch,
> HIVE-10176.11.patch, HIVE-10176.12.patch, HIVE-10176.13.patch,
> HIVE-10176.14.patch, HIVE-10176.15.patch, HIVE-10176.2.patch,
> HIVE-10176.3.patch, HIVE-10176.4.patch, HIVE-10176.5.patch,
> HIVE-10176.6.patch, HIVE-10176.7.patch, HIVE-10176.8.patch,
> HIVE-10176.9.patch, data
>
>
> When inserting values in to tables with TBLPROPERTIES
> ("skip.header.line.count"="1") the first value listed is also skipped.
> create table test (row int, name string) TBLPROPERTIES
> ("skip.header.line.count"="1");
> load data local inpath '/root/data' into table test;
> insert into table test values (1, 'a'), (2, 'b'), (3, 'c');
> (1, 'a') isn't inserted into the table.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-11417) Create shims for the row by row read path that is backed by VectorizedRowBatch
[ https://issues.apache.org/jira/browse/HIVE-11417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15277208#comment-15277208 ] Owen O'Malley commented on HIVE-11417: -- Published as https://reviews.apache.org/r/47137/ > Create shims for the row by row read path that is backed by VectorizedRowBatch > -- > > Key: HIVE-11417 > URL: https://issues.apache.org/jira/browse/HIVE-11417 > Project: Hive > Issue Type: Sub-task >Reporter: Owen O'Malley >Assignee: Owen O'Malley > Fix For: 2.1.0 > > Attachments: HIVE-11417.patch, HIVE-11417.patch, HIVE-11417.patch, > HIVE-11417.patch, HIVE-11417.patch, HIVE-11417.patch > > > I'd like to make the default path for reading and writing ORC files to be > vectorized. To ensure that Hive can still read row by row, we'll need shims > to support the old API. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12634) Add command to kill an ACID transacton
[ https://issues.apache.org/jira/browse/HIVE-12634?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Zheng updated HIVE-12634: - Attachment: HIVE-12634.7.patch patch 7, rebase > Add command to kill an ACID transacton > -- > > Key: HIVE-12634 > URL: https://issues.apache.org/jira/browse/HIVE-12634 > Project: Hive > Issue Type: New Feature > Components: Transactions >Affects Versions: 1.0.0 >Reporter: Eugene Koifman >Assignee: Wei Zheng > Labels: TODOC2.1 > Attachments: HIVE-12634.1.patch, HIVE-12634.2.patch, > HIVE-12634.3.patch, HIVE-12634.4.patch, HIVE-12634.5.patch, > HIVE-12634.6.patch, HIVE-12634.7.patch > > > Should add a CLI command to abort a (runaway) transaction. > This should clean up all state related to this txn. > The initiator of this (if still alive) will get an error trying to > heartbeat/commit, i.e. will become aware that the txn is dead. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13670) Improve Beeline connect/reconnect semantics
[ https://issues.apache.org/jira/browse/HIVE-13670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15277199#comment-15277199 ] Sushanth Sowmyan commented on HIVE-13670: - That's definitely workable, I'll make a trivial update. > Improve Beeline connect/reconnect semantics > --- > > Key: HIVE-13670 > URL: https://issues.apache.org/jira/browse/HIVE-13670 > Project: Hive > Issue Type: Improvement >Affects Versions: 2.1.0 >Reporter: Sushanth Sowmyan >Assignee: Sushanth Sowmyan > Attachments: HIVE-13670.2.patch, HIVE-13670.3.patch, HIVE-13670.patch > > > For most users of beeline, chances are that they will be using it with a > single HS2 instance most of the time. In this scenario, having them type out > a jdbc uri for HS2 every single time to !connect can get tiresome. Thus, we > should improve semantics so that if a user does a successful !connect, then > we must store the last-connected-to-url, so that if they do a !close, and > then a !reconnect, then !reconnect should attempt to connect to the last > successfully used url. > Also, if they then do a !save, then that last-successfully-used url must be > saved, so that in subsequent sessions, they can simply do !reconnect rather > than specifying a url for !connect. > In addition, it would be useful to introduce a new way of doing !connect that > does involve typing out a jdbc url every time (since this is highly likely to > be error-prone) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11417) Create shims for the row by row read path that is backed by VectorizedRowBatch
[ https://issues.apache.org/jira/browse/HIVE-11417?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Owen O'Malley updated HIVE-11417: - Attachment: HIVE-11417.patch Fix some of the schema evolution interactions. Also include --binary when generating the patch so that we get the binary diffs for orc-file-11-format.orc. > Create shims for the row by row read path that is backed by VectorizedRowBatch > -- > > Key: HIVE-11417 > URL: https://issues.apache.org/jira/browse/HIVE-11417 > Project: Hive > Issue Type: Sub-task >Reporter: Owen O'Malley >Assignee: Owen O'Malley > Fix For: 2.1.0 > > Attachments: HIVE-11417.patch, HIVE-11417.patch, HIVE-11417.patch, > HIVE-11417.patch, HIVE-11417.patch, HIVE-11417.patch > > > I'd like to make the default path for reading and writing ORC files to be > vectorized. To ensure that Hive can still read row by row, we'll need shims > to support the old API. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13565) thrift change
[ https://issues.apache.org/jira/browse/HIVE-13565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-13565: --- Status: Patch Available (was: Open) > thrift change > - > > Key: HIVE-13565 > URL: https://issues.apache.org/jira/browse/HIVE-13565 > Project: Hive > Issue Type: Sub-task >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Attachments: HIVE-13565.01.patch, HIVE-13565.02.patch, > HIVE-13565.03.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13565) thrift change
[ https://issues.apache.org/jira/browse/HIVE-13565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-13565: --- Status: Open (was: Patch Available) > thrift change > - > > Key: HIVE-13565 > URL: https://issues.apache.org/jira/browse/HIVE-13565 > Project: Hive > Issue Type: Sub-task >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Attachments: HIVE-13565.01.patch, HIVE-13565.02.patch, > HIVE-13565.03.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13565) thrift change
[ https://issues.apache.org/jira/browse/HIVE-13565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-13565: --- Status: Open (was: Patch Available) address sessionHiveMetaStore. > thrift change > - > > Key: HIVE-13565 > URL: https://issues.apache.org/jira/browse/HIVE-13565 > Project: Hive > Issue Type: Sub-task >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Attachments: HIVE-13565.01.patch, HIVE-13565.02.patch, > HIVE-13565.03.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13249) Hard upper bound on number of open transactions
[
https://issues.apache.org/jira/browse/HIVE-13249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15277151#comment-15277151
]
Eugene Koifman commented on HIVE-13249:
---
I think maxOpenTxns numOpenTxns tooManyOpenTxns should all be volatile since
they are accessed by different threads
I would make LOG.warn("Transaction database not properly configured ... be
"error"
You don't need to add "OpenTxnsCounter" - nothing bad can happen if
AcidOpenTxnsCounterService runs in parallel - it's just a count(*) query
Because TxnHandler exists in standalone metastore (e.g. HS2), you probably need
to start AcidOpenTxnsCounterService from TxnHandler.setConf() but such that
there is only 1 instance of AcidOpenTxnsCounterService per JVM
> Hard upper bound on number of open transactions
> ---
>
> Key: HIVE-13249
> URL: https://issues.apache.org/jira/browse/HIVE-13249
> Project: Hive
> Issue Type: Improvement
> Components: Transactions
>Affects Versions: 2.0.0
>Reporter: Wei Zheng
>Assignee: Wei Zheng
> Attachments: HIVE-13249.1.patch, HIVE-13249.2.patch,
> HIVE-13249.3.patch, HIVE-13249.4.patch, HIVE-13249.5.patch,
> HIVE-13249.6.patch, HIVE-13249.7.patch
>
>
> We need to have a safeguard by adding an upper bound for open transactions to
> avoid huge number of open-transaction requests, usually due to improper
> configuration of clients such as Storm.
> Once that limit is reached, clients will start failing.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13565) thrift change
[ https://issues.apache.org/jira/browse/HIVE-13565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-13565: --- Status: Patch Available (was: Open) > thrift change > - > > Key: HIVE-13565 > URL: https://issues.apache.org/jira/browse/HIVE-13565 > Project: Hive > Issue Type: Sub-task >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Attachments: HIVE-13565.01.patch, HIVE-13565.02.patch, > HIVE-13565.03.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13722) Add flag to detect constants to CBO pull up rules
[ https://issues.apache.org/jira/browse/HIVE-13722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15277137#comment-15277137 ] Jesus Camacho Rodriguez commented on HIVE-13722: [~ashutoshc], could you take a look? The problem was that we were firing the rule and producing the same expression again and again, thus entering in an infinite loop. New exit condition checks whether a constant expression has been added on top of Union (or Sort) or not. > Add flag to detect constants to CBO pull up rules > - > > Key: HIVE-13722 > URL: https://issues.apache.org/jira/browse/HIVE-13722 > Project: Hive > Issue Type: Sub-task > Components: CBO, Logical Optimizer, Physical Optimizer >Affects Versions: 2.1.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > Attachments: HIVE-13722.patch > > > Add flag to avoid firing CBO pull up constants rules indefinitely. This issue > can be reproduced using e.g. union27.q, union_remove_19.q, unionDistinct_1.q. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11793) SHOW LOCKS with DbTxnManager ignores filter options
[ https://issues.apache.org/jira/browse/HIVE-11793?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Zheng updated HIVE-11793: - Attachment: HIVE-11793.7.patch > SHOW LOCKS with DbTxnManager ignores filter options > --- > > Key: HIVE-11793 > URL: https://issues.apache.org/jira/browse/HIVE-11793 > Project: Hive > Issue Type: Improvement > Components: Transactions >Affects Versions: 1.0.0 >Reporter: Eugene Koifman >Assignee: Wei Zheng >Priority: Minor > Attachments: HIVE-11793.1.patch, HIVE-11793.2.patch, > HIVE-11793.3.patch, HIVE-11793.4.patch, HIVE-11793.5.patch, > HIVE-11793.6.patch, HIVE-11793.7.patch > > > https://cwiki.apache.org/confluence/display/Hive/Locking and > https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL#LanguageManualDDL-ShowLocks > list various options that can be used with SHOW LOCKS, e.g. > When ACID is enabled, all these options are ignored and a full list is > returned. > (also only ext lock id is shown, int lock id is not). > see DDLTask.showLocks() and TxnHandler.showLocks() > requires extending ShowLocksRequest which is a Thrift object -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11793) SHOW LOCKS with DbTxnManager ignores filter options
[ https://issues.apache.org/jira/browse/HIVE-11793?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Zheng updated HIVE-11793: - Attachment: (was: HIVE-11793.7.patch) > SHOW LOCKS with DbTxnManager ignores filter options > --- > > Key: HIVE-11793 > URL: https://issues.apache.org/jira/browse/HIVE-11793 > Project: Hive > Issue Type: Improvement > Components: Transactions >Affects Versions: 1.0.0 >Reporter: Eugene Koifman >Assignee: Wei Zheng >Priority: Minor > Attachments: HIVE-11793.1.patch, HIVE-11793.2.patch, > HIVE-11793.3.patch, HIVE-11793.4.patch, HIVE-11793.5.patch, > HIVE-11793.6.patch, HIVE-11793.7.patch > > > https://cwiki.apache.org/confluence/display/Hive/Locking and > https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL#LanguageManualDDL-ShowLocks > list various options that can be used with SHOW LOCKS, e.g. > When ACID is enabled, all these options are ignored and a full list is > returned. > (also only ext lock id is shown, int lock id is not). > see DDLTask.showLocks() and TxnHandler.showLocks() > requires extending ShowLocksRequest which is a Thrift object -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13722) Add flag to detect constants to CBO pull up rules
[ https://issues.apache.org/jira/browse/HIVE-13722?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-13722: --- Attachment: HIVE-13722.patch > Add flag to detect constants to CBO pull up rules > - > > Key: HIVE-13722 > URL: https://issues.apache.org/jira/browse/HIVE-13722 > Project: Hive > Issue Type: Sub-task > Components: CBO, Logical Optimizer, Physical Optimizer >Affects Versions: 2.1.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > Attachments: HIVE-13722.patch > > > Add flag to avoid firing CBO pull up constants rules indefinitely. This issue > can be reproduced using e.g. union27.q, union_remove_19.q, unionDistinct_1.q. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Work started] (HIVE-13722) Add flag to detect constants to CBO pull up rules
[ https://issues.apache.org/jira/browse/HIVE-13722?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-13722 started by Jesus Camacho Rodriguez. -- > Add flag to detect constants to CBO pull up rules > - > > Key: HIVE-13722 > URL: https://issues.apache.org/jira/browse/HIVE-13722 > Project: Hive > Issue Type: Sub-task > Components: CBO, Logical Optimizer, Physical Optimizer >Affects Versions: 2.1.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > > Add flag to avoid firing CBO pull up constants rules indefinitely. This issue > can be reproduced using e.g. union27.q, union_remove_19.q, unionDistinct_1.q. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13722) Add flag to detect constants to CBO pull up rules
[ https://issues.apache.org/jira/browse/HIVE-13722?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-13722: --- Status: Patch Available (was: In Progress) > Add flag to detect constants to CBO pull up rules > - > > Key: HIVE-13722 > URL: https://issues.apache.org/jira/browse/HIVE-13722 > Project: Hive > Issue Type: Sub-task > Components: CBO, Logical Optimizer, Physical Optimizer >Affects Versions: 2.1.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > > Add flag to avoid firing CBO pull up constants rules indefinitely. This issue > can be reproduced using e.g. union27.q, union_remove_19.q, unionDistinct_1.q. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13670) Improve Beeline connect/reconnect semantics
[
https://issues.apache.org/jira/browse/HIVE-13670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15277112#comment-15277112
]
Thejas M Nair commented on HIVE-13670:
--
In Beeline.java
how about initing the url 'if (cl.hasOption("reconnect"))' instead of
duplicating the code that is under 'if (url != null) {' ?
> Improve Beeline connect/reconnect semantics
> ---
>
> Key: HIVE-13670
> URL: https://issues.apache.org/jira/browse/HIVE-13670
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 2.1.0
>Reporter: Sushanth Sowmyan
>Assignee: Sushanth Sowmyan
> Attachments: HIVE-13670.2.patch, HIVE-13670.3.patch, HIVE-13670.patch
>
>
> For most users of beeline, chances are that they will be using it with a
> single HS2 instance most of the time. In this scenario, having them type out
> a jdbc uri for HS2 every single time to !connect can get tiresome. Thus, we
> should improve semantics so that if a user does a successful !connect, then
> we must store the last-connected-to-url, so that if they do a !close, and
> then a !reconnect, then !reconnect should attempt to connect to the last
> successfully used url.
> Also, if they then do a !save, then that last-successfully-used url must be
> saved, so that in subsequent sessions, they can simply do !reconnect rather
> than specifying a url for !connect.
> In addition, it would be useful to introduce a new way of doing !connect that
> does involve typing out a jdbc url every time (since this is highly likely to
> be error-prone)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (HIVE-13617) LLAP: support non-vectorized execution in IO
[ https://issues.apache.org/jira/browse/HIVE-13617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin reassigned HIVE-13617: --- Assignee: Sergey Shelukhin > LLAP: support non-vectorized execution in IO > > > Key: HIVE-13617 > URL: https://issues.apache.org/jira/browse/HIVE-13617 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > > Two approaches - a separate decoding path, into rows instead of VRBs; or > decoding VRBs into rows on a higher level (the original LlapInputFormat). I > think the latter might be better - it's not a hugely important path, and perf > in non-vectorized case is not the best anyway, so it's better to make do with > much less new code and architectural disruption. > Some ORC patches in progress introduce an easy to reuse (or so I hope, > anyway) VRB-to-row conversion, so we should just use that. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12634) Add command to kill an ACID transacton
[ https://issues.apache.org/jira/browse/HIVE-12634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15277055#comment-15277055 ] Eugene Koifman commented on HIVE-12634: --- +1 > Add command to kill an ACID transacton > -- > > Key: HIVE-12634 > URL: https://issues.apache.org/jira/browse/HIVE-12634 > Project: Hive > Issue Type: New Feature > Components: Transactions >Affects Versions: 1.0.0 >Reporter: Eugene Koifman >Assignee: Wei Zheng > Labels: TODOC2.1 > Attachments: HIVE-12634.1.patch, HIVE-12634.2.patch, > HIVE-12634.3.patch, HIVE-12634.4.patch, HIVE-12634.5.patch, HIVE-12634.6.patch > > > Should add a CLI command to abort a (runaway) transaction. > This should clean up all state related to this txn. > The initiator of this (if still alive) will get an error trying to > heartbeat/commit, i.e. will become aware that the txn is dead. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13642) Update GUESS_FIELDS option in .reviewboardrc to support current version of RBTools.
[
https://issues.apache.org/jira/browse/HIVE-13642?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15277042#comment-15277042
]
Jesus Camacho Rodriguez commented on HIVE-13642:
+1
> Update GUESS_FIELDS option in .reviewboardrc to support current version of
> RBTools.
> ---
>
> Key: HIVE-13642
> URL: https://issues.apache.org/jira/browse/HIVE-13642
> Project: Hive
> Issue Type: Bug
>Reporter: Takuma Wakamori
>Assignee: Takuma Wakamori
>Priority: Trivial
> Attachments: HIVE-13642.1.patch
>
>
> According to the instruction of using [Review
> Board|https://cwiki.apache.org/confluence/display/Hive/Review+Board], we can
> create a review request by running the following command:
> {code}
> $ rbt post
> {code}
> However, if we use current version of RBTools (0.7.6), above command doesn't
> work because the values accepted to the option "--guess-fields" was updated
> since 0.6.
> {code}
> $ rbt --version
> RBTools 0.7.6
> $ rbt post
> ERROR: Invalid value "true" for argument "--guess-fields"
> {code}
> It is better to update the value of GUESS_FIELDS option in .reviewboardrc to
> avoid this error.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13688) Variable substitutions not working for shell commands in beeline
[
https://issues.apache.org/jira/browse/HIVE-13688?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15277017#comment-15277017
]
Naveen Gangam commented on HIVE-13688:
--
Hi [~xuefuz],
Do you know why these are being treated as session-specific variables as
opposed to beeline-instance specific variables? What if we treat them similar
to a Unix shell where anything that is set using "export or set" is specific to
that shell and is not visible from another shell (that not a child of this
original shell).
In other words, should these variables, for substitution, be stored locally in
the beeline JVM and be applicable for all connections, current and future,
initiated from this beeline instance? Thanks
> Variable substitutions not working for shell commands in beeline
>
>
> Key: HIVE-13688
> URL: https://issues.apache.org/jira/browse/HIVE-13688
> Project: Hive
> Issue Type: Bug
> Components: Beeline
>Affects Versions: 1.1.0
>Reporter: Naveen Gangam
>Assignee: Naveen Gangam
>
> Appears that the variable substitutions work fine for DML and DDL commands.
> > set hivevar:v1=databases;
> No rows affected (0.004 seconds)
> > show ${v1};
> ++--+
> | database_name |
> ++--+
> | default|
> ++--+
> 1 row selected (0.243 seconds)
> > set hivevar:v1=tmp;
> No rows affected (0.061 seconds)
> > create table ${v1} (col1 string);
> No rows affected (0.065 seconds)
> > describe ${v1};
> INFO : Compiling
> command(queryId=hive_20160504105151_cb6ab7f1-26ac-4733-80ed-13a5134ba2b5):
> describe tmp
> INFO : Executing
> command(queryId=hive_20160504105151_cb6ab7f1-26ac-4733-80ed-13a5134ba2b5):
> describe tmp
> +---++--+--+
> | col_name | data_type | comment |
> +---++--+--+
> | col1 | string | |
> +---++--+--+
> 1 row selected (0.121 seconds)
> But it does not appear to work for shell commands (introduced in HIVE-6791 )
> > !sh sudo -u hdfs hdfs dfs -mkdir /user/root/${v1}
> mkdir: `/user/root/${v1}': No such file or directory <== problem
> Command failed with exit code = 1
> > !sh echo ${v1};
> ${v1};
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13338) Differences in vectorized_casts.q output for vectorized and non-vectorized runs
[ https://issues.apache.org/jira/browse/HIVE-13338?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-13338: Status: Patch Available (was: Open) > Differences in vectorized_casts.q output for vectorized and non-vectorized > runs > --- > > Key: HIVE-13338 > URL: https://issues.apache.org/jira/browse/HIVE-13338 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-13338.01.patch > > > Turn off vectorization and you get different results. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13338) Differences in vectorized_casts.q output for vectorized and non-vectorized runs
[ https://issues.apache.org/jira/browse/HIVE-13338?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-13338: Attachment: HIVE-13338.01.patch > Differences in vectorized_casts.q output for vectorized and non-vectorized > runs > --- > > Key: HIVE-13338 > URL: https://issues.apache.org/jira/browse/HIVE-13338 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-13338.01.patch > > > Turn off vectorization and you get different results. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13705) Insert into table removes existing data
[ https://issues.apache.org/jira/browse/HIVE-13705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15276980#comment-15276980 ] Ashutosh Chauhan commented on HIVE-13705: - Also, you may want to create HDFS ticket for this bug in LocalFileSystem. > Insert into table removes existing data > --- > > Key: HIVE-13705 > URL: https://issues.apache.org/jira/browse/HIVE-13705 > Project: Hive > Issue Type: Bug > Components: Query Processor >Affects Versions: 2.1.0 >Reporter: Aihua Xu >Assignee: Aihua Xu > Attachments: HIVE-13705.1.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13705) Insert into table removes existing data
[
https://issues.apache.org/jira/browse/HIVE-13705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15276977#comment-15276977
]
Ashutosh Chauhan commented on HIVE-13705:
-
yeah.. we already have similar arrangement for {{NullScanFileSystem}} in
ql/src/main/resources/META-INF/services/org.apache.hadoop.fs.FileSystem
> Insert into table removes existing data
> ---
>
> Key: HIVE-13705
> URL: https://issues.apache.org/jira/browse/HIVE-13705
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Affects Versions: 2.1.0
>Reporter: Aihua Xu
>Assignee: Aihua Xu
> Attachments: HIVE-13705.1.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13425) Fix partition addition in MSCK REPAIR TABLE command
[
https://issues.apache.org/jira/browse/HIVE-13425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15276950#comment-15276950
]
Hive QA commented on HIVE-13425:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12802825/HIVE-13425.3.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 14 failed/errored test(s), 9941 tests
executed
*Failed tests:*
{noformat}
TestCliDriver-gen_udf_example_add10.q-ppd_join4.q-union27.q-and-12-more - did
not produce a TEST-*.xml file
TestCliDriver-partition_timestamp.q-ppd_random.q-vector_outer_join5.q-and-12-more
- did not produce a TEST-*.xml file
TestCliDriver-ptf_general_queries.q-unionDistinct_1.q-groupby1_noskew.q-and-12-more
- did not produce a TEST-*.xml file
TestHWISessionManager - did not produce a TEST-*.xml file
TestMiniLlapCliDriver - did not produce a TEST-*.xml file
TestMiniTezCliDriver-enforce_order.q-vector_partition_diff_num_cols.q-unionDistinct_1.q-and-12-more
- did not produce a TEST-*.xml file
TestMiniTezCliDriver-join1.q-schema_evol_orc_nonvec_mapwork_part.q-mapjoin_decimal.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-bucketsortoptimize_insert_7.q-smb_mapjoin_15.q-mapreduce1.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-skewjoinopt3.q-union27.q-multigroupby_singlemr.q-and-12-more
- did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3
org.apache.hadoop.hive.llap.tez.TestConverters.testFragmentSpecToTaskSpec
org.apache.hadoop.hive.llap.tezplugins.TestLlapTaskCommunicator.testFinishableStateUpdateFailure
org.apache.hadoop.hive.ql.security.authorization.plugin.TestHiveOperationType.checkHiveOperationTypeMatch
org.apache.hive.service.cli.session.TestHiveSessionImpl.testLeakOperationHandle
{noformat}
Test results:
http://ec2-54-177-240-2.us-west-1.compute.amazonaws.com/job/PreCommit-HIVE-MASTER-Build/223/testReport
Console output:
http://ec2-54-177-240-2.us-west-1.compute.amazonaws.com/job/PreCommit-HIVE-MASTER-Build/223/console
Test logs:
http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-223/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 14 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12802825 - PreCommit-HIVE-MASTER-Build
> Fix partition addition in MSCK REPAIR TABLE command
> ---
>
> Key: HIVE-13425
> URL: https://issues.apache.org/jira/browse/HIVE-13425
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 2.0.0
>Reporter: Shinichi Yamashita
>Assignee: Shinichi Yamashita
>Priority: Minor
> Attachments: HIVE-13425.1.patch, HIVE-13425.2.patch,
> HIVE-13425.3.patch
>
>
> I create a table in following HiveQL.
> {code}
> hive> create table example (name string) partitioned by (id int);
> {code}
> And, I make some directories about this table in HDFS.
> {code}
> [root@hadoop ~]# hdfs dfs -ls -R /user/hive/warehouse/example
> drwxr-xr-x - root hadoop 0 2016-04-05 22:21
> /user/hive/warehouse/example/id=1
> drwxr-xr-x - root hadoop 0 2016-04-05 22:22
> /user/hive/warehouse/example/id=1/id=2
> -rw-r--r-- 1 root hadoop 8 2016-04-05 22:22
> /user/hive/warehouse/example/id=1/id=2/example.txt
> {code}
> Next I executed MSCK REPAIR TABLE command and added a partition. And this
> result became as follows.
> {code}
> [root@hadoop ~]# hive -e 'msck repair table example'
> OK
> Partitions not in metastore: example:id=1/id=2
> Repair: Added partition to metastore example:id=1/id=2
> Time taken: 1.243 seconds, Fetched: 2 row(s)
> [root@hadoop ~]# hive -e 'show partitions example'
> OK
> id=2
> {code}
> "id=1" should be a partition, but "id=2" was added. I will fix this problem.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13705) Insert into table removes existing data
[
https://issues.apache.org/jira/browse/HIVE-13705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15276949#comment-15276949
]
Aihua Xu commented on HIVE-13705:
-
[~ashutoshc] Want to double check with you your thoughts. I checked the logic
to plug in ProxyFileSystem. Seems we need to register ProxyFileSystem to let
ServiceLoader to load. Is that what you think?
{noformat}
private static void loadFileSystems() {
synchronized (FileSystem.class) {
if (!FILE_SYSTEMS_LOADED) {
ServiceLoader serviceLoader =
ServiceLoader.load(FileSystem.class);
for (FileSystem fs : serviceLoader) {
SERVICE_FILE_SYSTEMS.put(fs.getScheme(), fs.getClass());
}
FILE_SYSTEMS_LOADED = true;
}
}
}
{noformat}
> Insert into table removes existing data
> ---
>
> Key: HIVE-13705
> URL: https://issues.apache.org/jira/browse/HIVE-13705
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Affects Versions: 2.1.0
>Reporter: Aihua Xu
>Assignee: Aihua Xu
> Attachments: HIVE-13705.1.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13561) HiveServer2 is leaking ClassLoaders when add jar / temporary functions are used
[ https://issues.apache.org/jira/browse/HIVE-13561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15276943#comment-15276943 ] Trystan Leftwich commented on HIVE-13561: - [~vgumashta] Thanks for the notes. Sorry for the delay but i've attached an updated test based on your notes. > HiveServer2 is leaking ClassLoaders when add jar / temporary functions are > used > --- > > Key: HIVE-13561 > URL: https://issues.apache.org/jira/browse/HIVE-13561 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Affects Versions: 1.2.0, 1.2.1, 2.0.0 >Reporter: Trystan Leftwich >Assignee: Trystan Leftwich > Attachments: HIVE-13561-branch-1.2.3.patch, HIVE-13561.2.patch, > HIVE-13561.3.patch, HIVE-13561.4.patch > > > I can repo this on branch-1.2 and branch-2.0. > It looks to be the same issues as: HIVE-11408 > The patch from HIVE-11408 looks to fix the issue as well. > I've updated the patch from HIVE-11408 to be aligned with branch-1.2 and > master -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13561) HiveServer2 is leaking ClassLoaders when add jar / temporary functions are used
[ https://issues.apache.org/jira/browse/HIVE-13561?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Trystan Leftwich updated HIVE-13561: Attachment: HIVE-13561.4.patch > HiveServer2 is leaking ClassLoaders when add jar / temporary functions are > used > --- > > Key: HIVE-13561 > URL: https://issues.apache.org/jira/browse/HIVE-13561 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Affects Versions: 1.2.0, 1.2.1, 2.0.0 >Reporter: Trystan Leftwich >Assignee: Trystan Leftwich > Attachments: HIVE-13561-branch-1.2.3.patch, HIVE-13561.2.patch, > HIVE-13561.3.patch, HIVE-13561.4.patch > > > I can repo this on branch-1.2 and branch-2.0. > It looks to be the same issues as: HIVE-11408 > The patch from HIVE-11408 looks to fix the issue as well. > I've updated the patch from HIVE-11408 to be aligned with branch-1.2 and > master -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13343) Need to disable hybrid grace hash join in llap mode except for dynamically partitioned hash join
[ https://issues.apache.org/jira/browse/HIVE-13343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15276929#comment-15276929 ] Sergey Shelukhin commented on HIVE-13343: - They are because of Java version. > Need to disable hybrid grace hash join in llap mode except for dynamically > partitioned hash join > > > Key: HIVE-13343 > URL: https://issues.apache.org/jira/browse/HIVE-13343 > Project: Hive > Issue Type: Bug > Components: llap >Affects Versions: 2.1.0 >Reporter: Vikram Dixit K >Assignee: Vikram Dixit K > Attachments: HIVE-13343.1.patch, HIVE-13343.2.patch, > HIVE-13343.3.patch, HIVE-13343.4.patch, HIVE-13343.5.patch > > > Due to performance reasons, we should disable use of hybrid grace hash join > in llap when dynamic partition hash join is not used. With dynamic partition > hash join, we need hybrid grace hash join due to the possibility of skews. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13343) Need to disable hybrid grace hash join in llap mode except for dynamically partitioned hash join
[
https://issues.apache.org/jira/browse/HIVE-13343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15276909#comment-15276909
]
Vikram Dixit K commented on HIVE-13343:
---
It looks like many of the diffs are about slightly varying explain plans. I
don't believe they are because of my changes.
{code}
-COLUMN_STATS_ACCURATE
{"COLUMN_STATS":{"key":"true","value":"true"},"BASIC_STATS":"true"}
+COLUMN_STATS_ACCURATE
{"BASIC_STATS":"true","COLUMN_STATS":{"key":"true","value":"true"}}
{code}
> Need to disable hybrid grace hash join in llap mode except for dynamically
> partitioned hash join
>
>
> Key: HIVE-13343
> URL: https://issues.apache.org/jira/browse/HIVE-13343
> Project: Hive
> Issue Type: Bug
> Components: llap
>Affects Versions: 2.1.0
>Reporter: Vikram Dixit K
>Assignee: Vikram Dixit K
> Attachments: HIVE-13343.1.patch, HIVE-13343.2.patch,
> HIVE-13343.3.patch, HIVE-13343.4.patch, HIVE-13343.5.patch
>
>
> Due to performance reasons, we should disable use of hybrid grace hash join
> in llap when dynamic partition hash join is not used. With dynamic partition
> hash join, we need hybrid grace hash join due to the possibility of skews.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13597) revert HIVE-12892
[ https://issues.apache.org/jira/browse/HIVE-13597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-13597: Resolution: Fixed Fix Version/s: 2.1.0 Status: Resolved (was: Patch Available) Committed to master. > revert HIVE-12892 > - > > Key: HIVE-13597 > URL: https://issues.apache.org/jira/browse/HIVE-13597 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Fix For: 2.1.0 > > Attachments: HIVE-13597.01.patch, HIVE-13597.01.patch, > HIVE-13597.03.patch, HIVE-13597.nogen.patch, HIVE-13597.patch > > > Updating functions in LLAP via a metastore call is not going to be committed > in the current form due to other design considerations. > For now, we should revert the changes made in HIVE-12892; there's a DB change > involved there, and we don't want that to be included in a released version > before we decide on the final design of the update feature. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13693) Multi-insert query drops Filter before file output when there is a.val <> b.val
[
https://issues.apache.org/jira/browse/HIVE-13693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesus Camacho Rodriguez updated HIVE-13693:
---
Attachment: HIVE-13693.01.patch
> Multi-insert query drops Filter before file output when there is a.val <>
> b.val
> ---
>
> Key: HIVE-13693
> URL: https://issues.apache.org/jira/browse/HIVE-13693
> Project: Hive
> Issue Type: Bug
> Components: Logical Optimizer
>Affects Versions: 1.3.0, 2.0.0, 2.1.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
> Attachments: HIVE-13693.01.patch, HIVE-13693.patch
>
>
> To reproduce:
> {noformat}
> CREATE TABLE T_A ( id STRING, val STRING );
> CREATE TABLE T_B ( id STRING, val STRING );
> CREATE TABLE join_result_1 ( ida STRING, vala STRING, idb STRING, valb STRING
> );
> CREATE TABLE join_result_3 ( ida STRING, vala STRING, idb STRING, valb STRING
> );
> INSERT INTO TABLE T_A
> VALUES ('Id_1', 'val_101'), ('Id_2', 'val_102'), ('Id_3', 'val_103');
> INSERT INTO TABLE T_B
> VALUES ('Id_1', 'val_103'), ('Id_2', 'val_104');
> explain
> FROM T_A a LEFT JOIN T_B b ON a.id = b.id
> INSERT OVERWRITE TABLE join_result_1
> SELECT a.*, b.*
> WHERE b.id = 'Id_1' AND b.val = 'val_103'
> INSERT OVERWRITE TABLE join_result_3
> SELECT a.*, b.*
> WHERE b.val = 'val_104' AND b.id = 'Id_2' AND a.val <> b.val;
> {noformat}
> The (wrong) plan is the following:
> {noformat}
> STAGE DEPENDENCIES:
> Stage-2 is a root stage
> Stage-3 depends on stages: Stage-2
> Stage-0 depends on stages: Stage-3
> Stage-4 depends on stages: Stage-0
> Stage-1 depends on stages: Stage-3
> Stage-5 depends on stages: Stage-1
> STAGE PLANS:
> Stage: Stage-2
> Tez
> DagId: haha_20160504140944_174465c9-5d1a-42f9-9665-fae02eeb2767:2
> Edges:
> Reducer 2 <- Map 1 (SIMPLE_EDGE), Map 3 (SIMPLE_EDGE)
> DagName:
> Vertices:
> Map 1
> Map Operator Tree:
> TableScan
> alias: a
> Statistics: Num rows: 3 Data size: 36 Basic stats: COMPLETE
> Column stats: NONE
> Reduce Output Operator
> key expressions: id (type: string)
> sort order: +
> Map-reduce partition columns: id (type: string)
> Statistics: Num rows: 3 Data size: 36 Basic stats:
> COMPLETE Column stats: NONE
> value expressions: val (type: string)
> Map 3
> Map Operator Tree:
> TableScan
> alias: b
> Statistics: Num rows: 2 Data size: 24 Basic stats: COMPLETE
> Column stats: NONE
> Reduce Output Operator
> key expressions: id (type: string)
> sort order: +
> Map-reduce partition columns: id (type: string)
> Statistics: Num rows: 2 Data size: 24 Basic stats:
> COMPLETE Column stats: NONE
> value expressions: val (type: string)
> Reducer 2
> Reduce Operator Tree:
> Merge Join Operator
> condition map:
> Left Outer Join0 to 1
> keys:
> 0 id (type: string)
> 1 id (type: string)
> outputColumnNames: _col0, _col1, _col6
> Statistics: Num rows: 3 Data size: 39 Basic stats: COMPLETE
> Column stats: NONE
> Select Operator
> expressions: _col0 (type: string), _col1 (type: string),
> 'Id_1' (type: string), 'val_103' (type: string)
> outputColumnNames: _col0, _col1, _col2, _col3
> Statistics: Num rows: 3 Data size: 39 Basic stats: COMPLETE
> Column stats: NONE
> File Output Operator
> compressed: false
> Statistics: Num rows: 3 Data size: 39 Basic stats:
> COMPLETE Column stats: NONE
> table:
> input format: org.apache.hadoop.mapred.TextInputFormat
> output format:
> org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
> serde:
> org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
> name: bugtest2.join_result_1
> Filter Operator
> predicate: (_col1 <> _col6) (type: boolean)
> Statistics: Num rows: 3 Data size: 39 Basic stats: COMPLETE
> Column stats: NONE
> Select Operator
> expressions: _col0 (type: string), _col1 (type: string),
> 'Id_2' (type: string), 'val_104' (type: string)
> outputColumnNames: _col0, _col1,
[jira] [Updated] (HIVE-13693) Multi-insert query drops Filter before file output when there is a.val <> b.val
[
https://issues.apache.org/jira/browse/HIVE-13693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesus Camacho Rodriguez updated HIVE-13693:
---
Status: Open (was: Patch Available)
> Multi-insert query drops Filter before file output when there is a.val <>
> b.val
> ---
>
> Key: HIVE-13693
> URL: https://issues.apache.org/jira/browse/HIVE-13693
> Project: Hive
> Issue Type: Bug
> Components: Logical Optimizer
>Affects Versions: 2.0.0, 1.3.0, 2.1.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
> Attachments: HIVE-13693.patch
>
>
> To reproduce:
> {noformat}
> CREATE TABLE T_A ( id STRING, val STRING );
> CREATE TABLE T_B ( id STRING, val STRING );
> CREATE TABLE join_result_1 ( ida STRING, vala STRING, idb STRING, valb STRING
> );
> CREATE TABLE join_result_3 ( ida STRING, vala STRING, idb STRING, valb STRING
> );
> INSERT INTO TABLE T_A
> VALUES ('Id_1', 'val_101'), ('Id_2', 'val_102'), ('Id_3', 'val_103');
> INSERT INTO TABLE T_B
> VALUES ('Id_1', 'val_103'), ('Id_2', 'val_104');
> explain
> FROM T_A a LEFT JOIN T_B b ON a.id = b.id
> INSERT OVERWRITE TABLE join_result_1
> SELECT a.*, b.*
> WHERE b.id = 'Id_1' AND b.val = 'val_103'
> INSERT OVERWRITE TABLE join_result_3
> SELECT a.*, b.*
> WHERE b.val = 'val_104' AND b.id = 'Id_2' AND a.val <> b.val;
> {noformat}
> The (wrong) plan is the following:
> {noformat}
> STAGE DEPENDENCIES:
> Stage-2 is a root stage
> Stage-3 depends on stages: Stage-2
> Stage-0 depends on stages: Stage-3
> Stage-4 depends on stages: Stage-0
> Stage-1 depends on stages: Stage-3
> Stage-5 depends on stages: Stage-1
> STAGE PLANS:
> Stage: Stage-2
> Tez
> DagId: haha_20160504140944_174465c9-5d1a-42f9-9665-fae02eeb2767:2
> Edges:
> Reducer 2 <- Map 1 (SIMPLE_EDGE), Map 3 (SIMPLE_EDGE)
> DagName:
> Vertices:
> Map 1
> Map Operator Tree:
> TableScan
> alias: a
> Statistics: Num rows: 3 Data size: 36 Basic stats: COMPLETE
> Column stats: NONE
> Reduce Output Operator
> key expressions: id (type: string)
> sort order: +
> Map-reduce partition columns: id (type: string)
> Statistics: Num rows: 3 Data size: 36 Basic stats:
> COMPLETE Column stats: NONE
> value expressions: val (type: string)
> Map 3
> Map Operator Tree:
> TableScan
> alias: b
> Statistics: Num rows: 2 Data size: 24 Basic stats: COMPLETE
> Column stats: NONE
> Reduce Output Operator
> key expressions: id (type: string)
> sort order: +
> Map-reduce partition columns: id (type: string)
> Statistics: Num rows: 2 Data size: 24 Basic stats:
> COMPLETE Column stats: NONE
> value expressions: val (type: string)
> Reducer 2
> Reduce Operator Tree:
> Merge Join Operator
> condition map:
> Left Outer Join0 to 1
> keys:
> 0 id (type: string)
> 1 id (type: string)
> outputColumnNames: _col0, _col1, _col6
> Statistics: Num rows: 3 Data size: 39 Basic stats: COMPLETE
> Column stats: NONE
> Select Operator
> expressions: _col0 (type: string), _col1 (type: string),
> 'Id_1' (type: string), 'val_103' (type: string)
> outputColumnNames: _col0, _col1, _col2, _col3
> Statistics: Num rows: 3 Data size: 39 Basic stats: COMPLETE
> Column stats: NONE
> File Output Operator
> compressed: false
> Statistics: Num rows: 3 Data size: 39 Basic stats:
> COMPLETE Column stats: NONE
> table:
> input format: org.apache.hadoop.mapred.TextInputFormat
> output format:
> org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
> serde:
> org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
> name: bugtest2.join_result_1
> Filter Operator
> predicate: (_col1 <> _col6) (type: boolean)
> Statistics: Num rows: 3 Data size: 39 Basic stats: COMPLETE
> Column stats: NONE
> Select Operator
> expressions: _col0 (type: string), _col1 (type: string),
> 'Id_2' (type: string), 'val_104' (type: string)
> outputColumnNames: _col0, _col1, _col2, _col3
>
[jira] [Work started] (HIVE-13693) Multi-insert query drops Filter before file output when there is a.val <> b.val
[
https://issues.apache.org/jira/browse/HIVE-13693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HIVE-13693 started by Jesus Camacho Rodriguez.
--
> Multi-insert query drops Filter before file output when there is a.val <>
> b.val
> ---
>
> Key: HIVE-13693
> URL: https://issues.apache.org/jira/browse/HIVE-13693
> Project: Hive
> Issue Type: Bug
> Components: Logical Optimizer
>Affects Versions: 1.3.0, 2.0.0, 2.1.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
> Attachments: HIVE-13693.patch
>
>
> To reproduce:
> {noformat}
> CREATE TABLE T_A ( id STRING, val STRING );
> CREATE TABLE T_B ( id STRING, val STRING );
> CREATE TABLE join_result_1 ( ida STRING, vala STRING, idb STRING, valb STRING
> );
> CREATE TABLE join_result_3 ( ida STRING, vala STRING, idb STRING, valb STRING
> );
> INSERT INTO TABLE T_A
> VALUES ('Id_1', 'val_101'), ('Id_2', 'val_102'), ('Id_3', 'val_103');
> INSERT INTO TABLE T_B
> VALUES ('Id_1', 'val_103'), ('Id_2', 'val_104');
> explain
> FROM T_A a LEFT JOIN T_B b ON a.id = b.id
> INSERT OVERWRITE TABLE join_result_1
> SELECT a.*, b.*
> WHERE b.id = 'Id_1' AND b.val = 'val_103'
> INSERT OVERWRITE TABLE join_result_3
> SELECT a.*, b.*
> WHERE b.val = 'val_104' AND b.id = 'Id_2' AND a.val <> b.val;
> {noformat}
> The (wrong) plan is the following:
> {noformat}
> STAGE DEPENDENCIES:
> Stage-2 is a root stage
> Stage-3 depends on stages: Stage-2
> Stage-0 depends on stages: Stage-3
> Stage-4 depends on stages: Stage-0
> Stage-1 depends on stages: Stage-3
> Stage-5 depends on stages: Stage-1
> STAGE PLANS:
> Stage: Stage-2
> Tez
> DagId: haha_20160504140944_174465c9-5d1a-42f9-9665-fae02eeb2767:2
> Edges:
> Reducer 2 <- Map 1 (SIMPLE_EDGE), Map 3 (SIMPLE_EDGE)
> DagName:
> Vertices:
> Map 1
> Map Operator Tree:
> TableScan
> alias: a
> Statistics: Num rows: 3 Data size: 36 Basic stats: COMPLETE
> Column stats: NONE
> Reduce Output Operator
> key expressions: id (type: string)
> sort order: +
> Map-reduce partition columns: id (type: string)
> Statistics: Num rows: 3 Data size: 36 Basic stats:
> COMPLETE Column stats: NONE
> value expressions: val (type: string)
> Map 3
> Map Operator Tree:
> TableScan
> alias: b
> Statistics: Num rows: 2 Data size: 24 Basic stats: COMPLETE
> Column stats: NONE
> Reduce Output Operator
> key expressions: id (type: string)
> sort order: +
> Map-reduce partition columns: id (type: string)
> Statistics: Num rows: 2 Data size: 24 Basic stats:
> COMPLETE Column stats: NONE
> value expressions: val (type: string)
> Reducer 2
> Reduce Operator Tree:
> Merge Join Operator
> condition map:
> Left Outer Join0 to 1
> keys:
> 0 id (type: string)
> 1 id (type: string)
> outputColumnNames: _col0, _col1, _col6
> Statistics: Num rows: 3 Data size: 39 Basic stats: COMPLETE
> Column stats: NONE
> Select Operator
> expressions: _col0 (type: string), _col1 (type: string),
> 'Id_1' (type: string), 'val_103' (type: string)
> outputColumnNames: _col0, _col1, _col2, _col3
> Statistics: Num rows: 3 Data size: 39 Basic stats: COMPLETE
> Column stats: NONE
> File Output Operator
> compressed: false
> Statistics: Num rows: 3 Data size: 39 Basic stats:
> COMPLETE Column stats: NONE
> table:
> input format: org.apache.hadoop.mapred.TextInputFormat
> output format:
> org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
> serde:
> org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
> name: bugtest2.join_result_1
> Filter Operator
> predicate: (_col1 <> _col6) (type: boolean)
> Statistics: Num rows: 3 Data size: 39 Basic stats: COMPLETE
> Column stats: NONE
> Select Operator
> expressions: _col0 (type: string), _col1 (type: string),
> 'Id_2' (type: string), 'val_104' (type: string)
> outputColumnNames: _col0, _col1, _col2, _col3
>
[jira] [Resolved] (HIVE-13712) LLAP: LlapServiceDriver should package hadoop-metrics2-llapdaemon.properties when available
[ https://issues.apache.org/jira/browse/HIVE-13712?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran resolved HIVE-13712. -- Resolution: Fixed Fix Version/s: 2.1.0 Committed to master. > LLAP: LlapServiceDriver should package hadoop-metrics2-llapdaemon.properties > when available > --- > > Key: HIVE-13712 > URL: https://issues.apache.org/jira/browse/HIVE-13712 > Project: Hive > Issue Type: Bug > Components: llap >Affects Versions: 2.1.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran > Fix For: 2.1.0 > > Attachments: HIVE-13712.1.patch, HIVE-13712.2.patch > > > HIVE-13701 renamed hadoop-metrics2.properties to > hadoop-metrics2-llapdaemon.properties to avoid conflicts in classpath lookup. > MetricsSystem first looks for hadoop-metrics2-llapdaemon.properties file > first before falling back to hadoop-metrics2.properties. Make > LlapServiceDriver package hadoop-metrics2-llapdaemon.properties first and > fallback to hadoop-metrics2.properties. > NO PRECOMMIT TESTS -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12656) Turn hive.compute.query.using.stats on by default
[ https://issues.apache.org/jira/browse/HIVE-12656?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-12656: --- Status: Patch Available (was: Open) > Turn hive.compute.query.using.stats on by default > - > > Key: HIVE-12656 > URL: https://issues.apache.org/jira/browse/HIVE-12656 > Project: Hive > Issue Type: Bug >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Attachments: HIVE-12656.01.patch > > > We now have hive.compute.query.using.stats=false by default. We plan to turn > it on by default so that we can have better performance. We can also set it > to false in some test cases to maintain the original purpose of those tests.. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13712) LLAP: LlapServiceDriver should package hadoop-metrics2-llapdaemon.properties when available
[ https://issues.apache.org/jira/browse/HIVE-13712?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-13712: - Attachment: HIVE-13712.2.patch Fixed a small grammatical > LLAP: LlapServiceDriver should package hadoop-metrics2-llapdaemon.properties > when available > --- > > Key: HIVE-13712 > URL: https://issues.apache.org/jira/browse/HIVE-13712 > Project: Hive > Issue Type: Bug > Components: llap >Affects Versions: 2.1.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran > Attachments: HIVE-13712.1.patch, HIVE-13712.2.patch > > > HIVE-13701 renamed hadoop-metrics2.properties to > hadoop-metrics2-llapdaemon.properties to avoid conflicts in classpath lookup. > MetricsSystem first looks for hadoop-metrics2-llapdaemon.properties file > first before falling back to hadoop-metrics2.properties. Make > LlapServiceDriver package hadoop-metrics2-llapdaemon.properties first and > fallback to hadoop-metrics2.properties. > NO PRECOMMIT TESTS -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13712) LLAP: LlapServiceDriver should package hadoop-metrics2-llapdaemon.properties when available
[ https://issues.apache.org/jira/browse/HIVE-13712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15276759#comment-15276759 ] Sergey Shelukhin commented on HIVE-13712: - +1 > LLAP: LlapServiceDriver should package hadoop-metrics2-llapdaemon.properties > when available > --- > > Key: HIVE-13712 > URL: https://issues.apache.org/jira/browse/HIVE-13712 > Project: Hive > Issue Type: Bug > Components: llap >Affects Versions: 2.1.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran > Attachments: HIVE-13712.1.patch, HIVE-13712.2.patch > > > HIVE-13701 renamed hadoop-metrics2.properties to > hadoop-metrics2-llapdaemon.properties to avoid conflicts in classpath lookup. > MetricsSystem first looks for hadoop-metrics2-llapdaemon.properties file > first before falling back to hadoop-metrics2.properties. Make > LlapServiceDriver package hadoop-metrics2-llapdaemon.properties first and > fallback to hadoop-metrics2.properties. > NO PRECOMMIT TESTS -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11793) SHOW LOCKS with DbTxnManager ignores filter options
[ https://issues.apache.org/jira/browse/HIVE-11793?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Zheng updated HIVE-11793: - Attachment: HIVE-11793.7.patch patch 7 > SHOW LOCKS with DbTxnManager ignores filter options > --- > > Key: HIVE-11793 > URL: https://issues.apache.org/jira/browse/HIVE-11793 > Project: Hive > Issue Type: Improvement > Components: Transactions >Affects Versions: 1.0.0 >Reporter: Eugene Koifman >Assignee: Wei Zheng >Priority: Minor > Attachments: HIVE-11793.1.patch, HIVE-11793.2.patch, > HIVE-11793.3.patch, HIVE-11793.4.patch, HIVE-11793.5.patch, > HIVE-11793.6.patch, HIVE-11793.7.patch > > > https://cwiki.apache.org/confluence/display/Hive/Locking and > https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL#LanguageManualDDL-ShowLocks > list various options that can be used with SHOW LOCKS, e.g. > When ACID is enabled, all these options are ignored and a full list is > returned. > (also only ext lock id is shown, int lock id is not). > see DDLTask.showLocks() and TxnHandler.showLocks() > requires extending ShowLocksRequest which is a Thrift object -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13341) Stats state is not captured correctly: differentiate load table and create table
[ https://issues.apache.org/jira/browse/HIVE-13341?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-13341: --- Affects Version/s: 2.0.0 > Stats state is not captured correctly: differentiate load table and create > table > > > Key: HIVE-13341 > URL: https://issues.apache.org/jira/browse/HIVE-13341 > Project: Hive > Issue Type: Sub-task > Components: Logical Optimizer, Statistics >Affects Versions: 2.0.0 >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Fix For: 2.1.0 > > Attachments: HIVE-13341.01.patch, HIVE-13341.02.patch, > HIVE-13341.03.patch, HIVE-13341.04.patch, HIVE-13341.05.patch, > HIVE-13341.06.patch, HIVE-13341.07.patch, HIVE-13341.08.patch, > HIVE-13341.09.patch, HIVE-13341.final.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13341) Stats state is not captured correctly: differentiate load table and create table
[ https://issues.apache.org/jira/browse/HIVE-13341?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-13341: --- Attachment: HIVE-13341.final.patch Check all the test case failures one by one. Then generate new golden files one by one. Confirm every one works as expected. Update the golden files and pushed to master. Thanks [~ashutoshc] for the review. > Stats state is not captured correctly: differentiate load table and create > table > > > Key: HIVE-13341 > URL: https://issues.apache.org/jira/browse/HIVE-13341 > Project: Hive > Issue Type: Sub-task > Components: Logical Optimizer, Statistics >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Fix For: 2.1.0 > > Attachments: HIVE-13341.01.patch, HIVE-13341.02.patch, > HIVE-13341.03.patch, HIVE-13341.04.patch, HIVE-13341.05.patch, > HIVE-13341.06.patch, HIVE-13341.07.patch, HIVE-13341.08.patch, > HIVE-13341.09.patch, HIVE-13341.final.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13341) Stats state is not captured correctly: differentiate load table and create table
[ https://issues.apache.org/jira/browse/HIVE-13341?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-13341: --- Fix Version/s: 2.1.0 > Stats state is not captured correctly: differentiate load table and create > table > > > Key: HIVE-13341 > URL: https://issues.apache.org/jira/browse/HIVE-13341 > Project: Hive > Issue Type: Sub-task > Components: Logical Optimizer, Statistics >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Fix For: 2.1.0 > > Attachments: HIVE-13341.01.patch, HIVE-13341.02.patch, > HIVE-13341.03.patch, HIVE-13341.04.patch, HIVE-13341.05.patch, > HIVE-13341.06.patch, HIVE-13341.07.patch, HIVE-13341.08.patch, > HIVE-13341.09.patch, HIVE-13341.final.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13341) Stats state is not captured correctly: differentiate load table and create table
[ https://issues.apache.org/jira/browse/HIVE-13341?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-13341: --- Resolution: Fixed Status: Resolved (was: Patch Available) > Stats state is not captured correctly: differentiate load table and create > table > > > Key: HIVE-13341 > URL: https://issues.apache.org/jira/browse/HIVE-13341 > Project: Hive > Issue Type: Sub-task > Components: Logical Optimizer, Statistics >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Attachments: HIVE-13341.01.patch, HIVE-13341.02.patch, > HIVE-13341.03.patch, HIVE-13341.04.patch, HIVE-13341.05.patch, > HIVE-13341.06.patch, HIVE-13341.07.patch, HIVE-13341.08.patch, > HIVE-13341.09.patch, HIVE-13341.final.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13720) TestLlapTaskCommunicator fails on master
[
https://issues.apache.org/jira/browse/HIVE-13720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15276683#comment-15276683
]
Ashutosh Chauhan commented on HIVE-13720:
-
Stacktrace:
{code}
testFinishableStateUpdateFailure(org.apache.hadoop.hive.llap.tezplugins.TestLlapTaskCommunicator)
Time elapsed: 1.282 sec <<< ERROR!
java.lang.NullPointerException: hive.llap.daemon.service.hosts must be defined
at
com.google.common.base.Preconditions.checkNotNull(Preconditions.java:208)
at
org.apache.hadoop.hive.llap.registry.impl.LlapRegistryService.getClient(LlapRegistryService.java:57)
at
org.apache.hadoop.hive.llap.tezplugins.LlapTaskCommunicator.(LlapTaskCommunicator.java:136)
at
org.apache.hadoop.hive.llap.tezplugins.TestLlapTaskCommunicator$LlapTaskCommunicatorForTest.(TestLlapTaskCommunicator.java:385)
at
org.apache.hadoop.hive.llap.tezplugins.TestLlapTaskCommunicator$LlapTaskCommunicatorWrapperForTest.(TestLlapTaskCommunicator.java:293)
at
org.apache.hadoop.hive.llap.tezplugins.TestLlapTaskCommunicator.testFinishableStateUpdateFailure(TestLlapTaskCommunicator.java:135)
{code}
cc: [~sseth]
> TestLlapTaskCommunicator fails on master
>
>
> Key: HIVE-13720
> URL: https://issues.apache.org/jira/browse/HIVE-13720
> Project: Hive
> Issue Type: Bug
> Components: llap, Test
>Affects Versions: 2.1.0
>Reporter: Ashutosh Chauhan
>
> Can be reproduced locally as well
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
