[jira] [Commented] (HIVE-17834) Fix flaky triggers test
[ https://issues.apache.org/jira/browse/HIVE-17834?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16213746#comment-16213746 ] Prasanth Jayachandran commented on HIVE-17834: -- 1st run succeeded. GIving it another try to see it goes through without flakiness. > Fix flaky triggers test > --- > > Key: HIVE-17834 > URL: https://issues.apache.org/jira/browse/HIVE-17834 > Project: Hive > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran > Attachments: HIVE-17834.1.patch > > > https://issues.apache.org/jira/browse/HIVE-12631?focusedCommentId=16209803=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16209803 -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17828) Metastore: mysql upgrade scripts to 3.0.0 is broken
[
https://issues.apache.org/jira/browse/HIVE-17828?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Prasanth Jayachandran updated HIVE-17828:
-
Resolution: Fixed
Fix Version/s: 3.0.0
Status: Resolved (was: Patch Available)
Test failures are unrelated. Committed patch to master
> Metastore: mysql upgrade scripts to 3.0.0 is broken
> ---
>
> Key: HIVE-17828
> URL: https://issues.apache.org/jira/browse/HIVE-17828
> Project: Hive
> Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Gopal V
>Assignee: Prasanth Jayachandran
> Fix For: 3.0.0
>
> Attachments: HIVE-17828.1.patch, HIVE-17828.2.patch
>
>
> {code}
> +-+
> | |
> +-+
> | Finished upgrading MetaStore schema from 2.2.0 to 2.3.0 |
> +-+
> 1 row in set, 1 warning (0.00 sec)
> mysql> source upgrade-2.3.0-to-3.0.0.mysql.sql
> ++
> ||
> ++
> | Upgrading MetaStore schema from 2.3.0 to 3.0.0 |
> ++
> {code}
> {code}
> --
> CREATE TABLE WM_RESOURCEPLAN (
> `RP_ID` bigint(20) NOT NULL,
> `NAME` varchar(128) NOT NULL,
> `QUERY_PARALLELISM` int(11),
> `STATUS` varchar(20) NOT NULL,
> PRIMARY KEY (`RP_ID`),
> KEY `UNIQUE_WM_RESOURCEPLAN` (`NAME`),
> ) ENGINE=InnoDB DEFAULT CHARSET=latin1
> --
> ERROR 1064 (42000): You have an error in your SQL syntax; check the manual
> that corresponds to your MySQL server version for the right syntax to use
> near ') ENGINE=InnoDB DEFAULT CHARSET=latin1' at line 8
> --
> CREATE TABLE WM_POOL
> (
> `POOL_ID` bigint(20) NOT NULL,
> `RP_ID` bigint(20) NOT NULL,
> `PATH` varchar(1024) NOT NULL,
> `PARENT_POOL_ID` bigint(20),
> `ALLOC_FRACTION` DOUBLE,
> `QUERY_PARALLELISM` int(11),
> PRIMARY KEY (`POOL_ID`),
> KEY `UNIQUE_WM_POOL` (`RP_ID`, `PATH`),
> CONSTRAINT `WM_POOL_FK1` FOREIGN KEY (`RP_ID`) REFERENCES
> `WM_RESOURCEPLAN` (`RP_ID`),
> CONSTRAINT `WM_POOL_FK2` FOREIGN KEY (`PARENT_POOL_ID`) REFERENCES
> `WM_POOL` (`POOL_ID`)
> ) ENGINE=InnoDB DEFAULT CHARSET=latin1
> --
> ERROR 1071 (42000): Specified key was too long; max key length is 767 bytes
> --
> CREATE TABLE WM_TRIGGER
> (
> `TRIGGER_ID` bigint(20) NOT NULL,
> `RP_ID` bigint(20) NOT NULL,
> `NAME` varchar(128) NOT NULL,
> `TRIGGER_EXPRESSION` varchar(1024),
> `ACTION_EXPRESSION` varchar(1024),
> PRIMARY KEY (`TRIGGER_ID`),
> KEY `UNIQUE_WM_TRIGGER` (`RP_ID`, `NAME`),
> CONSTRAINT `WM_TRIGGER_FK1` FOREIGN KEY (`RP_ID`) REFERENCES
> `WM_RESOURCEPLAN` (`RP_ID`)
> ) ENGINE=InnoDB DEFAULT CHARSET=latin1
> --
> ERROR 1215 (HY000): Cannot add foreign key constraint
> --
> CREATE TABLE WM_POOL_TO_TRIGGER
> (
> `POOL_ID` bigint(20) NOT NULL,
> `TRIGGER_ID` bigint(20) NOT NULL,
> PRIMARY KEY (`POOL_ID`, `TRIGGER_ID`),
> CONSTRAINT `WM_POOL_TO_TRIGGER_FK1` FOREIGN KEY (`POOL_ID`) REFERENCES
> `WM_POOL` (`POOL_ID`),
> CONSTRAINT `WM_POOL_TO_TRIGGER_FK2` FOREIGN KEY (`TRIGGER_ID`) REFERENCES
> `WM_TRIGGER` (`TRIGGER_ID`)
> ) ENGINE=InnoDB DEFAULT CHARSET=latin1
> --
> ERROR 1215 (HY000): Cannot add foreign key constraint
> --
> CREATE TABLE WM_MAPPING
> (
> `MAPPING_ID` bigint(20) NOT NULL,
> `RP_ID` bigint(20) NOT NULL,
> `ENTITY_TYPE` varchar(10) NOT NULL,
> `ENTITY_NAME` varchar(128) NOT NULL,
> `POOL_ID` bigint(20) NOT NULL,
> `ORDERING int,
> PRIMARY KEY (`MAPPING_ID`),
> KEY `UNIQUE_WM_MAPPING` (`RP_ID`, `ENTITY_TYPE`, `ENTITY_NAME`),
> CONSTRAINT `WM_MAPPING_FK1` FOREIGN KEY (`RP_ID`) REFERENCES
> `WM_RESOURCEPLAN` (`RP_ID`),
> CONSTRAINT `WM_MAPPING_FK2` FOREIGN KEY (`POOL_ID`) REFERENCES `WM_POOL`
> (`POOL_ID`)
> ) ENGINE=InnoDB DEFAULT CHARSET=latin1;
> --
> ERROR 1064 (42000): You have an error in your SQL syntax; check the manual
> that corresponds to your MySQL server version for the right syntax to use
> near 'MAPPING_ID`),
> KEY `UNIQUE_WM_MAPPING` (`RP_ID`, `ENTITY_TYPE`, `ENTITY_NAME`' at line 8
> --
> UPDATE VERSION SET SCHEMA_VERSION='3.0.0', VERSION_COMMENT='Hive release
> version 3.0.0' where VER_ID=1
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17828) Metastore: mysql upgrade scripts to 3.0.0 is broken
[
https://issues.apache.org/jira/browse/HIVE-17828?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16213699#comment-16213699
]
Hive QA commented on HIVE-17828:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12893107/HIVE-17828.2.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 7 failed/errored test(s), 11315 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[optimize_nullscan]
(batchId=163)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[resourceplan]
(batchId=158)
org.apache.hadoop.hive.cli.control.TestDanglingQOuts.checkDanglingQOut
(batchId=204)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testConstraints
(batchId=221)
org.apache.hadoop.hive.ql.parse.authorization.plugin.sqlstd.TestOperation2Privilege.checkHiveOperationTypeMatch
(batchId=269)
org.apache.hive.jdbc.TestTriggersTezSessionPoolManager.testTriggerHighShuffleBytes
(batchId=228)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerHighShuffleBytes
(batchId=228)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/7417/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/7417/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-7417/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 7 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12893107 - PreCommit-HIVE-Build
> Metastore: mysql upgrade scripts to 3.0.0 is broken
> ---
>
> Key: HIVE-17828
> URL: https://issues.apache.org/jira/browse/HIVE-17828
> Project: Hive
> Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Gopal V
>Assignee: Prasanth Jayachandran
> Attachments: HIVE-17828.1.patch, HIVE-17828.2.patch
>
>
> {code}
> +-+
> | |
> +-+
> | Finished upgrading MetaStore schema from 2.2.0 to 2.3.0 |
> +-+
> 1 row in set, 1 warning (0.00 sec)
> mysql> source upgrade-2.3.0-to-3.0.0.mysql.sql
> ++
> ||
> ++
> | Upgrading MetaStore schema from 2.3.0 to 3.0.0 |
> ++
> {code}
> {code}
> --
> CREATE TABLE WM_RESOURCEPLAN (
> `RP_ID` bigint(20) NOT NULL,
> `NAME` varchar(128) NOT NULL,
> `QUERY_PARALLELISM` int(11),
> `STATUS` varchar(20) NOT NULL,
> PRIMARY KEY (`RP_ID`),
> KEY `UNIQUE_WM_RESOURCEPLAN` (`NAME`),
> ) ENGINE=InnoDB DEFAULT CHARSET=latin1
> --
> ERROR 1064 (42000): You have an error in your SQL syntax; check the manual
> that corresponds to your MySQL server version for the right syntax to use
> near ') ENGINE=InnoDB DEFAULT CHARSET=latin1' at line 8
> --
> CREATE TABLE WM_POOL
> (
> `POOL_ID` bigint(20) NOT NULL,
> `RP_ID` bigint(20) NOT NULL,
> `PATH` varchar(1024) NOT NULL,
> `PARENT_POOL_ID` bigint(20),
> `ALLOC_FRACTION` DOUBLE,
> `QUERY_PARALLELISM` int(11),
> PRIMARY KEY (`POOL_ID`),
> KEY `UNIQUE_WM_POOL` (`RP_ID`, `PATH`),
> CONSTRAINT `WM_POOL_FK1` FOREIGN KEY (`RP_ID`) REFERENCES
> `WM_RESOURCEPLAN` (`RP_ID`),
> CONSTRAINT `WM_POOL_FK2` FOREIGN KEY (`PARENT_POOL_ID`) REFERENCES
> `WM_POOL` (`POOL_ID`)
> ) ENGINE=InnoDB DEFAULT CHARSET=latin1
> --
> ERROR 1071 (42000): Specified key was too long; max key length is 767 bytes
> --
> CREATE TABLE WM_TRIGGER
> (
> `TRIGGER_ID` bigint(20) NOT NULL,
> `RP_ID` bigint(20) NOT NULL,
> `NAME` varchar(128) NOT NULL,
> `TRIGGER_EXPRESSION` varchar(1024),
> `ACTION_EXPRESSION` varchar(1024),
> PRIMARY KEY (`TRIGGER_ID`),
> KEY `UNIQUE_WM_TRIGGER` (`RP_ID`, `NAME`),
> CONSTRAINT `WM_TRIGGER_FK1` FOREIGN KEY (`RP_ID`) REFERENCES
> `WM_RESOURCEPLAN` (`RP_ID`)
> ) ENGINE=InnoDB DEFAULT CHARSET=latin1
> --
> ERROR 1215 (HY000): Cannot add foreign key constraint
> --
> CREATE TABLE WM_POOL_TO_TRIGGER
> (
> `POOL_ID` bigint(20) NOT NULL,
> `TRIGGER_ID` bigint(20) NOT NULL,
> PRIMARY KEY (`POOL_ID`, `TRIGGER_ID`),
> CONSTRAINT `WM_POOL_TO_TRIGGER_FK1` FOREIGN KEY (`POOL_ID`)
[jira] [Updated] (HIVE-17838) Make org.apache.hive.spark.client.rpc logging HoS specific and other logging cleanup
[
https://issues.apache.org/jira/browse/HIVE-17838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sahil Takiar updated HIVE-17838:
Attachment: HIVE-17838.1.patch
> Make org.apache.hive.spark.client.rpc logging HoS specific and other logging
> cleanup
>
>
> Key: HIVE-17838
> URL: https://issues.apache.org/jira/browse/HIVE-17838
> Project: Hive
> Issue Type: Sub-task
> Components: Hive
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
> Attachments: HIVE-17838.1.patch
>
>
> A lot of the logging in {{org.apache.hive.spark.client.rpc}} emits
> information that a generic RPC library would, but this package is only used
> by HoS. We should make the logging more HoS specific so users don't get
> confused as to what this RPC library is doing. Ideally, they aren't even
> aware that a RPC library is in place, it should be more transparent to the
> user.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-17838) Make org.apache.hive.spark.client.rpc logging HoS specific and other logging cleanup
[
https://issues.apache.org/jira/browse/HIVE-17838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sahil Takiar updated HIVE-17838:
Status: Patch Available (was: Open)
> Make org.apache.hive.spark.client.rpc logging HoS specific and other logging
> cleanup
>
>
> Key: HIVE-17838
> URL: https://issues.apache.org/jira/browse/HIVE-17838
> Project: Hive
> Issue Type: Sub-task
> Components: Hive
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
> Attachments: HIVE-17838.1.patch
>
>
> A lot of the logging in {{org.apache.hive.spark.client.rpc}} emits
> information that a generic RPC library would, but this package is only used
> by HoS. We should make the logging more HoS specific so users don't get
> confused as to what this RPC library is doing. Ideally, they aren't even
> aware that a RPC library is in place, it should be more transparent to the
> user.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17771) Implement commands to manage resource plan
[ https://issues.apache.org/jira/browse/HIVE-17771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16213672#comment-16213672 ] Lefty Leverenz commented on HIVE-17771: --- Doc note: This should be documented in the wiki, either in the LLAP design doc or in a new doc for LLAP workload management, with a cross-reference section in the DDL doc. The new keywords should also be documented in the DDL doc. * [LLAP | https://cwiki.apache.org/confluence/display/Hive/LLAP] * [DDL -- Keywords | https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL#LanguageManualDDL-Keywords,Non-reservedKeywordsandReservedKeywords] Added a TODOC3.0 label. > Implement commands to manage resource plan > -- > > Key: HIVE-17771 > URL: https://issues.apache.org/jira/browse/HIVE-17771 > Project: Hive > Issue Type: Sub-task >Reporter: Harish Jaiprakash >Assignee: Harish Jaiprakash > Labels: TODOC3.0 > Fix For: 3.0.0 > > Attachments: HIVE-17771.01.patch, HIVE-17771.02.patch, > HIVE-17771.03.patch, HIVE-17771.04.patch > > > Please see parent jira about llap workload management. > This jira is to implement create and show resource plan commands in hive to > configure resource plans for llap workload. The following are the proposed > commands implemented as part of the jira: > CREATE RESOURCE PLAN plan_name WITH QUERY_PARALLELISM parallelism; > SHOW RESOURCE PLAN plan_name; > SHOW RESOURCE PLANS; > ALTER RESOURCE PLAN plan_name SET QUERY_PARALLELISM = parallelism; > ALTER RESOURCE PLAN plan_name RENAME TO new_name; > ALTER RESOURCE PLAN plan_name ACTIVATE; > ALTER RESOURCE PLAN plan_name DISABLE; > ALTER RESOURCE PLAN plan_name ENABLE; > DROP RESOURCE PLAN; > It will be followed up with more jiras to manage pools, triggers and copy > resource plans. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17771) Implement commands to manage resource plan
[ https://issues.apache.org/jira/browse/HIVE-17771?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lefty Leverenz updated HIVE-17771: -- Labels: TODOC3.0 (was: ) > Implement commands to manage resource plan > -- > > Key: HIVE-17771 > URL: https://issues.apache.org/jira/browse/HIVE-17771 > Project: Hive > Issue Type: Sub-task >Reporter: Harish Jaiprakash >Assignee: Harish Jaiprakash > Labels: TODOC3.0 > Fix For: 3.0.0 > > Attachments: HIVE-17771.01.patch, HIVE-17771.02.patch, > HIVE-17771.03.patch, HIVE-17771.04.patch > > > Please see parent jira about llap workload management. > This jira is to implement create and show resource plan commands in hive to > configure resource plans for llap workload. The following are the proposed > commands implemented as part of the jira: > CREATE RESOURCE PLAN plan_name WITH QUERY_PARALLELISM parallelism; > SHOW RESOURCE PLAN plan_name; > SHOW RESOURCE PLANS; > ALTER RESOURCE PLAN plan_name SET QUERY_PARALLELISM = parallelism; > ALTER RESOURCE PLAN plan_name RENAME TO new_name; > ALTER RESOURCE PLAN plan_name ACTIVATE; > ALTER RESOURCE PLAN plan_name DISABLE; > ALTER RESOURCE PLAN plan_name ENABLE; > DROP RESOURCE PLAN; > It will be followed up with more jiras to manage pools, triggers and copy > resource plans. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17834) Fix flaky triggers test
[
https://issues.apache.org/jira/browse/HIVE-17834?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16213652#comment-16213652
]
Hive QA commented on HIVE-17834:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12892958/HIVE-17834.1.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 6 failed/errored test(s), 11314 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid]
(batchId=164)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast]
(batchId=156)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[optimize_nullscan]
(batchId=163)
org.apache.hadoop.hive.cli.control.TestDanglingQOuts.checkDanglingQOut
(batchId=204)
org.apache.hadoop.hive.llap.security.TestLlapSignerImpl.testSigning
(batchId=293)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testConstraints
(batchId=221)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/7416/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/7416/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-7416/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 6 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12892958 - PreCommit-HIVE-Build
> Fix flaky triggers test
> ---
>
> Key: HIVE-17834
> URL: https://issues.apache.org/jira/browse/HIVE-17834
> Project: Hive
> Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
> Attachments: HIVE-17834.1.patch
>
>
> https://issues.apache.org/jira/browse/HIVE-12631?focusedCommentId=16209803=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16209803
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-17858) MM - some union cases are broken
[ https://issues.apache.org/jira/browse/HIVE-17858?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-17858: Status: Patch Available (was: Open) [~ekoifman] can you take a look? Most of the patch is the .out file, java changes are relatively small. > MM - some union cases are broken > > > Key: HIVE-17858 > URL: https://issues.apache.org/jira/browse/HIVE-17858 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Labels: mm-gap-1 > Attachments: HIVE-17858.patch > > > mm_all test no longer runs on LLAP; if it's executed in LLAP, one can see > that some union cases no longer work. > Queries on partunion_mm, skew_dp_union_mm produce no results. > I'm not sure what part of "integration" broke it. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Comment Edited] (HIVE-17858) MM - some union cases are broken
[ https://issues.apache.org/jira/browse/HIVE-17858?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16213642#comment-16213642 ] Sergey Shelukhin edited comment on HIVE-17858 at 10/21/17 1:32 AM: --- There are a couple of issues, the main one being that DP directories cannot be located correctly due to some statement ID issues. I will file a followup jira, for now it looks like stmtId for MoveTask is not needed. It's set in load and import cases only, not for insert; moreover for union there are several statement IDs for the same query (one per side of the union), and the load needs to load all of them at once. I'm also enabling mm_all test for MiniLlap was (Author: sershe): There are a couple of issues, the main one being that DP directories cannot be located correctly due to some statement ID issues. I will file a followup jira, for now it looks like stmtId for MoveTask is not needed. It's set in load and import cases only, not for insert; moreover for union there are several statement IDs for the same query (one per side of the union), and the load needs to load all of them at once. > MM - some union cases are broken > > > Key: HIVE-17858 > URL: https://issues.apache.org/jira/browse/HIVE-17858 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Labels: mm-gap-1 > Attachments: HIVE-17858.patch > > > mm_all test no longer runs on LLAP; if it's executed in LLAP, one can see > that some union cases no longer work. > Queries on partunion_mm, skew_dp_union_mm produce no results. > I'm not sure what part of "integration" broke it. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17858) MM - some union cases are broken
[ https://issues.apache.org/jira/browse/HIVE-17858?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-17858: Attachment: HIVE-17858.patch There are a couple of issues, the main one being that DP directories cannot be located correctly due to some statement ID issues. I will file a followup jira, for now it looks like stmtId for MoveTask is not needed. It's set in load and import cases only, not for insert; moreover for union there are several statement IDs for the same query (one per side of the union), and the load needs to load all of them at once. > MM - some union cases are broken > > > Key: HIVE-17858 > URL: https://issues.apache.org/jira/browse/HIVE-17858 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Labels: mm-gap-1 > Attachments: HIVE-17858.patch > > > mm_all test no longer runs on LLAP; if it's executed in LLAP, one can see > that some union cases no longer work. > Queries on partunion_mm, skew_dp_union_mm produce no results. > I'm not sure what part of "integration" broke it. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17871) Add non nullability flag to druid time column
[
https://issues.apache.org/jira/browse/HIVE-17871?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16213627#comment-16213627
]
slim bouguerra commented on HIVE-17871:
---
[~ashutoshc] working on that but for some reason the rule is not firing as i
expected and working in calcite
https://issues.apache.org/jira/browse/CALCITE-2019
> Add non nullability flag to druid time column
> -
>
> Key: HIVE-17871
> URL: https://issues.apache.org/jira/browse/HIVE-17871
> Project: Hive
> Issue Type: Improvement
> Components: Druid integration
>Reporter: slim bouguerra
>Assignee: Jesus Camacho Rodriguez
> Attachments: HIVE-17871.patch
>
>
> Druid time column is non null all the time.
> Adding the non nullability flag will enable extra calcite goodness like
> transforming
> {code} select count(`__time`) from table {code} to {code} select count(*)
> from table {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-17771) Implement commands to manage resource plan
[ https://issues.apache.org/jira/browse/HIVE-17771?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-17771: Resolution: Fixed Fix Version/s: 3.0.0 Status: Resolved (was: Patch Available) Fixed the operations enum and committed to master. Thanks for the patch! > Implement commands to manage resource plan > -- > > Key: HIVE-17771 > URL: https://issues.apache.org/jira/browse/HIVE-17771 > Project: Hive > Issue Type: Sub-task >Reporter: Harish Jaiprakash >Assignee: Harish Jaiprakash > Fix For: 3.0.0 > > Attachments: HIVE-17771.01.patch, HIVE-17771.02.patch, > HIVE-17771.03.patch, HIVE-17771.04.patch > > > Please see parent jira about llap workload management. > This jira is to implement create and show resource plan commands in hive to > configure resource plans for llap workload. The following are the proposed > commands implemented as part of the jira: > CREATE RESOURCE PLAN plan_name WITH QUERY_PARALLELISM parallelism; > SHOW RESOURCE PLAN plan_name; > SHOW RESOURCE PLANS; > ALTER RESOURCE PLAN plan_name SET QUERY_PARALLELISM = parallelism; > ALTER RESOURCE PLAN plan_name RENAME TO new_name; > ALTER RESOURCE PLAN plan_name ACTIVATE; > ALTER RESOURCE PLAN plan_name DISABLE; > ALTER RESOURCE PLAN plan_name ENABLE; > DROP RESOURCE PLAN; > It will be followed up with more jiras to manage pools, triggers and copy > resource plans. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17771) Implement commands to manage resource plan
[ https://issues.apache.org/jira/browse/HIVE-17771?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-17771: Summary: Implement commands to manage resource plan (was: Implement commands to manage resource plan.) > Implement commands to manage resource plan > -- > > Key: HIVE-17771 > URL: https://issues.apache.org/jira/browse/HIVE-17771 > Project: Hive > Issue Type: Sub-task >Reporter: Harish Jaiprakash >Assignee: Harish Jaiprakash > Attachments: HIVE-17771.01.patch, HIVE-17771.02.patch, > HIVE-17771.03.patch, HIVE-17771.04.patch > > > Please see parent jira about llap workload management. > This jira is to implement create and show resource plan commands in hive to > configure resource plans for llap workload. The following are the proposed > commands implemented as part of the jira: > CREATE RESOURCE PLAN plan_name WITH QUERY_PARALLELISM parallelism; > SHOW RESOURCE PLAN plan_name; > SHOW RESOURCE PLANS; > ALTER RESOURCE PLAN plan_name SET QUERY_PARALLELISM = parallelism; > ALTER RESOURCE PLAN plan_name RENAME TO new_name; > ALTER RESOURCE PLAN plan_name ACTIVATE; > ALTER RESOURCE PLAN plan_name DISABLE; > ALTER RESOURCE PLAN plan_name ENABLE; > DROP RESOURCE PLAN; > It will be followed up with more jiras to manage pools, triggers and copy > resource plans. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17850) can VectorizedOrcAcidRowReader be removed once HIVE-17458 is done?
[ https://issues.apache.org/jira/browse/HIVE-17850?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16213612#comment-16213612 ] Eugene Koifman commented on HIVE-17850: --- OrcInputFormat.getRecordReader() does some checks based on whether it has OrcSplit or just FileSplit. So does VectorizedOrcAcidRowBatchReader.canCreateVectorizedAcidRowBatchReaderOnSplit(). Under what circumstances would getRecordReader() see FileSplit which is not OrcSplit? VectorizedOrcAcidRowReader is used w/o HIVE-17458. Even with HIVE-17458 it will be used if the split is not OrcSplit. I'm trying to understand if that is actually possible. > can VectorizedOrcAcidRowReader be removed once HIVE-17458 is done? > -- > > Key: HIVE-17850 > URL: https://issues.apache.org/jira/browse/HIVE-17850 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Affects Versions: 3.0.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman >Priority: Minor > > I don't think it will ever even be created once HIVE-17458 is done -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-14152) datanucleus.autoStartMechanismMode should set to 'Ignored' to allow rolling downgrade
[
https://issues.apache.org/jira/browse/HIVE-14152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16213611#comment-16213611
]
Alexander Kolbasov commented on HIVE-14152:
---
I was wrong and the {{datanucleus.autoStartMechanismMode}} is correct one.
> datanucleus.autoStartMechanismMode should set to 'Ignored' to allow rolling
> downgrade
> --
>
> Key: HIVE-14152
> URL: https://issues.apache.org/jira/browse/HIVE-14152
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Reporter: Daniel Dai
>Assignee: Thejas M Nair
> Labels: TODOC2.1.1, TODOC2.2
> Fix For: 2.1.1, 2.2.0
>
> Attachments: HIVE-14152.1.patch, HIVE-14152.2.patch,
> HIVE-14152.3.patch, HIVE-14152.addendum.patch
>
>
> We see the following issue when downgrading metastore:
> 1. Run some query using new tables
> 2. Downgrade metastore
> 3. Restart metastore will complain the new table does not exist
> In particular, constaints tables does not exist in branch-1. If we run Hive 2
> and create a constraint, then downgrade metastore to Hive 1, datanucleus will
> complain:
> {code}
> javax.jdo.JDOFatalUserException: Error starting up DataNucleus : a class
> "org.apache.hadoop.hive.metastore.model.MConstraint" was listed as being
> persisted previously in this datastore, yet the class wasnt found. Perhaps it
> is used by a different DataNucleus-enabled application in this datastore, or
> you have changed your class names.
> at
> org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:528)
> at
> org.datanucleus.api.jdo.JDOPersistenceManagerFactory.freezeConfiguration(JDOPersistenceManagerFactory.java:788)
> at
> org.datanucleus.api.jdo.JDOPersistenceManagerFactory.createPersistenceManagerFactory(JDOPersistenceManagerFactory.java:333)
> at
> org.datanucleus.api.jdo.JDOPersistenceManagerFactory.getPersistenceManagerFactory(JDOPersistenceManagerFactory.java:202)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at javax.jdo.JDOHelper$16.run(JDOHelper.java:1965)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.jdo.JDOHelper.invoke(JDOHelper.java:1960)
> at
> javax.jdo.JDOHelper.invokeGetPersistenceManagerFactoryOnImplementation(JDOHelper.java:1166)
> at javax.jdo.JDOHelper.getPersistenceManagerFactory(JDOHelper.java:808)
> at javax.jdo.JDOHelper.getPersistenceManagerFactory(JDOHelper.java:701)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.getPMF(ObjectStore.java:377)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.getPersistenceManager(ObjectStore.java:406)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.initialize(ObjectStore.java:299)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.setConf(ObjectStore.java:266)
> at
> org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:76)
> at
> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:136)
> at
> org.apache.hadoop.hive.metastore.RawStoreProxy.(RawStoreProxy.java:60)
> at
> org.apache.hadoop.hive.metastore.RawStoreProxy.getProxy(RawStoreProxy.java:69)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.newRawStore(HiveMetaStore.java:650)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMS(HiveMetaStore.java:628)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.createDefaultDB(HiveMetaStore.java:677)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:484)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.(RetryingHMSHandler.java:77)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.getProxy(RetryingHMSHandler.java:83)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore.newRetryingHMSHandler(HiveMetaStore.java:5905)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore.newRetryingHMSHandler(HiveMetaStore.java:5900)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore.startMetaStore(HiveMetaStore.java:6159)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore.main(HiveMetaStore.java:6084)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[jira] [Resolved] (HIVE-17872) Ignoring schema autostart doesn't work (HIVE-14152 used the wrong setting)
[
https://issues.apache.org/jira/browse/HIVE-17872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alexander Kolbasov resolved HIVE-17872.
---
Resolution: Not A Problem
Fix Version/s: 3.0.0
Target Version/s: (was: 3.0.0, 2.4.0)
> Ignoring schema autostart doesn't work (HIVE-14152 used the wrong setting)
> --
>
> Key: HIVE-17872
> URL: https://issues.apache.org/jira/browse/HIVE-17872
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 3.0.0, 2.4.0
>Reporter: Alexander Kolbasov
>Assignee: Alexander Kolbasov
> Fix For: 3.0.0
>
>
> The fix for HIVE-14152 used the wrong datanucleus property. Correct one is
> {{datanucleus.autoStartMechanism}} and the patch uses
> {{datanucleus.autoStartMechanismMode}} which is not supported by DataNucleus.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17872) Ignoring schema autostart doesn't work (HIVE-14152 used the wrong setting)
[
https://issues.apache.org/jira/browse/HIVE-17872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16213610#comment-16213610
]
Alexander Kolbasov commented on HIVE-17872:
---
Actually there are two settings, datanucleus.autoStartMechanism and
datanucleus.autoStartMechanismMode. The mode is the one that should be set to
'ignore'.
The datanucleus.autoStartMechanism should be set to non-null to initialize
auto-start and looks like it should be class name.
So I was wrong and HIVE-14152 is correct.
> Ignoring schema autostart doesn't work (HIVE-14152 used the wrong setting)
> --
>
> Key: HIVE-17872
> URL: https://issues.apache.org/jira/browse/HIVE-17872
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 3.0.0, 2.4.0
>Reporter: Alexander Kolbasov
>Assignee: Alexander Kolbasov
> Fix For: 3.0.0
>
>
> The fix for HIVE-14152 used the wrong datanucleus property. Correct one is
> {{datanucleus.autoStartMechanism}} and the patch uses
> {{datanucleus.autoStartMechanismMode}} which is not supported by DataNucleus.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-17453) Missing ASF headers 2 classes
[ https://issues.apache.org/jira/browse/HIVE-17453?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-17453: Resolution: Fixed Fix Version/s: 3.0.0 Status: Resolved (was: Patch Available) Already fixed via other commits. Thanks, ZhangBing! > Missing ASF headers 2 classes > - > > Key: HIVE-17453 > URL: https://issues.apache.org/jira/browse/HIVE-17453 > Project: Hive > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: ZhangBing Lin >Assignee: ZhangBing Lin >Priority: Trivial > Fix For: 3.0.0 > > Attachments: HIVE-17453.1.patch > > > TestTxnCommandsBase.java and TestTxnNoBuckets.java are missing the ASF header > that should be added. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17850) can VectorizedOrcAcidRowReader be removed once HIVE-17458 is done?
[ https://issues.apache.org/jira/browse/HIVE-17850?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16213592#comment-16213592 ] Sergey Shelukhin commented on HIVE-17850: - Oh, ic. Anyway, I guess it's never used then? I've little context on this > can VectorizedOrcAcidRowReader be removed once HIVE-17458 is done? > -- > > Key: HIVE-17850 > URL: https://issues.apache.org/jira/browse/HIVE-17850 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Affects Versions: 3.0.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman >Priority: Minor > > I don't think it will ever even be created once HIVE-17458 is done -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17850) can VectorizedOrcAcidRowReader be removed once HIVE-17458 is done?
[ https://issues.apache.org/jira/browse/HIVE-17850?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16213589#comment-16213589 ] Eugene Koifman commented on HIVE-17850: --- FileSplit is not OrcSplit > can VectorizedOrcAcidRowReader be removed once HIVE-17458 is done? > -- > > Key: HIVE-17850 > URL: https://issues.apache.org/jira/browse/HIVE-17850 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Affects Versions: 3.0.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman >Priority: Minor > > I don't think it will ever even be created once HIVE-17458 is done -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17771) Implement commands to manage resource plan.
[
https://issues.apache.org/jira/browse/HIVE-17771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16213588#comment-16213588
]
Hive QA commented on HIVE-17771:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12893328/HIVE-17771.04.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 7 failed/errored test(s), 11315 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[optimize_nullscan]
(batchId=163)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_2]
(batchId=101)
org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query39]
(batchId=243)
org.apache.hadoop.hive.cli.control.TestDanglingQOuts.checkDanglingQOut
(batchId=204)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testConstraints
(batchId=221)
org.apache.hadoop.hive.ql.security.authorization.plugin.TestHiveOperationType.checkHiveOperationTypeMatch
(batchId=284)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerHighShuffleBytes
(batchId=228)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/7415/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/7415/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-7415/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 7 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12893328 - PreCommit-HIVE-Build
> Implement commands to manage resource plan.
> ---
>
> Key: HIVE-17771
> URL: https://issues.apache.org/jira/browse/HIVE-17771
> Project: Hive
> Issue Type: Sub-task
>Reporter: Harish Jaiprakash
>Assignee: Harish Jaiprakash
> Attachments: HIVE-17771.01.patch, HIVE-17771.02.patch,
> HIVE-17771.03.patch, HIVE-17771.04.patch
>
>
> Please see parent jira about llap workload management.
> This jira is to implement create and show resource plan commands in hive to
> configure resource plans for llap workload. The following are the proposed
> commands implemented as part of the jira:
> CREATE RESOURCE PLAN plan_name WITH QUERY_PARALLELISM parallelism;
> SHOW RESOURCE PLAN plan_name;
> SHOW RESOURCE PLANS;
> ALTER RESOURCE PLAN plan_name SET QUERY_PARALLELISM = parallelism;
> ALTER RESOURCE PLAN plan_name RENAME TO new_name;
> ALTER RESOURCE PLAN plan_name ACTIVATE;
> ALTER RESOURCE PLAN plan_name DISABLE;
> ALTER RESOURCE PLAN plan_name ENABLE;
> DROP RESOURCE PLAN;
> It will be followed up with more jiras to manage pools, triggers and copy
> resource plans.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17850) can VectorizedOrcAcidRowReader be removed once HIVE-17458 is done?
[ https://issues.apache.org/jira/browse/HIVE-17850?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16213587#comment-16213587 ] Sergey Shelukhin commented on HIVE-17850: - OrcSplit is FileSplit > can VectorizedOrcAcidRowReader be removed once HIVE-17458 is done? > -- > > Key: HIVE-17850 > URL: https://issues.apache.org/jira/browse/HIVE-17850 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Affects Versions: 3.0.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman >Priority: Minor > > I don't think it will ever even be created once HIVE-17458 is done -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-15016) Run tests with Hadoop 3.0.0-beta1
[ https://issues.apache.org/jira/browse/HIVE-15016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-15016: Status: Patch Available (was: Open) > Run tests with Hadoop 3.0.0-beta1 > - > > Key: HIVE-15016 > URL: https://issues.apache.org/jira/browse/HIVE-15016 > Project: Hive > Issue Type: Sub-task > Components: Hive >Reporter: Sergio Peña >Assignee: Aihua Xu > Attachments: HIVE-15016.2.patch, HIVE-15016.3.patch, > HIVE-15016.4.patch, HIVE-15016.5.patch, HIVE-15016.6.patch, HIVE-15016.patch, > Hadoop3Upstream.patch > > > Hadoop 3.0.0-alpha1 was released back on Sep/16 to allow other components run > tests against this new version before GA. > We should start running tests with Hive to validate compatibility against > Hadoop 3.0. > NOTE: The patch used to test must not be committed to Hive until Hadoop 3.0 > GA is released. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-15016) Run tests with Hadoop 3.0.0-beta1
[ https://issues.apache.org/jira/browse/HIVE-15016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-15016: Attachment: HIVE-15016.6.patch With updated tez version. > Run tests with Hadoop 3.0.0-beta1 > - > > Key: HIVE-15016 > URL: https://issues.apache.org/jira/browse/HIVE-15016 > Project: Hive > Issue Type: Sub-task > Components: Hive >Reporter: Sergio Peña >Assignee: Aihua Xu > Attachments: HIVE-15016.2.patch, HIVE-15016.3.patch, > HIVE-15016.4.patch, HIVE-15016.5.patch, HIVE-15016.6.patch, HIVE-15016.patch, > Hadoop3Upstream.patch > > > Hadoop 3.0.0-alpha1 was released back on Sep/16 to allow other components run > tests against this new version before GA. > We should start running tests with Hive to validate compatibility against > Hadoop 3.0. > NOTE: The patch used to test must not be committed to Hive until Hadoop 3.0 > GA is released. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-15016) Run tests with Hadoop 3.0.0-beta1
[ https://issues.apache.org/jira/browse/HIVE-15016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-15016: Status: Open (was: Patch Available) > Run tests with Hadoop 3.0.0-beta1 > - > > Key: HIVE-15016 > URL: https://issues.apache.org/jira/browse/HIVE-15016 > Project: Hive > Issue Type: Sub-task > Components: Hive >Reporter: Sergio Peña >Assignee: Aihua Xu > Attachments: HIVE-15016.2.patch, HIVE-15016.3.patch, > HIVE-15016.4.patch, HIVE-15016.5.patch, HIVE-15016.patch, > Hadoop3Upstream.patch > > > Hadoop 3.0.0-alpha1 was released back on Sep/16 to allow other components run > tests against this new version before GA. > We should start running tests with Hive to validate compatibility against > Hadoop 3.0. > NOTE: The patch used to test must not be committed to Hive until Hadoop 3.0 > GA is released. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17850) can VectorizedOrcAcidRowReader be removed once HIVE-17458 is done?
[ https://issues.apache.org/jira/browse/HIVE-17850?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16213579#comment-16213579 ] Eugene Koifman commented on HIVE-17850: --- OrcInputFormat uses this when the split is FileSplit but not OrcSplit - when does this even happen? > can VectorizedOrcAcidRowReader be removed once HIVE-17458 is done? > -- > > Key: HIVE-17850 > URL: https://issues.apache.org/jira/browse/HIVE-17850 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Affects Versions: 3.0.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman >Priority: Minor > > I don't think it will ever even be created once HIVE-17458 is done -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17458) VectorizedOrcAcidRowBatchReader doesn't handle 'original' files
[ https://issues.apache.org/jira/browse/HIVE-17458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16213578#comment-16213578 ] Eugene Koifman commented on HIVE-17458: --- patch 4 adds support for delete events need a test with multiple stripes, ppd, etc - make sure ids assigned correctly test make sure compactor assigns them the same way disable LLAP cache for original read that needs ROW__IDs > VectorizedOrcAcidRowBatchReader doesn't handle 'original' files > --- > > Key: HIVE-17458 > URL: https://issues.apache.org/jira/browse/HIVE-17458 > Project: Hive > Issue Type: Improvement >Affects Versions: 2.2.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman >Priority: Critical > Attachments: HIVE-17458.01.patch, HIVE-17458.02.patch, > HIVE-17458.03.patch, HIVE-17458.04.patch > > > VectorizedOrcAcidRowBatchReader will not be used for original files. This > will likely look like a perf regression when converting a table from non-acid > to acid until it runs through a major compaction. > With Load Data support, if large files are added via Load Data, the read ops > will not vectorize until major compaction. > There is no reason why this should be the case. Just like > OrcRawRecordMerger, VectorizedOrcAcidRowBatchReader can look at the other > files in the logical tranche/bucket and calculate the offset for the RowBatch > of the split. (Presumably getRecordReader().getRowNumber() works the same in > vector mode). > In this case we don't even need OrcSplit.isOriginal() - the reader can infer > it from file path... which in particular simplifies > OrcInputFormat.determineSplitStrategies() -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17458) VectorizedOrcAcidRowBatchReader doesn't handle 'original' files
[ https://issues.apache.org/jira/browse/HIVE-17458?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-17458: -- Attachment: HIVE-17458.04.patch > VectorizedOrcAcidRowBatchReader doesn't handle 'original' files > --- > > Key: HIVE-17458 > URL: https://issues.apache.org/jira/browse/HIVE-17458 > Project: Hive > Issue Type: Improvement >Affects Versions: 2.2.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman >Priority: Critical > Attachments: HIVE-17458.01.patch, HIVE-17458.02.patch, > HIVE-17458.03.patch, HIVE-17458.04.patch > > > VectorizedOrcAcidRowBatchReader will not be used for original files. This > will likely look like a perf regression when converting a table from non-acid > to acid until it runs through a major compaction. > With Load Data support, if large files are added via Load Data, the read ops > will not vectorize until major compaction. > There is no reason why this should be the case. Just like > OrcRawRecordMerger, VectorizedOrcAcidRowBatchReader can look at the other > files in the logical tranche/bucket and calculate the offset for the RowBatch > of the split. (Presumably getRecordReader().getRowNumber() works the same in > vector mode). > In this case we don't even need OrcSplit.isOriginal() - the reader can infer > it from file path... which in particular simplifies > OrcInputFormat.determineSplitStrategies() -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17684) HoS memory issues with MapJoinMemoryExhaustionHandler
[
https://issues.apache.org/jira/browse/HIVE-17684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16213573#comment-16213573
]
Sahil Takiar commented on HIVE-17684:
-
[~csun], [~xuefuz] have you seen any issues related to this? We see it happen
pretty often; the {{MapJoinMemoryExhaustionHandler}} throws an exception and
the task fails, even though a simple GC call would reclaim a lot of heap space.
Usually task retries solve the issue, but occasionally the job fails.
> HoS memory issues with MapJoinMemoryExhaustionHandler
> -
>
> Key: HIVE-17684
> URL: https://issues.apache.org/jira/browse/HIVE-17684
> Project: Hive
> Issue Type: Bug
> Components: Spark
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>
> We have seen a number of memory issues due the {{HashSinkOperator}} use of
> the {{MapJoinMemoryExhaustionHandler}}. This handler is meant to detect
> scenarios where the small table is taking too much space in memory, in which
> case a {{MapJoinMemoryExhaustionError}} is thrown.
> The configs to control this logic are:
> {{hive.mapjoin.localtask.max.memory.usage}} (default 0.90)
> {{hive.mapjoin.followby.gby.localtask.max.memory.usage}} (default 0.55)
> The handler works by using the {{MemoryMXBean}} and uses the following logic
> to estimate how much memory the {{HashMap}} is consuming:
> {{MemoryMXBean#getHeapMemoryUsage().getUsed() /
> MemoryMXBean#getHeapMemoryUsage().getMax()}}
> The issue is that {{MemoryMXBean#getHeapMemoryUsage().getUsed()}} can be
> inaccurate. The value returned by this method returns all reachable and
> unreachable memory on the heap, so there may be a bunch of garbage data, and
> the JVM just hasn't taken the time to reclaim it all. This can lead to
> intermittent failures of this check even though a simple GC would have
> reclaimed enough space for the process to continue working.
> We should re-think the usage of {{MapJoinMemoryExhaustionHandler}} for HoS.
> In Hive-on-MR this probably made sense to use because every Hive task was run
> in a dedicated container, so a Hive Task could assume it created most of the
> data on the heap. However, in Hive-on-Spark there can be multiple Hive Tasks
> running in a single executor, each doing different things.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17856) MM tables - IOW is not ACID compliant
[
https://issues.apache.org/jira/browse/HIVE-17856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16213569#comment-16213569
]
Steve Yeom commented on HIVE-17856:
---
Talked with Sergey.
Sergey indicated that the current IOW execution code/path for MM tables is not
ACID compliant
because first it deletes directories as mentioned above by an IOW
writer/updater.
For one case, to support Snapshot isolation by concurrent open reader
transactions, we should not delete them with
the writers when possibly open transactions may exist, but through a mechanism
which is sync with open transactions,
as in the case of ACID tables, where delta directories are merged/deleted by
compactors with TxnList
with open and aborted xacts.
> MM tables - IOW is not ACID compliant
> -
>
> Key: HIVE-17856
> URL: https://issues.apache.org/jira/browse/HIVE-17856
> Project: Hive
> Issue Type: Sub-task
> Components: Transactions
>Reporter: Sergey Shelukhin
>Assignee: Steve Yeom
> Labels: mm-gap-1
>
> The following tests were removed from mm_all during "integration"... I should
> have never allowed such manner of intergration.
> MM logic should have been kept intact until ACID logic could catch up. Alas,
> here we are.
> {noformat}
> drop table iow0_mm;
> create table iow0_mm(key int) tblproperties("transactional"="true",
> "transactional_properties"="insert_only");
> insert overwrite table iow0_mm select key from intermediate;
> insert into table iow0_mm select key + 1 from intermediate;
> select * from iow0_mm order by key;
> insert overwrite table iow0_mm select key + 2 from intermediate;
> select * from iow0_mm order by key;
> drop table iow0_mm;
> drop table iow1_mm;
> create table iow1_mm(key int) partitioned by (key2 int)
> tblproperties("transactional"="true",
> "transactional_properties"="insert_only");
> insert overwrite table iow1_mm partition (key2)
> select key as k1, key from intermediate union all select key as k1, key from
> intermediate;
> insert into table iow1_mm partition (key2)
> select key + 1 as k1, key from intermediate union all select key as k1, key
> from intermediate;
> select * from iow1_mm order by key, key2;
> insert overwrite table iow1_mm partition (key2)
> select key + 3 as k1, key from intermediate union all select key + 4 as k1,
> key from intermediate;
> select * from iow1_mm order by key, key2;
> insert overwrite table iow1_mm partition (key2)
> select key + 3 as k1, key + 3 from intermediate union all select key + 2 as
> k1, key + 2 from intermediate;
> select * from iow1_mm order by key, key2;
> drop table iow1_mm;
> {noformat}
> {noformat}
> drop table simple_mm;
> create table simple_mm(key int) stored as orc tblproperties
> ("transactional"="true", "transactional_properties"="insert_only");
> insert into table simple_mm select key from intermediate;
> -insert overwrite table simple_mm select key from intermediate;
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17871) Add non nullability flag to druid time column
[
https://issues.apache.org/jira/browse/HIVE-17871?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16213570#comment-16213570
]
Ashutosh Chauhan commented on HIVE-17871:
-
[~bslim] Worth adding a qtest for this.
> Add non nullability flag to druid time column
> -
>
> Key: HIVE-17871
> URL: https://issues.apache.org/jira/browse/HIVE-17871
> Project: Hive
> Issue Type: Improvement
> Components: Druid integration
>Reporter: slim bouguerra
>Assignee: Jesus Camacho Rodriguez
> Attachments: HIVE-17871.patch
>
>
> Druid time column is non null all the time.
> Adding the non nullability flag will enable extra calcite goodness like
> transforming
> {code} select count(`__time`) from table {code} to {code} select count(*)
> from table {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Assigned] (HIVE-17871) Add non nullability flag to druid time column
[
https://issues.apache.org/jira/browse/HIVE-17871?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
slim bouguerra reassigned HIVE-17871:
-
Assignee: Jesus Camacho Rodriguez
> Add non nullability flag to druid time column
> -
>
> Key: HIVE-17871
> URL: https://issues.apache.org/jira/browse/HIVE-17871
> Project: Hive
> Issue Type: Improvement
> Components: Druid integration
>Reporter: slim bouguerra
>Assignee: Jesus Camacho Rodriguez
> Attachments: HIVE-17871.patch
>
>
> Druid time column is non null all the time.
> Adding the non nullability flag will enable extra calcite goodness like
> transforming
> {code} select count(`__time`) from table {code} to {code} select count(*)
> from table {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-17240) Function ACOS(2) and ASIN(2) should be null
[
https://issues.apache.org/jira/browse/HIVE-17240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashutosh Chauhan updated HIVE-17240:
Resolution: Fixed
Fix Version/s: 3.0.0
Status: Resolved (was: Patch Available)
Pushed to master. Thanks, Yuming!
> Function ACOS(2) and ASIN(2) should be null
> ---
>
> Key: HIVE-17240
> URL: https://issues.apache.org/jira/browse/HIVE-17240
> Project: Hive
> Issue Type: Bug
> Components: UDF
>Affects Versions: 1.1.1, 1.2.2, 2.2.0
>Reporter: Yuming Wang
>Assignee: Yuming Wang
> Fix For: 3.0.0
>
> Attachments: HIVE-17240.1.patch, HIVE-17240.2.patch,
> HIVE-17240.3.patch, HIVE-17240.4.patch, HIVE-17240.5.patch, HIVE-17240.6.patch
>
>
> {{acos(2)}} should be null, same as MySQL:
> {code:sql}
> hive> desc function extended acos;
> OK
> acos(x) - returns the arc cosine of x if -1<=x<=1 or NULL otherwise
> Example:
> > SELECT acos(1) FROM src LIMIT 1;
> 0
> > SELECT acos(2) FROM src LIMIT 1;
> NULL
> Time taken: 0.009 seconds, Fetched: 6 row(s)
> hive> select acos(2);
> OK
> NaN
> Time taken: 0.437 seconds, Fetched: 1 row(s)
> {code}
> {code:sql}
> mysql> select acos(2);
> +-+
> | acos(2) |
> +-+
> |NULL |
> +-+
> 1 row in set (0.00 sec)
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17872) Ignoring schema autostart doesn't work (HIVE-14152 used the wrong setting)
[
https://issues.apache.org/jira/browse/HIVE-17872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16213558#comment-16213558
]
Alexander Kolbasov commented on HIVE-17872:
---
[~vihangk1] [~stakiar] FYI
> Ignoring schema autostart doesn't work (HIVE-14152 used the wrong setting)
> --
>
> Key: HIVE-17872
> URL: https://issues.apache.org/jira/browse/HIVE-17872
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 3.0.0, 2.4.0
>Reporter: Alexander Kolbasov
>Assignee: Alexander Kolbasov
>
> The fix for HIVE-14152 used the wrong datanucleus property. Correct one is
> {{datanucleus.autoStartMechanism}} and the patch uses
> {{datanucleus.autoStartMechanismMode}} which is not supported by DataNucleus.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-14152) datanucleus.autoStartMechanismMode should set to 'Ignored' to allow rolling downgrade
[
https://issues.apache.org/jira/browse/HIVE-14152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16213544#comment-16213544
]
Alexander Kolbasov commented on HIVE-14152:
---
This patch uses the wrong configuration string. See HIVE-17872 for more
information.
> datanucleus.autoStartMechanismMode should set to 'Ignored' to allow rolling
> downgrade
> --
>
> Key: HIVE-14152
> URL: https://issues.apache.org/jira/browse/HIVE-14152
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Reporter: Daniel Dai
>Assignee: Thejas M Nair
> Labels: TODOC2.1.1, TODOC2.2
> Fix For: 2.1.1, 2.2.0
>
> Attachments: HIVE-14152.1.patch, HIVE-14152.2.patch,
> HIVE-14152.3.patch, HIVE-14152.addendum.patch
>
>
> We see the following issue when downgrading metastore:
> 1. Run some query using new tables
> 2. Downgrade metastore
> 3. Restart metastore will complain the new table does not exist
> In particular, constaints tables does not exist in branch-1. If we run Hive 2
> and create a constraint, then downgrade metastore to Hive 1, datanucleus will
> complain:
> {code}
> javax.jdo.JDOFatalUserException: Error starting up DataNucleus : a class
> "org.apache.hadoop.hive.metastore.model.MConstraint" was listed as being
> persisted previously in this datastore, yet the class wasnt found. Perhaps it
> is used by a different DataNucleus-enabled application in this datastore, or
> you have changed your class names.
> at
> org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:528)
> at
> org.datanucleus.api.jdo.JDOPersistenceManagerFactory.freezeConfiguration(JDOPersistenceManagerFactory.java:788)
> at
> org.datanucleus.api.jdo.JDOPersistenceManagerFactory.createPersistenceManagerFactory(JDOPersistenceManagerFactory.java:333)
> at
> org.datanucleus.api.jdo.JDOPersistenceManagerFactory.getPersistenceManagerFactory(JDOPersistenceManagerFactory.java:202)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at javax.jdo.JDOHelper$16.run(JDOHelper.java:1965)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.jdo.JDOHelper.invoke(JDOHelper.java:1960)
> at
> javax.jdo.JDOHelper.invokeGetPersistenceManagerFactoryOnImplementation(JDOHelper.java:1166)
> at javax.jdo.JDOHelper.getPersistenceManagerFactory(JDOHelper.java:808)
> at javax.jdo.JDOHelper.getPersistenceManagerFactory(JDOHelper.java:701)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.getPMF(ObjectStore.java:377)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.getPersistenceManager(ObjectStore.java:406)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.initialize(ObjectStore.java:299)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.setConf(ObjectStore.java:266)
> at
> org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:76)
> at
> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:136)
> at
> org.apache.hadoop.hive.metastore.RawStoreProxy.(RawStoreProxy.java:60)
> at
> org.apache.hadoop.hive.metastore.RawStoreProxy.getProxy(RawStoreProxy.java:69)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.newRawStore(HiveMetaStore.java:650)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMS(HiveMetaStore.java:628)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.createDefaultDB(HiveMetaStore.java:677)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:484)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.(RetryingHMSHandler.java:77)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.getProxy(RetryingHMSHandler.java:83)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore.newRetryingHMSHandler(HiveMetaStore.java:5905)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore.newRetryingHMSHandler(HiveMetaStore.java:5900)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore.startMetaStore(HiveMetaStore.java:6159)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore.main(HiveMetaStore.java:6084)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
>
[jira] [Assigned] (HIVE-17872) Ignoring schema autostart doesn't work (HIVE-14152 used the wrong setting)
[
https://issues.apache.org/jira/browse/HIVE-17872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alexander Kolbasov reassigned HIVE-17872:
-
> Ignoring schema autostart doesn't work (HIVE-14152 used the wrong setting)
> --
>
> Key: HIVE-17872
> URL: https://issues.apache.org/jira/browse/HIVE-17872
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 3.0.0, 2.4.0
>Reporter: Alexander Kolbasov
>Assignee: Alexander Kolbasov
>
> The fix for HIVE-14152 used the wrong datanucleus property. Correct one is
> {{datanucleus.autoStartMechanism}} and the patch uses
> {{datanucleus.autoStartMechanismMode}} which is not supported by DataNucleus.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-17871) Add non nullability flag to druid time column
[
https://issues.apache.org/jira/browse/HIVE-17871?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
slim bouguerra updated HIVE-17871:
--
Attachment: HIVE-17871.patch
> Add non nullability flag to druid time column
> -
>
> Key: HIVE-17871
> URL: https://issues.apache.org/jira/browse/HIVE-17871
> Project: Hive
> Issue Type: Improvement
> Components: Druid integration
>Reporter: slim bouguerra
> Attachments: HIVE-17871.patch
>
>
> Druid time column is non null all the time.
> Adding the non nullability flag will enable extra calcite goodness like
> transforming
> {code} select count(`__time`) from table {code} to {code} select count(*)
> from table {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-17871) Add non nullability flag to druid time column
[
https://issues.apache.org/jira/browse/HIVE-17871?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
slim bouguerra updated HIVE-17871:
--
Status: Patch Available (was: Open)
> Add non nullability flag to druid time column
> -
>
> Key: HIVE-17871
> URL: https://issues.apache.org/jira/browse/HIVE-17871
> Project: Hive
> Issue Type: Improvement
> Components: Druid integration
>Reporter: slim bouguerra
>
> Druid time column is non null all the time.
> Adding the non nullability flag will enable extra calcite goodness like
> transforming
> {code} select count(`__time`) from table {code} to {code} select count(*)
> from table {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Comment Edited] (HIVE-17809) Implement per pool trigger validation
[ https://issues.apache.org/jira/browse/HIVE-17809?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16213526#comment-16213526 ] Prasanth Jayachandran edited comment on HIVE-17809 at 10/20/17 11:55 PM: - [~sershe] patch adds per pool triggers, move to pool and action prioritization. Kept the action prioritization simple for now as it is easy to reason about. Patch should have comments about the priorities. Can you please review the patch? was (Author: prasanth_j): [~sershe] patch adds per pool triggers, move to pool and action prioritization. Kept the action prioritization simple for now as it is easy to reason about. Patch should have comments about the priorities. > Implement per pool trigger validation > - > > Key: HIVE-17809 > URL: https://issues.apache.org/jira/browse/HIVE-17809 > Project: Hive > Issue Type: Sub-task >Affects Versions: 3.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran > Attachments: HIVE-17809.1.patch > > > HIVE-17508 trigger validation is applied for all pools at once. This is > follow up to implement trigger validation at per pool level. > This should also implement resolution for multiple applicable actions, as per > the RB discussion -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17743) Add InterfaceAudience and InterfaceStability annotations for Thrift generated APIs
[
https://issues.apache.org/jira/browse/HIVE-17743?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16213528#comment-16213528
]
Sahil Takiar commented on HIVE-17743:
-
[~aihuaxu] can you take a look?
> Add InterfaceAudience and InterfaceStability annotations for Thrift generated
> APIs
> --
>
> Key: HIVE-17743
> URL: https://issues.apache.org/jira/browse/HIVE-17743
> Project: Hive
> Issue Type: Sub-task
> Components: Thrift API
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
> Attachments: HIVE-17743.1.patch, HIVE-17743.2.patch
>
>
> The Thrift generated files don't have {{InterfaceAudience}} or
> {{InterfaceStability}} annotations on them, mainly because all the files are
> auto-generated.
> We should add some code that auto-tags all the Java Thrift generated files
> with these annotations. This way even when they are re-generated, they still
> contain the annotations.
> We should be able to do this using the
> {{com.google.code.maven-replacer-plugin}} similar to what we do in
> {{standalone-metastore/pom.xml}}.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-17809) Implement per pool trigger validation
[ https://issues.apache.org/jira/browse/HIVE-17809?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-17809: - Status: Patch Available (was: Open) > Implement per pool trigger validation > - > > Key: HIVE-17809 > URL: https://issues.apache.org/jira/browse/HIVE-17809 > Project: Hive > Issue Type: Sub-task >Affects Versions: 3.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran > Attachments: HIVE-17809.1.patch > > > HIVE-17508 trigger validation is applied for all pools at once. This is > follow up to implement trigger validation at per pool level. > This should also implement resolution for multiple applicable actions, as per > the RB discussion -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17809) Implement per pool trigger validation
[ https://issues.apache.org/jira/browse/HIVE-17809?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-17809: - Attachment: HIVE-17809.1.patch > Implement per pool trigger validation > - > > Key: HIVE-17809 > URL: https://issues.apache.org/jira/browse/HIVE-17809 > Project: Hive > Issue Type: Sub-task >Affects Versions: 3.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran > Attachments: HIVE-17809.1.patch > > > HIVE-17508 trigger validation is applied for all pools at once. This is > follow up to implement trigger validation at per pool level. > This should also implement resolution for multiple applicable actions, as per > the RB discussion -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17258) Incorrect log messages in the Hive.java
[ https://issues.apache.org/jira/browse/HIVE-17258?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-17258: Resolution: Fixed Assignee: Oleg Danilov Fix Version/s: 3.0.0 Status: Resolved (was: Patch Available) Pushed to master. Thanks, Oleg! > Incorrect log messages in the Hive.java > --- > > Key: HIVE-17258 > URL: https://issues.apache.org/jira/browse/HIVE-17258 > Project: Hive > Issue Type: Bug >Reporter: Oleg Danilov >Assignee: Oleg Danilov >Priority: Trivial > Fix For: 3.0.0 > > Attachments: HIVE-17258.patch > > > There are couple of typos when using LOG methods in the > org.apache.hadoop.hive.ql.metadata.Hive class. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17240) Function ACOS(2) and ASIN(2) should be null
[
https://issues.apache.org/jira/browse/HIVE-17240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16213473#comment-16213473
]
Ashutosh Chauhan commented on HIVE-17240:
-
+1
> Function ACOS(2) and ASIN(2) should be null
> ---
>
> Key: HIVE-17240
> URL: https://issues.apache.org/jira/browse/HIVE-17240
> Project: Hive
> Issue Type: Bug
> Components: UDF
>Affects Versions: 1.1.1, 1.2.2, 2.2.0
>Reporter: Yuming Wang
>Assignee: Yuming Wang
> Attachments: HIVE-17240.1.patch, HIVE-17240.2.patch,
> HIVE-17240.3.patch, HIVE-17240.4.patch, HIVE-17240.5.patch, HIVE-17240.6.patch
>
>
> {{acos(2)}} should be null, same as MySQL:
> {code:sql}
> hive> desc function extended acos;
> OK
> acos(x) - returns the arc cosine of x if -1<=x<=1 or NULL otherwise
> Example:
> > SELECT acos(1) FROM src LIMIT 1;
> 0
> > SELECT acos(2) FROM src LIMIT 1;
> NULL
> Time taken: 0.009 seconds, Fetched: 6 row(s)
> hive> select acos(2);
> OK
> NaN
> Time taken: 0.437 seconds, Fetched: 1 row(s)
> {code}
> {code:sql}
> mysql> select acos(2);
> +-+
> | acos(2) |
> +-+
> |NULL |
> +-+
> 1 row in set (0.00 sec)
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17696) Vectorized reader does not seem to be pushing down projection columns in certain code paths
[
https://issues.apache.org/jira/browse/HIVE-17696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16213467#comment-16213467
]
Vihang Karajgaonkar commented on HIVE-17696:
+1 Thanks for the patch [~Ferd]
> Vectorized reader does not seem to be pushing down projection columns in
> certain code paths
> ---
>
> Key: HIVE-17696
> URL: https://issues.apache.org/jira/browse/HIVE-17696
> Project: Hive
> Issue Type: Sub-task
>Reporter: Vihang Karajgaonkar
>Assignee: Ferdinand Xu
> Attachments: HIVE-17696.patch
>
>
> This is the code snippet from {{VectorizedParquetRecordReader.java}}
> {noformat}
> MessageType tableSchema;
> if (indexAccess) {
> List indexSequence = new ArrayList<>();
> // Generates a sequence list of indexes
> for(int i = 0; i < columnNamesList.size(); i++) {
> indexSequence.add(i);
> }
> tableSchema = DataWritableReadSupport.getSchemaByIndex(fileSchema,
> columnNamesList,
> indexSequence);
> } else {
> tableSchema = DataWritableReadSupport.getSchemaByName(fileSchema,
> columnNamesList,
> columnTypesList);
> }
> indexColumnsWanted =
> ColumnProjectionUtils.getReadColumnIDs(configuration);
> if (!ColumnProjectionUtils.isReadAllColumns(configuration) &&
> !indexColumnsWanted.isEmpty()) {
> requestedSchema =
> DataWritableReadSupport.getSchemaByIndex(tableSchema,
> columnNamesList, indexColumnsWanted);
> } else {
> requestedSchema = fileSchema;
> }
> this.reader = new ParquetFileReader(
> configuration, footer.getFileMetaData(), file, blocks,
> requestedSchema.getColumns());
> {noformat}
> Couple of things to notice here:
> Most of this code is duplicated from {{DataWritableReadSupport.init()}}
> method.
> the else condition passes in fileSchema instead of using tableSchema like we
> do in DataWritableReadSupport.init() method. Does this cause projection
> columns to be missed when we read parquet files? We should probably just
> reuse ReadContext returned from {{DataWritableReadSupport.init()}} method
> here.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17240) Function ACOS(2) and ASIN(2) should be null
[
https://issues.apache.org/jira/browse/HIVE-17240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16213459#comment-16213459
]
Ashutosh Chauhan commented on HIVE-17240:
-
+1
> Function ACOS(2) and ASIN(2) should be null
> ---
>
> Key: HIVE-17240
> URL: https://issues.apache.org/jira/browse/HIVE-17240
> Project: Hive
> Issue Type: Bug
> Components: UDF
>Affects Versions: 1.1.1, 1.2.2, 2.2.0
>Reporter: Yuming Wang
>Assignee: Yuming Wang
> Attachments: HIVE-17240.1.patch, HIVE-17240.2.patch,
> HIVE-17240.3.patch, HIVE-17240.4.patch, HIVE-17240.5.patch, HIVE-17240.6.patch
>
>
> {{acos(2)}} should be null, same as MySQL:
> {code:sql}
> hive> desc function extended acos;
> OK
> acos(x) - returns the arc cosine of x if -1<=x<=1 or NULL otherwise
> Example:
> > SELECT acos(1) FROM src LIMIT 1;
> 0
> > SELECT acos(2) FROM src LIMIT 1;
> NULL
> Time taken: 0.009 seconds, Fetched: 6 row(s)
> hive> select acos(2);
> OK
> NaN
> Time taken: 0.437 seconds, Fetched: 1 row(s)
> {code}
> {code:sql}
> mysql> select acos(2);
> +-+
> | acos(2) |
> +-+
> |NULL |
> +-+
> 1 row in set (0.00 sec)
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-17867) Exception in windowing functions with TIMESTAMP WITH LOCAL TIME ZONE type
[
https://issues.apache.org/jira/browse/HIVE-17867?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesus Camacho Rodriguez updated HIVE-17867:
---
Attachment: HIVE-17867.patch
> Exception in windowing functions with TIMESTAMP WITH LOCAL TIME ZONE type
> -
>
> Key: HIVE-17867
> URL: https://issues.apache.org/jira/browse/HIVE-17867
> Project: Hive
> Issue Type: Bug
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
> Attachments: HIVE-17867.patch
>
>
> The following query where column {{ts}} is of type {{TIMESTAMP WITH LOCAL
> TIME ZONE}}:
> {code}
> select ts, i, sum(f) over (partition by i order by ts)
> from over10k_2
> limit 100;
> {code}
> fails with the following stacktrace:
> {code}
> org.apache.hadoop.hive.ql.parse.SemanticException: Failed to breakup
> Windowing invocations into Groups. At least 1 group must only depend on input
> columns. Also check for circular dependencies.
> Underlying error: Primitive type TIMESTAMPLOCALTZ not supported in Value
> Boundary expression
> at
> org.apache.hadoop.hive.ql.parse.WindowingComponentizer.next(WindowingComponentizer.java:97)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genWindowingPlan(SemanticAnalyzer.java:13508)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPostGroupByBodyPlan(SemanticAnalyzer.java:9912)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genBodyPlan(SemanticAnalyzer.java:9871)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:10784)
> ...
> {code}
> The list of supported types for boundaries expressions in PTFTranslator needs
> to be updated.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17304) ThreadMXBean based memory allocation monitory for hash table loader
[ https://issues.apache.org/jira/browse/HIVE-17304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16213456#comment-16213456 ] Prasanth Jayachandran commented on HIVE-17304: -- No. This needs further testing. I need to do some in-depth analysis to see which one is more accurate to the actual value based on heapdumps. So it's not ready for commit yet. > ThreadMXBean based memory allocation monitory for hash table loader > --- > > Key: HIVE-17304 > URL: https://issues.apache.org/jira/browse/HIVE-17304 > Project: Hive > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran > Attachments: HIVE-17304.1.patch, HIVE-17304.2.patch > > > Hash table memory monitoring is based on java data model which can be > unreliable because of various reasons (wrong object size estimation, adding > new variables to any class without accounting its size for memory monitoring, > etc.). We can use allocation size per thread that is provided by ThreadMXBean > and fallback to DataModel in case if JDK doesn't support thread based > allocations. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17856) MM tables - IOW is not ACID compliant
[
https://issues.apache.org/jira/browse/HIVE-17856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16213453#comment-16213453
]
Sergey Shelukhin commented on HIVE-17856:
-
Ok I noticed this while looking at the union bug. I think the reason IOW
"worked for me" in some of the tests and works for [~steveyeom2017] when he
runs the above test is that it's still using old MM logic that is not ACID
compliant.
This call in Hive.java:
{noformat}
deleteOldPathForReplace(newPartPath, oldPartPath, getConf(), isAutoPurge,
new JavaUtils.IdPathFilter(txnId, stmtId, false, true), true,
tbl.isStoredAsSubDirectories() ? tbl.getSkewedColNames().size() :
0);
{noformat}
Deletes old delta directories by means of IdPathFilter with 3rd arg being
false, which means, return ALL delta directories that don't match txnId.
So, all the other data in the table gets nuked.
This is the implementation is incorrect for ACID integration.
So, delete for MM table codepath needs to be removed (easy to find by looking
where IdPathFilter is used with isMatch == false, meaning "find every txn
except this one").
Then, IOW will probably stop working w.r.t. "overwrite" because old deltas will
stick around.
After that Eugene can comment on where and how ACID uses base directories to
implement IOW.
I suspect that in IOW case, it will be as simple as instead of creating
delta_ dir for output, creating base_ directory; and that would be
enough for all the logic shared with ACID, e.g. compactor, to handle it
correctly. The code that finds what to read in HiveInputFormat would also need
to be updated to take committed base-s into account. But I am not familiar with
ACID IOW, so it may not be as simple.
cc [~hagleitn] [~steveyeom2017] [~ekoifman]
> MM tables - IOW is not ACID compliant
> -
>
> Key: HIVE-17856
> URL: https://issues.apache.org/jira/browse/HIVE-17856
> Project: Hive
> Issue Type: Sub-task
> Components: Transactions
>Reporter: Sergey Shelukhin
>Assignee: Steve Yeom
> Labels: mm-gap-1
>
> The following tests were removed from mm_all during "integration"... I should
> have never allowed such manner of intergration.
> MM logic should have been kept intact until ACID logic could catch up. Alas,
> here we are.
> {noformat}
> drop table iow0_mm;
> create table iow0_mm(key int) tblproperties("transactional"="true",
> "transactional_properties"="insert_only");
> insert overwrite table iow0_mm select key from intermediate;
> insert into table iow0_mm select key + 1 from intermediate;
> select * from iow0_mm order by key;
> insert overwrite table iow0_mm select key + 2 from intermediate;
> select * from iow0_mm order by key;
> drop table iow0_mm;
> drop table iow1_mm;
> create table iow1_mm(key int) partitioned by (key2 int)
> tblproperties("transactional"="true",
> "transactional_properties"="insert_only");
> insert overwrite table iow1_mm partition (key2)
> select key as k1, key from intermediate union all select key as k1, key from
> intermediate;
> insert into table iow1_mm partition (key2)
> select key + 1 as k1, key from intermediate union all select key as k1, key
> from intermediate;
> select * from iow1_mm order by key, key2;
> insert overwrite table iow1_mm partition (key2)
> select key + 3 as k1, key from intermediate union all select key + 4 as k1,
> key from intermediate;
> select * from iow1_mm order by key, key2;
> insert overwrite table iow1_mm partition (key2)
> select key + 3 as k1, key + 3 from intermediate union all select key + 2 as
> k1, key + 2 from intermediate;
> select * from iow1_mm order by key, key2;
> drop table iow1_mm;
> {noformat}
> {noformat}
> drop table simple_mm;
> create table simple_mm(key int) stored as orc tblproperties
> ("transactional"="true", "transactional_properties"="insert_only");
> insert into table simple_mm select key from intermediate;
> -insert overwrite table simple_mm select key from intermediate;
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17258) Incorrect log messages in the Hive.java
[ https://issues.apache.org/jira/browse/HIVE-17258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16213452#comment-16213452 ] Ashutosh Chauhan commented on HIVE-17258: - +1 > Incorrect log messages in the Hive.java > --- > > Key: HIVE-17258 > URL: https://issues.apache.org/jira/browse/HIVE-17258 > Project: Hive > Issue Type: Bug >Reporter: Oleg Danilov >Priority: Trivial > Attachments: HIVE-17258.patch > > > There are couple of typos when using LOG methods in the > org.apache.hadoop.hive.ql.metadata.Hive class. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17259) Hive JDBC does not recognize UNIONTYPE columns
[
https://issues.apache.org/jira/browse/HIVE-17259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16213449#comment-16213449
]
Ashutosh Chauhan commented on HIVE-17259:
-
[~pvillard] Can you please attach a patch here so our unit tests can run on it?
See:
https://cwiki.apache.org/confluence/display/Hive/Hive+PreCommit+Patch+Testing
> Hive JDBC does not recognize UNIONTYPE columns
> --
>
> Key: HIVE-17259
> URL: https://issues.apache.org/jira/browse/HIVE-17259
> Project: Hive
> Issue Type: Bug
> Components: Beeline, JDBC
> Environment: Hive 1.2.1000.2.6.1.0-129
> Beeline version 1.2.1000.2.6.1.0-129 by Apache Hive
>Reporter: Pierre Villard
>Assignee: Pierre Villard
>
> Hive JDBC does not recognize UNIONTYPE columns.
> I've an external table backed by an avro schema containing a union type field.
> {noformat}
> "name" : "value",
> "type" : [ "int", "string", "null" ]
> {noformat}
> When describing the table I've:
> {noformat}
> describe test_table;
> +---+---+--+--+
> | col_name | data_type
> | comment |
> +---+---+--+--+
> | description | string
> | |
> | name | string
> | |
> | value | uniontype
> | |
> +---+---+--+--+
> {noformat}
> When doing a select query over the data using the Hive CLI, it works:
> {noformat}
> hive> select value from test_table;
> OK
> {0:10}
> {0:10}
> {0:9}
> {0:9}
> ...
> {noformat}
> But when using beeline, it fails:
> {noformat}
> 0: jdbc:hive2://> select * from test_table;
> Error: Unrecognized column type: UNIONTYPE (state=,code=0)
> {noformat}
> By applying the patch provided with this JIRA, the command succeeds and
> return the expected output.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-17856) MM tables - IOW is not ACID compliant
[
https://issues.apache.org/jira/browse/HIVE-17856?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Shelukhin updated HIVE-17856:
Summary: MM tables - IOW is not ACID compliant (was: MM tables - IOW is
broken)
> MM tables - IOW is not ACID compliant
> -
>
> Key: HIVE-17856
> URL: https://issues.apache.org/jira/browse/HIVE-17856
> Project: Hive
> Issue Type: Sub-task
> Components: Transactions
>Reporter: Sergey Shelukhin
>Assignee: Steve Yeom
> Labels: mm-gap-1
>
> The following tests were removed from mm_all during "integration"... I should
> have never allowed such manner of intergration.
> MM logic should have been kept intact until ACID logic could catch up. Alas,
> here we are.
> {noformat}
> drop table iow0_mm;
> create table iow0_mm(key int) tblproperties("transactional"="true",
> "transactional_properties"="insert_only");
> insert overwrite table iow0_mm select key from intermediate;
> insert into table iow0_mm select key + 1 from intermediate;
> select * from iow0_mm order by key;
> insert overwrite table iow0_mm select key + 2 from intermediate;
> select * from iow0_mm order by key;
> drop table iow0_mm;
> drop table iow1_mm;
> create table iow1_mm(key int) partitioned by (key2 int)
> tblproperties("transactional"="true",
> "transactional_properties"="insert_only");
> insert overwrite table iow1_mm partition (key2)
> select key as k1, key from intermediate union all select key as k1, key from
> intermediate;
> insert into table iow1_mm partition (key2)
> select key + 1 as k1, key from intermediate union all select key as k1, key
> from intermediate;
> select * from iow1_mm order by key, key2;
> insert overwrite table iow1_mm partition (key2)
> select key + 3 as k1, key from intermediate union all select key + 4 as k1,
> key from intermediate;
> select * from iow1_mm order by key, key2;
> insert overwrite table iow1_mm partition (key2)
> select key + 3 as k1, key + 3 from intermediate union all select key + 2 as
> k1, key + 2 from intermediate;
> select * from iow1_mm order by key, key2;
> drop table iow1_mm;
> {noformat}
> {noformat}
> drop table simple_mm;
> create table simple_mm(key int) stored as orc tblproperties
> ("transactional"="true", "transactional_properties"="insert_only");
> insert into table simple_mm select key from intermediate;
> -insert overwrite table simple_mm select key from intermediate;
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17304) ThreadMXBean based memory allocation monitory for hash table loader
[ https://issues.apache.org/jira/browse/HIVE-17304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16213445#comment-16213445 ] Ashutosh Chauhan commented on HIVE-17304: - [~prasanth_j] Shall we commit this? > ThreadMXBean based memory allocation monitory for hash table loader > --- > > Key: HIVE-17304 > URL: https://issues.apache.org/jira/browse/HIVE-17304 > Project: Hive > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran > Attachments: HIVE-17304.1.patch, HIVE-17304.2.patch > > > Hash table memory monitoring is based on java data model which can be > unreliable because of various reasons (wrong object size estimation, adding > new variables to any class without accounting its size for memory monitoring, > etc.). We can use allocation size per thread that is provided by ThreadMXBean > and fallback to DataModel in case if JDK doesn't support thread based > allocations. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17331) Path must be used as key type of the pathToAlises
[
https://issues.apache.org/jira/browse/HIVE-17331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16213437#comment-16213437
]
Ashutosh Chauhan commented on HIVE-17331:
-
[~olegd] Can you update patch as per Zoltan's suggestion?
> Path must be used as key type of the pathToAlises
> -
>
> Key: HIVE-17331
> URL: https://issues.apache.org/jira/browse/HIVE-17331
> Project: Hive
> Issue Type: Bug
>Reporter: Oleg Danilov
>Assignee: Oleg Danilov
>Priority: Minor
> Attachments: HIVE-17331.patch
>
>
> This code uses String instead of Path as key type of the pathToAliases map,
> so seems like get(String) always null.
> +*GenMapRedUtils.java*+
> {code:java}
> for (int pos = 0; pos < size; pos++) {
> String taskTmpDir = taskTmpDirLst.get(pos);
> TableDesc tt_desc = tt_descLst.get(pos);
> MapWork mWork = plan.getMapWork();
> if (mWork.getPathToAliases().get(taskTmpDir) == null) {
> taskTmpDir = taskTmpDir.intern();
> Path taskTmpDirPath =
> StringInternUtils.internUriStringsInPath(new Path(taskTmpDir));
> mWork.removePathToAlias(taskTmpDirPath);
> mWork.addPathToAlias(taskTmpDirPath, taskTmpDir);
> mWork.addPathToPartitionInfo(taskTmpDirPath, new
> PartitionDesc(tt_desc, null));
> mWork.getAliasToWork().put(taskTmpDir, topOperators.get(pos));
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-16601) Display Session Id and Query Name / Id in Spark UI
[
https://issues.apache.org/jira/browse/HIVE-16601?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sahil Takiar updated HIVE-16601:
Attachment: HIVE-16601.8.patch
Rebased patch.
> Display Session Id and Query Name / Id in Spark UI
> --
>
> Key: HIVE-16601
> URL: https://issues.apache.org/jira/browse/HIVE-16601
> Project: Hive
> Issue Type: Bug
> Components: Spark
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
> Attachments: HIVE-16601.1.patch, HIVE-16601.2.patch,
> HIVE-16601.3.patch, HIVE-16601.4.patch, HIVE-16601.5.patch,
> HIVE-16601.6.patch, HIVE-16601.7.patch, HIVE-16601.8.patch, Spark UI
> Applications List.png, Spark UI Jobs List.png
>
>
> We should display the session id for each HoS Application Launched, and the
> Query Name / Id and Dag Id for each Spark job launched. Hive-on-MR does
> something similar via the {{mapred.job.name}} parameter. The query name is
> displayed in the Job Name of the MR app.
> The changes here should also allow us to leverage the config
> {{hive.query.name}} for HoS.
> This should help with debuggability of HoS applications. The Hive-on-Tez UI
> does something similar.
> Related issues for Hive-on-Tez: HIVE-12357, HIVE-12523
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17453) Missing ASF headers 2 classes
[ https://issues.apache.org/jira/browse/HIVE-17453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16213429#comment-16213429 ] Ashutosh Chauhan commented on HIVE-17453: - +1 > Missing ASF headers 2 classes > - > > Key: HIVE-17453 > URL: https://issues.apache.org/jira/browse/HIVE-17453 > Project: Hive > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: ZhangBing Lin >Assignee: ZhangBing Lin >Priority: Trivial > Attachments: HIVE-17453.1.patch > > > TestTxnCommandsBase.java and TestTxnNoBuckets.java are missing the ASF header > that should be added. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17495) CachedStore: prewarm improvement (avoid multiple sql calls to read partition column stats), refactoring and caching some aggregate stats
[ https://issues.apache.org/jira/browse/HIVE-17495?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-17495: Target Version/s: 3.0.0 > CachedStore: prewarm improvement (avoid multiple sql calls to read partition > column stats), refactoring and caching some aggregate stats > > > Key: HIVE-17495 > URL: https://issues.apache.org/jira/browse/HIVE-17495 > Project: Hive > Issue Type: Bug > Components: Metastore >Reporter: Vaibhav Gumashta >Assignee: Vaibhav Gumashta > Attachments: HIVE-17495.1.patch, HIVE-17495.2.patch, > HIVE-17495.3.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17799) Add Ellipsis For Truncated Query In Hive Lock
[
https://issues.apache.org/jira/browse/HIVE-17799?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashutosh Chauhan updated HIVE-17799:
Resolution: Fixed
Fix Version/s: 3.0.0
Status: Resolved (was: Patch Available)
Pushed to master. Thanks, Beluga!
> Add Ellipsis For Truncated Query In Hive Lock
> -
>
> Key: HIVE-17799
> URL: https://issues.apache.org/jira/browse/HIVE-17799
> Project: Hive
> Issue Type: Improvement
> Components: Hive
>Affects Versions: 3.0.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Trivial
> Fix For: 3.0.0
>
> Attachments: HIVE-17799.1.patch
>
>
> [HIVE-16334] introduced truncation for storing queries in ZK lock nodes.
> This Jira is to add ellipsis into the query to let the operator know that
> truncation has occurred and therefore they will not find the specific query
> in their logs, only a prefix match will work.
> {code:sql}
> -- Truncation of query may be confusing to operator
> -- Without truncation
> SELECT * FROM TABLE WHERE COL=1
> -- With truncation (operator will not find this query in workload)
> SELECT * FROM TABLE
> -- With truncation (operator will know this is only a prefix match)
> SELECT * FROM TABLE...
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-17635) Add unit tests to CompactionTxnHandler and use PreparedStatements for queries
[ https://issues.apache.org/jira/browse/HIVE-17635?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Sherman updated HIVE-17635: -- Attachment: HIVE-17635.6.patch patch again now that tests are running again > Add unit tests to CompactionTxnHandler and use PreparedStatements for queries > - > > Key: HIVE-17635 > URL: https://issues.apache.org/jira/browse/HIVE-17635 > Project: Hive > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Andrew Sherman >Assignee: Andrew Sherman > Attachments: HIVE-17635.1.patch, HIVE-17635.2.patch, > HIVE-17635.3.patch, HIVE-17635.4.patch, HIVE-17635.6.patch > > > It is better for jdbc code that runs against the HMS database to use > PreparedStatements. Convert CompactionTxnHandler queries to use > PreparedStatement and add tests to TestCompactionTxnHandler to test these > queries, and improve code coverage. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17868) Make queries in spark_local_queries.q have deterministic output
[
https://issues.apache.org/jira/browse/HIVE-17868?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16213374#comment-16213374
]
Andrew Sherman commented on HIVE-17868:
---
Thanks [~xuefuz] for the suggestion. I looked at --SORT_QUERY_RESULTS and it
seems it sorts the output after the query has run.
So with a query like
{noformat}
select key, count(*) from src group by key limit 10
{noformat}
--SORT_QUERY_RESULTS will sort the output, but the results of the query are not
sorted before the limit is applied, so this is not enough to make the query
deterministic. But using
{noformat}
select key, count(*) from src group by key order by key limit 10
{noformat}
is, I think deterministic.
The queries in spark_local_queries.q are very small and adding the 'order by'
does not seem to make a significant difference to elapsed time.
> Make queries in spark_local_queries.q have deterministic output
> ---
>
> Key: HIVE-17868
> URL: https://issues.apache.org/jira/browse/HIVE-17868
> Project: Hive
> Issue Type: Bug
>Reporter: Andrew Sherman
>Assignee: Andrew Sherman
>
> Add 'order by' to queries so that output is always the same
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-16601) Display Session Id and Query Name / Id in Spark UI
[
https://issues.apache.org/jira/browse/HIVE-16601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16213370#comment-16213370
]
Hive QA commented on HIVE-16601:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12893294/Spark%20UI%20Jobs%20List.png
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/7414/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/7414/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-7414/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2017-10-20 22:18:16.898
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-7414/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2017-10-20 22:18:16.900
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at b958c75 HIVE-17806 : Create directory for metrics file if it
doesn't exist (Alexander Kolbasov, reviewed by Vihang Karajgaonkar)
+ git clean -f -d
Removing standalone-metastore/src/gen/org/
+ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
+ git reset --hard origin/master
HEAD is now at b958c75 HIVE-17806 : Create directory for metrics file if it
doesn't exist (Alexander Kolbasov, reviewed by Vihang Karajgaonkar)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2017-10-20 22:18:17.872
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
patch: Only garbage was found in the patch input.
patch: Only garbage was found in the patch input.
patch: Only garbage was found in the patch input.
fatal: unrecognized input
The patch does not appear to apply with p0, p1, or p2
+ exit 1
'
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12893294 - PreCommit-HIVE-Build
> Display Session Id and Query Name / Id in Spark UI
> --
>
> Key: HIVE-16601
> URL: https://issues.apache.org/jira/browse/HIVE-16601
> Project: Hive
> Issue Type: Bug
> Components: Spark
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
> Attachments: HIVE-16601.1.patch, HIVE-16601.2.patch,
> HIVE-16601.3.patch, HIVE-16601.4.patch, HIVE-16601.5.patch,
> HIVE-16601.6.patch, HIVE-16601.7.patch, Spark UI Applications List.png, Spark
> UI Jobs List.png
>
>
> We should display the session id for each HoS Application Launched, and the
> Query Name / Id and Dag Id for each Spark job launched. Hive-on-MR does
> something similar via the {{mapred.job.name}} parameter. The query name is
> displayed in the Job Name of the MR app.
> The changes here should also allow us to leverage the config
> {{hive.query.name}} for HoS.
> This should help with debuggability of HoS applications. The Hive-on-Tez UI
> does something similar.
> Related issues for Hive-on-Tez: HIVE-12357, HIVE-12523
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17368) DBTokenStore fails to connect in Kerberos enabled remote HMS environment
[
https://issues.apache.org/jira/browse/HIVE-17368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16213227#comment-16213227
]
Hive QA commented on HIVE-17368:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12893313/HIVE-17368.02.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 6 failed/errored test(s), 11314 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_sortmerge_join_2]
(batchId=47)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast]
(batchId=156)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[optimize_nullscan]
(batchId=163)
org.apache.hadoop.hive.cli.control.TestDanglingQOuts.checkDanglingQOut
(batchId=204)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testConstraints
(batchId=221)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerHighShuffleBytes
(batchId=228)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/7413/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/7413/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-7413/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 6 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12893313 - PreCommit-HIVE-Build
> DBTokenStore fails to connect in Kerberos enabled remote HMS environment
>
>
> Key: HIVE-17368
> URL: https://issues.apache.org/jira/browse/HIVE-17368
> Project: Hive
> Issue Type: Bug
>Affects Versions: 1.1.0, 2.0.0, 2.1.0, 2.2.0
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
> Attachments: HIVE-17368.01-branch-2.patch, HIVE-17368.01.patch,
> HIVE-17368.02-branch-2.patch, HIVE-17368.02.patch,
> HIVE-17368.03-branch-2.patch, HIVE-17368.04-branch-2.patch,
> HIVE-17368.05-branch-2.patch, HIVE-17368.06-branch-2.patch
>
>
> In setups where HMS is running as a remote process secured using Kerberos,
> and when {{DBTokenStore}} is configured as the token store, the HS2 Thrift
> API call {{GetDelegationToken}} fail with exception trace seen below. HS2 is
> not able to invoke HMS APIs needed to add/remove/renew tokens from the DB
> since it is possible that the user which is issue the {{GetDelegationToken}}
> is not kerberos enabled.
> Eg. Oozie submits a job on behalf of user "Joe". When Oozie opens a session
> with HS2 it uses Oozie's principal and creates a proxy UGI with Hive. This
> principal can establish a transport authenticated using Kerberos. It stores
> the HMS delegation token string in the sessionConf and sessionToken. Now,
> lets say Oozie issues a {{GetDelegationToken}} which has {{Joe}} as the owner
> and {{oozie}} as the renewer in {{GetDelegationTokenReq}}. This API call
> cannot instantiate a HMSClient and open transport to HMS using the HMSToken
> string available in the sessionConf, since DBTokenStore uses server HiveConf
> instead of sessionConf. It tries to establish transport using Kerberos and it
> fails since user Joe is not Kerberos enabled.
> I see the following exception trace in HS2 logs.
> {noformat}
> 2017-08-21T18:07:19,644 ERROR [HiveServer2-Handler-Pool: Thread-61]
> transport.TSaslTransport: SASL negotiation failure
> javax.security.sasl.SaslException: GSS initiate failed
> at
> com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:211)
> ~[?:1.8.0_121]
> at
> org.apache.thrift.transport.TSaslClientTransport.handleSaslStartMessage(TSaslClientTransport.java:94)
> ~[libthrift-0.9.3.jar:0.9.3]
> at
> org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:271)
> [libthrift-0.9.3.jar:0.9.3]
> at
> org.apache.thrift.transport.TSaslClientTransport.open(TSaslClientTransport.java:37)
> [libthrift-0.9.3.jar:0.9.3]
> at
> org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:52)
> [hive-shims-common-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:49)
> [hive-shims-common-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT]
> at java.security.AccessController.doPrivileged(Native Method)
> ~[?:1.8.0_121]
> at javax.security.auth.Subject.doAs(Subject.java:422) [?:1.8.0_121]
> at
>
[jira] [Commented] (HIVE-17868) Make queries in spark_local_queries.q have deterministic output
[ https://issues.apache.org/jira/browse/HIVE-17868?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16213178#comment-16213178 ] Xuefu Zhang commented on HIVE-17868: Adding order by might slow down the test. Preferable way is to add tag "--sort_query_result" (something like that). > Make queries in spark_local_queries.q have deterministic output > --- > > Key: HIVE-17868 > URL: https://issues.apache.org/jira/browse/HIVE-17868 > Project: Hive > Issue Type: Bug >Reporter: Andrew Sherman >Assignee: Andrew Sherman > > Add 'order by' to queries so that output is always the same -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Comment Edited] (HIVE-17820) Add buckets.q test for blobstores
[
https://issues.apache.org/jira/browse/HIVE-17820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16213163#comment-16213163
]
Vihang Karajgaonkar edited comment on HIVE-17820 at 10/20/17 8:18 PM:
--
Thanks [~rangu] for the patch. Changed the assignee to you Ran since you
already submitted the patch. Can you change the {{drop table}} commands to
{{drop table if exists}} to make the test more reliable. LGTM +1
was (Author: vihangk1):
Thanks [~rangu] for the patch. Changed the assignee to you Ran since you
already submitted the patch. LGTM +1
> Add buckets.q test for blobstores
> -
>
> Key: HIVE-17820
> URL: https://issues.apache.org/jira/browse/HIVE-17820
> Project: Hive
> Issue Type: Test
> Components: Tests
>Reporter: Ran Gu
>Assignee: Ran Gu
> Attachments: HIVE-17820.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Assigned] (HIVE-17868) Make queries in spark_local_queries.q have deterministic output
[ https://issues.apache.org/jira/browse/HIVE-17868?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Sherman reassigned HIVE-17868: - > Make queries in spark_local_queries.q have deterministic output > --- > > Key: HIVE-17868 > URL: https://issues.apache.org/jira/browse/HIVE-17868 > Project: Hive > Issue Type: Bug >Reporter: Andrew Sherman >Assignee: Andrew Sherman > > Add 'order by' to queries so that output is always the same -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Comment Edited] (HIVE-17820) Add buckets.q test for blobstores
[ https://issues.apache.org/jira/browse/HIVE-17820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16213163#comment-16213163 ] Vihang Karajgaonkar edited comment on HIVE-17820 at 10/20/17 8:16 PM: -- Thanks [~rangu] for the patch. Changed the assignee to you Ran since you already submitted the patch. LGTM +1 was (Author: vihangk1): Thanks [~rangu] for the patch. LGTM +1 > Add buckets.q test for blobstores > - > > Key: HIVE-17820 > URL: https://issues.apache.org/jira/browse/HIVE-17820 > Project: Hive > Issue Type: Test > Components: Tests >Reporter: Ran Gu >Assignee: Ran Gu > Attachments: HIVE-17820.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17820) Add buckets.q test for blobstores
[ https://issues.apache.org/jira/browse/HIVE-17820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16213163#comment-16213163 ] Vihang Karajgaonkar commented on HIVE-17820: Thanks [~rangu] for the patch. LGTM +1 > Add buckets.q test for blobstores > - > > Key: HIVE-17820 > URL: https://issues.apache.org/jira/browse/HIVE-17820 > Project: Hive > Issue Type: Test > Components: Tests >Reporter: Ran Gu > Attachments: HIVE-17820.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (HIVE-17820) Add buckets.q test for blobstores
[ https://issues.apache.org/jira/browse/HIVE-17820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vihang Karajgaonkar reassigned HIVE-17820: -- Assignee: Ran Gu > Add buckets.q test for blobstores > - > > Key: HIVE-17820 > URL: https://issues.apache.org/jira/browse/HIVE-17820 > Project: Hive > Issue Type: Test > Components: Tests >Reporter: Ran Gu >Assignee: Ran Gu > Attachments: HIVE-17820.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17617) Rollup of an empty resultset should contain the grouping of the empty grouping set
[
https://issues.apache.org/jira/browse/HIVE-17617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltan Haindrich updated HIVE-17617:
Attachment: HIVE-17617.07.patch
> Rollup of an empty resultset should contain the grouping of the empty
> grouping set
> --
>
> Key: HIVE-17617
> URL: https://issues.apache.org/jira/browse/HIVE-17617
> Project: Hive
> Issue Type: Sub-task
> Components: SQL
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
> Attachments: HIVE-17617.01.patch, HIVE-17617.03.patch,
> HIVE-17617.04.patch, HIVE-17617.05.patch, HIVE-17617.06.patch,
> HIVE-17617.07.patch, HIVE-17617.07.patch
>
>
> running
> {code}
> drop table if exists tx1;
> create table tx1 (a integer,b integer,c integer);
> select sum(c),
> grouping(b)
> fromtx1
> group by rollup (b);
> {code}
> returns 0 rows; however
> according to the standard:
> The is regarded as the shortest such initial sublist.
> For example, “ROLLUP ( (A, B), (C, D) )”
> is equivalent to “GROUPING SETS ( (A, B, C, D), (A, B), () )”.
> so I think the totals row (the grouping for {{()}} should be present) - psql
> returns it.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-17806) Create directory for metrics file if it doesn't exist
[ https://issues.apache.org/jira/browse/HIVE-17806?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vihang Karajgaonkar updated HIVE-17806: --- Resolution: Fixed Fix Version/s: 3.0.0 Status: Resolved (was: Patch Available) > Create directory for metrics file if it doesn't exist > - > > Key: HIVE-17806 > URL: https://issues.apache.org/jira/browse/HIVE-17806 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.0.0 >Reporter: Alexander Kolbasov >Assignee: Alexander Kolbasov > Fix For: 3.0.0 > > Attachments: HIVE-17806.04.patch > > > HIVE-17563 changed metrics code to use local file system operations instead > of Hadoop local file system operations. There is an unintended side effect - > hadoop file systems create the directory if it doesn't exist and java nio > interfaces don't. The purpose of this fix is to revert the behavior to the > original one to avoid surprises. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17839) Cannot generate thrift definitions in standalone-metastore.
[ https://issues.apache.org/jira/browse/HIVE-17839?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16213147#comment-16213147 ] Vihang Karajgaonkar commented on HIVE-17839: Hi [~alangates] You might have to attach the patch again since there was a ptest issue yesterday. > Cannot generate thrift definitions in standalone-metastore. > --- > > Key: HIVE-17839 > URL: https://issues.apache.org/jira/browse/HIVE-17839 > Project: Hive > Issue Type: Bug >Reporter: Harish Jaiprakash >Assignee: Alan Gates > Attachments: HIVE-17839.patch > > > mvn clean install -Pthriftif -Dthrift.home=... does not regenerate the thrift > sources. This is after the https://issues.apache.org/jira/browse/HIVE-17506 > fix. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (HIVE-17771) Implement commands to manage resource plan.
[ https://issues.apache.org/jira/browse/HIVE-17771?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin reassigned HIVE-17771: --- Assignee: Harish Jaiprakash (was: Sergey Shelukhin) > Implement commands to manage resource plan. > --- > > Key: HIVE-17771 > URL: https://issues.apache.org/jira/browse/HIVE-17771 > Project: Hive > Issue Type: Sub-task >Reporter: Harish Jaiprakash >Assignee: Harish Jaiprakash > Attachments: HIVE-17771.01.patch, HIVE-17771.02.patch, > HIVE-17771.03.patch, HIVE-17771.04.patch > > > Please see parent jira about llap workload management. > This jira is to implement create and show resource plan commands in hive to > configure resource plans for llap workload. The following are the proposed > commands implemented as part of the jira: > CREATE RESOURCE PLAN plan_name WITH QUERY_PARALLELISM parallelism; > SHOW RESOURCE PLAN plan_name; > SHOW RESOURCE PLANS; > ALTER RESOURCE PLAN plan_name SET QUERY_PARALLELISM = parallelism; > ALTER RESOURCE PLAN plan_name RENAME TO new_name; > ALTER RESOURCE PLAN plan_name ACTIVATE; > ALTER RESOURCE PLAN plan_name DISABLE; > ALTER RESOURCE PLAN plan_name ENABLE; > DROP RESOURCE PLAN; > It will be followed up with more jiras to manage pools, triggers and copy > resource plans. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17771) Implement commands to manage resource plan.
[ https://issues.apache.org/jira/browse/HIVE-17771?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-17771: Attachment: HIVE-17771.04.patch The same patch; looks like the tests didn't run > Implement commands to manage resource plan. > --- > > Key: HIVE-17771 > URL: https://issues.apache.org/jira/browse/HIVE-17771 > Project: Hive > Issue Type: Sub-task >Reporter: Harish Jaiprakash >Assignee: Sergey Shelukhin > Attachments: HIVE-17771.01.patch, HIVE-17771.02.patch, > HIVE-17771.03.patch, HIVE-17771.04.patch > > > Please see parent jira about llap workload management. > This jira is to implement create and show resource plan commands in hive to > configure resource plans for llap workload. The following are the proposed > commands implemented as part of the jira: > CREATE RESOURCE PLAN plan_name WITH QUERY_PARALLELISM parallelism; > SHOW RESOURCE PLAN plan_name; > SHOW RESOURCE PLANS; > ALTER RESOURCE PLAN plan_name SET QUERY_PARALLELISM = parallelism; > ALTER RESOURCE PLAN plan_name RENAME TO new_name; > ALTER RESOURCE PLAN plan_name ACTIVATE; > ALTER RESOURCE PLAN plan_name DISABLE; > ALTER RESOURCE PLAN plan_name ENABLE; > DROP RESOURCE PLAN; > It will be followed up with more jiras to manage pools, triggers and copy > resource plans. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (HIVE-17771) Implement commands to manage resource plan.
[ https://issues.apache.org/jira/browse/HIVE-17771?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin reassigned HIVE-17771: --- Assignee: Sergey Shelukhin (was: Harish Jaiprakash) > Implement commands to manage resource plan. > --- > > Key: HIVE-17771 > URL: https://issues.apache.org/jira/browse/HIVE-17771 > Project: Hive > Issue Type: Sub-task >Reporter: Harish Jaiprakash >Assignee: Sergey Shelukhin > Attachments: HIVE-17771.01.patch, HIVE-17771.02.patch, > HIVE-17771.03.patch, HIVE-17771.04.patch > > > Please see parent jira about llap workload management. > This jira is to implement create and show resource plan commands in hive to > configure resource plans for llap workload. The following are the proposed > commands implemented as part of the jira: > CREATE RESOURCE PLAN plan_name WITH QUERY_PARALLELISM parallelism; > SHOW RESOURCE PLAN plan_name; > SHOW RESOURCE PLANS; > ALTER RESOURCE PLAN plan_name SET QUERY_PARALLELISM = parallelism; > ALTER RESOURCE PLAN plan_name RENAME TO new_name; > ALTER RESOURCE PLAN plan_name ACTIVATE; > ALTER RESOURCE PLAN plan_name DISABLE; > ALTER RESOURCE PLAN plan_name ENABLE; > DROP RESOURCE PLAN; > It will be followed up with more jiras to manage pools, triggers and copy > resource plans. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17840) HiveMetaStore eats exception if transactionalListeners.notifyEvent fail
[
https://issues.apache.org/jira/browse/HIVE-17840?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16213096#comment-16213096
]
Alexander Kolbasov commented on HIVE-17840:
---
The exception is consumed in SentryStore.addNotificationEvent():
{code}
try {
this.openTransaction();
this.lockForUpdate();
...
commited = this.commitTransaction();
} catch (Exception var11) {
LOG.error("couldnot get lock for update", var11); // Here
} finally {
this.rollbackAndCleanup(commited, (Query)query);
}
{code}
> HiveMetaStore eats exception if transactionalListeners.notifyEvent fail
> ---
>
> Key: HIVE-17840
> URL: https://issues.apache.org/jira/browse/HIVE-17840
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Reporter: Daniel Dai
>Assignee: Daniel Dai
>
> For example, in add_partitions_core, if there's exception in
> MetaStoreListenerNotifier.notifyEvent(transactionalListeners,),
> transaction rollback but no exception thrown. Client will assume add
> partition is successful and take a positive path.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17844) Hive JDBC driver converts floats to doubles, changing values slightly
[ https://issues.apache.org/jira/browse/HIVE-17844?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16213072#comment-16213072 ] Zoltan Ivanfi commented on HIVE-17844: -- [~akolb], yes, that's true, but it doesn't represent the initial user-supplied number faithfully any more. If the user supplies the value 125.32, it can not be represented as a float precisely, but rounding rules during displaying that float will still display 125.32. However, when converting that float to double, different, more precise rounding rules start to apply, resulting in 125.3169482422 getting displayed instead of 125.32. It is true that the double value is precisely the same as the float value, yet although none of them is precisely the same as the user-supplied value, the float gets displayed as the original value and the double isn't. If 125.32 is stored directly to a double, then it is approximated more precisely and will display correctly. When it is stored as a float and then converted to a double, this is no longer true. > Hive JDBC driver converts floats to doubles, changing values slightly > - > > Key: HIVE-17844 > URL: https://issues.apache.org/jira/browse/HIVE-17844 > Project: Hive > Issue Type: Bug > Components: JDBC >Reporter: Zoltan Ivanfi > > When querying data through Hive's JDBC driver, float values are automatically > converted to doubles and will slightly differ from values seen when directly > querying the data using a textual SQL interface. > Please note that by slight difference I don't only mean additional zeroes at > the end, but an actual numeric difference in the displayed value. This is the > result of the different precision used when displaying double values, as > discussed > [here|https://stackoverflow.com/questions/17504833/why-converting-from-float-to-double-changes-the-value], > for example. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17867) Exception in windowing functions with TIMESTAMP WITH LOCAL TIME ZONE type
[
https://issues.apache.org/jira/browse/HIVE-17867?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesus Camacho Rodriguez updated HIVE-17867:
---
Description:
The following query where column {{ts}} is of type {{TIMESTAMP WITH LOCAL TIME
ZONE}}:
{code}
select ts, i, sum(f) over (partition by i order by ts)
from over10k_2
limit 100;
{code}
fails with the following stacktrace:
{code}
org.apache.hadoop.hive.ql.parse.SemanticException: Failed to breakup Windowing
invocations into Groups. At least 1 group must only depend on input columns.
Also check for circular dependencies.
Underlying error: Primitive type TIMESTAMPLOCALTZ not supported in Value
Boundary expression
at
org.apache.hadoop.hive.ql.parse.WindowingComponentizer.next(WindowingComponentizer.java:97)
at
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genWindowingPlan(SemanticAnalyzer.java:13508)
at
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPostGroupByBodyPlan(SemanticAnalyzer.java:9912)
at
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genBodyPlan(SemanticAnalyzer.java:9871)
at
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:10784)
...
{code}
The list of supported types for boundaries expressions in PTFTranslator needs
to be updated.
was:
The following query:
{code}
select ts, i, sum(f) over (partition by i order by ts)
from over10k_2
limit 100;
{code}
fails with the following stacktrace:
{code}
org.apache.hadoop.hive.ql.parse.SemanticException: Failed to breakup Windowing
invocations into Groups. At least 1 group must only depend on input columns.
Also check for circular dependencies.
Underlying error: Primitive type TIMESTAMPLOCALTZ not supported in Value
Boundary expression
at
org.apache.hadoop.hive.ql.parse.WindowingComponentizer.next(WindowingComponentizer.java:97)
at
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genWindowingPlan(SemanticAnalyzer.java:13508)
at
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPostGroupByBodyPlan(SemanticAnalyzer.java:9912)
at
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genBodyPlan(SemanticAnalyzer.java:9871)
at
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:10784)
...
{code}
The list of supported types for boundaries expressions in PTFTranslator needs
to be updated.
> Exception in windowing functions with TIMESTAMP WITH LOCAL TIME ZONE type
> -
>
> Key: HIVE-17867
> URL: https://issues.apache.org/jira/browse/HIVE-17867
> Project: Hive
> Issue Type: Bug
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>
> The following query where column {{ts}} is of type {{TIMESTAMP WITH LOCAL
> TIME ZONE}}:
> {code}
> select ts, i, sum(f) over (partition by i order by ts)
> from over10k_2
> limit 100;
> {code}
> fails with the following stacktrace:
> {code}
> org.apache.hadoop.hive.ql.parse.SemanticException: Failed to breakup
> Windowing invocations into Groups. At least 1 group must only depend on input
> columns. Also check for circular dependencies.
> Underlying error: Primitive type TIMESTAMPLOCALTZ not supported in Value
> Boundary expression
> at
> org.apache.hadoop.hive.ql.parse.WindowingComponentizer.next(WindowingComponentizer.java:97)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genWindowingPlan(SemanticAnalyzer.java:13508)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPostGroupByBodyPlan(SemanticAnalyzer.java:9912)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genBodyPlan(SemanticAnalyzer.java:9871)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:10784)
> ...
> {code}
> The list of supported types for boundaries expressions in PTFTranslator needs
> to be updated.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-17867) Exception in windowing functions with TIMESTAMP WITH LOCAL TIME ZONE type
[
https://issues.apache.org/jira/browse/HIVE-17867?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesus Camacho Rodriguez updated HIVE-17867:
---
Status: Patch Available (was: In Progress)
> Exception in windowing functions with TIMESTAMP WITH LOCAL TIME ZONE type
> -
>
> Key: HIVE-17867
> URL: https://issues.apache.org/jira/browse/HIVE-17867
> Project: Hive
> Issue Type: Bug
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>
> The following query:
> {code}
> select ts, i, sum(f) over (partition by i order by ts)
> from over10k_2
> limit 100;
> {code}
> fails with the following stacktrace:
> {code}
> org.apache.hadoop.hive.ql.parse.SemanticException: Failed to breakup
> Windowing invocations into Groups. At least 1 group must only depend on input
> columns. Also check for circular dependencies.
> Underlying error: Primitive type TIMESTAMPLOCALTZ not supported in Value
> Boundary expression
> at
> org.apache.hadoop.hive.ql.parse.WindowingComponentizer.next(WindowingComponentizer.java:97)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genWindowingPlan(SemanticAnalyzer.java:13508)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPostGroupByBodyPlan(SemanticAnalyzer.java:9912)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genBodyPlan(SemanticAnalyzer.java:9871)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:10784)
> ...
> {code}
> The list of supported types for boundaries expressions in PTFTranslator needs
> to be updated.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-17867) Exception in windowing functions with TIMESTAMP WITH LOCAL TIME ZONE type
[
https://issues.apache.org/jira/browse/HIVE-17867?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesus Camacho Rodriguez updated HIVE-17867:
---
Target Version/s: 3.0.0
> Exception in windowing functions with TIMESTAMP WITH LOCAL TIME ZONE type
> -
>
> Key: HIVE-17867
> URL: https://issues.apache.org/jira/browse/HIVE-17867
> Project: Hive
> Issue Type: Bug
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>
> The following query:
> {code}
> select ts, i, sum(f) over (partition by i order by ts)
> from over10k_2
> limit 100;
> {code}
> fails with the following stacktrace:
> {code}
> org.apache.hadoop.hive.ql.parse.SemanticException: Failed to breakup
> Windowing invocations into Groups. At least 1 group must only depend on input
> columns. Also check for circular dependencies.
> Underlying error: Primitive type TIMESTAMPLOCALTZ not supported in Value
> Boundary expression
> at
> org.apache.hadoop.hive.ql.parse.WindowingComponentizer.next(WindowingComponentizer.java:97)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genWindowingPlan(SemanticAnalyzer.java:13508)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPostGroupByBodyPlan(SemanticAnalyzer.java:9912)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genBodyPlan(SemanticAnalyzer.java:9871)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:10784)
> ...
> {code}
> The list of supported types for boundaries expressions in PTFTranslator needs
> to be updated.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Assigned] (HIVE-17867) Exception in windowing functions with TIMESTAMP WITH LOCAL TIME ZONE type
[
https://issues.apache.org/jira/browse/HIVE-17867?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesus Camacho Rodriguez reassigned HIVE-17867:
--
Assignee: Jesus Camacho Rodriguez
> Exception in windowing functions with TIMESTAMP WITH LOCAL TIME ZONE type
> -
>
> Key: HIVE-17867
> URL: https://issues.apache.org/jira/browse/HIVE-17867
> Project: Hive
> Issue Type: Bug
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>
> The following query:
> {code}
> select ts, i, sum(f) over (partition by i order by ts)
> from over10k_2
> limit 100;
> {code}
> fails with the following stacktrace:
> {code}
> org.apache.hadoop.hive.ql.parse.SemanticException: Failed to breakup
> Windowing invocations into Groups. At least 1 group must only depend on input
> columns. Also check for circular dependencies.
> Underlying error: Primitive type TIMESTAMPLOCALTZ not supported in Value
> Boundary expression
> at
> org.apache.hadoop.hive.ql.parse.WindowingComponentizer.next(WindowingComponentizer.java:97)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genWindowingPlan(SemanticAnalyzer.java:13508)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPostGroupByBodyPlan(SemanticAnalyzer.java:9912)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genBodyPlan(SemanticAnalyzer.java:9871)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:10784)
> ...
> {code}
> The list of supported types for boundaries expressions in PTFTranslator needs
> to be updated.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Work started] (HIVE-17867) Exception in windowing functions with TIMESTAMP WITH LOCAL TIME ZONE type
[
https://issues.apache.org/jira/browse/HIVE-17867?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HIVE-17867 started by Jesus Camacho Rodriguez.
--
> Exception in windowing functions with TIMESTAMP WITH LOCAL TIME ZONE type
> -
>
> Key: HIVE-17867
> URL: https://issues.apache.org/jira/browse/HIVE-17867
> Project: Hive
> Issue Type: Bug
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>
> The following query:
> {code}
> select ts, i, sum(f) over (partition by i order by ts)
> from over10k_2
> limit 100;
> {code}
> fails with the following stacktrace:
> {code}
> org.apache.hadoop.hive.ql.parse.SemanticException: Failed to breakup
> Windowing invocations into Groups. At least 1 group must only depend on input
> columns. Also check for circular dependencies.
> Underlying error: Primitive type TIMESTAMPLOCALTZ not supported in Value
> Boundary expression
> at
> org.apache.hadoop.hive.ql.parse.WindowingComponentizer.next(WindowingComponentizer.java:97)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genWindowingPlan(SemanticAnalyzer.java:13508)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPostGroupByBodyPlan(SemanticAnalyzer.java:9912)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genBodyPlan(SemanticAnalyzer.java:9871)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:10784)
> ...
> {code}
> The list of supported types for boundaries expressions in PTFTranslator needs
> to be updated.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17806) Create directory for metrics file if it doesn't exist
[ https://issues.apache.org/jira/browse/HIVE-17806?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16213003#comment-16213003 ] Vihang Karajgaonkar commented on HIVE-17806: Patch merged to master. Thanks for your contribution [~akolb]. Does this need to be merged in branch-2 as well? > Create directory for metrics file if it doesn't exist > - > > Key: HIVE-17806 > URL: https://issues.apache.org/jira/browse/HIVE-17806 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.0.0 >Reporter: Alexander Kolbasov >Assignee: Alexander Kolbasov > Attachments: HIVE-17806.04.patch > > > HIVE-17563 changed metrics code to use local file system operations instead > of Hadoop local file system operations. There is an unintended side effect - > hadoop file systems create the directory if it doesn't exist and java nio > interfaces don't. The purpose of this fix is to revert the behavior to the > original one to avoid surprises. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17828) Metastore: mysql upgrade scripts to 3.0.0 is broken
[
https://issues.apache.org/jira/browse/HIVE-17828?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16212987#comment-16212987
]
Prasanth Jayachandran commented on HIVE-17828:
--
Patch picked up by precommit but did not run any tests. Will retry
> Metastore: mysql upgrade scripts to 3.0.0 is broken
> ---
>
> Key: HIVE-17828
> URL: https://issues.apache.org/jira/browse/HIVE-17828
> Project: Hive
> Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Gopal V
>Assignee: Prasanth Jayachandran
> Attachments: HIVE-17828.1.patch, HIVE-17828.2.patch
>
>
> {code}
> +-+
> | |
> +-+
> | Finished upgrading MetaStore schema from 2.2.0 to 2.3.0 |
> +-+
> 1 row in set, 1 warning (0.00 sec)
> mysql> source upgrade-2.3.0-to-3.0.0.mysql.sql
> ++
> ||
> ++
> | Upgrading MetaStore schema from 2.3.0 to 3.0.0 |
> ++
> {code}
> {code}
> --
> CREATE TABLE WM_RESOURCEPLAN (
> `RP_ID` bigint(20) NOT NULL,
> `NAME` varchar(128) NOT NULL,
> `QUERY_PARALLELISM` int(11),
> `STATUS` varchar(20) NOT NULL,
> PRIMARY KEY (`RP_ID`),
> KEY `UNIQUE_WM_RESOURCEPLAN` (`NAME`),
> ) ENGINE=InnoDB DEFAULT CHARSET=latin1
> --
> ERROR 1064 (42000): You have an error in your SQL syntax; check the manual
> that corresponds to your MySQL server version for the right syntax to use
> near ') ENGINE=InnoDB DEFAULT CHARSET=latin1' at line 8
> --
> CREATE TABLE WM_POOL
> (
> `POOL_ID` bigint(20) NOT NULL,
> `RP_ID` bigint(20) NOT NULL,
> `PATH` varchar(1024) NOT NULL,
> `PARENT_POOL_ID` bigint(20),
> `ALLOC_FRACTION` DOUBLE,
> `QUERY_PARALLELISM` int(11),
> PRIMARY KEY (`POOL_ID`),
> KEY `UNIQUE_WM_POOL` (`RP_ID`, `PATH`),
> CONSTRAINT `WM_POOL_FK1` FOREIGN KEY (`RP_ID`) REFERENCES
> `WM_RESOURCEPLAN` (`RP_ID`),
> CONSTRAINT `WM_POOL_FK2` FOREIGN KEY (`PARENT_POOL_ID`) REFERENCES
> `WM_POOL` (`POOL_ID`)
> ) ENGINE=InnoDB DEFAULT CHARSET=latin1
> --
> ERROR 1071 (42000): Specified key was too long; max key length is 767 bytes
> --
> CREATE TABLE WM_TRIGGER
> (
> `TRIGGER_ID` bigint(20) NOT NULL,
> `RP_ID` bigint(20) NOT NULL,
> `NAME` varchar(128) NOT NULL,
> `TRIGGER_EXPRESSION` varchar(1024),
> `ACTION_EXPRESSION` varchar(1024),
> PRIMARY KEY (`TRIGGER_ID`),
> KEY `UNIQUE_WM_TRIGGER` (`RP_ID`, `NAME`),
> CONSTRAINT `WM_TRIGGER_FK1` FOREIGN KEY (`RP_ID`) REFERENCES
> `WM_RESOURCEPLAN` (`RP_ID`)
> ) ENGINE=InnoDB DEFAULT CHARSET=latin1
> --
> ERROR 1215 (HY000): Cannot add foreign key constraint
> --
> CREATE TABLE WM_POOL_TO_TRIGGER
> (
> `POOL_ID` bigint(20) NOT NULL,
> `TRIGGER_ID` bigint(20) NOT NULL,
> PRIMARY KEY (`POOL_ID`, `TRIGGER_ID`),
> CONSTRAINT `WM_POOL_TO_TRIGGER_FK1` FOREIGN KEY (`POOL_ID`) REFERENCES
> `WM_POOL` (`POOL_ID`),
> CONSTRAINT `WM_POOL_TO_TRIGGER_FK2` FOREIGN KEY (`TRIGGER_ID`) REFERENCES
> `WM_TRIGGER` (`TRIGGER_ID`)
> ) ENGINE=InnoDB DEFAULT CHARSET=latin1
> --
> ERROR 1215 (HY000): Cannot add foreign key constraint
> --
> CREATE TABLE WM_MAPPING
> (
> `MAPPING_ID` bigint(20) NOT NULL,
> `RP_ID` bigint(20) NOT NULL,
> `ENTITY_TYPE` varchar(10) NOT NULL,
> `ENTITY_NAME` varchar(128) NOT NULL,
> `POOL_ID` bigint(20) NOT NULL,
> `ORDERING int,
> PRIMARY KEY (`MAPPING_ID`),
> KEY `UNIQUE_WM_MAPPING` (`RP_ID`, `ENTITY_TYPE`, `ENTITY_NAME`),
> CONSTRAINT `WM_MAPPING_FK1` FOREIGN KEY (`RP_ID`) REFERENCES
> `WM_RESOURCEPLAN` (`RP_ID`),
> CONSTRAINT `WM_MAPPING_FK2` FOREIGN KEY (`POOL_ID`) REFERENCES `WM_POOL`
> (`POOL_ID`)
> ) ENGINE=InnoDB DEFAULT CHARSET=latin1;
> --
> ERROR 1064 (42000): You have an error in your SQL syntax; check the manual
> that corresponds to your MySQL server version for the right syntax to use
> near 'MAPPING_ID`),
> KEY `UNIQUE_WM_MAPPING` (`RP_ID`, `ENTITY_TYPE`, `ENTITY_NAME`' at line 8
> --
> UPDATE VERSION SET SCHEMA_VERSION='3.0.0', VERSION_COMMENT='Hive release
> version 3.0.0' where VER_ID=1
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17778) Add support for custom counters in trigger expression
[ https://issues.apache.org/jira/browse/HIVE-17778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16212986#comment-16212986 ] Prasanth Jayachandran commented on HIVE-17778: -- Patch not picked up by precommit > Add support for custom counters in trigger expression > - > > Key: HIVE-17778 > URL: https://issues.apache.org/jira/browse/HIVE-17778 > Project: Hive > Issue Type: Sub-task >Affects Versions: 3.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran > Attachments: HIVE-17778.1.patch, HIVE-17778.2.patch, > HIVE-17778.3.patch, HIVE-17778.4.patch > > > HIVE-17508 only supports limited counters. This ticket is to extend it to > support custom counters (counters that are not supported by execution engine > will be dropped). -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17834) Fix flaky triggers test
[ https://issues.apache.org/jira/browse/HIVE-17834?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16212984#comment-16212984 ] Prasanth Jayachandran commented on HIVE-17834: -- Patch not picked up by precommit > Fix flaky triggers test > --- > > Key: HIVE-17834 > URL: https://issues.apache.org/jira/browse/HIVE-17834 > Project: Hive > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran > Attachments: HIVE-17834.1.patch > > > https://issues.apache.org/jira/browse/HIVE-12631?focusedCommentId=16209803=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16209803 -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17368) DBTokenStore fails to connect in Kerberos enabled remote HMS environment
[
https://issues.apache.org/jira/browse/HIVE-17368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vihang Karajgaonkar updated HIVE-17368:
---
Attachment: HIVE-17368.02.patch
Attaching patch for master branch.
> DBTokenStore fails to connect in Kerberos enabled remote HMS environment
>
>
> Key: HIVE-17368
> URL: https://issues.apache.org/jira/browse/HIVE-17368
> Project: Hive
> Issue Type: Bug
>Affects Versions: 1.1.0, 2.0.0, 2.1.0, 2.2.0
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
> Attachments: HIVE-17368.01-branch-2.patch, HIVE-17368.01.patch,
> HIVE-17368.02-branch-2.patch, HIVE-17368.02.patch,
> HIVE-17368.03-branch-2.patch, HIVE-17368.04-branch-2.patch,
> HIVE-17368.05-branch-2.patch, HIVE-17368.06-branch-2.patch
>
>
> In setups where HMS is running as a remote process secured using Kerberos,
> and when {{DBTokenStore}} is configured as the token store, the HS2 Thrift
> API call {{GetDelegationToken}} fail with exception trace seen below. HS2 is
> not able to invoke HMS APIs needed to add/remove/renew tokens from the DB
> since it is possible that the user which is issue the {{GetDelegationToken}}
> is not kerberos enabled.
> Eg. Oozie submits a job on behalf of user "Joe". When Oozie opens a session
> with HS2 it uses Oozie's principal and creates a proxy UGI with Hive. This
> principal can establish a transport authenticated using Kerberos. It stores
> the HMS delegation token string in the sessionConf and sessionToken. Now,
> lets say Oozie issues a {{GetDelegationToken}} which has {{Joe}} as the owner
> and {{oozie}} as the renewer in {{GetDelegationTokenReq}}. This API call
> cannot instantiate a HMSClient and open transport to HMS using the HMSToken
> string available in the sessionConf, since DBTokenStore uses server HiveConf
> instead of sessionConf. It tries to establish transport using Kerberos and it
> fails since user Joe is not Kerberos enabled.
> I see the following exception trace in HS2 logs.
> {noformat}
> 2017-08-21T18:07:19,644 ERROR [HiveServer2-Handler-Pool: Thread-61]
> transport.TSaslTransport: SASL negotiation failure
> javax.security.sasl.SaslException: GSS initiate failed
> at
> com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:211)
> ~[?:1.8.0_121]
> at
> org.apache.thrift.transport.TSaslClientTransport.handleSaslStartMessage(TSaslClientTransport.java:94)
> ~[libthrift-0.9.3.jar:0.9.3]
> at
> org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:271)
> [libthrift-0.9.3.jar:0.9.3]
> at
> org.apache.thrift.transport.TSaslClientTransport.open(TSaslClientTransport.java:37)
> [libthrift-0.9.3.jar:0.9.3]
> at
> org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:52)
> [hive-shims-common-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:49)
> [hive-shims-common-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT]
> at java.security.AccessController.doPrivileged(Native Method)
> ~[?:1.8.0_121]
> at javax.security.auth.Subject.doAs(Subject.java:422) [?:1.8.0_121]
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
> [hadoop-common-2.7.2.jar:?]
> at
> org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport.open(TUGIAssumingTransport.java:49)
> [hive-shims-common-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:488)
> [hive-metastore-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.(HiveMetaStoreClient.java:255)
> [hive-metastore-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.(SessionHiveMetaStoreClient.java:70)
> [hive-exec-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT]
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method) ~[?:1.8.0_121]
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> [?:1.8.0_121]
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> [?:1.8.0_121]
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> [?:1.8.0_121]
> at
> org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1699)
> [hive-metastore-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.(RetryingMetaStoreClient.java:83)
> [hive-metastore-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT]
> at
[jira] [Commented] (HIVE-17844) Hive JDBC driver converts floats to doubles, changing values slightly
[ https://issues.apache.org/jira/browse/HIVE-17844?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16212942#comment-16212942 ] Alexander Kolbasov commented on HIVE-17844: --- According to the discussion that Zoltan refers to, the double represents float precisely, but the printed representation is different. > Hive JDBC driver converts floats to doubles, changing values slightly > - > > Key: HIVE-17844 > URL: https://issues.apache.org/jira/browse/HIVE-17844 > Project: Hive > Issue Type: Bug > Components: JDBC >Reporter: Zoltan Ivanfi > > When querying data through Hive's JDBC driver, float values are automatically > converted to doubles and will slightly differ from values seen when directly > querying the data using a textual SQL interface. > Please note that by slight difference I don't only mean additional zeroes at > the end, but an actual numeric difference in the displayed value. This is the > result of the different precision used when displaying double values, as > discussed > [here|https://stackoverflow.com/questions/17504833/why-converting-from-float-to-double-changes-the-value], > for example. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17831) HiveSemanticAnalyzerHookContext does not update the HiveOperation after sem.analyze() is called
[ https://issues.apache.org/jira/browse/HIVE-17831?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aihua Xu updated HIVE-17831: Fix Version/s: 2.2.1 2.4.0 3.0.0 2.1.2 > HiveSemanticAnalyzerHookContext does not update the HiveOperation after > sem.analyze() is called > --- > > Key: HIVE-17831 > URL: https://issues.apache.org/jira/browse/HIVE-17831 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.0.0, 2.4.0, 2.2.1, 2.3.1 >Reporter: Sergio Peña >Assignee: Aihua Xu > Fix For: 2.1.2, 3.0.0, 2.4.0, 2.2.1 > > Attachments: HIVE-17831.1.patch > > > The SemanticAnalyzer.analyze() called on the Driver.compile() method updates > the HiveOperation based on the analysis this does. However, the patch done on > HIVE-17048 does not update such operation and is send an invalid operation to > the postAnalyze() call. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-16601) Display Session Id and Query Name / Id in Spark UI
[
https://issues.apache.org/jira/browse/HIVE-16601?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sahil Takiar updated HIVE-16601:
Attachment: Spark UI Jobs List.png
> Display Session Id and Query Name / Id in Spark UI
> --
>
> Key: HIVE-16601
> URL: https://issues.apache.org/jira/browse/HIVE-16601
> Project: Hive
> Issue Type: Bug
> Components: Spark
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
> Attachments: HIVE-16601.1.patch, HIVE-16601.2.patch,
> HIVE-16601.3.patch, HIVE-16601.4.patch, HIVE-16601.5.patch,
> HIVE-16601.6.patch, HIVE-16601.7.patch, Spark UI Applications List.png, Spark
> UI Jobs List.png
>
>
> We should display the session id for each HoS Application Launched, and the
> Query Name / Id and Dag Id for each Spark job launched. Hive-on-MR does
> something similar via the {{mapred.job.name}} parameter. The query name is
> displayed in the Job Name of the MR app.
> The changes here should also allow us to leverage the config
> {{hive.query.name}} for HoS.
> This should help with debuggability of HoS applications. The Hive-on-Tez UI
> does something similar.
> Related issues for Hive-on-Tez: HIVE-12357, HIVE-12523
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-16601) Display Session Id and Query Name / Id in Spark UI
[
https://issues.apache.org/jira/browse/HIVE-16601?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sahil Takiar updated HIVE-16601:
Attachment: (was: Spark UI Jobs List.png)
> Display Session Id and Query Name / Id in Spark UI
> --
>
> Key: HIVE-16601
> URL: https://issues.apache.org/jira/browse/HIVE-16601
> Project: Hive
> Issue Type: Bug
> Components: Spark
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
> Attachments: HIVE-16601.1.patch, HIVE-16601.2.patch,
> HIVE-16601.3.patch, HIVE-16601.4.patch, HIVE-16601.5.patch,
> HIVE-16601.6.patch, HIVE-16601.7.patch, Spark UI Applications List.png
>
>
> We should display the session id for each HoS Application Launched, and the
> Query Name / Id and Dag Id for each Spark job launched. Hive-on-MR does
> something similar via the {{mapred.job.name}} parameter. The query name is
> displayed in the Job Name of the MR app.
> The changes here should also allow us to leverage the config
> {{hive.query.name}} for HoS.
> This should help with debuggability of HoS applications. The Hive-on-Tez UI
> does something similar.
> Related issues for Hive-on-Tez: HIVE-12357, HIVE-12523
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-16601) Display Session Id and Query Name / Id in Spark UI
[
https://issues.apache.org/jira/browse/HIVE-16601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16212822#comment-16212822
]
Sahil Takiar commented on HIVE-16601:
-
[~xuefuz] yeah, I was thinking of doing that, but decide not to for the sake of
brevity. If you think it is useful though, then I'll add it back in.
Attached new patch, where the Job Description is now "(queryId =
[hive-query-id])". Attached updated screenshots too.
> Display Session Id and Query Name / Id in Spark UI
> --
>
> Key: HIVE-16601
> URL: https://issues.apache.org/jira/browse/HIVE-16601
> Project: Hive
> Issue Type: Bug
> Components: Spark
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
> Attachments: HIVE-16601.1.patch, HIVE-16601.2.patch,
> HIVE-16601.3.patch, HIVE-16601.4.patch, HIVE-16601.5.patch,
> HIVE-16601.6.patch, HIVE-16601.7.patch, Spark UI Applications List.png
>
>
> We should display the session id for each HoS Application Launched, and the
> Query Name / Id and Dag Id for each Spark job launched. Hive-on-MR does
> something similar via the {{mapred.job.name}} parameter. The query name is
> displayed in the Job Name of the MR app.
> The changes here should also allow us to leverage the config
> {{hive.query.name}} for HoS.
> This should help with debuggability of HoS applications. The Hive-on-Tez UI
> does something similar.
> Related issues for Hive-on-Tez: HIVE-12357, HIVE-12523
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-16601) Display Session Id and Query Name / Id in Spark UI
[
https://issues.apache.org/jira/browse/HIVE-16601?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sahil Takiar updated HIVE-16601:
Attachment: HIVE-16601.7.patch
> Display Session Id and Query Name / Id in Spark UI
> --
>
> Key: HIVE-16601
> URL: https://issues.apache.org/jira/browse/HIVE-16601
> Project: Hive
> Issue Type: Bug
> Components: Spark
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
> Attachments: HIVE-16601.1.patch, HIVE-16601.2.patch,
> HIVE-16601.3.patch, HIVE-16601.4.patch, HIVE-16601.5.patch,
> HIVE-16601.6.patch, HIVE-16601.7.patch, Spark UI Applications List.png, Spark
> UI Jobs List.png
>
>
> We should display the session id for each HoS Application Launched, and the
> Query Name / Id and Dag Id for each Spark job launched. Hive-on-MR does
> something similar via the {{mapred.job.name}} parameter. The query name is
> displayed in the Job Name of the MR app.
> The changes here should also allow us to leverage the config
> {{hive.query.name}} for HoS.
> This should help with debuggability of HoS applications. The Hive-on-Tez UI
> does something similar.
> Related issues for Hive-on-Tez: HIVE-12357, HIVE-12523
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-17864) PTestClient cannot start during Precommit tests
[
https://issues.apache.org/jira/browse/HIVE-17864?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashutosh Chauhan updated HIVE-17864:
Resolution: Fixed
Fix Version/s: 3.0.0
Status: Resolved (was: Patch Available)
I see that this patch has already been committed. Thanks, Adam!
> PTestClient cannot start during Precommit tests
> ---
>
> Key: HIVE-17864
> URL: https://issues.apache.org/jira/browse/HIVE-17864
> Project: Hive
> Issue Type: Bug
>Reporter: Adam Szita
>Assignee: Adam Szita
> Fix For: 3.0.0
>
> Attachments: HIVE-17864.0.patch
>
>
> HIVE-17807 has bumped version number in testutils/ptest2/pom.xml from 1.0 to
> 3.0 resulting failure to start PTestClient during Precommit runs:
> {code}
> java -cp
> '/home/jenkins/jenkins-slave/workspace/PreCommit-HIVE-Build/hive/build/hive/testutils/ptest2/target/hive-ptest-1.0-classes.jar:/home/jenkins/jenkins-slave/workspace/PreCommit-HIVE-Build/hive/build/hive/testutils/ptest2/target/lib/*'
> org.apache.hive.ptest.api.client.PTestClient --command testStart --outputDir
> /home/jenkins/jenkins-slave/workspace/PreCommit-HIVE-Build/hive/build/hive/testutils/ptest2/target
> --password '[***]' --testHandle PreCommit-HIVE-Build-7389 --endpoint
> http://104.198.109.242:8080/hive-ptest-1.0 --logsEndpoint
> http://104.198.109.242/logs/ --profile master-mr2 --patch
> https://issues.apache.org/jira/secure/attachment/12893016/HIVE-17842.0.patch
> --jira HIVE-17842
> Error: Could not find or load main class
> org.apache.hive.ptest.api.client.PTestClient
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17864) PTestClient cannot start during Precommit tests
[
https://issues.apache.org/jira/browse/HIVE-17864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16212720#comment-16212720
]
Hive QA commented on HIVE-17864:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12893214/HIVE-17864.0.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/7412/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/7412/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-7412/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2017-10-20 14:38:39.228
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-7412/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2017-10-20 14:38:39.231
+ cd apache-github-source-source
+ git fetch origin
>From https://github.com/apache/hive
ea53ed8..76d8809 master -> origin/master
a93e99a..2106bf7 branch-2 -> origin/branch-2
92d3070..bc52f36 branch-2.3 -> origin/branch-2.3
+ git reset --hard HEAD
HEAD is now at ea53ed8 HIVE-17054: Expose SQL database constraints to Calcite
(Jesus Camacho Rodriguez reviewed by Ashutosh Chauhan) (addendum)
+ git clean -f -d
Removing standalone-metastore/src/gen/org/
+ git checkout master
Already on 'master'
Your branch is behind 'origin/master' by 7 commits, and can be fast-forwarded.
(use "git pull" to update your local branch)
+ git reset --hard origin/master
HEAD is now at 76d8809 HIVE-17864: PTestClient cannot start during Precommit
tests (Adam Szita, via Peter Vary)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2017-10-20 14:38:45.444
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
error: patch failed: dev-support/jenkins-execute-build.sh:48
error: dev-support/jenkins-execute-build.sh: patch does not apply
The patch does not appear to apply with p0, p1, or p2
+ exit 1
'
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12893214 - PreCommit-HIVE-Build
> PTestClient cannot start during Precommit tests
> ---
>
> Key: HIVE-17864
> URL: https://issues.apache.org/jira/browse/HIVE-17864
> Project: Hive
> Issue Type: Bug
>Reporter: Adam Szita
>Assignee: Adam Szita
> Attachments: HIVE-17864.0.patch
>
>
> HIVE-17807 has bumped version number in testutils/ptest2/pom.xml from 1.0 to
> 3.0 resulting failure to start PTestClient during Precommit runs:
> {code}
> java -cp
> '/home/jenkins/jenkins-slave/workspace/PreCommit-HIVE-Build/hive/build/hive/testutils/ptest2/target/hive-ptest-1.0-classes.jar:/home/jenkins/jenkins-slave/workspace/PreCommit-HIVE-Build/hive/build/hive/testutils/ptest2/target/lib/*'
> org.apache.hive.ptest.api.client.PTestClient --command testStart --outputDir
> /home/jenkins/jenkins-slave/workspace/PreCommit-HIVE-Build/hive/build/hive/testutils/ptest2/target
> --password '[***]' --testHandle PreCommit-HIVE-Build-7389 --endpoint
> http://104.198.109.242:8080/hive-ptest-1.0 --logsEndpoint
> http://104.198.109.242/logs/ --profile master-mr2 --patch
> https://issues.apache.org/jira/secure/attachment/12893016/HIVE-17842.0.patch
> --jira HIVE-17842
> Error: Could not find or load main class
> org.apache.hive.ptest.api.client.PTestClient
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-17866) MetaStoreEventListener - AlterTableEvent should include the partitions affected by the ALTER TABLE statement
[
https://issues.apache.org/jira/browse/HIVE-17866?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel del Castillo updated HIVE-17866:
---
Summary: MetaStoreEventListener - AlterTableEvent should include the
partitions affected by the ALTER TABLE statement (was: MetastoreEventListener
- {{AlterTableEvent}} should include the partitions affected by the {{ALTER
TABLE}} statement)
> MetaStoreEventListener - AlterTableEvent should include the partitions
> affected by the ALTER TABLE statement
>
>
> Key: HIVE-17866
> URL: https://issues.apache.org/jira/browse/HIVE-17866
> Project: Hive
> Issue Type: Improvement
> Components: Metastore
>Affects Versions: 2.3.0
>Reporter: Daniel del Castillo
>Priority: Minor
>
> Extend {{AlterTableEvent}} to include the set of partitions that have been
> modified by {{HiveAlterHandle}} during the execution of {{alterTable()}},
> e.g. if the statement includes the {{CASCADE}} option.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-17640) Comparison of date return null if time part is provided in string.
[ https://issues.apache.org/jira/browse/HIVE-17640?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yongzhi Chen updated HIVE-17640: Resolution: Fixed Status: Resolved (was: Patch Available) Push the fix to branch-2 > Comparison of date return null if time part is provided in string. > -- > > Key: HIVE-17640 > URL: https://issues.apache.org/jira/browse/HIVE-17640 > Project: Hive > Issue Type: Bug >Reporter: Yongzhi Chen >Assignee: Yongzhi Chen > Fix For: 2.4.0 > > Attachments: HIVE-17640.01-branch-2.patch > > > Reproduce: > select '2017-01-01 00:00:00' < current_date; > INFO : OK > ... > 1 row selected (18.324 seconds) > ... > NULL -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17865) MetaStoreEventListener - OnUpdate and OnDelete events required for capturing the event on UPDATE and DELETE statements respectively
[
https://issues.apache.org/jira/browse/HIVE-17865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel del Castillo updated HIVE-17865:
---
Summary: MetaStoreEventListener - OnUpdate and OnDelete events required for
capturing the event on UPDATE and DELETE statements respectively (was:
MetastoreEventListener - OnUpdate and OnDelete events required for capturing
the event on {{UPDATE}} and {{DELETE}} statements respectively)
> MetaStoreEventListener - OnUpdate and OnDelete events required for capturing
> the event on UPDATE and DELETE statements respectively
> ---
>
> Key: HIVE-17865
> URL: https://issues.apache.org/jira/browse/HIVE-17865
> Project: Hive
> Issue Type: New Feature
> Components: Metastore
>Affects Versions: 2.3.0
>Reporter: Daniel del Castillo
>Priority: Minor
>
> Extend {{MetaStoreEventListener}} with 2 new methods to capture the execution
> of {{UPDATE}} and {{DELETE}} statements, e.g.:
> {code:java}
> public void onUpdate(UpdateEvent updateEvent) throws MetaException {
> }
> public void onDelete(DeleteEvent deleteEvent) throws MetaException {
> }
> {code}
> In a similar fashion to {{InsertEvent}}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-17865) MetastoreEventListener - OnUpdate and OnDelete events required for capturing the event on {{UPDATE}} and {{DELETE}} statements respectively
[
https://issues.apache.org/jira/browse/HIVE-17865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel del Castillo updated HIVE-17865:
---
Component/s: Metastore
> MetastoreEventListener - OnUpdate and OnDelete events required for capturing
> the event on {{UPDATE}} and {{DELETE}} statements respectively
> ---
>
> Key: HIVE-17865
> URL: https://issues.apache.org/jira/browse/HIVE-17865
> Project: Hive
> Issue Type: New Feature
> Components: Metastore
>Affects Versions: 2.3.0
>Reporter: Daniel del Castillo
>Priority: Minor
>
> Extend {{MetaStoreEventListener}} with 2 new methods to capture the execution
> of {{UPDATE}} and {{DELETE}} statements, e.g.:
> {code:java}
> public void onUpdate(UpdateEvent updateEvent) throws MetaException {
> }
> public void onDelete(DeleteEvent deleteEvent) throws MetaException {
> }
> {code}
> In a similar fashion to {{InsertEvent}}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-17845) insert fails if target table columns are not lowercase
[ https://issues.apache.org/jira/browse/HIVE-17845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Naresh P R updated HIVE-17845: -- Attachment: HIVE-17845.patch > insert fails if target table columns are not lowercase > -- > > Key: HIVE-17845 > URL: https://issues.apache.org/jira/browse/HIVE-17845 > Project: Hive > Issue Type: Bug >Affects Versions: 1.2.1 >Reporter: Naresh P R >Assignee: Naresh P R >Priority: Minor > Fix For: 2.3.0 > > Attachments: HIVE-17845.patch > > > eg., > INSERT INTO TABLE EMP(ID,NAME) select * FROM SRC; > FAILED: SemanticException 1:27 '[ID,NAME]' in insert schema specification are > not found among regular columns of default.EMP nor dynamic partition > columns.. Error encountered near token 'NAME' > Whereas below insert is successful: > INSERT INTO TABLE EMP(id,name) select * FROM SRC; -- This message was sent by Atlassian JIRA (v6.4.14#64029)
