[jira] [Commented] (HIVE-17606) Improve security for DB notification related APIs
[ https://issues.apache.org/jira/browse/HIVE-17606?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16183695#comment-16183695 ] Lefty Leverenz commented on HIVE-17606: --- Thanks for fixing the parameter description. It looks good. > Improve security for DB notification related APIs > - > > Key: HIVE-17606 > URL: https://issues.apache.org/jira/browse/HIVE-17606 > Project: Hive > Issue Type: Improvement > Components: Metastore >Reporter: Tao Li >Assignee: Tao Li > Attachments: HIVE-17606.1.patch, HIVE-17606.2.patch, > HIVE-17606.3.patch, HIVE-17606.4.patch, HIVE-17606.5.patch > > > The purpose is to make sure only the superusers which are specified in the > proxyuser settings can make the db notification related API calls, since this > is supposed to be called by superuser/admin instead of any end user. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Comment Edited] (HIVE-17630) RESIGNAL:actual results are inconsistent with expectations at hplsql
[ https://issues.apache.org/jira/browse/HIVE-17630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16183687#comment-16183687 ] ZhangBing Lin edited comment on HIVE-17630 at 9/28/17 5:32 AM: --- At this page [http://www.hplsql.org/resignal],Example 1 and Example 2, actual results and sample results are inconsistent. was (Author: linzhangbing): At this page [http://www.hplsql.org/resignal],Example 1 and Example, actual results and sample results are inconsistent. > RESIGNAL:actual results are inconsistent with expectations at hplsql > > > Key: HIVE-17630 > URL: https://issues.apache.org/jira/browse/HIVE-17630 > Project: Hive > Issue Type: Bug > Components: hpl/sql >Affects Versions: 2.2.0, 3.0.0 >Reporter: ZhangBing Lin >Assignee: Dmitry Tolpeko >Priority: Minor > > when I execute example 3 at [http://www.hplsql.org/resignal]: > BEGIN > DECLARE EXIT HANDLER FOR SQLEXCEPTION > BEGIN > GET DIAGNOSTICS EXCEPTION 1 text = MESSAGE_TEXT; > PRINT 'SQLSTATE: ' || SQLSTATE; > PRINT 'Text: ' || text; > END; > > BEGIN > DECLARE EXIT HANDLER FOR SQLEXCEPTION > RESIGNAL SQLSTATE '02031' SET MESSAGE_TEXT = 'Some error'; > > SELECT * FROM abc.abc;-- Table does not exist, raise an exception > END; > END; > Actual results: > SQLSTATE: 42S02 > Text: Error while compiling statement: FAILED: SemanticException [Error > 10001]: Line 1:14 Table not found 'abc' > > The official result is： > SQLSTATE: 02031 > Text: Some error -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17630) RESIGNAL:actual results are inconsistent with expectations at hplsql
[ https://issues.apache.org/jira/browse/HIVE-17630?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ZhangBing Lin updated HIVE-17630: - Affects Version/s: 3.0.0 2.2.0 > RESIGNAL:actual results are inconsistent with expectations at hplsql > > > Key: HIVE-17630 > URL: https://issues.apache.org/jira/browse/HIVE-17630 > Project: Hive > Issue Type: Bug > Components: hpl/sql >Affects Versions: 2.2.0, 3.0.0 >Reporter: ZhangBing Lin >Assignee: Dmitry Tolpeko >Priority: Minor > > when I execute example 3 at [http://www.hplsql.org/resignal]: > BEGIN > DECLARE EXIT HANDLER FOR SQLEXCEPTION > BEGIN > GET DIAGNOSTICS EXCEPTION 1 text = MESSAGE_TEXT; > PRINT 'SQLSTATE: ' || SQLSTATE; > PRINT 'Text: ' || text; > END; > > BEGIN > DECLARE EXIT HANDLER FOR SQLEXCEPTION > RESIGNAL SQLSTATE '02031' SET MESSAGE_TEXT = 'Some error'; > > SELECT * FROM abc.abc;-- Table does not exist, raise an exception > END; > END; > Actual results: > SQLSTATE: 42S02 > Text: Error while compiling statement: FAILED: SemanticException [Error > 10001]: Line 1:14 Table not found 'abc' > > The official result is： > SQLSTATE: 02031 > Text: Some error -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17630) RESIGNAL:actual results are inconsistent with expectations at hplsql
[ https://issues.apache.org/jira/browse/HIVE-17630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16183687#comment-16183687 ] ZhangBing Lin commented on HIVE-17630: -- At this page [http://www.hplsql.org/resignal],Example 1 and Example, actual results and sample results are inconsistent. > RESIGNAL:actual results are inconsistent with expectations at hplsql > > > Key: HIVE-17630 > URL: https://issues.apache.org/jira/browse/HIVE-17630 > Project: Hive > Issue Type: Bug > Components: hpl/sql >Reporter: ZhangBing Lin >Assignee: Dmitry Tolpeko >Priority: Minor > > when I execute example 3 at [http://www.hplsql.org/resignal]: > BEGIN > DECLARE EXIT HANDLER FOR SQLEXCEPTION > BEGIN > GET DIAGNOSTICS EXCEPTION 1 text = MESSAGE_TEXT; > PRINT 'SQLSTATE: ' || SQLSTATE; > PRINT 'Text: ' || text; > END; > > BEGIN > DECLARE EXIT HANDLER FOR SQLEXCEPTION > RESIGNAL SQLSTATE '02031' SET MESSAGE_TEXT = 'Some error'; > > SELECT * FROM abc.abc;-- Table does not exist, raise an exception > END; > END; > Actual results: > SQLSTATE: 42S02 > Text: Error while compiling statement: FAILED: SemanticException [Error > 10001]: Line 1:14 Table not found 'abc' > > The official result is： > SQLSTATE: 02031 > Text: Some error -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17623) Fix Select query Fix Double column serde and some refactoring
[ https://issues.apache.org/jira/browse/HIVE-17623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16183682#comment-16183682 ] Ashutosh Chauhan commented on HIVE-17623: - TestDruidSerDe failure looks related. > Fix Select query Fix Double column serde and some refactoring > - > > Key: HIVE-17623 > URL: https://issues.apache.org/jira/browse/HIVE-17623 > Project: Hive > Issue Type: Bug > Components: Druid integration >Affects Versions: 3.0.0 >Reporter: slim bouguerra >Assignee: Ashutosh Chauhan > Attachments: HIVE-17623.2.patch, HIVE-17623.patch > > > This PR has 2 fixes. > First, fixes the limit of results returned by Select query that used to be > limited to 16K rows > Second fixes the type inference for the double type newly added to druid. > Use Jackson polymorphism to infer types and parse results from druid nodes. > Removes duplicate codes form RecordReaders. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17630) RESIGNAL:actual results are inconsistent with expectations at hplsql
[ https://issues.apache.org/jira/browse/HIVE-17630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16183680#comment-16183680 ] Dmitry Tolpeko commented on HIVE-17630: --- I will check this. > RESIGNAL:actual results are inconsistent with expectations at hplsql > > > Key: HIVE-17630 > URL: https://issues.apache.org/jira/browse/HIVE-17630 > Project: Hive > Issue Type: Bug > Components: hpl/sql >Reporter: ZhangBing Lin >Assignee: Dmitry Tolpeko >Priority: Minor > > when I execute example 3 at [http://www.hplsql.org/resignal]: > BEGIN > DECLARE EXIT HANDLER FOR SQLEXCEPTION > BEGIN > GET DIAGNOSTICS EXCEPTION 1 text = MESSAGE_TEXT; > PRINT 'SQLSTATE: ' || SQLSTATE; > PRINT 'Text: ' || text; > END; > > BEGIN > DECLARE EXIT HANDLER FOR SQLEXCEPTION > RESIGNAL SQLSTATE '02031' SET MESSAGE_TEXT = 'Some error'; > > SELECT * FROM abc.abc;-- Table does not exist, raise an exception > END; > END; > Actual results: > SQLSTATE: 42S02 > Text: Error while compiling statement: FAILED: SemanticException [Error > 10001]: Line 1:14 Table not found 'abc' > > The official result is： > SQLSTATE: 02031 > Text: Some error -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17625) Replication: update hive.repl.partitions.dump.parallelism to 100
[ https://issues.apache.org/jira/browse/HIVE-17625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16183676#comment-16183676 ] Thejas M Nair commented on HIVE-17625: -- +1 > Replication: update hive.repl.partitions.dump.parallelism to 100 > > > Key: HIVE-17625 > URL: https://issues.apache.org/jira/browse/HIVE-17625 > Project: Hive > Issue Type: Bug > Components: repl >Reporter: Vaibhav Gumashta >Assignee: Vaibhav Gumashta > Fix For: 3.0.0 > > Attachments: HIVE-17625.1.patch > > > Set hive.repl.partitions.dump.parallelism=100 -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-12734) Remove redundancy in HiveConfs serialized to UDFContext
[
https://issues.apache.org/jira/browse/HIVE-12734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16183670#comment-16183670
]
Hive QA commented on HIVE-12734:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12889360/HIVE-12734.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 7 failed/errored test(s), 11088 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_predicate_pushdown]
(batchId=232)
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_single_sourced_multi_insert]
(batchId=232)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[optimize_nullscan]
(batchId=162)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[union_fast_stats]
(batchId=157)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_explainuser_1]
(batchId=171)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14]
(batchId=236)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query23]
(batchId=236)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/7015/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/7015/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-7015/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 7 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12889360 - PreCommit-HIVE-Build
> Remove redundancy in HiveConfs serialized to UDFContext
> ---
>
> Key: HIVE-12734
> URL: https://issues.apache.org/jira/browse/HIVE-12734
> Project: Hive
> Issue Type: Bug
>Affects Versions: 1.2.1, 2.0.0, 2.2.0, 3.0.0
>Reporter: Mithun Radhakrishnan
>Assignee: Mithun Radhakrishnan
> Attachments: HIVE-12734.1.patch
>
>
> {{HCatLoader}} lands up serializing one {{HiveConf}} instance per
> table-alias, to Pig's {{UDFContext}}. This lands up bloating the
> {{UDFContext}}.
> To reduce the footprint, it makes sense to serialize a default-constructed
> {{HiveConf}} once, and one "diff" per {{HCatLoader}}. This should reduce the
> time taken to kick off jobs from {{pig -useHCatalog}} scripts.
> (Note_to_self: YHIVE-540).
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Assigned] (HIVE-17630) RESIGNAL:actual results are inconsistent with expectations at hplsql
[ https://issues.apache.org/jira/browse/HIVE-17630?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dmitry Tolpeko reassigned HIVE-17630: - Assignee: Dmitry Tolpeko > RESIGNAL:actual results are inconsistent with expectations at hplsql > > > Key: HIVE-17630 > URL: https://issues.apache.org/jira/browse/HIVE-17630 > Project: Hive > Issue Type: Bug > Components: hpl/sql >Reporter: ZhangBing Lin >Assignee: Dmitry Tolpeko >Priority: Minor > > when I execute example 3 at [http://www.hplsql.org/resignal]: > BEGIN > DECLARE EXIT HANDLER FOR SQLEXCEPTION > BEGIN > GET DIAGNOSTICS EXCEPTION 1 text = MESSAGE_TEXT; > PRINT 'SQLSTATE: ' || SQLSTATE; > PRINT 'Text: ' || text; > END; > > BEGIN > DECLARE EXIT HANDLER FOR SQLEXCEPTION > RESIGNAL SQLSTATE '02031' SET MESSAGE_TEXT = 'Some error'; > > SELECT * FROM abc.abc;-- Table does not exist, raise an exception > END; > END; > Actual results: > SQLSTATE: 42S02 > Text: Error while compiling statement: FAILED: SemanticException [Error > 10001]: Line 1:14 Table not found 'abc' > > The official result is： > SQLSTATE: 02031 > Text: Some error -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17606) Improve security for DB notification related APIs
[
https://issues.apache.org/jira/browse/HIVE-17606?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16183645#comment-16183645
]
Hive QA commented on HIVE-17606:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12889347/HIVE-17606.5.patch
{color:green}SUCCESS:{color} +1 due to 8 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 18 failed/errored test(s), 11089 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_predicate_pushdown]
(batchId=232)
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_single_sourced_multi_insert]
(batchId=232)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[optimize_nullscan]
(batchId=162)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[union_fast_stats]
(batchId=157)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_explainuser_1]
(batchId=171)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_2]
(batchId=101)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14]
(batchId=236)
org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager2.checkExpectedLocks
(batchId=285)
org.apache.hadoop.hive.ql.parse.TestReplicationScenariosAcrossInstances.parallelExecutionOfReplicationBootStrapLoad
(batchId=220)
org.apache.hadoop.hive.ql.parse.TestReplicationScenariosAcrossInstances.testBootstrapFunctionReplication
(batchId=220)
org.apache.hadoop.hive.ql.parse.TestReplicationScenariosAcrossInstances.testCreateFunctionIncrementalReplication
(batchId=220)
org.apache.hadoop.hive.ql.parse.TestReplicationScenariosAcrossInstances.testCreateFunctionWithFunctionBinaryJarsOnHDFS
(batchId=220)
org.apache.hadoop.hive.ql.parse.TestReplicationScenariosAcrossInstances.testDropFunctionIncrementalReplication
(batchId=220)
org.apache.hadoop.hive.ql.parse.TestReplicationScenariosAcrossInstances.testMultipleStagesOfReplicationLoadTask
(batchId=220)
org.apache.hive.hcatalog.api.TestHCatClient.testReplicationTaskIter
(batchId=181)
org.apache.hive.hcatalog.api.repl.commands.TestCommands.testBasicReplEximCommands
(batchId=181)
org.apache.hive.hcatalog.api.repl.commands.TestCommands.testNoopReplEximCommands
(batchId=181)
org.apache.hive.jdbc.TestJdbcWithMiniHS2.testReplDumpResultSet (batchId=229)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/7014/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/7014/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-7014/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 18 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12889347 - PreCommit-HIVE-Build
> Improve security for DB notification related APIs
> -
>
> Key: HIVE-17606
> URL: https://issues.apache.org/jira/browse/HIVE-17606
> Project: Hive
> Issue Type: Improvement
> Components: Metastore
>Reporter: Tao Li
>Assignee: Tao Li
> Attachments: HIVE-17606.1.patch, HIVE-17606.2.patch,
> HIVE-17606.3.patch, HIVE-17606.4.patch, HIVE-17606.5.patch
>
>
> The purpose is to make sure only the superusers which are specified in the
> proxyuser settings can make the db notification related API calls, since this
> is supposed to be called by superuser/admin instead of any end user.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-17631) upgrade orc to 1.4.0
[ https://issues.apache.org/jira/browse/HIVE-17631?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saijin Huang updated HIVE-17631: Status: Patch Available (was: Open) > upgrade orc to 1.4.0 > > > Key: HIVE-17631 > URL: https://issues.apache.org/jira/browse/HIVE-17631 > Project: Hive > Issue Type: Improvement >Affects Versions: 3.0.0 >Reporter: Saijin Huang >Assignee: Saijin Huang > Attachments: HIVE-17631.1.patch > > > It seems like orc 1.4.0 has a latest and stable version: > https://orc.apache.org/docs/releases.html -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17631) upgrade orc to 1.4.0
[ https://issues.apache.org/jira/browse/HIVE-17631?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saijin Huang updated HIVE-17631: Description: It seems like orc 1.4.0 has a latest and stable version: https://orc.apache.org/docs/releases.html > upgrade orc to 1.4.0 > > > Key: HIVE-17631 > URL: https://issues.apache.org/jira/browse/HIVE-17631 > Project: Hive > Issue Type: Improvement >Affects Versions: 3.0.0 >Reporter: Saijin Huang >Assignee: Saijin Huang > Attachments: HIVE-17631.1.patch > > > It seems like orc 1.4.0 has a latest and stable version: > https://orc.apache.org/docs/releases.html -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17631) upgrade orc to 1.4.0
[ https://issues.apache.org/jira/browse/HIVE-17631?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saijin Huang updated HIVE-17631: Attachment: HIVE-17631.1.patch > upgrade orc to 1.4.0 > > > Key: HIVE-17631 > URL: https://issues.apache.org/jira/browse/HIVE-17631 > Project: Hive > Issue Type: Improvement >Affects Versions: 3.0.0 > Environment: It seems like orc 1.4.0 has a latest and stable version: > https://orc.apache.org/docs/releases.html >Reporter: Saijin Huang >Assignee: Saijin Huang > Attachments: HIVE-17631.1.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17631) upgrade orc to 1.4.0
[ https://issues.apache.org/jira/browse/HIVE-17631?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saijin Huang updated HIVE-17631: Environment: (was: It seems like orc 1.4.0 has a latest and stable version: https://orc.apache.org/docs/releases.html) > upgrade orc to 1.4.0 > > > Key: HIVE-17631 > URL: https://issues.apache.org/jira/browse/HIVE-17631 > Project: Hive > Issue Type: Improvement >Affects Versions: 3.0.0 >Reporter: Saijin Huang >Assignee: Saijin Huang > Attachments: HIVE-17631.1.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (HIVE-17631) upgrade orc to 1.4.0
[ https://issues.apache.org/jira/browse/HIVE-17631?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saijin Huang reassigned HIVE-17631: --- > upgrade orc to 1.4.0 > > > Key: HIVE-17631 > URL: https://issues.apache.org/jira/browse/HIVE-17631 > Project: Hive > Issue Type: Improvement >Affects Versions: 3.0.0 > Environment: It seems like orc 1.4.0 has a latest and stable version: > https://orc.apache.org/docs/releases.html >Reporter: Saijin Huang >Assignee: Saijin Huang > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17623) Fix Select query Fix Double column serde and some refactoring
[
https://issues.apache.org/jira/browse/HIVE-17623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16183591#comment-16183591
]
Hive QA commented on HIVE-17623:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12889394/HIVE-17623.2.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 8 failed/errored test(s), 11088 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_predicate_pushdown]
(batchId=232)
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_single_sourced_multi_insert]
(batchId=232)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[optimize_nullscan]
(batchId=162)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[union_fast_stats]
(batchId=157)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_explainuser_1]
(batchId=171)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_2]
(batchId=101)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14]
(batchId=236)
org.apache.hadoop.hive.druid.serde.TestDruidSerDe.testDruidDeserializer
(batchId=249)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/7013/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/7013/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-7013/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 8 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12889394 - PreCommit-HIVE-Build
> Fix Select query Fix Double column serde and some refactoring
> -
>
> Key: HIVE-17623
> URL: https://issues.apache.org/jira/browse/HIVE-17623
> Project: Hive
> Issue Type: Bug
> Components: Druid integration
>Affects Versions: 3.0.0
>Reporter: slim bouguerra
>Assignee: Ashutosh Chauhan
> Attachments: HIVE-17623.2.patch, HIVE-17623.patch
>
>
> This PR has 2 fixes.
> First, fixes the limit of results returned by Select query that used to be
> limited to 16K rows
> Second fixes the type inference for the double type newly added to druid.
> Use Jackson polymorphism to infer types and parse results from druid nodes.
> Removes duplicate codes form RecordReaders.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-17486) Enable SharedWorkOptimizer in tez on HOS
[ https://issues.apache.org/jira/browse/HIVE-17486?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liyunzhang_intel updated HIVE-17486: Description: in HIVE-16602, Implement shared scans with Tez. Given a query plan, the goal is to identify scans on input tables that can be merged so the data is read only once. Optimization will be carried out at the physical level. In Hive on Spark, it caches the result ofsSpark work if the spark work is used by more than 1 child spark work. After sharedWorkOptimizer is enabled in physical plan in HoS, the identical table scans are merged to 1 table scan. This result of table scan will be used by more 1 child spark work. Thus we need not do the same computation because of cache mechanism. was: in HIVE-16602, Implement shared scans with Tez. Given a query plan, the goal is to identify scans on input tables that can be merged so the data is read only once. Optimization will be carried out at the physical level. > Enable SharedWorkOptimizer in tez on HOS > > > Key: HIVE-17486 > URL: https://issues.apache.org/jira/browse/HIVE-17486 > Project: Hive > Issue Type: Bug >Reporter: liyunzhang_intel >Assignee: liyunzhang_intel > > in HIVE-16602, Implement shared scans with Tez. > Given a query plan, the goal is to identify scans on input tables that can be > merged so the data is read only once. Optimization will be carried out at the > physical level. In Hive on Spark, it caches the result ofsSpark work if the > spark work is used by more than 1 child spark work. After sharedWorkOptimizer > is enabled in physical plan in HoS, the identical table scans are merged to 1 > table scan. This result of table scan will be used by more 1 child spark > work. Thus we need not do the same computation because of cache mechanism. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17486) Enable SharedWorkOptimizer in tez on HOS
[ https://issues.apache.org/jira/browse/HIVE-17486?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liyunzhang_intel updated HIVE-17486: Description: in HIVE-16602, Implement shared scans with Tez. Given a query plan, the goal is to identify scans on input tables that can be merged so the data is read only once. Optimization will be carried out at the physical level. In Hive on Spark, it caches the result of spark work if the spark work is used by more than 1 child spark work. After sharedWorkOptimizer is enabled in physical plan in HoS, the identical table scans are merged to 1 table scan. This result of table scan will be used by more 1 child spark work. Thus we need not do the same computation because of cache mechanism. was: in HIVE-16602, Implement shared scans with Tez. Given a query plan, the goal is to identify scans on input tables that can be merged so the data is read only once. Optimization will be carried out at the physical level. In Hive on Spark, it caches the result ofsSpark work if the spark work is used by more than 1 child spark work. After sharedWorkOptimizer is enabled in physical plan in HoS, the identical table scans are merged to 1 table scan. This result of table scan will be used by more 1 child spark work. Thus we need not do the same computation because of cache mechanism. > Enable SharedWorkOptimizer in tez on HOS > > > Key: HIVE-17486 > URL: https://issues.apache.org/jira/browse/HIVE-17486 > Project: Hive > Issue Type: Bug >Reporter: liyunzhang_intel >Assignee: liyunzhang_intel > > in HIVE-16602, Implement shared scans with Tez. > Given a query plan, the goal is to identify scans on input tables that can be > merged so the data is read only once. Optimization will be carried out at the > physical level. In Hive on Spark, it caches the result of spark work if the > spark work is used by more than 1 child spark work. After sharedWorkOptimizer > is enabled in physical plan in HoS, the identical table scans are merged to 1 > table scan. This result of table scan will be used by more 1 child spark > work. Thus we need not do the same computation because of cache mechanism. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17630) RESIGNAL:actual results are inconsistent with expectations at hplsql
[ https://issues.apache.org/jira/browse/HIVE-17630?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ZhangBing Lin updated HIVE-17630: - Description: when I execute example 3 at [http://www.hplsql.org/resignal]: BEGIN DECLARE EXIT HANDLER FOR SQLEXCEPTION BEGIN GET DIAGNOSTICS EXCEPTION 1 text = MESSAGE_TEXT; PRINT 'SQLSTATE: ' || SQLSTATE; PRINT 'Text: ' || text; END; BEGIN DECLARE EXIT HANDLER FOR SQLEXCEPTION RESIGNAL SQLSTATE '02031' SET MESSAGE_TEXT = 'Some error'; SELECT * FROM abc.abc;-- Table does not exist, raise an exception END; END; Actual results: SQLSTATE: 42S02 Text: Error while compiling statement: FAILED: SemanticException [Error 10001]: Line 1:14 Table not found 'abc' The official result is： SQLSTATE: 02031 Text: Some error was: when I execute example 3 at [http://www.hplsql.org/resignal]: BEGIN DECLARE EXIT HANDLER FOR SQLEXCEPTION BEGIN GET DIAGNOSTICS EXCEPTION 1 text = MESSAGE_TEXT; PRINT 'SQLSTATE: ' || SQLSTATE; PRINT 'Text: ' || text; END; BEGIN DECLARE EXIT HANDLER FOR SQLEXCEPTION RESIGNAL SQLSTATE '02031' SET MESSAGE_TEXT = 'Some error'; SELECT * FROM abc.abc;-- Table does not exist, raise an exception END; END; result: SQLSTATE: 42S02 Text: Error while compiling statement: FAILED: SemanticException [Error 10001]: Line 1:14 Table not found 'abc' > RESIGNAL:actual results are inconsistent with expectations at hplsql > > > Key: HIVE-17630 > URL: https://issues.apache.org/jira/browse/HIVE-17630 > Project: Hive > Issue Type: Bug > Components: hpl/sql >Reporter: ZhangBing Lin >Priority: Minor > > when I execute example 3 at [http://www.hplsql.org/resignal]: > BEGIN > DECLARE EXIT HANDLER FOR SQLEXCEPTION > BEGIN > GET DIAGNOSTICS EXCEPTION 1 text = MESSAGE_TEXT; > PRINT 'SQLSTATE: ' || SQLSTATE; > PRINT 'Text: ' || text; > END; > > BEGIN > DECLARE EXIT HANDLER FOR SQLEXCEPTION > RESIGNAL SQLSTATE '02031' SET MESSAGE_TEXT = 'Some error'; > > SELECT * FROM abc.abc;-- Table does not exist, raise an exception > END; > END; > Actual results: > SQLSTATE: 42S02 > Text: Error while compiling statement: FAILED: SemanticException [Error > 10001]: Line 1:14 Table not found 'abc' > > The official result is： > SQLSTATE: 02031 > Text: Some error -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17630) RESIGNAL:actual results are inconsistent with expectations at hplsql
[ https://issues.apache.org/jira/browse/HIVE-17630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16183571#comment-16183571 ] ZhangBing Lin commented on HIVE-17630: -- [~dmtolpeko],can you help me to answer my doubts?Thank you > RESIGNAL:actual results are inconsistent with expectations at hplsql > > > Key: HIVE-17630 > URL: https://issues.apache.org/jira/browse/HIVE-17630 > Project: Hive > Issue Type: Bug > Components: hpl/sql >Reporter: ZhangBing Lin >Priority: Minor > > when I execute example 3 at [http://www.hplsql.org/resignal]: > BEGIN > DECLARE EXIT HANDLER FOR SQLEXCEPTION > BEGIN > GET DIAGNOSTICS EXCEPTION 1 text = MESSAGE_TEXT; > PRINT 'SQLSTATE: ' || SQLSTATE; > PRINT 'Text: ' || text; > END; > > BEGIN > DECLARE EXIT HANDLER FOR SQLEXCEPTION > RESIGNAL SQLSTATE '02031' SET MESSAGE_TEXT = 'Some error'; > > SELECT * FROM abc.abc;-- Table does not exist, raise an exception > END; > END; > result: > SQLSTATE: 42S02 > Text: Error while compiling statement: FAILED: SemanticException [Error > 10001]: Line 1:14 Table not found 'abc' -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17630) RESIGNAL:actual results are inconsistent with expectations at hplsql
[ https://issues.apache.org/jira/browse/HIVE-17630?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ZhangBing Lin updated HIVE-17630: - Description: when I execute example 3 at [http://www.hplsql.org/resignal]: BEGIN DECLARE EXIT HANDLER FOR SQLEXCEPTION BEGIN GET DIAGNOSTICS EXCEPTION 1 text = MESSAGE_TEXT; PRINT 'SQLSTATE: ' || SQLSTATE; PRINT 'Text: ' || text; END; BEGIN DECLARE EXIT HANDLER FOR SQLEXCEPTION RESIGNAL SQLSTATE '02031' SET MESSAGE_TEXT = 'Some error'; SELECT * FROM abc.abc;-- Table does not exist, raise an exception END; END; result: SQLSTATE: 42S02 Text: Error while compiling statement: FAILED: SemanticException [Error 10001]: Line 1:14 Table not found 'abc' > RESIGNAL:actual results are inconsistent with expectations at hplsql > > > Key: HIVE-17630 > URL: https://issues.apache.org/jira/browse/HIVE-17630 > Project: Hive > Issue Type: Bug > Components: hpl/sql >Reporter: ZhangBing Lin >Priority: Minor > > when I execute example 3 at [http://www.hplsql.org/resignal]: > BEGIN > DECLARE EXIT HANDLER FOR SQLEXCEPTION > BEGIN > GET DIAGNOSTICS EXCEPTION 1 text = MESSAGE_TEXT; > PRINT 'SQLSTATE: ' || SQLSTATE; > PRINT 'Text: ' || text; > END; > > BEGIN > DECLARE EXIT HANDLER FOR SQLEXCEPTION > RESIGNAL SQLSTATE '02031' SET MESSAGE_TEXT = 'Some error'; > > SELECT * FROM abc.abc;-- Table does not exist, raise an exception > END; > END; > result: > SQLSTATE: 42S02 > Text: Error while compiling statement: FAILED: SemanticException [Error > 10001]: Line 1:14 Table not found 'abc' -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17629) CachedStore - wait for prewarm at use time, not init time
[ https://issues.apache.org/jira/browse/HIVE-17629?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-17629: Attachment: HIVE-17629.patch Preliminary patch, need to add config check (need to run now), probably some tests will fail. This moves wait on usage side, so the init is not blocked and if you don't use specific methods you never wait. Also initializes in HS2 (need to also add to standalone metastore), and prewarm is moved to background thread. Also cleaned up a little bit - some stuff that's shared was not static even though shared state is all static, etc. > CachedStore - wait for prewarm at use time, not init time > - > > Key: HIVE-17629 > URL: https://issues.apache.org/jira/browse/HIVE-17629 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-17629.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17621) Hive-site settings are ignored during HCatInputFormat split-calculation
[
https://issues.apache.org/jira/browse/HIVE-17621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16183545#comment-16183545
]
Hive QA commented on HIVE-17621:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12889332/HIVE-17621.1-branch-2.2.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 60 failed/errored test(s), 9936 tests
executed
*Failed tests:*
{noformat}
TestDerbyConnector - did not produce a TEST-*.xml file (likely timed out)
(batchId=244)
TestJdbcDriver2 - did not produce a TEST-*.xml file (likely timed out)
(batchId=225)
TestMiniLlapLocalCliDriver - did not produce a TEST-*.xml file (likely timed
out) (batchId=167)
[acid_globallimit.q,alter_merge_2_orc.q]
TestMiniSparkOnYarnCliDriver - did not produce a TEST-*.xml file (likely timed
out) (batchId=173)
[infer_bucket_sort_reducers_power_two.q,list_bucket_dml_10.q,orc_merge9.q,orc_merge6.q,leftsemijoin_mr.q,bucket6.q,bucketmapjoin7.q,uber_reduce.q,empty_dir_in_table.q,vector_outer_join3.q,index_bitmap_auto.q,vector_outer_join2.q,vector_outer_join1.q,orc_merge1.q,orc_merge_diff_fs.q,load_hdfs_file_with_space_in_the_name.q,scriptfile1_win.q,quotedid_smb.q,truncate_column_buckets.q,orc_merge3.q]
TestMiniSparkOnYarnCliDriver - did not produce a TEST-*.xml file (likely timed
out) (batchId=174)
[infer_bucket_sort_num_buckets.q,gen_udf_example_add10.q,insert_overwrite_directory2.q,orc_merge5.q,bucketmapjoin6.q,import_exported_table.q,vector_outer_join0.q,orc_merge4.q,temp_table_external.q,orc_merge_incompat1.q,root_dir_external_table.q,constprog_semijoin.q,auto_sortmerge_join_16.q,schemeAuthority.q,index_bitmap3.q,external_table_with_space_in_location_path.q,parallel_orderby.q,infer_bucket_sort_map_operators.q,bucketizedhiveinputformat.q,remote_script.q]
TestMiniSparkOnYarnCliDriver - did not produce a TEST-*.xml file (likely timed
out) (batchId=175)
[scriptfile1.q,vector_outer_join5.q,file_with_header_footer.q,bucket4.q,input16_cc.q,bucket5.q,infer_bucket_sort_merge.q,constprog_partitioner.q,orc_merge2.q,reduce_deduplicate.q,schemeAuthority2.q,load_fs2.q,orc_merge8.q,orc_merge_incompat2.q,infer_bucket_sort_bucketed_table.q,vector_outer_join4.q,disable_merge_for_bucketing.q,vector_inner_join.q,orc_merge7.q]
TestSparkCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=118)
[bucketmapjoin4.q,bucket_map_join_spark4.q,union21.q,groupby2_noskew.q,timestamp_2.q,date_join1.q,mergejoins.q,smb_mapjoin_11.q,auto_sortmerge_join_3.q,mapjoin_test_outer.q,vectorization_9.q,merge2.q,groupby6_noskew.q,auto_join_without_localtask.q,multi_join_union.q]
TestSparkCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=119)
[join_cond_pushdown_unqual4.q,union_remove_7.q,join13.q,join_vc.q,groupby_cube1.q,bucket_map_join_spark2.q,sample3.q,smb_mapjoin_19.q,stats16.q,union23.q,union.q,union31.q,cbo_udf_udaf.q,ptf_decimal.q,bucketmapjoin2.q]
TestSparkCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=120)
[parallel_join1.q,union27.q,union12.q,groupby7_map_multi_single_reducer.q,varchar_join1.q,join7.q,join_reorder4.q,skewjoinopt2.q,bucketsortoptimize_insert_2.q,smb_mapjoin_17.q,script_env_var1.q,groupby7_map.q,groupby3.q,bucketsortoptimize_insert_8.q,union20.q]
TestSparkCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=121)
[ptf_general_queries.q,auto_join_reordering_values.q,sample2.q,join1.q,decimal_join.q,mapjoin_subquery2.q,join32_lessSize.q,mapjoin1.q,order2.q,skewjoinopt18.q,union_remove_18.q,join25.q,groupby9.q,bucketsortoptimize_insert_6.q,ctas.q]
TestSparkCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=122)
[groupby_map_ppr.q,nullgroup4_multi_distinct.q,join_rc.q,union14.q,smb_mapjoin_12.q,vector_cast_constant.q,union_remove_4.q,auto_join11.q,load_dyn_part7.q,udaf_collect_set.q,vectorization_12.q,groupby_sort_skew_1.q,groupby_sort_skew_1_23.q,smb_mapjoin_25.q,skewjoinopt12.q]
TestSparkCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=123)
[skewjoinopt15.q,auto_join18.q,list_bucket_dml_2.q,input1_limit.q,load_dyn_part3.q,union_remove_14.q,auto_sortmerge_join_14.q,auto_sortmerge_join_15.q,union10.q,bucket_map_join_tez2.q,groupby5_map_skew.q,join_reorder.q,sample1.q,bucketmapjoin8.q,union34.q]
TestSparkCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=124)
[avro_joins.q,skewjoinopt16.q,auto_join14.q,vectorization_14.q,auto_join26.q,stats1.q,cbo_stats.q,auto_sortmerge_join_6.q,union22.q,union_remove_24.q,union_view.q,smb_mapjoin_22.q,stats15.q,ptf_matchpath.q,transform_ppr1.q]
TestSparkCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=125)
[jira] [Updated] (HIVE-17628) always use fully qualified path for tables/partitions/etc.
[ https://issues.apache.org/jira/browse/HIVE-17628?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-17628: Description: # Different services, or the same one at different times, may have different default FS, so it doesn't make sense to persist a non-qualified path. # The logic to detect whether we are using default FS or not is anyway rather questionable e.g. it will run if the setting is set to the same value as the default fs, as long as it's set. # In fact might be more expensive that just making the path qualified as it iterates thru all the properties, including the ones added from getConfVarInputStream. # It also hits HADOOP-13500. was: # Different services, or the same one at different times, may have different default FS, so it doesn't make sense to persist a non-qualified path. # The logic to detect whether we are using default FS or not is anyway rather questionable e.g. if it will run if the setting is set to the same value as the default fs, as long as it's set; and in fact might be more expensive that just making the path qualified as it iterates thru all the properties, including the ones added from getConfVarInputStream. # It also hits HADOOP-13500. > always use fully qualified path for tables/partitions/etc. > -- > > Key: HIVE-17628 > URL: https://issues.apache.org/jira/browse/HIVE-17628 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-17628.patch > > > # Different services, or the same one at different times, may have different > default FS, so it doesn't make sense to persist a non-qualified path. > # The logic to detect whether we are using default FS or not is anyway rather > questionable e.g. it will run if the setting is set to the same value as the > default fs, as long as it's set. > # In fact might be more expensive that just making the path qualified as it > iterates thru all the properties, including the ones added from > getConfVarInputStream. > # It also hits HADOOP-13500. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17628) always use fully qualified path for tables/partitions/etc.
[ https://issues.apache.org/jira/browse/HIVE-17628?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-17628: Status: Patch Available (was: Open) > always use fully qualified path for tables/partitions/etc. > -- > > Key: HIVE-17628 > URL: https://issues.apache.org/jira/browse/HIVE-17628 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-17628.patch > > > # Different services, or the same one at different times, may have different > default FS, so it doesn't make sense to persist a non-qualified path. > # The logic to detect whether we are using default FS or not is anyway rather > questionable e.g. if it will run if the setting is set to the same value as > the default fs, as long as it's set; and in fact might be more expensive that > just making the path qualified as it iterates thru all the properties, > including the ones added from getConfVarInputStream. > # It also hits HADOOP-13500. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17628) always use fully qualified path for tables/partitions/etc.
[ https://issues.apache.org/jira/browse/HIVE-17628?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-17628: Attachment: HIVE-17628.patch [~ashutoshc] can you take a look? > always use fully qualified path for tables/partitions/etc. > -- > > Key: HIVE-17628 > URL: https://issues.apache.org/jira/browse/HIVE-17628 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-17628.patch > > > # Different services, or the same one at different times, may have different > default FS, so it doesn't make sense to persist a non-qualified path. > # The logic to detect whether we are using default FS or not is anyway rather > questionable e.g. if it will run if the setting is set to the same value as > the default fs, as long as it's set; and in fact might be more expensive that > just making the path qualified as it iterates thru all the properties, > including the ones added from getConfVarInputStream. > # It also hits HADOOP-13500. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (HIVE-17629) CachedStore - wait for prewarm at use time, not init time
[ https://issues.apache.org/jira/browse/HIVE-17629?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin reassigned HIVE-17629: --- > CachedStore - wait for prewarm at use time, not init time > - > > Key: HIVE-17629 > URL: https://issues.apache.org/jira/browse/HIVE-17629 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (HIVE-17628) always use fully qualified path for tables/partitions/etc.
[ https://issues.apache.org/jira/browse/HIVE-17628?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin reassigned HIVE-17628: --- > always use fully qualified path for tables/partitions/etc. > -- > > Key: HIVE-17628 > URL: https://issues.apache.org/jira/browse/HIVE-17628 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > > # Different services, or the same one at different times, may have different > default FS, so it doesn't make sense to persist a non-qualified path. > # The logic to detect whether we are using default FS or not is anyway rather > questionable e.g. if it will run if the setting is set to the same value as > the default fs, as long as it's set; and in fact might be more expensive that > just making the path qualified as it iterates thru all the properties, > including the ones added from getConfVarInputStream. > # It also hits HADOOP-13500. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17621) Hive-site settings are ignored during HCatInputFormat split-calculation
[
https://issues.apache.org/jira/browse/HIVE-17621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16183487#comment-16183487
]
Hive QA commented on HIVE-17621:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12889332/HIVE-17621.1-branch-2.2.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 58 failed/errored test(s), 9936 tests
executed
*Failed tests:*
{noformat}
TestDerbyConnector - did not produce a TEST-*.xml file (likely timed out)
(batchId=244)
TestJdbcDriver2 - did not produce a TEST-*.xml file (likely timed out)
(batchId=225)
TestMiniLlapLocalCliDriver - did not produce a TEST-*.xml file (likely timed
out) (batchId=167)
[acid_globallimit.q,alter_merge_2_orc.q]
TestMiniSparkOnYarnCliDriver - did not produce a TEST-*.xml file (likely timed
out) (batchId=173)
[infer_bucket_sort_reducers_power_two.q,list_bucket_dml_10.q,orc_merge9.q,orc_merge6.q,leftsemijoin_mr.q,bucket6.q,bucketmapjoin7.q,uber_reduce.q,empty_dir_in_table.q,vector_outer_join3.q,index_bitmap_auto.q,vector_outer_join2.q,vector_outer_join1.q,orc_merge1.q,orc_merge_diff_fs.q,load_hdfs_file_with_space_in_the_name.q,scriptfile1_win.q,quotedid_smb.q,truncate_column_buckets.q,orc_merge3.q]
TestMiniSparkOnYarnCliDriver - did not produce a TEST-*.xml file (likely timed
out) (batchId=174)
[infer_bucket_sort_num_buckets.q,gen_udf_example_add10.q,insert_overwrite_directory2.q,orc_merge5.q,bucketmapjoin6.q,import_exported_table.q,vector_outer_join0.q,orc_merge4.q,temp_table_external.q,orc_merge_incompat1.q,root_dir_external_table.q,constprog_semijoin.q,auto_sortmerge_join_16.q,schemeAuthority.q,index_bitmap3.q,external_table_with_space_in_location_path.q,parallel_orderby.q,infer_bucket_sort_map_operators.q,bucketizedhiveinputformat.q,remote_script.q]
TestMiniSparkOnYarnCliDriver - did not produce a TEST-*.xml file (likely timed
out) (batchId=175)
[scriptfile1.q,vector_outer_join5.q,file_with_header_footer.q,bucket4.q,input16_cc.q,bucket5.q,infer_bucket_sort_merge.q,constprog_partitioner.q,orc_merge2.q,reduce_deduplicate.q,schemeAuthority2.q,load_fs2.q,orc_merge8.q,orc_merge_incompat2.q,infer_bucket_sort_bucketed_table.q,vector_outer_join4.q,disable_merge_for_bucketing.q,vector_inner_join.q,orc_merge7.q]
TestSparkCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=118)
[bucketmapjoin4.q,bucket_map_join_spark4.q,union21.q,groupby2_noskew.q,timestamp_2.q,date_join1.q,mergejoins.q,smb_mapjoin_11.q,auto_sortmerge_join_3.q,mapjoin_test_outer.q,vectorization_9.q,merge2.q,groupby6_noskew.q,auto_join_without_localtask.q,multi_join_union.q]
TestSparkCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=119)
[join_cond_pushdown_unqual4.q,union_remove_7.q,join13.q,join_vc.q,groupby_cube1.q,bucket_map_join_spark2.q,sample3.q,smb_mapjoin_19.q,stats16.q,union23.q,union.q,union31.q,cbo_udf_udaf.q,ptf_decimal.q,bucketmapjoin2.q]
TestSparkCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=120)
[parallel_join1.q,union27.q,union12.q,groupby7_map_multi_single_reducer.q,varchar_join1.q,join7.q,join_reorder4.q,skewjoinopt2.q,bucketsortoptimize_insert_2.q,smb_mapjoin_17.q,script_env_var1.q,groupby7_map.q,groupby3.q,bucketsortoptimize_insert_8.q,union20.q]
TestSparkCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=121)
[ptf_general_queries.q,auto_join_reordering_values.q,sample2.q,join1.q,decimal_join.q,mapjoin_subquery2.q,join32_lessSize.q,mapjoin1.q,order2.q,skewjoinopt18.q,union_remove_18.q,join25.q,groupby9.q,bucketsortoptimize_insert_6.q,ctas.q]
TestSparkCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=122)
[groupby_map_ppr.q,nullgroup4_multi_distinct.q,join_rc.q,union14.q,smb_mapjoin_12.q,vector_cast_constant.q,union_remove_4.q,auto_join11.q,load_dyn_part7.q,udaf_collect_set.q,vectorization_12.q,groupby_sort_skew_1.q,groupby_sort_skew_1_23.q,smb_mapjoin_25.q,skewjoinopt12.q]
TestSparkCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=123)
[skewjoinopt15.q,auto_join18.q,list_bucket_dml_2.q,input1_limit.q,load_dyn_part3.q,union_remove_14.q,auto_sortmerge_join_14.q,auto_sortmerge_join_15.q,union10.q,bucket_map_join_tez2.q,groupby5_map_skew.q,join_reorder.q,sample1.q,bucketmapjoin8.q,union34.q]
TestSparkCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=124)
[avro_joins.q,skewjoinopt16.q,auto_join14.q,vectorization_14.q,auto_join26.q,stats1.q,cbo_stats.q,auto_sortmerge_join_6.q,union22.q,union_remove_24.q,union_view.q,smb_mapjoin_22.q,stats15.q,ptf_matchpath.q,transform_ppr1.q]
TestSparkCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=125)
[jira] [Commented] (HIVE-17623) Fix Select query Fix Double column serde and some refactoring
[ https://issues.apache.org/jira/browse/HIVE-17623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16183485#comment-16183485 ] Ashutosh Chauhan commented on HIVE-17623: - +1 > Fix Select query Fix Double column serde and some refactoring > - > > Key: HIVE-17623 > URL: https://issues.apache.org/jira/browse/HIVE-17623 > Project: Hive > Issue Type: Bug > Components: Druid integration >Affects Versions: 3.0.0 >Reporter: slim bouguerra >Assignee: Ashutosh Chauhan > Attachments: HIVE-17623.2.patch, HIVE-17623.patch > > > This PR has 2 fixes. > First, fixes the limit of results returned by Select query that used to be > limited to 16K rows > Second fixes the type inference for the double type newly added to druid. > Use Jackson polymorphism to infer types and parse results from druid nodes. > Removes duplicate codes form RecordReaders. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17545) Make HoS RDD Cacheing Optimization Configurable
[
https://issues.apache.org/jira/browse/HIVE-17545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16183483#comment-16183483
]
Rui Li commented on HIVE-17545:
---
[~stakiar] I think so. We actually have {{SplitSparkWorkResolver}} to clone the
works if they have multiple children. Besides, RDD caching is also used in
other places like parallel order by. If we want to control the behaviour, guess
we need to consolidate the usage a little bit.
> Make HoS RDD Cacheing Optimization Configurable
> ---
>
> Key: HIVE-17545
> URL: https://issues.apache.org/jira/browse/HIVE-17545
> Project: Hive
> Issue Type: Improvement
> Components: Physical Optimizer, Spark
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
> Attachments: HIVE-17545.1.patch, HIVE-17545.2.patch
>
>
> The RDD cacheing optimization add in HIVE-10550 is enabled by default. We
> should make it configurable in case users want to disable it. We can leave it
> on by default to preserve backwards compatibility.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-13843) Re-enable the HoS tests disabled in HIVE-13402
[ https://issues.apache.org/jira/browse/HIVE-13843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16183480#comment-16183480 ] Rui Li commented on HIVE-13843: --- Thanks [~stakiar] for offering the help. > Re-enable the HoS tests disabled in HIVE-13402 > -- > > Key: HIVE-13843 > URL: https://issues.apache.org/jira/browse/HIVE-13843 > Project: Hive > Issue Type: Test >Reporter: Rui Li >Assignee: Sahil Takiar > Attachments: HIVE-13843.1.patch > > > With HIVE-13525, we can now fix and re-enable the tests for Spark. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17523) Insert into druid table hangs Hive server2 in an infinit loop
[
https://issues.apache.org/jira/browse/HIVE-17523?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
slim bouguerra updated HIVE-17523:
--
Attachment: HIVE-17523.2.patch
> Insert into druid table hangs Hive server2 in an infinit loop
> --
>
> Key: HIVE-17523
> URL: https://issues.apache.org/jira/browse/HIVE-17523
> Project: Hive
> Issue Type: Bug
> Components: Druid integration
>Reporter: slim bouguerra
>Assignee: Nishant Bangarwa
> Labels: pull-request-available
> Attachments: HIVE-17523.2.patch, HIVE-17523.patch
>
>
> Inserting data via insert into table backed by druid can lead to a Hive
> server hang.
> This is due to some bug in the naming of druid segments partitions.
> To reproduce the issue
> {code}
> drop table login_hive;
> create table login_hive(`timecolumn` timestamp, `userid` string, `num_l`
> double);
> insert into login_hive values ('2015-01-01 00:00:00', 'user1', 5);
> insert into login_hive values ('2015-01-01 01:00:00', 'user2', 4);
> insert into login_hive values ('2015-01-01 02:00:00', 'user3', 2);
> insert into login_hive values ('2015-01-02 00:00:00', 'user1', 1);
> insert into login_hive values ('2015-01-02 01:00:00', 'user2', 2);
> insert into login_hive values ('2015-01-02 02:00:00', 'user3', 8);
> insert into login_hive values ('2015-01-03 00:00:00', 'user1', 5);
> insert into login_hive values ('2015-01-03 01:00:00', 'user2', 9);
> insert into login_hive values ('2015-01-03 04:00:00', 'user3', 2);
> insert into login_hive values ('2015-03-09 00:00:00', 'user3', 5);
> insert into login_hive values ('2015-03-09 01:00:00', 'user1', 0);
> insert into login_hive values ('2015-03-09 05:00:00', 'user2', 0);
> drop table login_druid;
> CREATE TABLE login_druid
> STORED BY 'org.apache.hadoop.hive.druid.DruidStorageHandler'
> TBLPROPERTIES ("druid.datasource" = "druid_login_test_tmp",
> "druid.segment.granularity" = "DAY", "druid.query.granularity" = "HOUR")
> AS
> select `timecolumn` as `__time`, `userid`, `num_l` FROM login_hive;
> select * FROM login_druid;
> insert into login_druid values ('2015-03-09 05:00:00', 'user4', 0);
> {code}
> This patch unifies the logic of pushing and segments naming by using Druid
> data segment pusher as much as possible.
> This patch also has some minor code refactoring and test enhancements.
>
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-17623) Fix Select query Fix Double column serde and some refactoring
[ https://issues.apache.org/jira/browse/HIVE-17623?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] slim bouguerra updated HIVE-17623: -- Attachment: HIVE-17623.2.patch > Fix Select query Fix Double column serde and some refactoring > - > > Key: HIVE-17623 > URL: https://issues.apache.org/jira/browse/HIVE-17623 > Project: Hive > Issue Type: Bug > Components: Druid integration >Affects Versions: 3.0.0 >Reporter: slim bouguerra >Assignee: Ashutosh Chauhan > Attachments: HIVE-17623.2.patch, HIVE-17623.patch > > > This PR has 2 fixes. > First, fixes the limit of results returned by Select query that used to be > limited to 16K rows > Second fixes the type inference for the double type newly added to druid. > Use Jackson polymorphism to infer types and parse results from druid nodes. > Removes duplicate codes form RecordReaders. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17611) Add new LazyBinary SerDe for faster writes
[ https://issues.apache.org/jira/browse/HIVE-17611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Dere updated HIVE-17611: -- Resolution: Fixed Fix Version/s: 3.0.0 Status: Resolved (was: Patch Available) committed to master > Add new LazyBinary SerDe for faster writes > -- > > Key: HIVE-17611 > URL: https://issues.apache.org/jira/browse/HIVE-17611 > Project: Hive > Issue Type: Improvement > Components: Serializers/Deserializers >Reporter: Jason Dere >Assignee: Jason Dere > Fix For: 3.0.0 > > Attachments: HIVE-17611.1.patch > > > LazyBinarySerDe.serialize() ends up making > getCategory()/getPrimitiveCategory() calls for every column of every row. > Tried some simple tests to eliminate these calls for the non-vectorized > version, this looks like it speeds up the writes by ~3x. > Adding a LazyBinarySerDe2 class with this new implementation. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (HIVE-17591) Getting error while select,insert,update,delete to hive from IBM SPSS Modeler server
[ https://issues.apache.org/jira/browse/HIVE-17591?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] slim bouguerra reassigned HIVE-17591: - Assignee: slim bouguerra > Getting error while select,insert,update,delete to hive from IBM SPSS > Modeler server > -- > > Key: HIVE-17591 > URL: https://issues.apache.org/jira/browse/HIVE-17591 > Project: Hive > Issue Type: Bug > Components: Hive, HiveServer2 >Affects Versions: 1.2.2 > Environment: Redhat 7.2 >Reporter: sudarshana >Assignee: slim bouguerra >Priority: Blocker > Fix For: 0.13.1 > > > We are getting while we are connecting from IBM SPSS Modeler server which > intern use SPSS analytical server. But we have everything(hive authorization, > privilege ..etc) done. > A SQL exception occurred. The error is: > org.apache.hive.service.cli.HiveSQLException: Error while complining > statement:FAILED: HiveAccessControlException Permission denied: > Principal[name=,type] does not have following previlegers for operation > QUERY[[SELECT] on Object[type=TABLE_OR_VIEW,name]] -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (HIVE-17591) Getting error while select,insert,update,delete to hive from IBM SPSS Modeler server
[ https://issues.apache.org/jira/browse/HIVE-17591?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] slim bouguerra reassigned HIVE-17591: - Assignee: (was: slim bouguerra) > Getting error while select,insert,update,delete to hive from IBM SPSS > Modeler server > -- > > Key: HIVE-17591 > URL: https://issues.apache.org/jira/browse/HIVE-17591 > Project: Hive > Issue Type: Bug > Components: Hive, HiveServer2 >Affects Versions: 1.2.2 > Environment: Redhat 7.2 >Reporter: sudarshana >Priority: Blocker > Fix For: 0.13.1 > > > We are getting while we are connecting from IBM SPSS Modeler server which > intern use SPSS analytical server. But we have everything(hive authorization, > privilege ..etc) done. > A SQL exception occurred. The error is: > org.apache.hive.service.cli.HiveSQLException: Error while complining > statement:FAILED: HiveAccessControlException Permission denied: > Principal[name=,type] does not have following previlegers for operation > QUERY[[SELECT] on Object[type=TABLE_OR_VIEW,name]] -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17626) Query reoptimization using cached runtime statistics
[ https://issues.apache.org/jira/browse/HIVE-17626?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-17626: - Attachment: runtimestats.patch cc [~ashutoshc] > Query reoptimization using cached runtime statistics > > > Key: HIVE-17626 > URL: https://issues.apache.org/jira/browse/HIVE-17626 > Project: Hive > Issue Type: New Feature > Components: Logical Optimizer >Affects Versions: 3.0.0 >Reporter: Prasanth Jayachandran > Attachments: runtimestats.patch > > > Something similar to "EXPLAIN ANALYZE" where we annotate explain plan with > actual and estimated statistics. The runtime stats can be cached at query > level and subsequent execution of the same query can make use of the cached > statistics from the previous run for better optimization. > Some use cases, > 1) re-planning join query (mapjoin failures can be converted to shuffle joins) > 2) better statistics for table scan operator if dynamic partition pruning is > involved > 3) Better estimates for bloom filter initialization (setting expected entries > during merge) > This can extended to support wider queries by caching fragments of operator > plans scanning same table(s) or matching some operator sequences. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17588) LlapRowRecordReader doing name-based field lookup for every column of every row
[ https://issues.apache.org/jira/browse/HIVE-17588?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Dere updated HIVE-17588: -- Resolution: Fixed Fix Version/s: 3.0.0 Status: Resolved (was: Patch Available) Committed to master > LlapRowRecordReader doing name-based field lookup for every column of every > row > --- > > Key: HIVE-17588 > URL: https://issues.apache.org/jira/browse/HIVE-17588 > Project: Hive > Issue Type: Bug >Reporter: Jason Dere >Assignee: Jason Dere > Fix For: 3.0.0 > > Attachments: HIVE-17588.1.patch > > > setRowFromStruct() is using > StructObjectInspector.getStructFieldRef(fieldName), which does a name-based > lookup - this can be changed to do an index-based lookup which should be > faster. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17523) Insert into druid table hangs Hive server2 in an infinit loop
[
https://issues.apache.org/jira/browse/HIVE-17523?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16183419#comment-16183419
]
Ashutosh Chauhan commented on HIVE-17523:
-
+1
> Insert into druid table hangs Hive server2 in an infinit loop
> --
>
> Key: HIVE-17523
> URL: https://issues.apache.org/jira/browse/HIVE-17523
> Project: Hive
> Issue Type: Bug
> Components: Druid integration
>Reporter: slim bouguerra
>Assignee: Nishant Bangarwa
> Labels: pull-request-available
> Attachments: HIVE-17523.patch
>
>
> Inserting data via insert into table backed by druid can lead to a Hive
> server hang.
> This is due to some bug in the naming of druid segments partitions.
> To reproduce the issue
> {code}
> drop table login_hive;
> create table login_hive(`timecolumn` timestamp, `userid` string, `num_l`
> double);
> insert into login_hive values ('2015-01-01 00:00:00', 'user1', 5);
> insert into login_hive values ('2015-01-01 01:00:00', 'user2', 4);
> insert into login_hive values ('2015-01-01 02:00:00', 'user3', 2);
> insert into login_hive values ('2015-01-02 00:00:00', 'user1', 1);
> insert into login_hive values ('2015-01-02 01:00:00', 'user2', 2);
> insert into login_hive values ('2015-01-02 02:00:00', 'user3', 8);
> insert into login_hive values ('2015-01-03 00:00:00', 'user1', 5);
> insert into login_hive values ('2015-01-03 01:00:00', 'user2', 9);
> insert into login_hive values ('2015-01-03 04:00:00', 'user3', 2);
> insert into login_hive values ('2015-03-09 00:00:00', 'user3', 5);
> insert into login_hive values ('2015-03-09 01:00:00', 'user1', 0);
> insert into login_hive values ('2015-03-09 05:00:00', 'user2', 0);
> drop table login_druid;
> CREATE TABLE login_druid
> STORED BY 'org.apache.hadoop.hive.druid.DruidStorageHandler'
> TBLPROPERTIES ("druid.datasource" = "druid_login_test_tmp",
> "druid.segment.granularity" = "DAY", "druid.query.granularity" = "HOUR")
> AS
> select `timecolumn` as `__time`, `userid`, `num_l` FROM login_hive;
> select * FROM login_druid;
> insert into login_druid values ('2015-03-09 05:00:00', 'user4', 0);
> {code}
> This patch unifies the logic of pushing and segments naming by using Druid
> data segment pusher as much as possible.
> This patch also has some minor code refactoring and test enhancements.
>
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17614) Notification_sequence initialization using SQL statement which is compatible with Mysql 5.1
[
https://issues.apache.org/jira/browse/HIVE-17614?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16183399#comment-16183399
]
Hive QA commented on HIVE-17614:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12889245/HIVE-17614.0.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 16 failed/errored test(s), 11083 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_predicate_pushdown]
(batchId=231)
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_single_sourced_multi_insert]
(batchId=231)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[cte_2]
(batchId=143)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[global_limit]
(batchId=143)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[llap_udf]
(batchId=143)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[orc_ppd_schema_evol_3a]
(batchId=143)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[rcfile_createas1]
(batchId=143)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[schemeAuthority]
(batchId=143)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[tez_union_dynamic_partition]
(batchId=143)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[tez_union_dynamic_partition_2]
(batchId=143)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[unionDistinct_1]
(batchId=143)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[optimize_nullscan]
(batchId=161)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[union_fast_stats]
(batchId=156)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_explainuser_1]
(batchId=170)
org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager2.createTable (batchId=284)
org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager2.testMerge3Way01
(batchId=284)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/7010/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/7010/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-7010/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 16 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12889245 - PreCommit-HIVE-Build
> Notification_sequence initialization using SQL statement which is compatible
> with Mysql 5.1
> ---
>
> Key: HIVE-17614
> URL: https://issues.apache.org/jira/browse/HIVE-17614
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Affects Versions: 3.0.0
>Reporter: anishek
>Assignee: anishek
> Fix For: 3.0.0
>
> Attachments: HIVE-17614.0.patch
>
>
> Since a lot of people are still using hive with 5.1 mysql as the rdbms for
> metastore, it will be helpful to have the initialization statement introduced
> as part of HIVE-16886 for mysql to be friendlier to these older versions.
> INSERT INTO `NOTIFICATION_SEQUENCE` (`NNI_ID`, `NEXT_EVENT_ID`) SELECT * from
> (select 1 as `NNI_ID`, 1 as `NOTIFICATION_SEQUENCE`) a WHERE (select
> count(*) from `NOTIFICATION_SEQUENCE`) = 0;
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-17482) External LLAP client: acquire locks for tables queried directly by LLAP
[ https://issues.apache.org/jira/browse/HIVE-17482?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Dere updated HIVE-17482: -- Attachment: HIVE-17482.5.patch Added a unit test to TestAcidOnTez, will post update to RB > External LLAP client: acquire locks for tables queried directly by LLAP > --- > > Key: HIVE-17482 > URL: https://issues.apache.org/jira/browse/HIVE-17482 > Project: Hive > Issue Type: Sub-task > Components: llap >Reporter: Jason Dere >Assignee: Jason Dere > Attachments: HIVE-17482.1.patch, HIVE-17482.2.patch, > HIVE-17482.3.patch, HIVE-17482.4.patch, HIVE-17482.5.patch > > > When using the LLAP external client with simple queries (filter/project of > single table), the appropriate locks should be taken on the table being read > like they are for normal Hive queries. This is important in the case of > transactional tables being queried, since the compactor relies on the > presence of table locks to determine whether it can safely delete old > versions of compacted files without affecting currently running queries. > This does not have to happen in the complex query case, since a query is used > (with the appropriate locking mechanisms) to create/populate the temp table > holding the results to the complex query. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-16395) ConcurrentModificationException on config object in HoS
[
https://issues.apache.org/jira/browse/HIVE-16395?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16183376#comment-16183376
]
Andrew Sherman commented on HIVE-16395:
---
In the case of the reported bug S3AUtils.propagateBucketOptions() is cloning a
Configuration, and then iterating over the properties in the source
Configuration which is where the exception happend. So we could fix this
particular bug (in Hadoop) by having S3AUtils.propagateBucketOptions() iterate
over the clone it has just made, adding any new properties after the operation
has finished. I have code that demonstrates the problem, and a fix.
The more general fix is to clone the JobConf. I think we would do this by
setting in spark.hadoop.cloneConf to true in HiveSparkClientFactory.
I did some toy benchmarks on cloning a Configuration using
{noformat}
Configuration clone = new Configuration(original);
{noformat}
The time it takes depends on the size of the Configuration.
* A Configuration with 1000 properties takes less than 1 ms.
* A Configuration with 1 properties takes ~ 3.5 ms.
What do you think is the best approach [~lirui] [~stakiar] ?
> ConcurrentModificationException on config object in HoS
> ---
>
> Key: HIVE-16395
> URL: https://issues.apache.org/jira/browse/HIVE-16395
> Project: Hive
> Issue Type: Task
> Components: Spark
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>
> Looks like this is happening inside spark executors, looks to be some race
> condition when modifying {{Configuration}} objects.
> Stack-Trace:
> {code}
> java.io.IOException: java.lang.reflect.InvocationTargetException
> at
> org.apache.hadoop.hive.io.HiveIOExceptionHandlerChain.handleRecordReaderCreationException(HiveIOExceptionHandlerChain.java:97)
> at
> org.apache.hadoop.hive.io.HiveIOExceptionHandlerUtil.handleRecordReaderCreationException(HiveIOExceptionHandlerUtil.java:57)
> at
> org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.initNextRecordReader(HadoopShimsSecure.java:267)
> at
> org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.(HadoopShimsSecure.java:213)
> at
> org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileInputFormatShim.getRecordReader(HadoopShimsSecure.java:334)
> at
> org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getRecordReader(CombineHiveInputFormat.java:682)
> at org.apache.spark.rdd.HadoopRDD$$anon$1.(HadoopRDD.scala:240)
> at org.apache.spark.rdd.HadoopRDD.compute(HadoopRDD.scala:211)
> at org.apache.spark.rdd.HadoopRDD.compute(HadoopRDD.scala:101)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
> at
> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
> at org.apache.spark.rdd.UnionRDD.compute(UnionRDD.scala:87)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
> at
> org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:73)
> at
> org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41)
> at org.apache.spark.scheduler.Task.run(Task.scala:89)
> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:242)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.reflect.InvocationTargetException
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
> at
> org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.initNextRecordReader(HadoopShimsSecure.java:253)
> ... 21 more
> Caused by: java.util.ConcurrentModificationException
> at java.util.Hashtable$Enumerator.next(Hashtable.java:1167)
> at
> org.apache.hadoop.conf.Configuration.iterator(Configuration.java:2455)
> at
> org.apache.hadoop.fs.s3a.S3AUtils.propagateBucketOptions(S3AUtils.java:716)
> at
> org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:181)
> at
> org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2815)
> at
[jira] [Updated] (HIVE-14813) Make TransactionBatchImpl.toString() include state of each txn: commit/abort
[ https://issues.apache.org/jira/browse/HIVE-14813?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-14813: -- Resolution: Fixed Fix Version/s: 3.0.0 Status: Resolved (was: Patch Available) committed to master (hive 3.0) thanks Alan for the review > Make TransactionBatchImpl.toString() include state of each txn: commit/abort > > > Key: HIVE-14813 > URL: https://issues.apache.org/jira/browse/HIVE-14813 > Project: Hive > Issue Type: Bug > Components: HCatalog, Transactions >Affects Versions: 1.2.1 >Reporter: Eugene Koifman >Assignee: Eugene Koifman > Fix For: 3.0.0 > > Attachments: HIVE-14813.01.patch, HIVE-14813.02.patch > > > perhaps include user abort vs system abort on batch close > also include if close() was called and if there was an exception > cc [~deepesh] -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17625) Replication: update hive.repl.partitions.dump.parallelism to 100
[ https://issues.apache.org/jira/browse/HIVE-17625?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vaibhav Gumashta updated HIVE-17625: Attachment: HIVE-17625.1.patch > Replication: update hive.repl.partitions.dump.parallelism to 100 > > > Key: HIVE-17625 > URL: https://issues.apache.org/jira/browse/HIVE-17625 > Project: Hive > Issue Type: Bug > Components: repl >Reporter: Vaibhav Gumashta >Assignee: Vaibhav Gumashta > Fix For: 3.0.0 > > Attachments: HIVE-17625.1.patch > > > Set hive.repl.partitions.dump.parallelism=100 -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17625) Replication: update hive.repl.partitions.dump.parallelism to 100
[ https://issues.apache.org/jira/browse/HIVE-17625?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vaibhav Gumashta updated HIVE-17625: Fix Version/s: 3.0.0 Status: Patch Available (was: Open) > Replication: update hive.repl.partitions.dump.parallelism to 100 > > > Key: HIVE-17625 > URL: https://issues.apache.org/jira/browse/HIVE-17625 > Project: Hive > Issue Type: Bug > Components: repl >Reporter: Vaibhav Gumashta >Assignee: Vaibhav Gumashta > Fix For: 3.0.0 > > Attachments: HIVE-17625.1.patch > > > Set hive.repl.partitions.dump.parallelism=100 -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (HIVE-17625) Replication: update hive.repl.partitions.dump.parallelism to 100
[ https://issues.apache.org/jira/browse/HIVE-17625?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vaibhav Gumashta reassigned HIVE-17625: --- Assignee: Vaibhav Gumashta > Replication: update hive.repl.partitions.dump.parallelism to 100 > > > Key: HIVE-17625 > URL: https://issues.apache.org/jira/browse/HIVE-17625 > Project: Hive > Issue Type: Bug > Components: repl >Reporter: Vaibhav Gumashta >Assignee: Vaibhav Gumashta > > Set hive.repl.partitions.dump.parallelism=100 -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17619) Exclude avatica-core.jar dependency from avatica shaded jar
[
https://issues.apache.org/jira/browse/HIVE-17619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16183311#comment-16183311
]
Hive QA commented on HIVE-17619:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12889321/HIVE-17619.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 9 failed/errored test(s), 11083 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_predicate_pushdown]
(batchId=231)
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_single_sourced_multi_insert]
(batchId=231)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[rcfile_format_nonpart]
(batchId=243)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[optimize_nullscan]
(batchId=161)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[union_fast_stats]
(batchId=156)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_explainuser_1]
(batchId=170)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_2]
(batchId=100)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14]
(batchId=235)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query23]
(batchId=235)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/7009/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/7009/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-7009/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 9 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12889321 - PreCommit-HIVE-Build
> Exclude avatica-core.jar dependency from avatica shaded jar
> ---
>
> Key: HIVE-17619
> URL: https://issues.apache.org/jira/browse/HIVE-17619
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.1.1
>Reporter: Aihua Xu
>Assignee: Aihua Xu
> Attachments: HIVE-17619.1.patch
>
>
> avatica.jar is included in the project but this jar has a dependency on
> avatica-core.jar and it's pulled into the project as well.
> If avatica-core.jar is included in the classpath in front of avatica.jar,
> then hive could run into missing class which is shaded inside avatica.jar.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-15212) merge branch into master
[ https://issues.apache.org/jira/browse/HIVE-15212?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-15212: Attachment: HIVE-15212.16.patch Updating the patch. Lots of merging, as well as final cleanup making this ready to commit conditional on the tests passing (I am going to disable MM-specific tests that fail and file bugs to fix them after merge). > merge branch into master > > > Key: HIVE-15212 > URL: https://issues.apache.org/jira/browse/HIVE-15212 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-15212.01.patch, HIVE-15212.02.patch, > HIVE-15212.03.patch, HIVE-15212.04.patch, HIVE-15212.05.patch, > HIVE-15212.06.patch, HIVE-15212.07.patch, HIVE-15212.08.patch, > HIVE-15212.09.patch, HIVE-15212.10.patch, HIVE-15212.11.patch, > HIVE-15212.12.patch, HIVE-15212.12.patch, HIVE-15212.13.patch, > HIVE-15212.13.patch, HIVE-15212.14.patch, HIVE-15212.15.patch, > HIVE-15212.16.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-14813) Make TransactionBatchImpl.toString() include state of each txn: commit/abort
[ https://issues.apache.org/jira/browse/HIVE-14813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16183276#comment-16183276 ] Alan Gates commented on HIVE-14813: --- +1 > Make TransactionBatchImpl.toString() include state of each txn: commit/abort > > > Key: HIVE-14813 > URL: https://issues.apache.org/jira/browse/HIVE-14813 > Project: Hive > Issue Type: Bug > Components: HCatalog, Transactions >Affects Versions: 1.2.1 >Reporter: Eugene Koifman >Assignee: Eugene Koifman > Attachments: HIVE-14813.01.patch, HIVE-14813.02.patch > > > perhaps include user abort vs system abort on batch close > also include if close() was called and if there was an exception > cc [~deepesh] -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17624) MapredLocakTask running in separate JVM could throw ClassNotFoundException
[
https://issues.apache.org/jira/browse/HIVE-17624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16183270#comment-16183270
]
Aihua Xu commented on HIVE-17624:
-
patch-1: pass added jars through -libjars option so the subjvm will have the
jar in the classpath. Modified the qtest to use a different jar since
hive-hcatalog-core.jar is included /lib and it's in the classpath even "add
jar" is not called.
> MapredLocakTask running in separate JVM could throw ClassNotFoundException
> ---
>
> Key: HIVE-17624
> URL: https://issues.apache.org/jira/browse/HIVE-17624
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Affects Versions: 2.1.1
>Reporter: Aihua Xu
>Assignee: Aihua Xu
> Attachments: HIVE-17624.1.patch
>
>
> {noformat}
> set hive.auto.convert.join=true;
> set hive.auto.convert.join.use.nonstaged=false;
> add jar hive-hcatalog-core.jar;
> drop table if exists t1;
> CREATE TABLE t1 (a string, b string)
> ROW FORMAT SERDE 'org.apache.hive.hcatalog.data.JsonSerDe';
> LOAD DATA LOCAL INPATH "data/files/sample.json" INTO TABLE t1;
> select * from t1 l join t1 r on l.a=r.a;
> {noformat}
> The join will use a MapJoin which uses MapredLocalTask in a separate JVM to
> load the table into a Hashmap. But hive doesn't pass added jar to the
> classpath in such JVM so the following exception is thrown.
> {noformat}
> org.apache.hadoop.hive.ql.metadata.HiveException: Failed with exception
> java.lang.ClassNotFoundException:
> org.apache.hive.hcatalog.data.JsonSerDejava.lang.RuntimeException:
> java.lang.ClassNotFoundException: org.apache.hive.hcatalog.data.JsonSerDe
> at
> org.apache.hadoop.hive.ql.plan.TableDesc.getDeserializerClass(TableDesc.java:72)
> at
> org.apache.hadoop.hive.ql.plan.TableDesc.getDeserializer(TableDesc.java:92)
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.setupOutputObjectInspector(FetchOperator.java:564)
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.initialize(FetchOperator.java:172)
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.(FetchOperator.java:140)
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.(FetchOperator.java:127)
> at
> org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask.initializeOperators(MapredLocalTask.java:462)
> at
> org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask.startForward(MapredLocalTask.java:390)
> at
> org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask.executeInProcess(MapredLocalTask.java:370)
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecDriver.main(ExecDriver.java:756)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
> Caused by: java.lang.ClassNotFoundException:
> org.apache.hive.hcatalog.data.JsonSerDe
> at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:270)
> at
> org.apache.hadoop.hive.ql.plan.TableDesc.getDeserializerClass(TableDesc.java:69)
> ... 15 more
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.setupOutputObjectInspector(FetchOperator.java:586)
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.initialize(FetchOperator.java:172)
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.(FetchOperator.java:140)
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.(FetchOperator.java:127)
> at
> org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask.initializeOperators(MapredLocalTask.java:462)
> at
> org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask.startForward(MapredLocalTask.java:390)
> at
> org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask.executeInProcess(MapredLocalTask.java:370)
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecDriver.main(ExecDriver.java:756)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
>
[jira] [Updated] (HIVE-17624) MapredLocakTask running in separate JVM could throw ClassNotFoundException
[
https://issues.apache.org/jira/browse/HIVE-17624?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aihua Xu updated HIVE-17624:
Status: Patch Available (was: Open)
> MapredLocakTask running in separate JVM could throw ClassNotFoundException
> ---
>
> Key: HIVE-17624
> URL: https://issues.apache.org/jira/browse/HIVE-17624
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Affects Versions: 2.1.1
>Reporter: Aihua Xu
>Assignee: Aihua Xu
> Attachments: HIVE-17624.1.patch
>
>
> {noformat}
> set hive.auto.convert.join=true;
> set hive.auto.convert.join.use.nonstaged=false;
> add jar hive-hcatalog-core.jar;
> drop table if exists t1;
> CREATE TABLE t1 (a string, b string)
> ROW FORMAT SERDE 'org.apache.hive.hcatalog.data.JsonSerDe';
> LOAD DATA LOCAL INPATH "data/files/sample.json" INTO TABLE t1;
> select * from t1 l join t1 r on l.a=r.a;
> {noformat}
> The join will use a MapJoin which uses MapredLocalTask in a separate JVM to
> load the table into a Hashmap. But hive doesn't pass added jar to the
> classpath in such JVM so the following exception is thrown.
> {noformat}
> org.apache.hadoop.hive.ql.metadata.HiveException: Failed with exception
> java.lang.ClassNotFoundException:
> org.apache.hive.hcatalog.data.JsonSerDejava.lang.RuntimeException:
> java.lang.ClassNotFoundException: org.apache.hive.hcatalog.data.JsonSerDe
> at
> org.apache.hadoop.hive.ql.plan.TableDesc.getDeserializerClass(TableDesc.java:72)
> at
> org.apache.hadoop.hive.ql.plan.TableDesc.getDeserializer(TableDesc.java:92)
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.setupOutputObjectInspector(FetchOperator.java:564)
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.initialize(FetchOperator.java:172)
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.(FetchOperator.java:140)
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.(FetchOperator.java:127)
> at
> org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask.initializeOperators(MapredLocalTask.java:462)
> at
> org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask.startForward(MapredLocalTask.java:390)
> at
> org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask.executeInProcess(MapredLocalTask.java:370)
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecDriver.main(ExecDriver.java:756)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
> Caused by: java.lang.ClassNotFoundException:
> org.apache.hive.hcatalog.data.JsonSerDe
> at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:270)
> at
> org.apache.hadoop.hive.ql.plan.TableDesc.getDeserializerClass(TableDesc.java:69)
> ... 15 more
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.setupOutputObjectInspector(FetchOperator.java:586)
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.initialize(FetchOperator.java:172)
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.(FetchOperator.java:140)
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.(FetchOperator.java:127)
> at
> org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask.initializeOperators(MapredLocalTask.java:462)
> at
> org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask.startForward(MapredLocalTask.java:390)
> at
> org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask.executeInProcess(MapredLocalTask.java:370)
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecDriver.main(ExecDriver.java:756)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
> at
[jira] [Updated] (HIVE-17624) MapredLocakTask running in separate JVM could throw ClassNotFoundException
[
https://issues.apache.org/jira/browse/HIVE-17624?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aihua Xu updated HIVE-17624:
Attachment: HIVE-17624.1.patch
> MapredLocakTask running in separate JVM could throw ClassNotFoundException
> ---
>
> Key: HIVE-17624
> URL: https://issues.apache.org/jira/browse/HIVE-17624
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Affects Versions: 2.1.1
>Reporter: Aihua Xu
>Assignee: Aihua Xu
> Attachments: HIVE-17624.1.patch
>
>
> {noformat}
> set hive.auto.convert.join=true;
> set hive.auto.convert.join.use.nonstaged=false;
> add jar hive-hcatalog-core.jar;
> drop table if exists t1;
> CREATE TABLE t1 (a string, b string)
> ROW FORMAT SERDE 'org.apache.hive.hcatalog.data.JsonSerDe';
> LOAD DATA LOCAL INPATH "data/files/sample.json" INTO TABLE t1;
> select * from t1 l join t1 r on l.a=r.a;
> {noformat}
> The join will use a MapJoin which uses MapredLocalTask in a separate JVM to
> load the table into a Hashmap. But hive doesn't pass added jar to the
> classpath in such JVM so the following exception is thrown.
> {noformat}
> org.apache.hadoop.hive.ql.metadata.HiveException: Failed with exception
> java.lang.ClassNotFoundException:
> org.apache.hive.hcatalog.data.JsonSerDejava.lang.RuntimeException:
> java.lang.ClassNotFoundException: org.apache.hive.hcatalog.data.JsonSerDe
> at
> org.apache.hadoop.hive.ql.plan.TableDesc.getDeserializerClass(TableDesc.java:72)
> at
> org.apache.hadoop.hive.ql.plan.TableDesc.getDeserializer(TableDesc.java:92)
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.setupOutputObjectInspector(FetchOperator.java:564)
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.initialize(FetchOperator.java:172)
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.(FetchOperator.java:140)
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.(FetchOperator.java:127)
> at
> org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask.initializeOperators(MapredLocalTask.java:462)
> at
> org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask.startForward(MapredLocalTask.java:390)
> at
> org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask.executeInProcess(MapredLocalTask.java:370)
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecDriver.main(ExecDriver.java:756)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
> Caused by: java.lang.ClassNotFoundException:
> org.apache.hive.hcatalog.data.JsonSerDe
> at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:270)
> at
> org.apache.hadoop.hive.ql.plan.TableDesc.getDeserializerClass(TableDesc.java:69)
> ... 15 more
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.setupOutputObjectInspector(FetchOperator.java:586)
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.initialize(FetchOperator.java:172)
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.(FetchOperator.java:140)
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.(FetchOperator.java:127)
> at
> org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask.initializeOperators(MapredLocalTask.java:462)
> at
> org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask.startForward(MapredLocalTask.java:390)
> at
> org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask.executeInProcess(MapredLocalTask.java:370)
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecDriver.main(ExecDriver.java:756)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
> at
[jira] [Assigned] (HIVE-17624) MapredLocakTask running in separate JVM could throw ClassNotFoundException
[
https://issues.apache.org/jira/browse/HIVE-17624?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aihua Xu reassigned HIVE-17624:
---
> MapredLocakTask running in separate JVM could throw ClassNotFoundException
> ---
>
> Key: HIVE-17624
> URL: https://issues.apache.org/jira/browse/HIVE-17624
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Affects Versions: 2.1.1
>Reporter: Aihua Xu
>Assignee: Aihua Xu
>
> {noformat}
> set hive.auto.convert.join=true;
> set hive.auto.convert.join.use.nonstaged=false;
> add jar hive-hcatalog-core.jar;
> drop table if exists t1;
> CREATE TABLE t1 (a string, b string)
> ROW FORMAT SERDE 'org.apache.hive.hcatalog.data.JsonSerDe';
> LOAD DATA LOCAL INPATH "data/files/sample.json" INTO TABLE t1;
> select * from t1 l join t1 r on l.a=r.a;
> {noformat}
> The join will use a MapJoin which uses MapredLocalTask in a separate JVM to
> load the table into a Hashmap. But hive doesn't pass added jar to the
> classpath in such JVM so the following exception is thrown.
> {noformat}
> org.apache.hadoop.hive.ql.metadata.HiveException: Failed with exception
> java.lang.ClassNotFoundException:
> org.apache.hive.hcatalog.data.JsonSerDejava.lang.RuntimeException:
> java.lang.ClassNotFoundException: org.apache.hive.hcatalog.data.JsonSerDe
> at
> org.apache.hadoop.hive.ql.plan.TableDesc.getDeserializerClass(TableDesc.java:72)
> at
> org.apache.hadoop.hive.ql.plan.TableDesc.getDeserializer(TableDesc.java:92)
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.setupOutputObjectInspector(FetchOperator.java:564)
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.initialize(FetchOperator.java:172)
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.(FetchOperator.java:140)
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.(FetchOperator.java:127)
> at
> org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask.initializeOperators(MapredLocalTask.java:462)
> at
> org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask.startForward(MapredLocalTask.java:390)
> at
> org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask.executeInProcess(MapredLocalTask.java:370)
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecDriver.main(ExecDriver.java:756)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
> Caused by: java.lang.ClassNotFoundException:
> org.apache.hive.hcatalog.data.JsonSerDe
> at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:270)
> at
> org.apache.hadoop.hive.ql.plan.TableDesc.getDeserializerClass(TableDesc.java:69)
> ... 15 more
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.setupOutputObjectInspector(FetchOperator.java:586)
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.initialize(FetchOperator.java:172)
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.(FetchOperator.java:140)
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.(FetchOperator.java:127)
> at
> org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask.initializeOperators(MapredLocalTask.java:462)
> at
> org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask.startForward(MapredLocalTask.java:390)
> at
> org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask.executeInProcess(MapredLocalTask.java:370)
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecDriver.main(ExecDriver.java:756)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
> {noformat}
--
This message was sent by Atlassian JIRA
[jira] [Assigned] (HIVE-15016) Run tests with Hadoop 3.0.0-alpha1
[ https://issues.apache.org/jira/browse/HIVE-15016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergio Peña reassigned HIVE-15016: -- Assignee: Aihua Xu (was: Sergio Peña) > Run tests with Hadoop 3.0.0-alpha1 > -- > > Key: HIVE-15016 > URL: https://issues.apache.org/jira/browse/HIVE-15016 > Project: Hive > Issue Type: Sub-task > Components: Hive >Reporter: Sergio Peña >Assignee: Aihua Xu > Attachments: Hadoop3Upstream.patch > > > Hadoop 3.0.0-alpha1 was released back on Sep/16 to allow other components run > tests against this new version before GA. > We should start running tests with Hive to validate compatibility against > Hadoop 3.0. > NOTE: The patch used to test must not be committed to Hive until Hadoop 3.0 > GA is released. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-15016) Run tests with Hadoop 3.0.0-alpha1
[ https://issues.apache.org/jira/browse/HIVE-15016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16183257#comment-16183257 ] Aihua Xu commented on HIVE-15016: - OK. Thanks [~ashutoshc] I will start to work on it. > Run tests with Hadoop 3.0.0-alpha1 > -- > > Key: HIVE-15016 > URL: https://issues.apache.org/jira/browse/HIVE-15016 > Project: Hive > Issue Type: Sub-task > Components: Hive >Reporter: Sergio Peña >Assignee: Sergio Peña > Attachments: Hadoop3Upstream.patch > > > Hadoop 3.0.0-alpha1 was released back on Sep/16 to allow other components run > tests against this new version before GA. > We should start running tests with Hive to validate compatibility against > Hadoop 3.0. > NOTE: The patch used to test must not be committed to Hive until Hadoop 3.0 > GA is released. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-13843) Re-enable the HoS tests disabled in HIVE-13402
[ https://issues.apache.org/jira/browse/HIVE-13843?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sahil Takiar updated HIVE-13843: Target Version/s: 3.0.0 Status: Patch Available (was: Open) > Re-enable the HoS tests disabled in HIVE-13402 > -- > > Key: HIVE-13843 > URL: https://issues.apache.org/jira/browse/HIVE-13843 > Project: Hive > Issue Type: Test >Reporter: Rui Li >Assignee: Sahil Takiar > Attachments: HIVE-13843.1.patch > > > With HIVE-13525, we can now fix and re-enable the tests for Spark. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-13843) Re-enable the HoS tests disabled in HIVE-13402
[ https://issues.apache.org/jira/browse/HIVE-13843?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sahil Takiar updated HIVE-13843: Attachment: HIVE-13843.1.patch Attaching patch with the q-files added back in, and re-generated the golden files. Haven't gone through the new golden files to check if they match the old ones, but will double check. > Re-enable the HoS tests disabled in HIVE-13402 > -- > > Key: HIVE-13843 > URL: https://issues.apache.org/jira/browse/HIVE-13843 > Project: Hive > Issue Type: Test >Reporter: Rui Li >Assignee: Sahil Takiar > Attachments: HIVE-13843.1.patch > > > With HIVE-13525, we can now fix and re-enable the tests for Spark. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (HIVE-13843) Re-enable the HoS tests disabled in HIVE-13402
[ https://issues.apache.org/jira/browse/HIVE-13843?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sahil Takiar reassigned HIVE-13843: --- Assignee: Sahil Takiar (was: Rui Li) > Re-enable the HoS tests disabled in HIVE-13402 > -- > > Key: HIVE-13843 > URL: https://issues.apache.org/jira/browse/HIVE-13843 > Project: Hive > Issue Type: Test >Reporter: Rui Li >Assignee: Sahil Takiar > Attachments: HIVE-13843.1.patch > > > With HIVE-13525, we can now fix and re-enable the tests for Spark. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-13843) Re-enable the HoS tests disabled in HIVE-13402
[ https://issues.apache.org/jira/browse/HIVE-13843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16183252#comment-16183252 ] Sahil Takiar commented on HIVE-13843: - Hey [~lirui], if you don't mind, I would like to pick up this JIRA. > Re-enable the HoS tests disabled in HIVE-13402 > -- > > Key: HIVE-13843 > URL: https://issues.apache.org/jira/browse/HIVE-13843 > Project: Hive > Issue Type: Test >Reporter: Rui Li >Assignee: Rui Li > > With HIVE-13525, we can now fix and re-enable the tests for Spark. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-15016) Run tests with Hadoop 3.0.0-alpha1
[ https://issues.apache.org/jira/browse/HIVE-15016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16183250#comment-16183250 ] Ashutosh Chauhan commented on HIVE-15016: - IMHO we can just change hadoop version on master. > Run tests with Hadoop 3.0.0-alpha1 > -- > > Key: HIVE-15016 > URL: https://issues.apache.org/jira/browse/HIVE-15016 > Project: Hive > Issue Type: Sub-task > Components: Hive >Reporter: Sergio Peña >Assignee: Sergio Peña > Attachments: Hadoop3Upstream.patch > > > Hadoop 3.0.0-alpha1 was released back on Sep/16 to allow other components run > tests against this new version before GA. > We should start running tests with Hive to validate compatibility against > Hadoop 3.0. > NOTE: The patch used to test must not be committed to Hive until Hadoop 3.0 > GA is released. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-15016) Run tests with Hadoop 3.0.0-alpha1
[ https://issues.apache.org/jira/browse/HIVE-15016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16183244#comment-16183244 ] Aihua Xu commented on HIVE-15016: - Should I fork a new branch for hadoop3 support? > Run tests with Hadoop 3.0.0-alpha1 > -- > > Key: HIVE-15016 > URL: https://issues.apache.org/jira/browse/HIVE-15016 > Project: Hive > Issue Type: Sub-task > Components: Hive >Reporter: Sergio Peña >Assignee: Sergio Peña > Attachments: Hadoop3Upstream.patch > > > Hadoop 3.0.0-alpha1 was released back on Sep/16 to allow other components run > tests against this new version before GA. > We should start running tests with Hive to validate compatibility against > Hadoop 3.0. > NOTE: The patch used to test must not be committed to Hive until Hadoop 3.0 > GA is released. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-15016) Run tests with Hadoop 3.0.0-alpha1
[ https://issues.apache.org/jira/browse/HIVE-15016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16183238#comment-16183238 ] Sergio Peña commented on HIVE-15016: Hadoop is on alpha4 right now, so we could change the subject name to reflect the latest release. > Run tests with Hadoop 3.0.0-alpha1 > -- > > Key: HIVE-15016 > URL: https://issues.apache.org/jira/browse/HIVE-15016 > Project: Hive > Issue Type: Sub-task > Components: Hive >Reporter: Sergio Peña >Assignee: Sergio Peña > Attachments: Hadoop3Upstream.patch > > > Hadoop 3.0.0-alpha1 was released back on Sep/16 to allow other components run > tests against this new version before GA. > We should start running tests with Hive to validate compatibility against > Hadoop 3.0. > NOTE: The patch used to test must not be committed to Hive until Hadoop 3.0 > GA is released. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-15016) Run tests with Hadoop 3.0.0-alpha1
[ https://issues.apache.org/jira/browse/HIVE-15016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16183236#comment-16183236 ] Sergio Peña commented on HIVE-15016: The idea was to keep this jira alive with required changes to hadoop3 until the official release is done. But I'm open to other ideas if this the patch is getting larger. > Run tests with Hadoop 3.0.0-alpha1 > -- > > Key: HIVE-15016 > URL: https://issues.apache.org/jira/browse/HIVE-15016 > Project: Hive > Issue Type: Sub-task > Components: Hive >Reporter: Sergio Peña >Assignee: Sergio Peña > Attachments: Hadoop3Upstream.patch > > > Hadoop 3.0.0-alpha1 was released back on Sep/16 to allow other components run > tests against this new version before GA. > We should start running tests with Hive to validate compatibility against > Hadoop 3.0. > NOTE: The patch used to test must not be committed to Hive until Hadoop 3.0 > GA is released. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-15016) Run tests with Hadoop 3.0.0-alpha1
[ https://issues.apache.org/jira/browse/HIVE-15016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16183227#comment-16183227 ] Aihua Xu commented on HIVE-15016: - I'm wondering how we should contribute hadoop3 support upstream. Should we update dependency to point to hadoop 3 in Hive? > Run tests with Hadoop 3.0.0-alpha1 > -- > > Key: HIVE-15016 > URL: https://issues.apache.org/jira/browse/HIVE-15016 > Project: Hive > Issue Type: Sub-task > Components: Hive >Reporter: Sergio Peña >Assignee: Sergio Peña > Attachments: Hadoop3Upstream.patch > > > Hadoop 3.0.0-alpha1 was released back on Sep/16 to allow other components run > tests against this new version before GA. > We should start running tests with Hive to validate compatibility against > Hadoop 3.0. > NOTE: The patch used to test must not be committed to Hive until Hadoop 3.0 > GA is released. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-12734) Remove redundancy in HiveConfs serialized to UDFContext
[
https://issues.apache.org/jira/browse/HIVE-12734?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mithun Radhakrishnan updated HIVE-12734:
Attachment: HIVE-12734.1.patch
Initial patch. Includes the fix to HIVE-12734 *and* HIVE-17621. Testing both at
a go.
> Remove redundancy in HiveConfs serialized to UDFContext
> ---
>
> Key: HIVE-12734
> URL: https://issues.apache.org/jira/browse/HIVE-12734
> Project: Hive
> Issue Type: Bug
>Affects Versions: 1.2.1, 2.0.0, 2.2.0, 3.0.0
>Reporter: Mithun Radhakrishnan
>Assignee: Mithun Radhakrishnan
> Attachments: HIVE-12734.1.patch
>
>
> {{HCatLoader}} lands up serializing one {{HiveConf}} instance per
> table-alias, to Pig's {{UDFContext}}. This lands up bloating the
> {{UDFContext}}.
> To reduce the footprint, it makes sense to serialize a default-constructed
> {{HiveConf}} once, and one "diff" per {{HCatLoader}}. This should reduce the
> time taken to kick off jobs from {{pig -useHCatalog}} scripts.
> (Note_to_self: YHIVE-540).
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-12734) Remove redundancy in HiveConfs serialized to UDFContext
[
https://issues.apache.org/jira/browse/HIVE-12734?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mithun Radhakrishnan updated HIVE-12734:
Status: Patch Available (was: Open)
Submitting for tests.
> Remove redundancy in HiveConfs serialized to UDFContext
> ---
>
> Key: HIVE-12734
> URL: https://issues.apache.org/jira/browse/HIVE-12734
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.2.0, 2.0.0, 1.2.1, 3.0.0
>Reporter: Mithun Radhakrishnan
>Assignee: Mithun Radhakrishnan
> Attachments: HIVE-12734.1.patch
>
>
> {{HCatLoader}} lands up serializing one {{HiveConf}} instance per
> table-alias, to Pig's {{UDFContext}}. This lands up bloating the
> {{UDFContext}}.
> To reduce the footprint, it makes sense to serialize a default-constructed
> {{HiveConf}} once, and one "diff" per {{HCatLoader}}. This should reduce the
> time taken to kick off jobs from {{pig -useHCatalog}} scripts.
> (Note_to_self: YHIVE-540).
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17526) Disable conversion to ACID if table has _copy_N files on branch-1
[
https://issues.apache.org/jira/browse/HIVE-17526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16183223#comment-16183223
]
Hive QA commented on HIVE-17526:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12886882/HIVE-17526.1-branch-1.patch
{color:green}SUCCESS:{color} +1 due to 3 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 163 failed/errored test(s), 8125 tests
executed
*Failed tests:*
{noformat}
TestAcidOnTez - did not produce a TEST-*.xml file (likely timed out)
(batchId=376)
TestAdminUser - did not produce a TEST-*.xml file (likely timed out)
(batchId=358)
TestAuthorizationPreEventListener - did not produce a TEST-*.xml file (likely
timed out) (batchId=391)
TestAuthzApiEmbedAuthorizerInEmbed - did not produce a TEST-*.xml file (likely
timed out) (batchId=368)
TestAuthzApiEmbedAuthorizerInRemote - did not produce a TEST-*.xml file (likely
timed out) (batchId=374)
TestBeeLineWithArgs - did not produce a TEST-*.xml file (likely timed out)
(batchId=398)
TestCLIAuthzSessionContext - did not produce a TEST-*.xml file (likely timed
out) (batchId=416)
TestClearDanglingScratchDir - did not produce a TEST-*.xml file (likely timed
out) (batchId=383)
TestClientSideAuthorizationProvider - did not produce a TEST-*.xml file (likely
timed out) (batchId=390)
TestCompactor - did not produce a TEST-*.xml file (likely timed out)
(batchId=379)
TestCreateUdfEntities - did not produce a TEST-*.xml file (likely timed out)
(batchId=378)
TestCustomAuthentication - did not produce a TEST-*.xml file (likely timed out)
(batchId=399)
TestDBTokenStore - did not produce a TEST-*.xml file (likely timed out)
(batchId=342)
TestDDLWithRemoteMetastoreSecondNamenode - did not produce a TEST-*.xml file
(likely timed out) (batchId=377)
TestDynamicSerDe - did not produce a TEST-*.xml file (likely timed out)
(batchId=345)
TestEmbeddedHiveMetaStore - did not produce a TEST-*.xml file (likely timed
out) (batchId=355)
TestEmbeddedThriftBinaryCLIService - did not produce a TEST-*.xml file (likely
timed out) (batchId=402)
TestFilterHooks - did not produce a TEST-*.xml file (likely timed out)
(batchId=350)
TestFolderPermissions - did not produce a TEST-*.xml file (likely timed out)
(batchId=385)
TestHS2AuthzContext - did not produce a TEST-*.xml file (likely timed out)
(batchId=419)
TestHS2AuthzSessionContext - did not produce a TEST-*.xml file (likely timed
out) (batchId=420)
TestHS2ClearDanglingScratchDir - did not produce a TEST-*.xml file (likely
timed out) (batchId=406)
TestHS2ImpersonationWithRemoteMS - did not produce a TEST-*.xml file (likely
timed out) (batchId=407)
TestHiveAuthorizerCheckInvocation - did not produce a TEST-*.xml file (likely
timed out) (batchId=394)
TestHiveAuthorizerShowFilters - did not produce a TEST-*.xml file (likely timed
out) (batchId=393)
TestHiveHistory - did not produce a TEST-*.xml file (likely timed out)
(batchId=396)
TestHiveMetaStoreTxns - did not produce a TEST-*.xml file (likely timed out)
(batchId=370)
TestHiveMetaStoreWithEnvironmentContext - did not produce a TEST-*.xml file
(likely timed out) (batchId=360)
TestHiveMetaTool - did not produce a TEST-*.xml file (likely timed out)
(batchId=373)
TestHiveServer2 - did not produce a TEST-*.xml file (likely timed out)
(batchId=422)
TestHiveServer2SessionTimeout - did not produce a TEST-*.xml file (likely timed
out) (batchId=423)
TestHiveSessionImpl - did not produce a TEST-*.xml file (likely timed out)
(batchId=403)
TestHs2Hooks - did not produce a TEST-*.xml file (likely timed out)
(batchId=375)
TestHs2HooksWithMiniKdc - did not produce a TEST-*.xml file (likely timed out)
(batchId=451)
TestJdbcDriver2 - did not produce a TEST-*.xml file (likely timed out)
(batchId=410)
TestJdbcMetadataApiAuth - did not produce a TEST-*.xml file (likely timed out)
(batchId=421)
TestJdbcWithLocalClusterSpark - did not produce a TEST-*.xml file (likely timed
out) (batchId=415)
TestJdbcWithMiniHS2 - did not produce a TEST-*.xml file (likely timed out)
(batchId=412)
TestJdbcWithMiniKdc - did not produce a TEST-*.xml file (likely timed out)
(batchId=448)
TestJdbcWithMiniKdcCookie - did not produce a TEST-*.xml file (likely timed
out) (batchId=447)
TestJdbcWithMiniKdcSQLAuthBinary - did not produce a TEST-*.xml file (likely
timed out) (batchId=445)
TestJdbcWithMiniKdcSQLAuthHttp - did not produce a TEST-*.xml file (likely
timed out) (batchId=450)
TestJdbcWithMiniMr - did not produce a TEST-*.xml file (likely timed out)
(batchId=411)
TestJdbcWithSQLAuthUDFBlacklist - did not produce a TEST-*.xml file (likely
timed out) (batchId=417)
TestJdbcWithSQLAuthorization - did not produce a TEST-*.xml file (likely timed
out) (batchId=418)
TestLocationQueries - did not produce a TEST-*.xml file (likely timed out)
(batchId=382)
TestMTQueries - did not produce a TEST-*.xml file (likely timed out)
[jira] [Commented] (HIVE-17508) Implement pool rules and triggers based on counters
[
https://issues.apache.org/jira/browse/HIVE-17508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16183188#comment-16183188
]
Prasanth Jayachandran commented on HIVE-17508:
--
Sure. Will fix the findFirst so that it can handle multiple rules for same
variable/counter.
.2 patch just did that. The only parsing it does it figure out what type of
counter it is (time vs fs), also validate the value which has to happen even if
we just split by '>'.
.3 patch added more sugar to counter value to make it more generic which also
uses the HiveConf's size and time validators. So I would say it is already
primitive :) I agree we should use antlr for much better validation and
parsing. Will definitely take it up in the follow up.
> Implement pool rules and triggers based on counters
> ---
>
> Key: HIVE-17508
> URL: https://issues.apache.org/jira/browse/HIVE-17508
> Project: Hive
> Issue Type: Sub-task
>Affects Versions: 3.0.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
> Attachments: HIVE-17508.1.patch, HIVE-17508.2.patch,
> HIVE-17508.3.patch, HIVE-17508.3.patch, HIVE-17508.WIP.2.patch,
> HIVE-17508.WIP.patch
>
>
> Workload management can defined Rules that are bound to a resource plan. Each
> rule can have a trigger expression and an action associated with it. Trigger
> expressions are evaluated at runtime after configurable check interval, based
> on which actions like killing a query, moving a query to different pool etc.
> will get invoked. Simple rule could be something like
> {code}
> CREATE RULE slow_query IN resource_plan_name
> WHEN execution_time_ms > 1
> MOVE TO slow_queue
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-17562) ACID 1.0 + ETL strategy should treat empty compacted files as uncovered deltas
[ https://issues.apache.org/jira/browse/HIVE-17562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-17562: - Resolution: Fixed Fix Version/s: 2.4.0 Status: Resolved (was: Patch Available) Committed to branch-2. Thanks for the review! > ACID 1.0 + ETL strategy should treat empty compacted files as uncovered deltas > -- > > Key: HIVE-17562 > URL: https://issues.apache.org/jira/browse/HIVE-17562 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 2.4.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran > Fix For: 2.4.0 > > Attachments: HIVE-17562.1.branch-2.patch, HIVE-17562-branch-2.patch, > HIVE-17562-branch-2.patch > > > In branch-2, with ACID 1.0, following sequence will result in incorrect > results > 1) Set split strategy to ETL > 2) Insert some rows > 3) Delete all rows > 4) Alter table compact MAJOR > 5) Insert some rows > 6) Select * query will not return any rows that is written at last (step 5) > The reason for that, compaction essentially voids the first insert in step 2. > Now when ETL split strategy is chosen, there will not be any stripes in the > base files. So no split gets generated and any subsequent deltas gets ignored. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17606) Improve security for DB notification related APIs
[ https://issues.apache.org/jira/browse/HIVE-17606?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Li updated HIVE-17606: -- Status: Patch Available (was: Open) > Improve security for DB notification related APIs > - > > Key: HIVE-17606 > URL: https://issues.apache.org/jira/browse/HIVE-17606 > Project: Hive > Issue Type: Improvement > Components: Metastore >Reporter: Tao Li >Assignee: Tao Li > Attachments: HIVE-17606.1.patch, HIVE-17606.2.patch, > HIVE-17606.3.patch, HIVE-17606.4.patch, HIVE-17606.5.patch > > > The purpose is to make sure only the superusers which are specified in the > proxyuser settings can make the db notification related API calls, since this > is supposed to be called by superuser/admin instead of any end user. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17606) Improve security for DB notification related APIs
[ https://issues.apache.org/jira/browse/HIVE-17606?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Li updated HIVE-17606: -- Attachment: HIVE-17606.5.patch > Improve security for DB notification related APIs > - > > Key: HIVE-17606 > URL: https://issues.apache.org/jira/browse/HIVE-17606 > Project: Hive > Issue Type: Improvement > Components: Metastore >Reporter: Tao Li >Assignee: Tao Li > Attachments: HIVE-17606.1.patch, HIVE-17606.2.patch, > HIVE-17606.3.patch, HIVE-17606.4.patch, HIVE-17606.5.patch > > > The purpose is to make sure only the superusers which are specified in the > proxyuser settings can make the db notification related API calls, since this > is supposed to be called by superuser/admin instead of any end user. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17606) Improve security for DB notification related APIs
[ https://issues.apache.org/jira/browse/HIVE-17606?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Li updated HIVE-17606: -- Attachment: (was: HIVE-17606.5.patch) > Improve security for DB notification related APIs > - > > Key: HIVE-17606 > URL: https://issues.apache.org/jira/browse/HIVE-17606 > Project: Hive > Issue Type: Improvement > Components: Metastore >Reporter: Tao Li >Assignee: Tao Li > Attachments: HIVE-17606.1.patch, HIVE-17606.2.patch, > HIVE-17606.3.patch, HIVE-17606.4.patch, HIVE-17606.5.patch > > > The purpose is to make sure only the superusers which are specified in the > proxyuser settings can make the db notification related API calls, since this > is supposed to be called by superuser/admin instead of any end user. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17606) Improve security for DB notification related APIs
[ https://issues.apache.org/jira/browse/HIVE-17606?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Li updated HIVE-17606: -- Attachment: HIVE-17606.5.patch > Improve security for DB notification related APIs > - > > Key: HIVE-17606 > URL: https://issues.apache.org/jira/browse/HIVE-17606 > Project: Hive > Issue Type: Improvement > Components: Metastore >Reporter: Tao Li >Assignee: Tao Li > Attachments: HIVE-17606.1.patch, HIVE-17606.2.patch, > HIVE-17606.3.patch, HIVE-17606.4.patch, HIVE-17606.5.patch > > > The purpose is to make sure only the superusers which are specified in the > proxyuser settings can make the db notification related API calls, since this > is supposed to be called by superuser/admin instead of any end user. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17606) Improve security for DB notification related APIs
[ https://issues.apache.org/jira/browse/HIVE-17606?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Li updated HIVE-17606: -- Status: Open (was: Patch Available) > Improve security for DB notification related APIs > - > > Key: HIVE-17606 > URL: https://issues.apache.org/jira/browse/HIVE-17606 > Project: Hive > Issue Type: Improvement > Components: Metastore >Reporter: Tao Li >Assignee: Tao Li > Attachments: HIVE-17606.1.patch, HIVE-17606.2.patch, > HIVE-17606.3.patch, HIVE-17606.4.patch > > > The purpose is to make sure only the superusers which are specified in the > proxyuser settings can make the db notification related API calls, since this > is supposed to be called by superuser/admin instead of any end user. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17606) Improve security for DB notification related APIs
[ https://issues.apache.org/jira/browse/HIVE-17606?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Li updated HIVE-17606: -- Attachment: HIVE-17606.5.patch > Improve security for DB notification related APIs > - > > Key: HIVE-17606 > URL: https://issues.apache.org/jira/browse/HIVE-17606 > Project: Hive > Issue Type: Improvement > Components: Metastore >Reporter: Tao Li >Assignee: Tao Li > Attachments: HIVE-17606.1.patch, HIVE-17606.2.patch, > HIVE-17606.3.patch, HIVE-17606.4.patch, HIVE-17606.5.patch > > > The purpose is to make sure only the superusers which are specified in the > proxyuser settings can make the db notification related API calls, since this > is supposed to be called by superuser/admin instead of any end user. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17606) Improve security for DB notification related APIs
[ https://issues.apache.org/jira/browse/HIVE-17606?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Li updated HIVE-17606: -- Attachment: (was: HIVE-17606.5.patch) > Improve security for DB notification related APIs > - > > Key: HIVE-17606 > URL: https://issues.apache.org/jira/browse/HIVE-17606 > Project: Hive > Issue Type: Improvement > Components: Metastore >Reporter: Tao Li >Assignee: Tao Li > Attachments: HIVE-17606.1.patch, HIVE-17606.2.patch, > HIVE-17606.3.patch, HIVE-17606.4.patch > > > The purpose is to make sure only the superusers which are specified in the > proxyuser settings can make the db notification related API calls, since this > is supposed to be called by superuser/admin instead of any end user. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (HIVE-17623) Fix Select query Fix Double column serde and some refactoring
[ https://issues.apache.org/jira/browse/HIVE-17623?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] slim bouguerra reassigned HIVE-17623: - Assignee: Ashutosh Chauhan > Fix Select query Fix Double column serde and some refactoring > - > > Key: HIVE-17623 > URL: https://issues.apache.org/jira/browse/HIVE-17623 > Project: Hive > Issue Type: Bug > Components: Druid integration >Affects Versions: 3.0.0 >Reporter: slim bouguerra >Assignee: Ashutosh Chauhan > Attachments: HIVE-17623.patch > > > This PR has 2 fixes. > First, fixes the limit of results returned by Select query that used to be > limited to 16K rows > Second fixes the type inference for the double type newly added to druid. > Use Jackson polymorphism to infer types and parse results from druid nodes. > Removes duplicate codes form RecordReaders. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17623) Fix Select query Fix Double column serde and some refactoring
[ https://issues.apache.org/jira/browse/HIVE-17623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16183127#comment-16183127 ] slim bouguerra commented on HIVE-17623: --- https://reviews.apache.org/r/62625/ > Fix Select query Fix Double column serde and some refactoring > - > > Key: HIVE-17623 > URL: https://issues.apache.org/jira/browse/HIVE-17623 > Project: Hive > Issue Type: Bug > Components: Druid integration >Affects Versions: 3.0.0 >Reporter: slim bouguerra > Attachments: HIVE-17623.patch > > > This PR has 2 fixes. > First, fixes the limit of results returned by Select query that used to be > limited to 16K rows > Second fixes the type inference for the double type newly added to druid. > Use Jackson polymorphism to infer types and parse results from druid nodes. > Removes duplicate codes form RecordReaders. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17623) Fix Select query Fix Double column serde and some refactoring
[ https://issues.apache.org/jira/browse/HIVE-17623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16183125#comment-16183125 ] Ashutosh Chauhan commented on HIVE-17623: - Can you please also create a RB for this? > Fix Select query Fix Double column serde and some refactoring > - > > Key: HIVE-17623 > URL: https://issues.apache.org/jira/browse/HIVE-17623 > Project: Hive > Issue Type: Bug > Components: Druid integration >Affects Versions: 3.0.0 >Reporter: slim bouguerra > Attachments: HIVE-17623.patch > > > This PR has 2 fixes. > First, fixes the limit of results returned by Select query that used to be > limited to 16K rows > Second fixes the type inference for the double type newly added to druid. > Use Jackson polymorphism to infer types and parse results from druid nodes. > Removes duplicate codes form RecordReaders. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17623) Fix Select query Fix Double column serde and some refactoring
[ https://issues.apache.org/jira/browse/HIVE-17623?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] slim bouguerra updated HIVE-17623: -- Attachment: HIVE-17623.patch > Fix Select query Fix Double column serde and some refactoring > - > > Key: HIVE-17623 > URL: https://issues.apache.org/jira/browse/HIVE-17623 > Project: Hive > Issue Type: Bug > Components: Druid integration >Affects Versions: 3.0.0 >Reporter: slim bouguerra > Attachments: HIVE-17623.patch > > > This PR has 2 fixes. > First, fixes the limit of results returned by Select query that used to be > limited to 16K rows > Second fixes the type inference for the double type newly added to druid. > Use Jackson polymorphism to infer types and parse results from druid nodes. > Removes duplicate codes form RecordReaders. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17606) Improve security for DB notification related APIs
[
https://issues.apache.org/jira/browse/HIVE-17606?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16183122#comment-16183122
]
Hive QA commented on HIVE-17606:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12889255/HIVE-17606.4.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 145 failed/errored test(s), 11084 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_predicate_pushdown]
(batchId=231)
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_single_sourced_multi_insert]
(batchId=231)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[import_addpartition_blobstore_to_blobstore]
(batchId=243)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[import_addpartition_blobstore_to_local]
(batchId=243)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[import_addpartition_blobstore_to_warehouse]
(batchId=243)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[import_addpartition_local_to_blobstore]
(batchId=243)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[import_blobstore_to_blobstore]
(batchId=243)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[import_blobstore_to_blobstore_nonpart]
(batchId=243)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[import_blobstore_to_local]
(batchId=243)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[import_blobstore_to_warehouse]
(batchId=243)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[import_blobstore_to_warehouse_nonpart]
(batchId=243)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[import_local_to_blobstore]
(batchId=243)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[exim_00_nonpart_empty]
(batchId=14)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[exim_01_nonpart]
(batchId=52)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[exim_02_00_part_empty]
(batchId=64)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[exim_02_part]
(batchId=49)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[exim_03_nonpart_over_compat]
(batchId=5)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[exim_04_all_part]
(batchId=28)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[exim_04_evolved_parts]
(batchId=30)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[exim_05_some_part]
(batchId=71)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[exim_06_one_part]
(batchId=83)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[exim_07_all_part_over_nonoverlap]
(batchId=10)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[exim_08_nonpart_rename]
(batchId=59)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[exim_09_part_spec_nonoverlap]
(batchId=8)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[exim_10_external_managed]
(batchId=67)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[exim_11_managed_external]
(batchId=67)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[exim_12_external_location]
(batchId=53)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[exim_13_managed_location]
(batchId=37)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[exim_14_managed_location_over_existing]
(batchId=52)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[exim_15_external_part]
(batchId=38)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[exim_16_part_external]
(batchId=57)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[exim_17_part_managed]
(batchId=43)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[exim_18_part_external]
(batchId=69)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[exim_19_00_part_external_location]
(batchId=64)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[exim_19_part_external_location]
(batchId=26)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[exim_20_part_managed_location]
(batchId=37)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[exim_21_export_authsuccess]
(batchId=40)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[exim_22_import_exist_authsuccess]
(batchId=18)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[exim_23_import_part_authsuccess]
(batchId=20)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[exim_24_import_nonexist_authsuccess]
(batchId=18)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[exim_25_export_parentpath_has_inaccessible_children]
(batchId=63)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[exim_hidden_files]
(batchId=46)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[repl_2_exim_basic]
(batchId=75)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[repl_3_exim_metadata]
(batchId=55)
[jira] [Updated] (HIVE-17623) Fix Select query Fix Double column serde and some refactoring
[ https://issues.apache.org/jira/browse/HIVE-17623?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] slim bouguerra updated HIVE-17623: -- Status: Patch Available (was: Open) > Fix Select query Fix Double column serde and some refactoring > - > > Key: HIVE-17623 > URL: https://issues.apache.org/jira/browse/HIVE-17623 > Project: Hive > Issue Type: Bug > Components: Druid integration >Affects Versions: 3.0.0 >Reporter: slim bouguerra > > This PR has 2 fixes. > First, fixes the limit of results returned by Select query that used to be > limited to 16K rows > Second fixes the type inference for the double type newly added to druid. > Use Jackson polymorphism to infer types and parse results from druid nodes. > Removes duplicate codes form RecordReaders. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-15212) merge branch into master
[ https://issues.apache.org/jira/browse/HIVE-15212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16183114#comment-16183114 ] Sergey Shelukhin commented on HIVE-15212: - [~hagleitn] [~ekoifman] [~wei.zheng] given that this is pretty close to commit and the fundamentals won't change, do you want to start reviewing? Or give +1s given that most of this work has been collaborative so you are familiar with it :) +1 from my side. I think we are going to commit with some issues in MM tables (ensuring mainline Hive paths work), basically as if the work is done on master and not a feature branch, like with everyone else in Hive. Otherwise conflicts from people not using feature branches (there's another big one now) will prevent this from ever merging. Then fix it on master like everyone else :) > merge branch into master > > > Key: HIVE-15212 > URL: https://issues.apache.org/jira/browse/HIVE-15212 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-15212.01.patch, HIVE-15212.02.patch, > HIVE-15212.03.patch, HIVE-15212.04.patch, HIVE-15212.05.patch, > HIVE-15212.06.patch, HIVE-15212.07.patch, HIVE-15212.08.patch, > HIVE-15212.09.patch, HIVE-15212.10.patch, HIVE-15212.11.patch, > HIVE-15212.12.patch, HIVE-15212.12.patch, HIVE-15212.13.patch, > HIVE-15212.13.patch, HIVE-15212.14.patch, HIVE-15212.15.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17508) Implement pool rules and triggers based on counters
[
https://issues.apache.org/jira/browse/HIVE-17508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16183096#comment-16183096
]
Sergey Shelukhin commented on HIVE-17508:
-
Having multiple rules on the same variable can happen due to operator error (or
intentional design when the operator is too lazy to delete old rules)... it
should at least error out in this case. Using findFirst in some places seems to
pick one random rule.
I think the current factory pretends to be a parser :) If we want a parser we
should use antlr. Otherwise for now it should be more primitive, maybe
literally just split the string on '>' then. But it's a minor comment.
> Implement pool rules and triggers based on counters
> ---
>
> Key: HIVE-17508
> URL: https://issues.apache.org/jira/browse/HIVE-17508
> Project: Hive
> Issue Type: Sub-task
>Affects Versions: 3.0.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
> Attachments: HIVE-17508.1.patch, HIVE-17508.2.patch,
> HIVE-17508.3.patch, HIVE-17508.3.patch, HIVE-17508.WIP.2.patch,
> HIVE-17508.WIP.patch
>
>
> Workload management can defined Rules that are bound to a resource plan. Each
> rule can have a trigger expression and an action associated with it. Trigger
> expressions are evaluated at runtime after configurable check interval, based
> on which actions like killing a query, moving a query to different pool etc.
> will get invoked. Simple rule could be something like
> {code}
> CREATE RULE slow_query IN resource_plan_name
> WHEN execution_time_ms > 1
> MOVE TO slow_queue
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17562) ACID 1.0 + ETL strategy should treat empty compacted files as uncovered deltas
[ https://issues.apache.org/jira/browse/HIVE-17562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16183083#comment-16183083 ] Eugene Koifman commented on HIVE-17562: --- +1 for HIVE-17562-branch-2.patch > ACID 1.0 + ETL strategy should treat empty compacted files as uncovered deltas > -- > > Key: HIVE-17562 > URL: https://issues.apache.org/jira/browse/HIVE-17562 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 2.4.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran > Attachments: HIVE-17562.1.branch-2.patch, HIVE-17562-branch-2.patch, > HIVE-17562-branch-2.patch > > > In branch-2, with ACID 1.0, following sequence will result in incorrect > results > 1) Set split strategy to ETL > 2) Insert some rows > 3) Delete all rows > 4) Alter table compact MAJOR > 5) Insert some rows > 6) Select * query will not return any rows that is written at last (step 5) > The reason for that, compaction essentially voids the first insert in step 2. > Now when ETL split strategy is chosen, there will not be any stripes in the > base files. So no split gets generated and any subsequent deltas gets ignored. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17544) Provide all required info for Authorization
[ https://issues.apache.org/jira/browse/HIVE-17544?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Na Li updated HIVE-17544: - Description: Right now, for authorization 2, the HiveAuthorizationValidator.checkPrivileges(HiveOperationType var1, List var2, List var3, HiveAuthzContext var4) does not contain the parsed sql command string as input. Therefore, Sentry has to parse the command again. The API should be changed to include all required information as input, so Sentry does not need to parse the sql command string again. was: Right now, for authorization 2, the HiveAuthorizationValidator.checkPrivileges(HiveOperationType var1, List var2, List var3, HiveAuthzContext var4) does not contain the parsed sql command string as input. Therefore, Sentry has to parse the command again. The API should be changed to include the parsed result as input, so Sentry does not need to parse the sql command string again. > Provide all required info for Authorization > --- > > Key: HIVE-17544 > URL: https://issues.apache.org/jira/browse/HIVE-17544 > Project: Hive > Issue Type: Task > Components: Authorization >Affects Versions: 2.1.1 >Reporter: Na Li >Assignee: Aihua Xu >Priority: Critical > > Right now, for authorization 2, the > HiveAuthorizationValidator.checkPrivileges(HiveOperationType var1, > List var2, List var3, > HiveAuthzContext var4) does not contain the parsed sql command string as > input. Therefore, Sentry has to parse the command again. > The API should be changed to include all required information as input, so > Sentry does not need to parse the sql command string again. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17544) Provide all required info for Authorization
[ https://issues.apache.org/jira/browse/HIVE-17544?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Na Li updated HIVE-17544: - Summary: Provide all required info for Authorization (was: Add Parsed Tree as input for Authorization) > Provide all required info for Authorization > --- > > Key: HIVE-17544 > URL: https://issues.apache.org/jira/browse/HIVE-17544 > Project: Hive > Issue Type: Task > Components: Authorization >Affects Versions: 2.1.1 >Reporter: Na Li >Assignee: Aihua Xu >Priority: Critical > > Right now, for authorization 2, the > HiveAuthorizationValidator.checkPrivileges(HiveOperationType var1, > List var2, List var3, > HiveAuthzContext var4) does not contain the parsed sql command string as > input. Therefore, Sentry has to parse the command again. > The API should be changed to include the parsed result as input, so Sentry > does not need to parse the sql command string again. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Reopened] (HIVE-17544) Add Parsed Tree as input for Authorization
[ https://issues.apache.org/jira/browse/HIVE-17544?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Na Li reopened HIVE-17544: -- hive does not include UDF class name for create function. That will break the function authorization > Add Parsed Tree as input for Authorization > -- > > Key: HIVE-17544 > URL: https://issues.apache.org/jira/browse/HIVE-17544 > Project: Hive > Issue Type: Task > Components: Authorization >Affects Versions: 2.1.1 >Reporter: Na Li >Assignee: Aihua Xu >Priority: Critical > > Right now, for authorization 2, the > HiveAuthorizationValidator.checkPrivileges(HiveOperationType var1, > List var2, List var3, > HiveAuthzContext var4) does not contain the parsed sql command string as > input. Therefore, Sentry has to parse the command again. > The API should be changed to include the parsed result as input, so Sentry > does not need to parse the sql command string again. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-12734) Remove redundancy in HiveConfs serialized to UDFContext
[
https://issues.apache.org/jira/browse/HIVE-12734?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mithun Radhakrishnan updated HIVE-12734:
Affects Version/s: 3.0.0
2.2.0
> Remove redundancy in HiveConfs serialized to UDFContext
> ---
>
> Key: HIVE-12734
> URL: https://issues.apache.org/jira/browse/HIVE-12734
> Project: Hive
> Issue Type: Bug
>Affects Versions: 1.2.1, 2.0.0, 2.2.0, 3.0.0
>Reporter: Mithun Radhakrishnan
>Assignee: Mithun Radhakrishnan
>
> {{HCatLoader}} lands up serializing one {{HiveConf}} instance per
> table-alias, to Pig's {{UDFContext}}. This lands up bloating the
> {{UDFContext}}.
> To reduce the footprint, it makes sense to serialize a default-constructed
> {{HiveConf}} once, and one "diff" per {{HCatLoader}}. This should reduce the
> time taken to kick off jobs from {{pig -useHCatalog}} scripts.
> (Note_to_self: YHIVE-540).
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17621) Hive-site settings are ignored during HCatInputFormat split-calculation
[
https://issues.apache.org/jira/browse/HIVE-17621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16183064#comment-16183064
]
Mithun Radhakrishnan commented on HIVE-17621:
-
I'm +1 on this, pending tests.
> Hive-site settings are ignored during HCatInputFormat split-calculation
> ---
>
> Key: HIVE-17621
> URL: https://issues.apache.org/jira/browse/HIVE-17621
> Project: Hive
> Issue Type: Bug
> Components: HCatalog
>Affects Versions: 2.2.0, 3.0.0
>Reporter: Mithun Radhakrishnan
>Assignee: Chris Drome
> Attachments: HIVE-17621.1-branch-2.2.patch,
> HIVE-17621.1-branch-2.patch, HIVE-17621.1.patch
>
>
> Another one that [~selinazh] and [~cdrome] worked on.
> The production {{hive-site.xml}} could well contain settings that differ from
> the defaults in {{HiveConf.java}}. In our case, we introduced a custom ORC
> split-strategy, which we introduced as the site-wide default.
> We noticed that during {{HCatInputFormat::getSplits()}}, if the user-script
> did not contain the setting, the site-wide default was ignored in favour of
> the {{HiveConf}} default. HCat would not convey hive-site settings to the
> input-format (or anywhere downstream).
> The forthcoming patch fixes this problem.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-15016) Run tests with Hadoop 3.0.0-alpha1
[ https://issues.apache.org/jira/browse/HIVE-15016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16183063#comment-16183063 ] Sergio Peña commented on HIVE-15016: I stopped working on this while ago. Someone else can take it. [~aihuaxu] you were running hadoop3 tests as well. Do you have some results for this jira? > Run tests with Hadoop 3.0.0-alpha1 > -- > > Key: HIVE-15016 > URL: https://issues.apache.org/jira/browse/HIVE-15016 > Project: Hive > Issue Type: Sub-task > Components: Hive >Reporter: Sergio Peña >Assignee: Sergio Peña > Attachments: Hadoop3Upstream.patch > > > Hadoop 3.0.0-alpha1 was released back on Sep/16 to allow other components run > tests against this new version before GA. > We should start running tests with Hive to validate compatibility against > Hadoop 3.0. > NOTE: The patch used to test must not be committed to Hive until Hadoop 3.0 > GA is released. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17621) Hive-site settings are ignored during HCatInputFormat split-calculation
[
https://issues.apache.org/jira/browse/HIVE-17621?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mithun Radhakrishnan updated HIVE-17621:
Attachment: HIVE-17621.1-branch-2.2.patch
HIVE-17621.1-branch-2.patch
> Hive-site settings are ignored during HCatInputFormat split-calculation
> ---
>
> Key: HIVE-17621
> URL: https://issues.apache.org/jira/browse/HIVE-17621
> Project: Hive
> Issue Type: Bug
> Components: HCatalog
>Affects Versions: 2.2.0, 3.0.0
>Reporter: Mithun Radhakrishnan
>Assignee: Chris Drome
> Attachments: HIVE-17621.1-branch-2.2.patch,
> HIVE-17621.1-branch-2.patch, HIVE-17621.1.patch
>
>
> Another one that [~selinazh] and [~cdrome] worked on.
> The production {{hive-site.xml}} could well contain settings that differ from
> the defaults in {{HiveConf.java}}. In our case, we introduced a custom ORC
> split-strategy, which we introduced as the site-wide default.
> We noticed that during {{HCatInputFormat::getSplits()}}, if the user-script
> did not contain the setting, the site-wide default was ignored in favour of
> the {{HiveConf}} default. HCat would not convey hive-site settings to the
> input-format (or anywhere downstream).
> The forthcoming patch fixes this problem.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-15016) Run tests with Hadoop 3.0.0-alpha1
[ https://issues.apache.org/jira/browse/HIVE-15016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16183045#comment-16183045 ] Ashutosh Chauhan commented on HIVE-15016: - [~spena] Any progress on this? I think we shall do HIVE-15326 and HIVE-16532 first so that its we can get it to compile atleast : ) > Run tests with Hadoop 3.0.0-alpha1 > -- > > Key: HIVE-15016 > URL: https://issues.apache.org/jira/browse/HIVE-15016 > Project: Hive > Issue Type: Sub-task > Components: Hive >Reporter: Sergio Peña >Assignee: Sergio Peña > Attachments: Hadoop3Upstream.patch > > > Hadoop 3.0.0-alpha1 was released back on Sep/16 to allow other components run > tests against this new version before GA. > We should start running tests with Hive to validate compatibility against > Hadoop 3.0. > NOTE: The patch used to test must not be committed to Hive until Hadoop 3.0 > GA is released. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (HIVE-17622) Implement ANTLR based expressions for rule triggers
[ https://issues.apache.org/jira/browse/HIVE-17622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran reassigned HIVE-17622: > Implement ANTLR based expressions for rule triggers > --- > > Key: HIVE-17622 > URL: https://issues.apache.org/jira/browse/HIVE-17622 > Project: Hive > Issue Type: Sub-task >Affects Versions: 3.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran > > For more expressiveness of rule trigger expressions move manual expression > parsing in HIVE-17508 to ANTLR based grammar. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17621) Hive-site settings are ignored during HCatInputFormat split-calculation
[
https://issues.apache.org/jira/browse/HIVE-17621?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mithun Radhakrishnan updated HIVE-17621:
Status: Patch Available (was: Open)
> Hive-site settings are ignored during HCatInputFormat split-calculation
> ---
>
> Key: HIVE-17621
> URL: https://issues.apache.org/jira/browse/HIVE-17621
> Project: Hive
> Issue Type: Bug
> Components: HCatalog
>Affects Versions: 2.2.0, 3.0.0
>Reporter: Mithun Radhakrishnan
>Assignee: Chris Drome
> Attachments: HIVE-17621.1.patch
>
>
> Another one that [~selinazh] and [~cdrome] worked on.
> The production {{hive-site.xml}} could well contain settings that differ from
> the defaults in {{HiveConf.java}}. In our case, we introduced a custom ORC
> split-strategy, which we introduced as the site-wide default.
> We noticed that during {{HCatInputFormat::getSplits()}}, if the user-script
> did not contain the setting, the site-wide default was ignored in favour of
> the {{HiveConf}} default. HCat would not convey hive-site settings to the
> input-format (or anywhere downstream).
> The forthcoming patch fixes this problem.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17508) Implement pool rules and triggers based on counters
[
https://issues.apache.org/jira/browse/HIVE-17508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16183007#comment-16183007
]
Prasanth Jayachandran commented on HIVE-17508:
--
Like the idea of f(, ) -> , all applied in WM. The initial
issue was with the poll semantics for getting the progress in driver code which
will essentially loop in TezJobMonitor. I will probably add some listener for
counters, TezJobMonitor will then publish to those listeners (WM). I will
address your comments in the next patch.
As far as the grammar for expression goes, I don't think a full fledged grammar
would be required. Isn't there a jira already to handle the frontend? I added
this primarily by seeing the schema in the metastore which stores the
expression as varchar. If frontend grammar already exists then I can reuse,
else I will add the grammar in a follow up and integrate it with this patch.
The only expression of interest is greater than '>'. Also having >1 rule is
equivalent of disjunctive evaluation. Any rule violation will trigger action
instead of all rule violation. IMHO conjunctive evaluation can be added later
with full fledged grammar.
> Implement pool rules and triggers based on counters
> ---
>
> Key: HIVE-17508
> URL: https://issues.apache.org/jira/browse/HIVE-17508
> Project: Hive
> Issue Type: Sub-task
>Affects Versions: 3.0.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
> Attachments: HIVE-17508.1.patch, HIVE-17508.2.patch,
> HIVE-17508.3.patch, HIVE-17508.3.patch, HIVE-17508.WIP.2.patch,
> HIVE-17508.WIP.patch
>
>
> Workload management can defined Rules that are bound to a resource plan. Each
> rule can have a trigger expression and an action associated with it. Trigger
> expressions are evaluated at runtime after configurable check interval, based
> on which actions like killing a query, moving a query to different pool etc.
> will get invoked. Simple rule could be something like
> {code}
> CREATE RULE slow_query IN resource_plan_name
> WHEN execution_time_ms > 1
> MOVE TO slow_queue
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-17621) Hive-site settings are ignored during HCatInputFormat split-calculation
[
https://issues.apache.org/jira/browse/HIVE-17621?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mithun Radhakrishnan updated HIVE-17621:
Attachment: HIVE-17621.1.patch
> Hive-site settings are ignored during HCatInputFormat split-calculation
> ---
>
> Key: HIVE-17621
> URL: https://issues.apache.org/jira/browse/HIVE-17621
> Project: Hive
> Issue Type: Bug
> Components: HCatalog
>Affects Versions: 2.2.0, 3.0.0
>Reporter: Mithun Radhakrishnan
>Assignee: Chris Drome
> Attachments: HIVE-17621.1.patch
>
>
> Another one that [~selinazh] and [~cdrome] worked on.
> The production {{hive-site.xml}} could well contain settings that differ from
> the defaults in {{HiveConf.java}}. In our case, we introduced a custom ORC
> split-strategy, which we introduced as the site-wide default.
> We noticed that during {{HCatInputFormat::getSplits()}}, if the user-script
> did not contain the setting, the site-wide default was ignored in favour of
> the {{HiveConf}} default. HCat would not convey hive-site settings to the
> input-format (or anywhere downstream).
> The forthcoming patch fixes this problem.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-17604) Add druid properties to conf white list
[ https://issues.apache.org/jira/browse/HIVE-17604?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-17604: Resolution: Fixed Fix Version/s: 3.0.0 Status: Resolved (was: Patch Available) Pushed to master. > Add druid properties to conf white list > --- > > Key: HIVE-17604 > URL: https://issues.apache.org/jira/browse/HIVE-17604 > Project: Hive > Issue Type: Improvement > Components: Configuration, Druid integration >Affects Versions: 2.2.0, 2.3.0 >Reporter: Ashutosh Chauhan >Assignee: Ashutosh Chauhan > Fix For: 3.0.0 > > Attachments: HIVE-17604.2.patch, HIVE-17604.patch > > > Currently throws: > Error: Error while processing statement: Cannot modify > hive.druid.select.distribute at runtime. It is not in list of params that are > allowed to be modified at runtime (state=42000,code=1) -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (HIVE-17621) Hive-site settings are ignored during HCatInputFormat split-calculation
[
https://issues.apache.org/jira/browse/HIVE-17621?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mithun Radhakrishnan reassigned HIVE-17621:
---
> Hive-site settings are ignored during HCatInputFormat split-calculation
> ---
>
> Key: HIVE-17621
> URL: https://issues.apache.org/jira/browse/HIVE-17621
> Project: Hive
> Issue Type: Bug
> Components: HCatalog
>Affects Versions: 2.2.0, 3.0.0
>Reporter: Mithun Radhakrishnan
>Assignee: Chris Drome
>
> Another one that [~selinazh] and [~cdrome] worked on.
> The production {{hive-site.xml}} could well contain settings that differ from
> the defaults in {{HiveConf.java}}. In our case, we introduced a custom ORC
> split-strategy, which we introduced as the site-wide default.
> We noticed that during {{HCatInputFormat::getSplits()}}, if the user-script
> did not contain the setting, the site-wide default was ignored in favour of
> the {{HiveConf}} default. HCat would not convey hive-site settings to the
> input-format (or anywhere downstream).
> The forthcoming patch fixes this problem.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
