[jira] [Commented] (HIVE-9453) Initial patch [hbase-metastore branch]
[ https://issues.apache.org/jira/browse/HIVE-9453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14632301#comment-14632301 ] Lefty Leverenz commented on HIVE-9453: -- Since it isn't the 1.2 release after all, I'm linking this issue to HIVE-9752 (Documentation for HBase metastore). Also, the new tool to dump out the contents of the metastore should be documented (Alan's comment, 30/Jan/15). > Initial patch [hbase-metastore branch] > -- > > Key: HIVE-9453 > URL: https://issues.apache.org/jira/browse/HIVE-9453 > Project: Hive > Issue Type: Sub-task > Components: Metastore >Affects Versions: hbase-metastore-branch >Reporter: Alan Gates >Assignee: Alan Gates > Fix For: hbase-metastore-branch > > Attachments: HIVE-9453-reviewcomments.pdf, HIVE-9453.2.patch, > HIVE-9453.3.patch, HIVE-9453.patch > > > This initial patch has several important features: > # HBaseStore, a new implementation of RawStore that stores the data in HBase. > # Subclasses of the thrift metastore objects to remove the massive > duplication of data where every partition contains a nearly identical storage > descriptor. > # Caches for catalog objects and statistics so that repeated metastore calls > don't result in repeated calls against HBase. > Currently this "works" to the point that "load table" and "select" work. I > have not tested any other statements, and I suspect most fail. There is no > security, no authorization, and a no a lot of other things. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11306) Add a bloom-1 filter for Hybrid MapJoin spills
[ https://issues.apache.org/jira/browse/HIVE-11306?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gopal V updated HIVE-11306: --- Attachment: HIVE-11306.1.patch > Add a bloom-1 filter for Hybrid MapJoin spills > -- > > Key: HIVE-11306 > URL: https://issues.apache.org/jira/browse/HIVE-11306 > Project: Hive > Issue Type: Improvement > Components: Hive >Affects Versions: 1.3.0, 2.0.0 >Reporter: Gopal V >Assignee: Gopal V > Attachments: HIVE-11306.1.patch > > > HIVE-9277 implemented Spillable joins for Tez, which suffers from a > corner-case performance issue when joining wide small tables against a narrow > big table (like a user info table join events stream). > The fact that the wide table is spilled causes extra IO, even though the nDV > of the join key might be in the thousands. > A cheap bloom-1 filter would add a massive performance gain for such queries, > massively cutting down on the spill IO costs for the big-table spills. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9152) Dynamic Partition Pruning [Spark Branch]
[ https://issues.apache.org/jira/browse/HIVE-9152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14632293#comment-14632293 ] Lefty Leverenz commented on HIVE-9152: -- Doc note: This adds two configuration parameters (*hive.spark.dynamic.partition.pruning* and *hive.spark.dynamic.partition.pruning.max.data.size*) to HiveConf.java, so they should be documented in Hive on Spark: Getting Started and in the Spark section of Configuration Properties. * doc now or only after merge branch with master? -- [Hive on Spark: Getting Started -- Configuring Hive | https://cwiki.apache.org/confluence/display/Hive/Hive+on+Spark%3A+Getting+Started#HiveonSpark:GettingStarted-ConfiguringHive] * wait until merge with master -- [Configuration Properties -- Spark | https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-Spark] Hive on Spark: Getting Started might also need general information about dynamic partition pruning. > Dynamic Partition Pruning [Spark Branch] > > > Key: HIVE-9152 > URL: https://issues.apache.org/jira/browse/HIVE-9152 > Project: Hive > Issue Type: Sub-task > Components: Spark >Affects Versions: spark-branch >Reporter: Brock Noland >Assignee: Chao Sun > Labels: TODOC-SPARK > Fix For: spark-branch > > Attachments: HIVE-9152.1-spark.patch, HIVE-9152.10-spark.patch, > HIVE-9152.11-spark.patch, HIVE-9152.12-spark.patch, HIVE-9152.2-spark.patch, > HIVE-9152.3-spark.patch, HIVE-9152.4-spark.patch, HIVE-9152.5-spark.patch, > HIVE-9152.6-spark.patch, HIVE-9152.8-spark.patch, HIVE-9152.9-spark.patch > > > Tez implemented dynamic partition pruning in HIVE-7826. This is a nice > optimization and we should implement the same in HOS. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11305) LLAP: Hybrid Map-join cache returns invalid data
[
https://issues.apache.org/jira/browse/HIVE-11305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14632290#comment-14632290
]
Gopal V commented on HIVE-11305:
Running the same tests with {{./dist/hive/bin/hive --hiveconf
hive.execution.mode=container -f q55-test.sql}} works.
> LLAP: Hybrid Map-join cache returns invalid data
> -
>
> Key: HIVE-11305
> URL: https://issues.apache.org/jira/browse/HIVE-11305
> Project: Hive
> Issue Type: Sub-task
>Affects Versions: llap
> Environment: TPC-DS 200 scale data
>Reporter: Gopal V
>Assignee: Sergey Shelukhin
>Priority: Critical
> Fix For: llap
>
> Attachments: q55-test.sql
>
>
> Start a 1-node LLAP cluster with 16 executors and run attached test-case on
> the single node instance.
> {code}
> Caused by: java.lang.ClassCastException:
> org.apache.hadoop.hive.ql.exec.persistence.HybridHashTableContainer cannot be
> cast to
> org.apache.hadoop.hive.ql.exec.vector.mapjoin.hashtable.VectorMapJoinTableContainer
> at
> org.apache.hadoop.hive.ql.exec.vector.mapjoin.VectorMapJoinCommonOperator.loadHashTable(VectorMapJoinCommonOperator.java:648)
> at
> org.apache.hadoop.hive.ql.exec.MapJoinOperator.cleanUpInputFileChangedOp(MapJoinOperator.java:314)
> at
> org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1104)
> at
> org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1108)
> at
> org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1108)
> at
> org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1108)
> at
> org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1108)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.process(VectorMapOperator.java:37)
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.processRow(MapRecordSource.java:86)
> ... 17 more
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-11305) LLAP: Hybrid Map-join cache returns invalid data
[
https://issues.apache.org/jira/browse/HIVE-11305?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gopal V updated HIVE-11305:
---
Attachment: q55-test.sql
> LLAP: Hybrid Map-join cache returns invalid data
> -
>
> Key: HIVE-11305
> URL: https://issues.apache.org/jira/browse/HIVE-11305
> Project: Hive
> Issue Type: Sub-task
>Affects Versions: llap
> Environment: TPC-DS 200 scale data
>Reporter: Gopal V
>Assignee: Sergey Shelukhin
>Priority: Critical
> Fix For: llap
>
> Attachments: q55-test.sql
>
>
> Start a 1-node LLAP cluster with 16 executors and run attached test-case on
> the single node instance.
> {code}
> Caused by: java.lang.ClassCastException:
> org.apache.hadoop.hive.ql.exec.persistence.HybridHashTableContainer cannot be
> cast to
> org.apache.hadoop.hive.ql.exec.vector.mapjoin.hashtable.VectorMapJoinTableContainer
> at
> org.apache.hadoop.hive.ql.exec.vector.mapjoin.VectorMapJoinCommonOperator.loadHashTable(VectorMapJoinCommonOperator.java:648)
> at
> org.apache.hadoop.hive.ql.exec.MapJoinOperator.cleanUpInputFileChangedOp(MapJoinOperator.java:314)
> at
> org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1104)
> at
> org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1108)
> at
> org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1108)
> at
> org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1108)
> at
> org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1108)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.process(VectorMapOperator.java:37)
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.processRow(MapRecordSource.java:86)
> ... 17 more
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-9152) Dynamic Partition Pruning [Spark Branch]
[ https://issues.apache.org/jira/browse/HIVE-9152?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lefty Leverenz updated HIVE-9152: - Labels: TODOC-SPARK (was: ) > Dynamic Partition Pruning [Spark Branch] > > > Key: HIVE-9152 > URL: https://issues.apache.org/jira/browse/HIVE-9152 > Project: Hive > Issue Type: Sub-task > Components: Spark >Affects Versions: spark-branch >Reporter: Brock Noland >Assignee: Chao Sun > Labels: TODOC-SPARK > Fix For: spark-branch > > Attachments: HIVE-9152.1-spark.patch, HIVE-9152.10-spark.patch, > HIVE-9152.11-spark.patch, HIVE-9152.12-spark.patch, HIVE-9152.2-spark.patch, > HIVE-9152.3-spark.patch, HIVE-9152.4-spark.patch, HIVE-9152.5-spark.patch, > HIVE-9152.6-spark.patch, HIVE-9152.8-spark.patch, HIVE-9152.9-spark.patch > > > Tez implemented dynamic partition pruning in HIVE-7826. This is a nice > optimization and we should implement the same in HOS. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11077) Add support in parser and wire up to txn manager
[
https://issues.apache.org/jira/browse/HIVE-11077?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14632284#comment-14632284
]
Hive QA commented on HIVE-11077:
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12745919/HIVE-11077.6.patch
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 9242 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cbo_rp_join0
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/4647/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/4647/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-4647/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12745919 - PreCommit-HIVE-TRUNK-Build
> Add support in parser and wire up to txn manager
>
>
> Key: HIVE-11077
> URL: https://issues.apache.org/jira/browse/HIVE-11077
> Project: Hive
> Issue Type: Sub-task
> Components: SQL, Transactions
>Affects Versions: 1.2.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
> Attachments: HIVE-11077.3.patch, HIVE-11077.5.patch,
> HIVE-11077.6.patch, HIVE-11077.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (HIVE-10930) LLAP: Set java.io.tmpdir correctly for LLAP Slider instance
[ https://issues.apache.org/jira/browse/HIVE-10930?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gopal V resolved HIVE-10930. Resolution: Fixed Release Note: HIVE-10930: LLAP: Set java.io.tmpdir correctly for LLAP Slider instance (Gopal V) > LLAP: Set java.io.tmpdir correctly for LLAP Slider instance > --- > > Key: HIVE-10930 > URL: https://issues.apache.org/jira/browse/HIVE-10930 > Project: Hive > Issue Type: Sub-task >Affects Versions: llap >Reporter: Gopal V >Assignee: Gopal V > Fix For: llap > > Attachments: HIVE-10930.patch > > > LLAP's Hybrid Grace Hash is IO bound writing to /tmp. > Use the yarn local dirs instead of "/tmp" so that createTempFile works > correctly. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-10930) LLAP: Set java.io.tmpdir correctly for LLAP Slider instance
[ https://issues.apache.org/jira/browse/HIVE-10930?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gopal V updated HIVE-10930: --- Attachment: HIVE-10930.patch > LLAP: Set java.io.tmpdir correctly for LLAP Slider instance > --- > > Key: HIVE-10930 > URL: https://issues.apache.org/jira/browse/HIVE-10930 > Project: Hive > Issue Type: Sub-task >Affects Versions: llap >Reporter: Gopal V >Assignee: Gopal V > Fix For: llap > > Attachments: HIVE-10930.patch > > > LLAP's Hybrid Grace Hash is IO bound writing to /tmp. > Use the yarn local dirs instead of "/tmp" so that createTempFile works > correctly. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11303) Getting Tez LimitExceededException after dag execution on large query
[
https://issues.apache.org/jira/browse/HIVE-11303?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14632255#comment-14632255
]
Gopal V commented on HIVE-11303:
is this triggered by the {{hive.tez.exec.print.summary}} ?
> Getting Tez LimitExceededException after dag execution on large query
> -
>
> Key: HIVE-11303
> URL: https://issues.apache.org/jira/browse/HIVE-11303
> Project: Hive
> Issue Type: Bug
> Components: Tez
>Reporter: Jason Dere
>
> {noformat}
> 2015-07-17 18:18:11,830 INFO [main]: counters.Limits
> (Limits.java:ensureInitialized(59)) - Counter limits initialized with
> parameters: GROUP_NAME_MAX=256, MAX_GROUPS=500, COUNTER_NAME_MAX=64,
> MAX_COUNTERS=1200
> 2015-07-17 18:18:11,841 ERROR [main]: exec.Task (TezTask.java:execute(189)) -
> Failed to execute tez graph.
> org.apache.tez.common.counters.LimitExceededException: Too many counters:
> 1201 max=1200
> at org.apache.tez.common.counters.Limits.checkCounters(Limits.java:87)
> at org.apache.tez.common.counters.Limits.incrCounters(Limits.java:94)
> at
> org.apache.tez.common.counters.AbstractCounterGroup.addCounter(AbstractCounterGroup.java:76)
> at
> org.apache.tez.common.counters.AbstractCounterGroup.addCounterImpl(AbstractCounterGroup.java:93)
> at
> org.apache.tez.common.counters.AbstractCounterGroup.findCounter(AbstractCounterGroup.java:104)
> at
> org.apache.tez.dag.api.DagTypeConverters.convertTezCountersFromProto(DagTypeConverters.java:567)
> at
> org.apache.tez.dag.api.client.DAGStatus.getDAGCounters(DAGStatus.java:148)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezTask.execute(TezTask.java:175)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:160)
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:89)
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1673)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1432)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1213)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1064)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1054)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:213)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:165)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:376)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:311)
> at
> org.apache.hadoop.hive.cli.CliDriver.processReader(CliDriver.java:409)
> at
> org.apache.hadoop.hive.cli.CliDriver.processFile(CliDriver.java:425)
> at
> org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:714)
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:681)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:621)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-10799) Refactor the SearchArgumentFactory to remove the dependence on ExprNodeGenericFuncDesc
[
https://issues.apache.org/jira/browse/HIVE-10799?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14632252#comment-14632252
]
Hive QA commented on HIVE-10799:
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12745912/HIVE-10799.patch
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 9239 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cbo_rp_join0
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/4646/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/4646/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-4646/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12745912 - PreCommit-HIVE-TRUNK-Build
> Refactor the SearchArgumentFactory to remove the dependence on
> ExprNodeGenericFuncDesc
> --
>
> Key: HIVE-10799
> URL: https://issues.apache.org/jira/browse/HIVE-10799
> Project: Hive
> Issue Type: Sub-task
>Reporter: Owen O'Malley
>Assignee: Owen O'Malley
> Attachments: HIVE-10799.patch, HIVE-10799.patch, HIVE-10799.patch,
> HIVE-10799.patch
>
>
> SearchArgumentFactory and SearchArgumentImpl are high level and shouldn't
> depend on the internals of Hive's AST model.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-11210) Remove dependency on HiveConf from Orc reader & writer
[
https://issues.apache.org/jira/browse/HIVE-11210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1463#comment-1463
]
Hive QA commented on HIVE-11210:
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12745907/HIVE-11210.patch
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 9225 tests executed
*Failed tests:*
{noformat}
TestCustomAuthentication - did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cbo_rp_join0
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/4645/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/4645/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-4645/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12745907 - PreCommit-HIVE-TRUNK-Build
> Remove dependency on HiveConf from Orc reader & writer
> --
>
> Key: HIVE-11210
> URL: https://issues.apache.org/jira/browse/HIVE-11210
> Project: Hive
> Issue Type: Sub-task
>Reporter: Owen O'Malley
>Assignee: Owen O'Malley
> Attachments: HIVE-11210.patch, HIVE-11210.patch
>
>
> Currently the ORC reader and writer get their default values from HiveConf. I
> propose that we make the reader and writer have their own programatic
> defaults and the OrcInputFormat and OrcOutputFormat can use the version in
> HiveConf.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-11300) HBase metastore: Support token and master key methods
[ https://issues.apache.org/jira/browse/HIVE-11300?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alan Gates updated HIVE-11300: -- Attachment: HIVE-11300.patch > HBase metastore: Support token and master key methods > - > > Key: HIVE-11300 > URL: https://issues.apache.org/jira/browse/HIVE-11300 > Project: Hive > Issue Type: Improvement > Components: Metastore >Affects Versions: hbase-metastore-branch >Reporter: Alan Gates >Assignee: Alan Gates > Attachments: HIVE-11300.patch > > > The methods addToken, removeToken, getToken, getAllTokenIdentifiers, > addMasterKey, updateMasterKey, removeMasterKey, and getMasterKeys() need to > be implemented. They are all used in security. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-10937) LLAP: make ObjectCache for plans work properly in the daemon
[ https://issues.apache.org/jira/browse/HIVE-10937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-10937: Attachment: HIVE-10937.03.patch Rebased the patch and extended cleanup code for it > LLAP: make ObjectCache for plans work properly in the daemon > > > Key: HIVE-10937 > URL: https://issues.apache.org/jira/browse/HIVE-10937 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Fix For: llap > > Attachments: HIVE-10937.01.patch, HIVE-10937.02.patch, > HIVE-10937.03.patch, HIVE-10937.patch > > > There's perf hit otherwise, esp. when stupid planner creates 1009 reducers of > 4Mb each. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11303) Getting Tez LimitExceededException after dag execution on large query
[
https://issues.apache.org/jira/browse/HIVE-11303?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14632212#comment-14632212
]
Jason Dere commented on HIVE-11303:
---
Thing is, I've actually set tez.counters.max=16000 on the Hive command line. I
can also see in the Tez logs, that the counters are set to 16000 during the dag
execution.
The log messages in the Jira description (including the line where
MAX_COUNTERS=1200) is from my hive.log. So it seems like when we're trying to
fetch the counters after execution, that tez.counters.max is somehow not set in
the DagClient's configuration. I suspect the TezSessionState's conf is coming
from the SessionState's configuration, which might be getting initialized with
a blank config from SessionState.getSessionConf().
> Getting Tez LimitExceededException after dag execution on large query
> -
>
> Key: HIVE-11303
> URL: https://issues.apache.org/jira/browse/HIVE-11303
> Project: Hive
> Issue Type: Bug
> Components: Tez
>Reporter: Jason Dere
>
> {noformat}
> 2015-07-17 18:18:11,830 INFO [main]: counters.Limits
> (Limits.java:ensureInitialized(59)) - Counter limits initialized with
> parameters: GROUP_NAME_MAX=256, MAX_GROUPS=500, COUNTER_NAME_MAX=64,
> MAX_COUNTERS=1200
> 2015-07-17 18:18:11,841 ERROR [main]: exec.Task (TezTask.java:execute(189)) -
> Failed to execute tez graph.
> org.apache.tez.common.counters.LimitExceededException: Too many counters:
> 1201 max=1200
> at org.apache.tez.common.counters.Limits.checkCounters(Limits.java:87)
> at org.apache.tez.common.counters.Limits.incrCounters(Limits.java:94)
> at
> org.apache.tez.common.counters.AbstractCounterGroup.addCounter(AbstractCounterGroup.java:76)
> at
> org.apache.tez.common.counters.AbstractCounterGroup.addCounterImpl(AbstractCounterGroup.java:93)
> at
> org.apache.tez.common.counters.AbstractCounterGroup.findCounter(AbstractCounterGroup.java:104)
> at
> org.apache.tez.dag.api.DagTypeConverters.convertTezCountersFromProto(DagTypeConverters.java:567)
> at
> org.apache.tez.dag.api.client.DAGStatus.getDAGCounters(DAGStatus.java:148)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezTask.execute(TezTask.java:175)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:160)
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:89)
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1673)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1432)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1213)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1064)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1054)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:213)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:165)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:376)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:311)
> at
> org.apache.hadoop.hive.cli.CliDriver.processReader(CliDriver.java:409)
> at
> org.apache.hadoop.hive.cli.CliDriver.processFile(CliDriver.java:425)
> at
> org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:714)
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:681)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:621)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-11244) Beeline prompt info improvement for cluster mode
[
https://issues.apache.org/jira/browse/HIVE-11244?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nemon Lou updated HIVE-11244:
-
Attachment: HIVE-11244.2.patch
Code style fixed.
I do see the failed test with this patch ,but after revert it and patch
again，then fail to reproduce.
> Beeline prompt info improvement for cluster mode
>
>
> Key: HIVE-11244
> URL: https://issues.apache.org/jira/browse/HIVE-11244
> Project: Hive
> Issue Type: Improvement
> Components: Beeline
>Affects Versions: 1.2.0
>Reporter: Nemon Lou
>Assignee: Nemon Lou
>Priority: Minor
> Labels: TODOC2.0
> Attachments: HIVE-11244.1.patch, HIVE-11244.2.patch,
> HIVE-11244.patch, HIVE-11244.patch, HIVE-11244.patch
>
>
> Currently Beeline prompt info for Cluster mode is like this:
> {noformat}
> 0: jdbc:hive2://192.168.115.1:24002,192.168.1>
> {noformat}
> Using the very HiveServer2's IP that this beeline connect to is more helpful
> for users.
> Like this:
> {noformat}
> 0: jdbc:hive2://192.168.115.1:24002>
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-11077) Add support in parser and wire up to txn manager
[ https://issues.apache.org/jira/browse/HIVE-11077?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-11077: -- Attachment: HIVE-11077.6.patch > Add support in parser and wire up to txn manager > > > Key: HIVE-11077 > URL: https://issues.apache.org/jira/browse/HIVE-11077 > Project: Hive > Issue Type: Sub-task > Components: SQL, Transactions >Affects Versions: 1.2.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman > Attachments: HIVE-11077.3.patch, HIVE-11077.5.patch, > HIVE-11077.6.patch, HIVE-11077.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11301) thrift metastore issue when getting stats results in disconnect
[
https://issues.apache.org/jira/browse/HIVE-11301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14632175#comment-14632175
]
Sergey Shelukhin commented on HIVE-11301:
-
(I know what the problem is, I don't have cycles to fix it right now :)
> thrift metastore issue when getting stats results in disconnect
> ---
>
> Key: HIVE-11301
> URL: https://issues.apache.org/jira/browse/HIVE-11301
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Reporter: Sergey Shelukhin
>
> On metastore side it looks like this:
> {noformat}
> 2015-07-17 20:32:27,795 ERROR [pool-3-thread-150]: server.TThreadPoolServer
> (TThreadPoolServer.java:run(294)) - Thrift error occurred during processing
> of message.
> org.apache.thrift.protocol.TProtocolException: Required field 'colStats' is
> unset! Struct:AggrStats(colStats:null, partsFound:0)
> at
> org.apache.hadoop.hive.metastore.api.AggrStats.validate(AggrStats.java:389)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_aggr_stats_for_result.validate(ThriftHiveMetastore.java)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_aggr_stats_for_result$get_aggr_stats_for_resultStandardScheme.write(ThriftHiveMetastore.java)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_aggr_stats_for_result$get_aggr_stats_for_resultStandardScheme.write(ThriftHiveMetastore.java)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_aggr_stats_for_result.write(ThriftHiveMetastore.java)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:53)
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:110)
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:106)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor.process(TUGIBasedProcessor.java:118)
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:285)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
> and then
> {noformat}
> 2015-07-17 20:32:27,796 WARN [pool-3-thread-150]:
> transport.TIOStreamTransport (TIOStreamTransport.java:close(112)) - Error
> closing output stream.
> java.net.SocketException: Socket closed
> at
> java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:116)
> at java.net.SocketOutputStream.write(SocketOutputStream.java:153)
> at
> java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
> at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
> at java.io.FilterOutputStream.close(FilterOutputStream.java:158)
> at
> org.apache.thrift.transport.TIOStreamTransport.close(TIOStreamTransport.java:110)
> at org.apache.thrift.transport.TSocket.close(TSocket.java:196)
> at
> org.apache.hadoop.hive.thrift.TFilterTransport.close(TFilterTransport.java:52)
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:304)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
> Which on client manifests as
> {noformat}
> 2015-07-17 20:32:27,796 WARN [main()]: metastore.RetryingMetaStoreClient
> (RetryingMetaStoreClient.java:invoke(187)) - MetaStoreClient lost connection.
> Attempting to reconnect.
> org.apache.thrift.transport.TTransportException
> at
> org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)
> at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
> at
> org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429)
> at
> org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318)
> at
> org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219)
> at
> org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:69)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_aggr_stats_for(ThriftHiveMetastore.java:3029)
> at
> org.apache.hadoop.
[jira] [Commented] (HIVE-11299) Hive query planning allocating lots of memory and hitting OOMs
[
https://issues.apache.org/jira/browse/HIVE-11299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14632170#comment-14632170
]
Hive QA commented on HIVE-11299:
{color:red}Overall{color}: -1 no tests executed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12745898/HIVE-11299.patch
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/4644/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/4644/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-4644/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hive-ptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ [[ -n /usr/java/jdk1.7.0_45-cloudera ]]
+ export JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera
+ JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera
+ export
PATH=/usr/java/jdk1.7.0_45-cloudera/bin/:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.7.0_45-cloudera/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin
+
PATH=/usr/java/jdk1.7.0_45-cloudera/bin/:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.7.0_45-cloudera/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'M2_OPTS=-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost
-Dhttp.proxyPort=3128'
+ M2_OPTS='-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost
-Dhttp.proxyPort=3128'
+ cd /data/hive-ptest/working/
+ tee /data/hive-ptest/logs/PreCommit-HIVE-TRUNK-Build-4644/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at 7b17df1 HIVE-11291 : Avoid allocation storm while doing rule
matching on operator/expression trees (Ashutosh Chauhan via Hari Sankar)
+ git clean -f -d
+ git checkout master
Already on 'master'
+ git reset --hard origin/master
HEAD is now at 7b17df1 HIVE-11291 : Avoid allocation storm while doing rule
matching on operator/expression trees (Ashutosh Chauhan via Hari Sankar)
+ git merge --ff-only origin/master
Already up-to-date.
+ git gc
+ patchCommandPath=/data/hive-ptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hive-ptest/working/scratch/build.patch
+ [[ -f /data/hive-ptest/working/scratch/build.patch ]]
+ chmod +x /data/hive-ptest/working/scratch/smart-apply-patch.sh
+ /data/hive-ptest/working/scratch/smart-apply-patch.sh
/data/hive-ptest/working/scratch/build.patch
The patch does not appear to apply with p0, p1, or p2
+ exit 1
'
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12745898 - PreCommit-HIVE-TRUNK-Build
> Hive query planning allocating lots of memory and hitting OOMs
> ---
>
> Key: HIVE-11299
> URL: https://issues.apache.org/jira/browse/HIVE-11299
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 0.14.0
>Reporter: Mostafa Mokhtar
>Assignee: Mostafa Mokhtar
> Attachments: HIVE-11299.patch
>
>
> Plan generation for queries with lots of disjunct filters spends lots of time
> compiling.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-11209) Clean up dependencies in HiveDecimalWritable
[
https://issues.apache.org/jira/browse/HIVE-11209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14632168#comment-14632168
]
Hive QA commented on HIVE-11209:
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12745893/HIVE-11209.patch
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 9226 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cbo_rp_join0
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/4643/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/4643/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-4643/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12745893 - PreCommit-HIVE-TRUNK-Build
> Clean up dependencies in HiveDecimalWritable
>
>
> Key: HIVE-11209
> URL: https://issues.apache.org/jira/browse/HIVE-11209
> Project: Hive
> Issue Type: Sub-task
>Reporter: Owen O'Malley
>Assignee: Owen O'Malley
> Fix For: 2.0.0
>
> Attachments: HIVE-11209.patch, HIVE-11209.patch, HIVE-11209.patch,
> HIVE-11209.patch
>
>
> Currently HiveDecimalWritable depends on:
> * org.apache.hadoop.hive.serde2.ByteStream
> * org.apache.hadoop.hive.serde2.lazybinary.LazyBinaryUtils
> * org.apache.hadoop.hive.serde2.typeinfo.HiveDecimalUtils
> since we need HiveDecimalWritable for the decimal VectorizedColumnBatch,
> breaking these dependencies will improve things.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-11295) LLAP: clean up ORC dependencies on object pools
[ https://issues.apache.org/jira/browse/HIVE-11295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14632162#comment-14632162 ] Sergey Shelukhin commented on HIVE-11295: - Btw this officially makes Hive Enterprise software, cause it adds what amounts to a FactoryFactory. Maybe we should keep the interfaces, but remove the pools for objects that don't cross thread boundaries, and re-add them if we see problems again. Maybe the problems were only with objects that do cross thread boundaries. Maybe we also need pool for LlapDataBuffer but that would have to tie in with refcounting. > LLAP: clean up ORC dependencies on object pools > --- > > Key: HIVE-11295 > URL: https://issues.apache.org/jira/browse/HIVE-11295 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-11259.patch > > > Before there's storage handler module, we can clean some things up -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11295) LLAP: clean up ORC dependencies on object pools
[ https://issues.apache.org/jira/browse/HIVE-11295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14632136#comment-14632136 ] Sergey Shelukhin commented on HIVE-11295: - After the other one would be better. > LLAP: clean up ORC dependencies on object pools > --- > > Key: HIVE-11295 > URL: https://issues.apache.org/jira/browse/HIVE-11295 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-11259.patch > > > Before there's storage handler module, we can clean some things up -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10884) Enable some beeline tests and turn on HIVE-4239 by default
[
https://issues.apache.org/jira/browse/HIVE-10884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14632135#comment-14632135
]
Sergey Shelukhin commented on HIVE-10884:
-
Looks like it timed out without any output saved...
{noformat}
2015-07-16 10:21:04,139 INFO HostExecutor.executeTests:176 Starting isolated
execution on 54.80.208.185
2015-07-16 11:57:56,933 INFO LocalCommand.:35 Starting timeout 1h rsync
-vaPe "ssh -i /home/hiveptest/.ssh/hive-ptest-user-key" --timeout 600
hiveptest@107.22.49.60:/home/hiveptest/107.22.49.60-hiveptest-2/logs/
/data/hive-ptest/logs/PreCommit-HIVE-TRUNK-Build-4628/failed/TestBeeLineDriver
20
{noformat}
Would the above directory be preserved somewhere?
> Enable some beeline tests and turn on HIVE-4239 by default
> --
>
> Key: HIVE-10884
> URL: https://issues.apache.org/jira/browse/HIVE-10884
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-10884.01.patch, HIVE-10884.02.patch,
> HIVE-10884.03.patch, HIVE-10884.04.patch, HIVE-10884.05.patch,
> HIVE-10884.06.patch, HIVE-10884.07.patch, HIVE-10884.07.patch,
> HIVE-10884.patch
>
>
> See comments in HIVE-4239.
> Beeline tests with parallelism need to be enabled to turn compilation
> parallelism on by default.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-10799) Refactor the SearchArgumentFactory to remove the dependence on ExprNodeGenericFuncDesc
[ https://issues.apache.org/jira/browse/HIVE-10799?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Owen O'Malley updated HIVE-10799: - Attachment: HIVE-10799.patch Fixed a couple of problems. > Refactor the SearchArgumentFactory to remove the dependence on > ExprNodeGenericFuncDesc > -- > > Key: HIVE-10799 > URL: https://issues.apache.org/jira/browse/HIVE-10799 > Project: Hive > Issue Type: Sub-task >Reporter: Owen O'Malley >Assignee: Owen O'Malley > Attachments: HIVE-10799.patch, HIVE-10799.patch, HIVE-10799.patch, > HIVE-10799.patch > > > SearchArgumentFactory and SearchArgumentImpl are high level and shouldn't > depend on the internals of Hive's AST model. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11288) Avro SerDe InstanceCache returns incorrect schema
[
https://issues.apache.org/jira/browse/HIVE-11288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14632084#comment-14632084
]
Hive QA commented on HIVE-11288:
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12745875/HIVE-11288.2.patch
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 9226 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cbo_rp_join0
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/4642/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/4642/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-4642/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12745875 - PreCommit-HIVE-TRUNK-Build
> Avro SerDe InstanceCache returns incorrect schema
> -
>
> Key: HIVE-11288
> URL: https://issues.apache.org/jira/browse/HIVE-11288
> Project: Hive
> Issue Type: Bug
>Reporter: Greg Phillips
> Attachments: HIVE-11288.2.patch, HIVE-11288.patch
>
>
> To reproduce this error, take two fields in an avro schema document matching
> the following:
> "type" : { "type": "array", "items": [ "null", { "type": "map", "values": [
> "null", "string" ] } ] }
> "type" : { "type": "map", "values": [ "null" , { "type": "array", "items": [
> "null" , "string"] } ] }
> After creating two tables in hive with these schemas, the describe statement
> on each of them will only return the schema for the first one loaded. This
> is due to a hashCode() collision in the InstanceCache.
> A patch will be included in this ticket shortly which removes the hashCode
> call from the InstanceCache's internal HashMap, and instead provides the
> entire schema object.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-10953) Get partial stats instead of complete stats in some queries
[ https://issues.apache.org/jira/browse/HIVE-10953?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alan Gates updated HIVE-10953: -- Affects Version/s: hbase-metastore-branch > Get partial stats instead of complete stats in some queries > --- > > Key: HIVE-10953 > URL: https://issues.apache.org/jira/browse/HIVE-10953 > Project: Hive > Issue Type: Sub-task > Components: Metastore >Affects Versions: hbase-metastore-branch >Reporter: Daniel Dai >Assignee: Vaibhav Gumashta > Fix For: hbase-metastore-branch > > > In ppd_constant_where.q, the result is different than benchmark: > Result: > Statistics: Num rows: 0 Data size: 11624 Basic stats: PARTIAL Column stats: > NONE > Benchmark: > Statistics: Num rows: 1000 Data size: 10624 Basic stats: COMPLETE Column > stats: NONE > This might cause quite a few failures so we need to investigate it first. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-10950) Unit test against HBase Metastore
[ https://issues.apache.org/jira/browse/HIVE-10950?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alan Gates updated HIVE-10950: -- Affects Version/s: hbase-metastore-branch > Unit test against HBase Metastore > - > > Key: HIVE-10950 > URL: https://issues.apache.org/jira/browse/HIVE-10950 > Project: Hive > Issue Type: Sub-task > Components: Metastore >Affects Versions: hbase-metastore-branch >Reporter: Daniel Dai >Assignee: Daniel Dai > Fix For: hbase-metastore-branch > > Attachments: HIVE-10950-1.patch, HIVE-10950-2.patch > > > We need to run the entire Hive UT against HBase Metastore and make sure they > pass. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-10289) Support filter on non-first partition key and non-string partition key
[ https://issues.apache.org/jira/browse/HIVE-10289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alan Gates updated HIVE-10289: -- Affects Version/s: hbase-metastore-branch > Support filter on non-first partition key and non-string partition key > -- > > Key: HIVE-10289 > URL: https://issues.apache.org/jira/browse/HIVE-10289 > Project: Hive > Issue Type: Sub-task > Components: Metastore >Affects Versions: hbase-metastore-branch >Reporter: Daniel Dai >Assignee: Daniel Dai > Attachments: HIVE-10289.1.patch > > > Currently, partition filtering only handles the first partition key and the > type for this partition key must be string. In order to break this > limitation, several improvements are required: > 1. Change serialization format for partition key. Currently partition keys > are serialized into delimited string, which sorted on string order not with > regard to the actual type of the partition key. We use BinarySortableSerDe > for this purpose. > 2. For filter condition not on the initial partition keys, push it into HBase > RowFilter. RowFilter will deserialize the partition key and evaluate the > filter condition. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-10169) get metatool to work with hbase metastore
[ https://issues.apache.org/jira/browse/HIVE-10169?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alan Gates updated HIVE-10169: -- Affects Version/s: hbase-metastore-branch > get metatool to work with hbase metastore > - > > Key: HIVE-10169 > URL: https://issues.apache.org/jira/browse/HIVE-10169 > Project: Hive > Issue Type: Sub-task > Components: Metastore >Affects Versions: hbase-metastore-branch >Reporter: Thejas M Nair > > The metatool is used for enabling namenode HA, and it uses ObjectStore > directly. > There needs to be a way to support equivalent functionality with hbase > metastore. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HIVE-9905) Investigate ways to improve NDV calculations during stats aggregation [hbase-metastore branch]
[ https://issues.apache.org/jira/browse/HIVE-9905?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alan Gates resolved HIVE-9905. -- Resolution: Duplicate This isn't HBase metastore specific. Besides it looks like HIVE-9689 will do what we want, so closing this as a dup of that. > Investigate ways to improve NDV calculations during stats aggregation > [hbase-metastore branch] > -- > > Key: HIVE-9905 > URL: https://issues.apache.org/jira/browse/HIVE-9905 > Project: Hive > Issue Type: Sub-task > Components: Metastore >Reporter: Vaibhav Gumashta >Assignee: Vaibhav Gumashta > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9904) Perform stats aggregation in HBase co-processor [hbase-metastore branch]
[ https://issues.apache.org/jira/browse/HIVE-9904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alan Gates updated HIVE-9904: - Affects Version/s: hbase-metastore-branch > Perform stats aggregation in HBase co-processor [hbase-metastore branch] > - > > Key: HIVE-9904 > URL: https://issues.apache.org/jira/browse/HIVE-9904 > Project: Hive > Issue Type: Sub-task > Components: Metastore >Affects Versions: hbase-metastore-branch >Reporter: Vaibhav Gumashta >Assignee: Vaibhav Gumashta > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11210) Remove dependency on HiveConf from Orc reader & writer
[ https://issues.apache.org/jira/browse/HIVE-11210?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Owen O'Malley updated HIVE-11210: - Attachment: HIVE-11210.patch Sorry, I managed to get the wrong version of the patch. > Remove dependency on HiveConf from Orc reader & writer > -- > > Key: HIVE-11210 > URL: https://issues.apache.org/jira/browse/HIVE-11210 > Project: Hive > Issue Type: Sub-task >Reporter: Owen O'Malley >Assignee: Owen O'Malley > Attachments: HIVE-11210.patch, HIVE-11210.patch > > > Currently the ORC reader and writer get their default values from HiveConf. I > propose that we make the reader and writer have their own programatic > defaults and the OrcInputFormat and OrcOutputFormat can use the version in > HiveConf. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9578) Add support for getDatabases and alterDatabase calls [hbase-metastore branch]
[ https://issues.apache.org/jira/browse/HIVE-9578?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alan Gates updated HIVE-9578: - Affects Version/s: hbase-metastore-branch > Add support for getDatabases and alterDatabase calls [hbase-metastore branch] > - > > Key: HIVE-9578 > URL: https://issues.apache.org/jira/browse/HIVE-9578 > Project: Hive > Issue Type: Sub-task > Components: Metastore >Affects Versions: hbase-metastore-branch >Reporter: Alan Gates >Assignee: Alan Gates > Fix For: hbase-metastore-branch > > Attachments: HIVE-9578.2.patch, HIVE-9578.patch > > > The initial patch only supporting getting a single database, add database, > and drop database. Support needs to be added for alter database, getting all > the databases, and getting database names by pattern. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9453) Initial patch [hbase-metastore branch]
[ https://issues.apache.org/jira/browse/HIVE-9453?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alan Gates updated HIVE-9453: - Affects Version/s: hbase-metastore-branch > Initial patch [hbase-metastore branch] > -- > > Key: HIVE-9453 > URL: https://issues.apache.org/jira/browse/HIVE-9453 > Project: Hive > Issue Type: Sub-task > Components: Metastore >Affects Versions: hbase-metastore-branch >Reporter: Alan Gates >Assignee: Alan Gates > Fix For: hbase-metastore-branch > > Attachments: HIVE-9453-reviewcomments.pdf, HIVE-9453.2.patch, > HIVE-9453.3.patch, HIVE-9453.patch > > > This initial patch has several important features: > # HBaseStore, a new implementation of RawStore that stores the data in HBase. > # Subclasses of the thrift metastore objects to remove the massive > duplication of data where every partition contains a nearly identical storage > descriptor. > # Caches for catalog objects and statistics so that repeated metastore calls > don't result in repeated calls against HBase. > Currently this "works" to the point that "load table" and "select" work. I > have not tested any other statements, and I suspect most fail. There is no > security, no authorization, and a no a lot of other things. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11210) Remove dependency on HiveConf from Orc reader & writer
[ https://issues.apache.org/jira/browse/HIVE-11210?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Owen O'Malley updated HIVE-11210: - Attachment: (was: HIVE-11210.patch) > Remove dependency on HiveConf from Orc reader & writer > -- > > Key: HIVE-11210 > URL: https://issues.apache.org/jira/browse/HIVE-11210 > Project: Hive > Issue Type: Sub-task >Reporter: Owen O'Malley >Assignee: Owen O'Malley > Attachments: HIVE-11210.patch > > > Currently the ORC reader and writer get their default values from HiveConf. I > propose that we make the reader and writer have their own programatic > defaults and the OrcInputFormat and OrcOutputFormat can use the version in > HiveConf. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9452) Use HBase to store Hive metadata
[ https://issues.apache.org/jira/browse/HIVE-9452?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alan Gates updated HIVE-9452: - Affects Version/s: hbase-metastore-branch > Use HBase to store Hive metadata > > > Key: HIVE-9452 > URL: https://issues.apache.org/jira/browse/HIVE-9452 > Project: Hive > Issue Type: Improvement > Components: Metastore >Affects Versions: hbase-metastore-branch >Reporter: Alan Gates >Assignee: Alan Gates > Attachments: HBaseMetastoreApproach.pdf > > > This is an umbrella JIRA for a project to explore using HBase to store the > Hive data catalog (ie the metastore). This project has several goals: > # The current metastore implementation is slow when tables have thousands or > more partitions. With Tez and Spark engines we are pushing Hive to a point > where queries only take a few seconds to run. But planning the query can > take as long as running it. Much of this time is spent in metadata > operations. > # Due to scale limitations we have never allowed tasks to communicate > directly with the metastore. However, with the development of LLAP this > requirement will have to be relaxed. If we can relax this there are other > use cases that could benefit from this. > # Eating our own dogfood. Rather than using external systems to store our > metadata there are benefits to using other components in the Hadoop system. > The proposal is to create a new branch and work on the prototype there. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HIVE-11274) Support executing script file from hdfs in beeline
[
https://issues.apache.org/jira/browse/HIVE-11274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14632022#comment-14632022
]
Xuefu Zhang edited comment on HIVE-11274 at 7/17/15 10:38 PM:
--
[~jeffzhang] does Ferdinand's comment make sense to you? I'm closing this one
and feel free to reopen if otherwise.
was (Author: xuefuz):
[~jeffzhang] does Ferdinand's comments make sense to you? I'm closing this one
and feel free to reopen if otherwise.
> Support executing script file from hdfs in beeline
> --
>
> Key: HIVE-11274
> URL: https://issues.apache.org/jira/browse/HIVE-11274
> Project: Hive
> Issue Type: Improvement
>Reporter: Jeff Zhang
>Assignee: Ferdinand Xu
>
> It is supported for cli to execute hive script on hdfs, it should be
> straightforward to support it in beeline too.
> Beeline.java
> {code}
> private int executeFile(String fileName) {
> FileInputStream initStream = null;
> try {
> initStream = new FileInputStream(fileName);
> return execute(getConsoleReader(initStream), true);
> } catch (Throwable t) {
> handleException(t);
> return ERRNO_OTHER;
> } finally {
> IOUtils.closeStream(initStream);
> consoleReader = null;
> output(""); // dummy new line
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (HIVE-11274) Support executing script file from hdfs in beeline
[
https://issues.apache.org/jira/browse/HIVE-11274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14632022#comment-14632022
]
Xuefu Zhang edited comment on HIVE-11274 at 7/17/15 10:38 PM:
--
[~jeffzhang] does Ferdinand's comments make sense to you? I'm closing this one
and feel free to reopen if otherwise.
was (Author: xuefuz):
[~jeffzhang] doesn't Ferdinand's comments makes sense to you? I'm closing this
one and feel free to reopen if otherwise.
> Support executing script file from hdfs in beeline
> --
>
> Key: HIVE-11274
> URL: https://issues.apache.org/jira/browse/HIVE-11274
> Project: Hive
> Issue Type: Improvement
>Reporter: Jeff Zhang
>Assignee: Ferdinand Xu
>
> It is supported for cli to execute hive script on hdfs, it should be
> straightforward to support it in beeline too.
> Beeline.java
> {code}
> private int executeFile(String fileName) {
> FileInputStream initStream = null;
> try {
> initStream = new FileInputStream(fileName);
> return execute(getConsoleReader(initStream), true);
> } catch (Throwable t) {
> handleException(t);
> return ERRNO_OTHER;
> } finally {
> IOUtils.closeStream(initStream);
> consoleReader = null;
> output(""); // dummy new line
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-11294) Use HBase to cache aggregated stats
[ https://issues.apache.org/jira/browse/HIVE-11294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14632048#comment-14632048 ] ASF GitHub Bot commented on HIVE-11294: --- GitHub user alanfgates opened a pull request: https://github.com/apache/hive/pull/46 Hbase metastore HIVE-11294 Cache aggregated stats in hbase You can merge this pull request into a Git repository by running: $ git pull https://github.com/alanfgates/hive hbase-metastore Alternatively you can review and apply these changes as the patch at: https://github.com/apache/hive/pull/46.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #46 commit 58ca965ca58925e4846e14cf37d64638edb3e353 Author: Alan Gates Date: 2015-01-23T19:05:19Z Creating branch for hbase metastore development. git-svn-id: https://svn.apache.org/repos/asf/hive/branches/hbase-metastore@1654331 13f79535-47bb-0310-9956-ffa450edef68 commit e81704eeaf061e7152c63379b828debdfdfe9a8e Author: Alan Gates Date: 2015-02-04T20:00:49Z HIVE-9453 Initial patch [hbase-metastore branch] (Alan Gates, reviewed by Thejas Nair) git-svn-id: https://svn.apache.org/repos/asf/hive/branches/hbase-metastore@1657394 13f79535-47bb-0310-9956-ffa450edef68 commit 120047c308a011f17f94258779e4c47fcff191a5 Author: Alan Gates Date: 2015-02-10T21:41:19Z HIVE-9578 Add support for getDatabases and alterDatabase calls [hbase-metastore branch] git-svn-id: https://svn.apache.org/repos/asf/hive/branches/hbase-metastore@1658821 13f79535-47bb-0310-9956-ffa450edef68 commit c7dcf4a85d5cef4151d7b72115b4a1027af488ac Author: Alan Gates Date: 2015-02-10T22:47:53Z HIVE-9579 Support all get tables [hbase-metastore branch] git-svn-id: https://svn.apache.org/repos/asf/hive/branches/hbase-metastore@1658829 13f79535-47bb-0310-9956-ffa450edef68 commit cd82601f6e5c4aea5db72b74629b6aada23cf7c2 Author: Alan Gates Date: 2015-02-19T18:51:39Z HIVE-9606 Need a tool to export metadata from RDBMS based metastore into HBase [hbase-metastore branch] (Alan Gates, reviewed by Thejas Nair) git-svn-id: https://svn.apache.org/repos/asf/hive/branches/hbase-metastore@1660974 13f79535-47bb-0310-9956-ffa450edef68 commit 21ba0a93dcadcf8726ae7a18e81f758921552697 Author: Alan Gates Date: 2015-03-02T18:17:01Z HIVE-9641 Fill out remaining partition functions in HBaseStore (Alan Gates reviewed by Vaibhav Gumashta) git-svn-id: https://svn.apache.org/repos/asf/hive/branches/hbase-metastore@1663386 13f79535-47bb-0310-9956-ffa450edef68 commit bb0010caa8734af34ecac7b5b054afd95b49b0e0 Author: Alan Gates Date: 2015-03-05T03:46:28Z HIVE-9677 Implement privileges call in HBaseStore (Alan Gates) git-svn-id: https://svn.apache.org/repos/asf/hive/branches/hbase-metastore@1664220 13f79535-47bb-0310-9956-ffa450edef68 commit 408d467bffa0923714b8262427f1d1d72ba9932d Author: Alan Gates Date: 2015-03-05T04:09:31Z HIVE-9763 Remove M* classes from RawStore interface (Alan Gates) git-svn-id: https://svn.apache.org/repos/asf/hive/branches/hbase-metastore@1664226 13f79535-47bb-0310-9956-ffa450edef68 commit be575b84a8d5b4764550499c60bdccb7b0d1f82f Author: Alan Gates Date: 2015-03-05T04:35:53Z HIVE-9802 Refactor HBaseReadWrite to allow different implementations underneath, also includes code from HIVE-9783 Move serialization of objects in HBase to protocol buffers (Alan Gates) git-svn-id: https://svn.apache.org/repos/asf/hive/branches/hbase-metastore@1664227 13f79535-47bb-0310-9956-ffa450edef68 commit 646468fcdfd612524961751ba3575a26a1fb3fcb Author: Alan Gates Date: 2015-03-12T19:41:56Z HIVE-9885 HBase metastore protobufs need to check hasX() functions for values that are not auto set in serialization [hbase-metastore branch] (Alan Gates) git-svn-id: https://svn.apache.org/repos/asf/hive/branches/hbase-metastore@1666280 13f79535-47bb-0310-9956-ffa450edef68 commit d268b81bcd2fbaae6fdbfe2f980f644ce5e3d92d Author: Alan Gates Date: 2015-03-12T20:01:30Z HIVE-9874 Partition storage descriptors being set from table sd without copying [hbase-metastore branch] (Alan Gates) git-svn-id: https://svn.apache.org/repos/asf/hive/branches/hbase-metastore@1666285 13f79535-47bb-0310-9956-ffa450edef68 commit 24f224867b70bacc91a877242aee1b0b84efd127 Author: Alan Gates Date: 2015-03-17T15:50:31Z HIVE-9942 Implement functions methods in HBaseStore (Alan Gates) git-svn-id: https://svn.apache.org/repos/asf/hive/branches/hbase-metastore@1667338 13f79535-47bb-0310-9956-ffa450edef68 commit 35ac8b0257614523064c1b3647a91e123b78e999 Author: Alan Gates Date: 2015-03-18T17:41:50Z HIVE-10008 Ne
[jira] [Commented] (HIVE-11274) Support executing script file from hdfs in beeline
[
https://issues.apache.org/jira/browse/HIVE-11274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14632043#comment-14632043
]
Jeff Zhang commented on HIVE-11274:
---
Make sense.
> Support executing script file from hdfs in beeline
> --
>
> Key: HIVE-11274
> URL: https://issues.apache.org/jira/browse/HIVE-11274
> Project: Hive
> Issue Type: Improvement
>Reporter: Jeff Zhang
>Assignee: Ferdinand Xu
>
> It is supported for cli to execute hive script on hdfs, it should be
> straightforward to support it in beeline too.
> Beeline.java
> {code}
> private int executeFile(String fileName) {
> FileInputStream initStream = null;
> try {
> initStream = new FileInputStream(fileName);
> return execute(getConsoleReader(initStream), true);
> } catch (Throwable t) {
> handleException(t);
> return ERRNO_OTHER;
> } finally {
> IOUtils.closeStream(initStream);
> consoleReader = null;
> output(""); // dummy new line
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (HIVE-11274) Support executing script file from hdfs in beeline
[
https://issues.apache.org/jira/browse/HIVE-11274?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xuefu Zhang resolved HIVE-11274.
Resolution: Won't Fix
[~jeffzhang] doesn't Ferdinand's comments makes sense to you? I'm closing this
one and feel free to reopen if otherwise.
> Support executing script file from hdfs in beeline
> --
>
> Key: HIVE-11274
> URL: https://issues.apache.org/jira/browse/HIVE-11274
> Project: Hive
> Issue Type: Improvement
>Reporter: Jeff Zhang
>Assignee: Ferdinand Xu
>
> It is supported for cli to execute hive script on hdfs, it should be
> straightforward to support it in beeline too.
> Beeline.java
> {code}
> private int executeFile(String fileName) {
> FileInputStream initStream = null;
> try {
> initStream = new FileInputStream(fileName);
> return execute(getConsoleReader(initStream), true);
> } catch (Throwable t) {
> handleException(t);
> return ERRNO_OTHER;
> } finally {
> IOUtils.closeStream(initStream);
> consoleReader = null;
> output(""); // dummy new line
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-11299) Hive query planning allocating lots of memory and hitting OOMs
[ https://issues.apache.org/jira/browse/HIVE-11299?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mostafa Mokhtar updated HIVE-11299: --- Attachment: HIVE-11299.patch > Hive query planning allocating lots of memory and hitting OOMs > --- > > Key: HIVE-11299 > URL: https://issues.apache.org/jira/browse/HIVE-11299 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 0.14.0 >Reporter: Mostafa Mokhtar >Assignee: Mostafa Mokhtar > Attachments: HIVE-11299.patch > > > Plan generation for queries with lots of disjunct filters spends lots of time > compiling. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11299) Hive query planning allocating lots of memory and hitting OOMs
[ https://issues.apache.org/jira/browse/HIVE-11299?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mostafa Mokhtar updated HIVE-11299: --- Attachment: (was: HIVE-11299.patch) > Hive query planning allocating lots of memory and hitting OOMs > --- > > Key: HIVE-11299 > URL: https://issues.apache.org/jira/browse/HIVE-11299 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 0.14.0 >Reporter: Mostafa Mokhtar >Assignee: Mostafa Mokhtar > > Plan generation for queries with lots of disjunct filters spends lots of time > compiling. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11299) Hive query planning allocating lots of memory and hitting OOMs
[ https://issues.apache.org/jira/browse/HIVE-11299?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mostafa Mokhtar updated HIVE-11299: --- Attachment: HIVE-11299.patch > Hive query planning allocating lots of memory and hitting OOMs > --- > > Key: HIVE-11299 > URL: https://issues.apache.org/jira/browse/HIVE-11299 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 0.14.0 >Reporter: Mostafa Mokhtar >Assignee: Mostafa Mokhtar > Attachments: HIVE-11299.patch > > > Plan generation for queries with lots of disjunct filters spends lots of time > compiling. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11296) Merge from master to spark branch [Spark Branch]
[ https://issues.apache.org/jira/browse/HIVE-11296?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuefu Zhang updated HIVE-11296: --- Component/s: (was: spark-branch) Spark Summary: Merge from master to spark branch [Spark Branch] (was: Merge from master to spark branch) > Merge from master to spark branch [Spark Branch] > > > Key: HIVE-11296 > URL: https://issues.apache.org/jira/browse/HIVE-11296 > Project: Hive > Issue Type: Bug > Components: Spark >Reporter: Chao Sun >Assignee: Chao Sun > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11209) Clean up dependencies in HiveDecimalWritable
[ https://issues.apache.org/jira/browse/HIVE-11209?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Owen O'Malley updated HIVE-11209: - Attachment: HIVE-11209.patch Found and fixed a screwup in the code that I moved over. I also fixed some of the debugging code that got exercised when the unit tests failed by adding INTERVAL_YEAR_MONTH and INTERVAL_DAY_TIME support to the buildJSONString. > Clean up dependencies in HiveDecimalWritable > > > Key: HIVE-11209 > URL: https://issues.apache.org/jira/browse/HIVE-11209 > Project: Hive > Issue Type: Sub-task >Reporter: Owen O'Malley >Assignee: Owen O'Malley > Fix For: 2.0.0 > > Attachments: HIVE-11209.patch, HIVE-11209.patch, HIVE-11209.patch, > HIVE-11209.patch > > > Currently HiveDecimalWritable depends on: > * org.apache.hadoop.hive.serde2.ByteStream > * org.apache.hadoop.hive.serde2.lazybinary.LazyBinaryUtils > * org.apache.hadoop.hive.serde2.typeinfo.HiveDecimalUtils > since we need HiveDecimalWritable for the decimal VectorizedColumnBatch, > breaking these dependencies will improve things. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11294) Use HBase to cache aggregated stats
[
https://issues.apache.org/jira/browse/HIVE-11294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14631945#comment-14631945
]
Hive QA commented on HIVE-11294:
{color:red}Overall{color}: -1 no tests executed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12745874/HIVE-11294.patch
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/4641/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/4641/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-4641/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hive-ptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ [[ -n /usr/java/jdk1.7.0_45-cloudera ]]
+ export JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera
+ JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera
+ export
PATH=/usr/java/jdk1.7.0_45-cloudera/bin/:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.7.0_45-cloudera/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin
+
PATH=/usr/java/jdk1.7.0_45-cloudera/bin/:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.7.0_45-cloudera/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'M2_OPTS=-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost
-Dhttp.proxyPort=3128'
+ M2_OPTS='-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost
-Dhttp.proxyPort=3128'
+ cd /data/hive-ptest/working/
+ tee /data/hive-ptest/logs/PreCommit-HIVE-TRUNK-Build-4641/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ cd apache-github-source-source
+ git fetch origin
>From https://github.com/apache/hive
7fe23aa..8646c12 master -> origin/master
dddffe1..4221699 spark -> origin/spark
+ git reset --hard HEAD
HEAD is now at 7fe23aa HIVE-11282: CBO (Calcite Return Path): Inferring Hive
type char/varchar of length zero which is not allowed (Jesus Camacho Rodriguez,
reviewed by Ashutosh Chauhan)
+ git clean -f -d
+ git checkout master
Already on 'master'
Your branch is behind 'origin/master' by 2 commits, and can be fast-forwarded.
+ git reset --hard origin/master
HEAD is now at 8646c12 HIVE-11285 : ObjectInspector for partition columns in
FetchOperator in SMBJoin causes exception (Pengcheng Xiong via Ashutosh Chauhan)
+ git merge --ff-only origin/master
Already up-to-date.
+ git gc
+ patchCommandPath=/data/hive-ptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hive-ptest/working/scratch/build.patch
+ [[ -f /data/hive-ptest/working/scratch/build.patch ]]
+ chmod +x /data/hive-ptest/working/scratch/smart-apply-patch.sh
+ /data/hive-ptest/working/scratch/smart-apply-patch.sh
/data/hive-ptest/working/scratch/build.patch
The patch does not appear to apply with p0, p1, or p2
+ exit 1
'
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12745874 - PreCommit-HIVE-TRUNK-Build
> Use HBase to cache aggregated stats
> ---
>
> Key: HIVE-11294
> URL: https://issues.apache.org/jira/browse/HIVE-11294
> Project: Hive
> Issue Type: Improvement
> Components: Metastore
>Affects Versions: hbase-metastore-branch
>Reporter: Alan Gates
>Assignee: Alan Gates
> Fix For: hbase-metastore-branch
>
> Attachments: HIVE-11294.patch
>
>
> Currently stats are cached only in the memory of the client. Given that
> HBase can easily manage the scale of caching aggregated stats we should be
> using it to do so.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-11291) Avoid allocation storm while doing rule matching on operator/expression trees
[
https://issues.apache.org/jira/browse/HIVE-11291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14631937#comment-14631937
]
Hive QA commented on HIVE-11291:
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12745848/HIVE-11291.patch
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 9223 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cbo_rp_join0
org.apache.hive.hcatalog.hbase.TestPigHBaseStorageHandler.org.apache.hive.hcatalog.hbase.TestPigHBaseStorageHandler
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/4640/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/4640/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-4640/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12745848 - PreCommit-HIVE-TRUNK-Build
> Avoid allocation storm while doing rule matching on operator/expression trees
> -
>
> Key: HIVE-11291
> URL: https://issues.apache.org/jira/browse/HIVE-11291
> Project: Hive
> Issue Type: Improvement
>Reporter: Ashutosh Chauhan
>Assignee: Ashutosh Chauhan
> Attachments: HIVE-11291.patch
>
>
> RuleRegExMatch repeatedly allocates string while trying to find a matching
> pattern. This results in huge GC churn for large trees.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-11284) Fix cbo_rp_join0 failure on master
[ https://issues.apache.org/jira/browse/HIVE-11284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14631933#comment-14631933 ] Ashutosh Chauhan commented on HIVE-11284: - Cool. +1 > Fix cbo_rp_join0 failure on master > -- > > Key: HIVE-11284 > URL: https://issues.apache.org/jira/browse/HIVE-11284 > Project: Hive > Issue Type: Bug > Components: CBO, Query Planning >Affects Versions: 2.0.0 >Reporter: Ashutosh Chauhan >Assignee: Jesus Camacho Rodriguez > Attachments: HIVE-11284.patch > > > It first failed in this build : > http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/4621/ -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11284) Fix cbo_rp_join0 failure on master
[
https://issues.apache.org/jira/browse/HIVE-11284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14631924#comment-14631924
]
Jesus Camacho Rodriguez commented on HIVE-11284:
Sure.
==
Background:
When RT is on, HiveJoinToMultiJoinRule is responsible for creating MultiJoin
operators out of Join operators (if possible). But the plan might be the
following:
{noformat}
Join(...)
Project(...)
Join(...)
...
Filter(...)
...
{noformat}
Thus, first we need to pull up the Project operator(s) through Join(s) using
HiveJoinProjectTransposeRule, to obtain:
{noformat}
Project(...)
Join(...)
Join(...)
...
Filter(...)
...
{noformat}
Then, HiveJoinToMultiJoinRule will find both Join operators and merge them.
Finally, the Project operator(s) will be pushed down again throw the tree (this
is done by HiveRelTrimmer).
==
Problem:
HIVE-11223 enabled the return path for the cbo_rp_join0.q test.
At the same time, HIVE-11252 changed the application of the rules.
- Before HIVE-11252, HiveJoinToMultiJoinRule was applied first, then
HiveJoinProjectTransposeRule + HiveJoinToMultiJoinRule. This was merging the
operators correctly, because no projects were found in between the Joins.
- After HIVE-11252, only HiveJoinToMultiJoinRule + HiveJoinProjectTransposeRule
are applied (this is correct). The problem is that 1)
HiveJoinProjectTransposeRule.LEFT and HiveJoinProjectTransposeRule.RIGHT were
not enabled, and 2) even if they were enabled, they were not working as
expected (more on this below). That's why the test failed.
==
Solution:
HiveJoinProjectTransposeRule.LEFT and HiveJoinProjectTransposeRule.RIGHT get
enabled in the patch. In addition, JoinProjectTransposeRule was only working on
inner joins, but we need it to work also on outer joins in Hive. That's why I
had to bring so much code from the Calcite superclass
(JoinProjectTransposeRule) to the Hive subclass (HiveJoinProjectTransposeRule)
to solve the issue. I plan to create a Calcite patch so we can use
JoinProjectTransposeRule with all kind of Joins.
> Fix cbo_rp_join0 failure on master
> --
>
> Key: HIVE-11284
> URL: https://issues.apache.org/jira/browse/HIVE-11284
> Project: Hive
> Issue Type: Bug
> Components: CBO, Query Planning
>Affects Versions: 2.0.0
>Reporter: Ashutosh Chauhan
>Assignee: Jesus Camacho Rodriguez
> Attachments: HIVE-11284.patch
>
>
> It first failed in this build :
> http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/4621/
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-8678) Pig fails to correctly load DATE fields using HCatalog
[
https://issues.apache.org/jira/browse/HIVE-8678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14631906#comment-14631906
]
Sushanth Sowmyan commented on HIVE-8678:
Something seems weird here - looking at the code, it looks like the current
code, where it simply casts to Date should be the right way to do this, since
it should have called .getPrimitiveJavaObject() on the PrimitiveObjectInspector
to get this object, and DateObjectInspector.getPrimitiveJavaObject() should
have returned a Date. However, clearly, from your stack trace, you're getting a
string. I'll dig into this and update as I find more.
> Pig fails to correctly load DATE fields using HCatalog
> --
>
> Key: HIVE-8678
> URL: https://issues.apache.org/jira/browse/HIVE-8678
> Project: Hive
> Issue Type: Bug
> Components: HCatalog
>Affects Versions: 0.13.1
>Reporter: Michael McLellan
>Assignee: Sushanth Sowmyan
>
> Using:
> Hadoop 2.5.0-cdh5.2.0
> Pig 0.12.0-cdh5.2.0
> Hive 0.13.1-cdh5.2.0
> When using pig -useHCatalog to load a Hive table that has a DATE field, when
> trying to DUMP the field, the following error occurs:
> {code}
> 2014-10-30 22:58:05,469 [main] ERROR
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.Launcher -
> org.apache.pig.backend.executionengine.ExecException: ERROR 6018: Error
> converting read value to tuple
> at
> org.apache.hive.hcatalog.pig.HCatBaseLoader.getNext(HCatBaseLoader.java:76)
> at org.apache.hive.hcatalog.pig.HCatLoader.getNext(HCatLoader.java:58)
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigRecordReader.nextKeyValue(PigRecordReader.java:211)
> at
> org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(MapTask.java:553)
> at
> org.apache.hadoop.mapreduce.task.MapContextImpl.nextKeyValue(MapContextImpl.java:80)
> at
> org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.nextKeyValue(WrappedMapper.java:91)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:784)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)
> Caused by: java.lang.ClassCastException: java.lang.String cannot be cast to
> java.sql.Date
> at
> org.apache.hive.hcatalog.pig.PigHCatUtil.extractPigObject(PigHCatUtil.java:420)
> at
> org.apache.hive.hcatalog.pig.PigHCatUtil.transformToTuple(PigHCatUtil.java:457)
> at
> org.apache.hive.hcatalog.pig.PigHCatUtil.transformToTuple(PigHCatUtil.java:375)
> at
> org.apache.hive.hcatalog.pig.HCatBaseLoader.getNext(HCatBaseLoader.java:64)
> 2014-10-30 22:58:05,469 [main] ERROR
> org.apache.pig.tools.pigstats.SimplePigStats - ERROR 6018: Error converting
> read value to tuple
> {code}
> It seems to be occuring here:
> https://github.com/apache/hive/blob/trunk/hcatalog/hcatalog-pig-adapter/src/main/java/org/apache/hive/hcatalog/pig/PigHCatUtil.java#L433
> and that it should be:
> {code}Date d = Date.valueOf(o);{code}
> instead of
> {code}Date d = (Date) o;{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (HIVE-8678) Pig fails to correctly load DATE fields using HCatalog
[
https://issues.apache.org/jira/browse/HIVE-8678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sushanth Sowmyan reassigned HIVE-8678:
--
Assignee: Sushanth Sowmyan
> Pig fails to correctly load DATE fields using HCatalog
> --
>
> Key: HIVE-8678
> URL: https://issues.apache.org/jira/browse/HIVE-8678
> Project: Hive
> Issue Type: Bug
> Components: HCatalog
>Affects Versions: 0.13.1
>Reporter: Michael McLellan
>Assignee: Sushanth Sowmyan
>
> Using:
> Hadoop 2.5.0-cdh5.2.0
> Pig 0.12.0-cdh5.2.0
> Hive 0.13.1-cdh5.2.0
> When using pig -useHCatalog to load a Hive table that has a DATE field, when
> trying to DUMP the field, the following error occurs:
> {code}
> 2014-10-30 22:58:05,469 [main] ERROR
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.Launcher -
> org.apache.pig.backend.executionengine.ExecException: ERROR 6018: Error
> converting read value to tuple
> at
> org.apache.hive.hcatalog.pig.HCatBaseLoader.getNext(HCatBaseLoader.java:76)
> at org.apache.hive.hcatalog.pig.HCatLoader.getNext(HCatLoader.java:58)
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigRecordReader.nextKeyValue(PigRecordReader.java:211)
> at
> org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(MapTask.java:553)
> at
> org.apache.hadoop.mapreduce.task.MapContextImpl.nextKeyValue(MapContextImpl.java:80)
> at
> org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.nextKeyValue(WrappedMapper.java:91)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:784)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)
> Caused by: java.lang.ClassCastException: java.lang.String cannot be cast to
> java.sql.Date
> at
> org.apache.hive.hcatalog.pig.PigHCatUtil.extractPigObject(PigHCatUtil.java:420)
> at
> org.apache.hive.hcatalog.pig.PigHCatUtil.transformToTuple(PigHCatUtil.java:457)
> at
> org.apache.hive.hcatalog.pig.PigHCatUtil.transformToTuple(PigHCatUtil.java:375)
> at
> org.apache.hive.hcatalog.pig.HCatBaseLoader.getNext(HCatBaseLoader.java:64)
> 2014-10-30 22:58:05,469 [main] ERROR
> org.apache.pig.tools.pigstats.SimplePigStats - ERROR 6018: Error converting
> read value to tuple
> {code}
> It seems to be occuring here:
> https://github.com/apache/hive/blob/trunk/hcatalog/hcatalog-pig-adapter/src/main/java/org/apache/hive/hcatalog/pig/PigHCatUtil.java#L433
> and that it should be:
> {code}Date d = Date.valueOf(o);{code}
> instead of
> {code}Date d = (Date) o;{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-11077) Add support in parser and wire up to txn manager
[ https://issues.apache.org/jira/browse/HIVE-11077?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14631904#comment-14631904 ] Eugene Koifman commented on HIVE-11077: --- This is not quite ready for review. Hopefully by Monday. > Add support in parser and wire up to txn manager > > > Key: HIVE-11077 > URL: https://issues.apache.org/jira/browse/HIVE-11077 > Project: Hive > Issue Type: Sub-task > Components: SQL, Transactions >Affects Versions: 1.2.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman > Attachments: HIVE-11077.3.patch, HIVE-11077.5.patch, HIVE-11077.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11295) LLAP: clean up ORC dependencies on object pools
[ https://issues.apache.org/jira/browse/HIVE-11295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-11295: Attachment: HIVE-11259.patch this patch depends on the other patch. [~prasanth_j] can you review? > LLAP: clean up ORC dependencies on object pools > --- > > Key: HIVE-11295 > URL: https://issues.apache.org/jira/browse/HIVE-11295 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-11259.patch > > > Before there's storage handler module, we can clean some things up -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11281) Memoize toStringTree() for ASTNode
[ https://issues.apache.org/jira/browse/HIVE-11281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14631874#comment-14631874 ] Gunther Hagleitner commented on HIVE-11281: --- How many places do we have that transform nodes? Can we make the nodes immutable? > Memoize toStringTree() for ASTNode > -- > > Key: HIVE-11281 > URL: https://issues.apache.org/jira/browse/HIVE-11281 > Project: Hive > Issue Type: Bug > Components: Query Planning >Affects Versions: 1.0.0, 1.2.0, 1.1.0 >Reporter: Ashutosh Chauhan >Assignee: Ashutosh Chauhan > Attachments: HIVE-11281.2.patch, HIVE-11281.3.patch, HIVE-11281.patch > > > Currently it defaults to BaseTree::toStringTree() which always computes it > recursively. For very large trees it becomes expensive. On one particular > test query compilation time went down from 55 seconds to 2 seconds. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11284) Fix cbo_rp_join0 failure on master
[ https://issues.apache.org/jira/browse/HIVE-11284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14631813#comment-14631813 ] Ashutosh Chauhan commented on HIVE-11284: - Can you briefly explain what was the issue and how is patch fixing it? > Fix cbo_rp_join0 failure on master > -- > > Key: HIVE-11284 > URL: https://issues.apache.org/jira/browse/HIVE-11284 > Project: Hive > Issue Type: Bug > Components: CBO, Query Planning >Affects Versions: 2.0.0 >Reporter: Ashutosh Chauhan >Assignee: Jesus Camacho Rodriguez > Attachments: HIVE-11284.patch > > > It first failed in this build : > http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/4621/ -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11077) Add support in parser and wire up to txn manager
[ https://issues.apache.org/jira/browse/HIVE-11077?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14631809#comment-14631809 ] Alan Gates commented on HIVE-11077: --- This is a big patch. Could you create a review board entry? > Add support in parser and wire up to txn manager > > > Key: HIVE-11077 > URL: https://issues.apache.org/jira/browse/HIVE-11077 > Project: Hive > Issue Type: Sub-task > Components: SQL, Transactions >Affects Versions: 1.2.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman > Attachments: HIVE-11077.3.patch, HIVE-11077.5.patch, HIVE-11077.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11210) Remove dependency on HiveConf from Orc reader & writer
[
https://issues.apache.org/jira/browse/HIVE-11210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14631808#comment-14631808
]
Hive QA commented on HIVE-11210:
{color:red}Overall{color}: -1 no tests executed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12745845/HIVE-11210.patch
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/4639/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/4639/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-4639/
Messages:
{noformat}
This message was trimmed, see log for full details
[INFO]
[INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) @
spark-client ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 1 resource
[INFO] Copying 3 resources
[INFO]
[INFO] --- maven-antrun-plugin:1.7:run (setup-test-dirs) @ spark-client ---
[INFO] Executing tasks
main:
[mkdir] Created dir:
/data/hive-ptest/working/apache-github-source-source/spark-client/target/tmp
[mkdir] Created dir:
/data/hive-ptest/working/apache-github-source-source/spark-client/target/warehouse
[mkdir] Created dir:
/data/hive-ptest/working/apache-github-source-source/spark-client/target/tmp/conf
[copy] Copying 11 files to
/data/hive-ptest/working/apache-github-source-source/spark-client/target/tmp/conf
[INFO] Executed tasks
[INFO]
[INFO] --- maven-compiler-plugin:3.1:testCompile (default-testCompile) @
spark-client ---
[INFO] Compiling 5 source files to
/data/hive-ptest/working/apache-github-source-source/spark-client/target/test-classes
[INFO]
[INFO] --- maven-dependency-plugin:2.8:copy (copy-guava-14) @ spark-client ---
[INFO] Configured Artifact: com.google.guava:guava:14.0.1:jar
[INFO] Copying guava-14.0.1.jar to
/data/hive-ptest/working/apache-github-source-source/spark-client/target/dependency/guava-14.0.1.jar
[INFO]
[INFO] --- maven-surefire-plugin:2.16:test (default-test) @ spark-client ---
[INFO] Tests are skipped.
[INFO]
[INFO] --- maven-jar-plugin:2.2:jar (default-jar) @ spark-client ---
[INFO] Building jar:
/data/hive-ptest/working/apache-github-source-source/spark-client/target/spark-client-2.0.0-SNAPSHOT.jar
[INFO]
[INFO] --- maven-site-plugin:3.3:attach-descriptor (attach-descriptor) @
spark-client ---
[INFO]
[INFO] --- maven-install-plugin:2.4:install (default-install) @ spark-client ---
[INFO] Installing
/data/hive-ptest/working/apache-github-source-source/spark-client/target/spark-client-2.0.0-SNAPSHOT.jar
to
/home/hiveptest/.m2/repository/org/apache/hive/spark-client/2.0.0-SNAPSHOT/spark-client-2.0.0-SNAPSHOT.jar
[INFO] Installing
/data/hive-ptest/working/apache-github-source-source/spark-client/pom.xml to
/home/hiveptest/.m2/repository/org/apache/hive/spark-client/2.0.0-SNAPSHOT/spark-client-2.0.0-SNAPSHOT.pom
[INFO]
[INFO]
[INFO] Building Hive Query Language 2.0.0-SNAPSHOT
[INFO]
[INFO]
[INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ hive-exec ---
[INFO] Deleting /data/hive-ptest/working/apache-github-source-source/ql/target
[INFO] Deleting /data/hive-ptest/working/apache-github-source-source/ql
(includes = [datanucleus.log, derby.log], excludes = [])
[INFO]
[INFO] --- maven-enforcer-plugin:1.3.1:enforce (enforce-no-snapshots) @
hive-exec ---
[INFO]
[INFO] --- maven-antrun-plugin:1.7:run (generate-sources) @ hive-exec ---
[INFO] Executing tasks
main:
[mkdir] Created dir:
/data/hive-ptest/working/apache-github-source-source/ql/target/generated-sources/java/org/apache/hadoop/hive/ql/exec/vector/expressions/gen
[mkdir] Created dir:
/data/hive-ptest/working/apache-github-source-source/ql/target/generated-sources/java/org/apache/hadoop/hive/ql/exec/vector/expressions/aggregates/gen
[mkdir] Created dir:
/data/hive-ptest/working/apache-github-source-source/ql/target/generated-test-sources/java/org/apache/hadoop/hive/ql/exec/vector/expressions/gen
Generating vector expression code
Generating vector expression test code
[INFO] Executed tasks
[INFO]
[INFO] --- build-helper-maven-plugin:1.8:add-source (add-source) @ hive-exec ---
[INFO] Source directory:
/data/hive-ptest/working/apache-github-source-source/ql/src/gen/protobuf/gen-java
added.
[INFO] Source directory:
/data/hive-ptest/working/apache-github-source-source/ql/src/gen/thrift/gen-javabean
added.
[INFO] Source directory:
/data/hive-ptest/working/apache-github-source-source/ql/target/generated-sources/java
added.
[INFO]
[INFO] --- antlr3-maven-plugin:3.4:antlr (default) @ hive-exec ---
[IN
[jira] [Commented] (HIVE-11284) Fix cbo_rp_join0 failure on master
[
https://issues.apache.org/jira/browse/HIVE-11284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14631802#comment-14631802
]
Hive QA commented on HIVE-11284:
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12745846/HIVE-11284.patch
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 9225 tests executed
*Failed tests:*
{noformat}
org.apache.hive.hcatalog.streaming.TestStreaming.testTransactionBatchCommit_Json
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/4638/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/4638/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-4638/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12745846 - PreCommit-HIVE-TRUNK-Build
> Fix cbo_rp_join0 failure on master
> --
>
> Key: HIVE-11284
> URL: https://issues.apache.org/jira/browse/HIVE-11284
> Project: Hive
> Issue Type: Bug
> Components: CBO, Query Planning
>Affects Versions: 2.0.0
>Reporter: Ashutosh Chauhan
>Assignee: Jesus Camacho Rodriguez
> Attachments: HIVE-11284.patch
>
>
> It first failed in this build :
> http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/4621/
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-11288) Avro SerDe InstanceCache returns incorrect schema
[
https://issues.apache.org/jira/browse/HIVE-11288?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Greg Phillips updated HIVE-11288:
-
Attachment: HIVE-11288.2.patch
Implemented hashCode and equals methods for test class 'Foo' to allow it to
work properly within a HashMap.
> Avro SerDe InstanceCache returns incorrect schema
> -
>
> Key: HIVE-11288
> URL: https://issues.apache.org/jira/browse/HIVE-11288
> Project: Hive
> Issue Type: Bug
>Reporter: Greg Phillips
> Attachments: HIVE-11288.2.patch, HIVE-11288.patch
>
>
> To reproduce this error, take two fields in an avro schema document matching
> the following:
> "type" : { "type": "array", "items": [ "null", { "type": "map", "values": [
> "null", "string" ] } ] }
> "type" : { "type": "map", "values": [ "null" , { "type": "array", "items": [
> "null" , "string"] } ] }
> After creating two tables in hive with these schemas, the describe statement
> on each of them will only return the schema for the first one loaded. This
> is due to a hashCode() collision in the InstanceCache.
> A patch will be included in this ticket shortly which removes the hashCode
> call from the InstanceCache's internal HashMap, and instead provides the
> entire schema object.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-11294) Use HBase to cache aggregated stats
[ https://issues.apache.org/jira/browse/HIVE-11294?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alan Gates updated HIVE-11294: -- Attachment: HIVE-11294.patch This patch adds caching of the aggregates stats to HBase. It also fundamentally changes how cached entries are matched. Now only exact matches are taken, rather than partial matches as was done in the past. The key for entries in the cache is an md5 sum of the dbname, tablename, and sorted list of partition names. This allows for reasonable key sizes and fast lookup. A limited number of entries are still kept in memory (10K by default) for a limited time (1 min by default). This is to reduce back and forth to HBase. Entries in HBase are kept in the cache for 1 week or until a partition's stats are updated or the partition is dropped. Determining when an aggregate needs to be dropped is not straight forward. Since the key is an md5 sum we cannot determine from the key if an entry contains the partition that was updated or dropped. To deal with this each entry also contains a bloom filter of all the partition names. When a partition is updated or dropped it is added a queue. Every 5 seconds a separate thread takes all of the entries from the queue and does a full scan of the cache. It uses the bloom filters to determine if any of the entries in the queue match one of the partitions in the aggregate. If so, it drops the aggregate entry. Given that this is done by a bloom filter there will be some false positives (entries that get dropped that shouldn't) but the error rate was chosen to be very low (0.1%). This makes the bloom filter larger but the motivation in choosing the bloom filter was to minimize processing time rather than to save space. All of this means there will be lag between when a partition is dropped or updated and when the aggregate is dropped. It will be < 5 seconds if the drop was done on the same HS2 instance, or <65 seconds if done on another instance. Given that these are statistics I think that's acceptable. Ideally we would not drop an aggregate as soon as a single partition is dropped or updated. Instead we should be tracking the number of invalidated partitions and only drop the aggregate once it reaches a threshold like 5%. Doing this would require implementing the invalidation logic as a co-processor rather than as a filter, which is why I didn't do it this way to begin with. > Use HBase to cache aggregated stats > --- > > Key: HIVE-11294 > URL: https://issues.apache.org/jira/browse/HIVE-11294 > Project: Hive > Issue Type: Improvement > Components: Metastore >Affects Versions: hbase-metastore-branch >Reporter: Alan Gates >Assignee: Alan Gates > Attachments: HIVE-11294.patch > > > Currently stats are cached only in the memory of the client. Given that > HBase can easily manage the scale of caching aggregated stats we should be > using it to do so. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11284) Fix cbo_rp_join0 failure on master
[ https://issues.apache.org/jira/browse/HIVE-11284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14631778#comment-14631778 ] Jesus Camacho Rodriguez commented on HIVE-11284: [~ashutoshc], actually it was the interaction of HIVE-11252 and HIVE-11223, and I didn't realize as the QA run on both at the same time. This patch solves the issue. > Fix cbo_rp_join0 failure on master > -- > > Key: HIVE-11284 > URL: https://issues.apache.org/jira/browse/HIVE-11284 > Project: Hive > Issue Type: Bug > Components: CBO, Query Planning >Affects Versions: 2.0.0 >Reporter: Ashutosh Chauhan >Assignee: Jesus Camacho Rodriguez > Attachments: HIVE-11284.patch > > > It first failed in this build : > http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/4621/ -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11233) Support of binary sortable key and values for HBaseStorageHandler
[ https://issues.apache.org/jira/browse/HIVE-11233?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Svetozar Ivanov updated HIVE-11233: --- Attachment: HIVE-11233-branch-1.2.patch > Support of binary sortable key and values for HBaseStorageHandler > - > > Key: HIVE-11233 > URL: https://issues.apache.org/jira/browse/HIVE-11233 > Project: Hive > Issue Type: New Feature > Components: HBase Handler >Affects Versions: 1.2.1 >Reporter: Svetozar Ivanov >Assignee: Svetozar Ivanov > Labels: Binary, Hbase, Numeric, Phoenix, Sortable > Attachments: HIVE-11233-branch-1.2.patch, HIVE-11233.1-1.2.patch, > HIVE-11233.1.patch, HIVE-11233.patch > > > Currently HBaseStorageHandler doesn't provide mechanism for storage of binary > sortable key and values. It is necessary when given HBase table is used for > persistence by Apache Hive and Apache Phoenix. In that way all byte arrays > read or written by Hive will be compatible with binary sortable format used > by Phoenix. > It turns out the major difference is in all numeric data types accordingly > officially provided documentation - > https://phoenix.apache.org/language/datatypes.html. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11233) Support of binary sortable key and values for HBaseStorageHandler
[ https://issues.apache.org/jira/browse/HIVE-11233?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Svetozar Ivanov updated HIVE-11233: --- Attachment: HIVE-11233.1.patch > Support of binary sortable key and values for HBaseStorageHandler > - > > Key: HIVE-11233 > URL: https://issues.apache.org/jira/browse/HIVE-11233 > Project: Hive > Issue Type: New Feature > Components: HBase Handler >Affects Versions: 1.2.1 >Reporter: Svetozar Ivanov >Assignee: Svetozar Ivanov > Labels: Binary, Hbase, Numeric, Phoenix, Sortable > Attachments: HIVE-11233.1-1.2.patch, HIVE-11233.1.patch, > HIVE-11233.patch > > > Currently HBaseStorageHandler doesn't provide mechanism for storage of binary > sortable key and values. It is necessary when given HBase table is used for > persistence by Apache Hive and Apache Phoenix. In that way all byte arrays > read or written by Hive will be compatible with binary sortable format used > by Phoenix. > It turns out the major difference is in all numeric data types accordingly > officially provided documentation - > https://phoenix.apache.org/language/datatypes.html. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11233) Support of binary sortable key and values for HBaseStorageHandler
[ https://issues.apache.org/jira/browse/HIVE-11233?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Svetozar Ivanov updated HIVE-11233: --- Attachment: HIVE-11233.1-1.2.patch > Support of binary sortable key and values for HBaseStorageHandler > - > > Key: HIVE-11233 > URL: https://issues.apache.org/jira/browse/HIVE-11233 > Project: Hive > Issue Type: New Feature > Components: HBase Handler >Affects Versions: 1.2.1 >Reporter: Svetozar Ivanov >Assignee: Svetozar Ivanov > Labels: Binary, Hbase, Numeric, Phoenix, Sortable > Attachments: HIVE-11233.1-1.2.patch, HIVE-11233.patch > > > Currently HBaseStorageHandler doesn't provide mechanism for storage of binary > sortable key and values. It is necessary when given HBase table is used for > persistence by Apache Hive and Apache Phoenix. In that way all byte arrays > read or written by Hive will be compatible with binary sortable format used > by Phoenix. > It turns out the major difference is in all numeric data types accordingly > officially provided documentation - > https://phoenix.apache.org/language/datatypes.html. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11288) Avro SerDe InstanceCache returns incorrect schema
[
https://issues.apache.org/jira/browse/HIVE-11288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14631774#comment-14631774
]
Greg Phillips commented on HIVE-11288:
--
The test is wrong. It is relying on the java Object.equals() which requires
the exact same object to return true.
This is similar to the original problem with the InstanceCache. It is relying
on the hashCode to determine equality which breaks the hashCode contract. I
will provide the patch for the unit test shortly.
> Avro SerDe InstanceCache returns incorrect schema
> -
>
> Key: HIVE-11288
> URL: https://issues.apache.org/jira/browse/HIVE-11288
> Project: Hive
> Issue Type: Bug
>Reporter: Greg Phillips
> Attachments: HIVE-11288.patch
>
>
> To reproduce this error, take two fields in an avro schema document matching
> the following:
> "type" : { "type": "array", "items": [ "null", { "type": "map", "values": [
> "null", "string" ] } ] }
> "type" : { "type": "map", "values": [ "null" , { "type": "array", "items": [
> "null" , "string"] } ] }
> After creating two tables in hive with these schemas, the describe statement
> on each of them will only return the schema for the first one loaded. This
> is due to a hashCode() collision in the InstanceCache.
> A patch will be included in this ticket shortly which removes the hashCode
> call from the InstanceCache's internal HashMap, and instead provides the
> entire schema object.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-11233) Support of binary sortable key and values for HBaseStorageHandler
[ https://issues.apache.org/jira/browse/HIVE-11233?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Svetozar Ivanov updated HIVE-11233: --- Attachment: (was: HIVE-11233.1-branch-1.2.patch) > Support of binary sortable key and values for HBaseStorageHandler > - > > Key: HIVE-11233 > URL: https://issues.apache.org/jira/browse/HIVE-11233 > Project: Hive > Issue Type: New Feature > Components: HBase Handler >Affects Versions: 1.2.1 >Reporter: Svetozar Ivanov >Assignee: Svetozar Ivanov > Labels: Binary, Hbase, Numeric, Phoenix, Sortable > Attachments: HIVE-11233.patch > > > Currently HBaseStorageHandler doesn't provide mechanism for storage of binary > sortable key and values. It is necessary when given HBase table is used for > persistence by Apache Hive and Apache Phoenix. In that way all byte arrays > read or written by Hive will be compatible with binary sortable format used > by Phoenix. > It turns out the major difference is in all numeric data types accordingly > officially provided documentation - > https://phoenix.apache.org/language/datatypes.html. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11233) Support of binary sortable key and values for HBaseStorageHandler
[ https://issues.apache.org/jira/browse/HIVE-11233?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Svetozar Ivanov updated HIVE-11233: --- Attachment: (was: HIVE-11233.2-branch-1.2.patch) > Support of binary sortable key and values for HBaseStorageHandler > - > > Key: HIVE-11233 > URL: https://issues.apache.org/jira/browse/HIVE-11233 > Project: Hive > Issue Type: New Feature > Components: HBase Handler >Affects Versions: 1.2.1 >Reporter: Svetozar Ivanov >Assignee: Svetozar Ivanov > Labels: Binary, Hbase, Numeric, Phoenix, Sortable > Attachments: HIVE-11233.patch > > > Currently HBaseStorageHandler doesn't provide mechanism for storage of binary > sortable key and values. It is necessary when given HBase table is used for > persistence by Apache Hive and Apache Phoenix. In that way all byte arrays > read or written by Hive will be compatible with binary sortable format used > by Phoenix. > It turns out the major difference is in all numeric data types accordingly > officially provided documentation - > https://phoenix.apache.org/language/datatypes.html. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11282) CBO (Calcite Return Path): Inferring Hive type char/varchar of length zero which is not allowed
[ https://issues.apache.org/jira/browse/HIVE-11282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14631755#comment-14631755 ] Jesus Camacho Rodriguez commented on HIVE-11282: Fail is unrelated (HIVE-11284). > CBO (Calcite Return Path): Inferring Hive type char/varchar of length zero > which is not allowed > --- > > Key: HIVE-11282 > URL: https://issues.apache.org/jira/browse/HIVE-11282 > Project: Hive > Issue Type: Sub-task > Components: CBO >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > Fix For: 2.0.0 > > Attachments: HIVE-11282.patch > > > When RT is on, we try to infer the Hive type from the Calcite type for the > value '’ e.g. in udf3.q, and we end up with char (length=0) as a result. The > min length of char/varchar in Hive is 1, thus an Exception is thrown. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11269) Intermittent bug with kryo serializers
[ https://issues.apache.org/jira/browse/HIVE-11269?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14631680#comment-14631680 ] Soundararajan Velu commented on HIVE-11269: --- Cool the later two issues were due to non serializable fields in the UDF and UDAF, made them transient and it seems to proceed. the first problem still remains, I will provide you the sample table and data to reproduce this problem, > Intermittent bug with kryo serializers > -- > > Key: HIVE-11269 > URL: https://issues.apache.org/jira/browse/HIVE-11269 > Project: Hive > Issue Type: Bug >Affects Versions: 1.3.0 >Reporter: Soundararajan Velu > > getting kryo serialization exception when running large queries, this works > fine on hive 0.14, > and works fine on 1.3.0 with the following flags, > set hive.plan.serialization.format=kryo; > set hive.exec.parallel=false; > set hive.limit.optimize.enable=false; > set hive.optimize.metadataonly=false; > set hive.optimize.reducededuplication=false; > set hive.optimize.sort.dynamic.partition=false; > set hive.stats.fetch.partition.stats=false; > set hive.vectorized.execution.enabled=false; > set hive.vectorized.execution.reduce.enabled=false; > set hive.cbo.enable=false; > set hive.compute.query.using.stats=false; > set hive.multigroupby.singlereducer=false; > set hive.optimize.ppd=false; > set hive.optimize.skewjoin.compiletime=false; > set hive.optimize.skewjoin=false; > set hive.optimize.union.remove=false; > set hive.mapred.mode=nonstrict; > set hive.auto.convert.join.noconditionaltask=false; > set hive.optimize.sort.dynamic.partition=false; > set hive.rpc.query.plan=true; > StackTrace > colExprMap (org.apache.hadoop.hive.ql.exec.SelectOperator) > childOperators (org.apache.hadoop.hive.ql.exec.JoinOperator) > reducer (org.apache.hadoop.hive.ql.plan.ReduceWork) > at > org.apache.hadoop.hive.ql.exec.Utilities.getBaseWork(Utilities.java:462) > at > org.apache.hadoop.hive.ql.exec.Utilities.getReduceWork(Utilities.java:309) > at > org.apache.hadoop.hive.ql.exec.mr.ExecReducer.configure(ExecReducer.java:112) > ... 14 more > Caused by: org.apache.hive.com.esotericsoftware.kryo.KryoException: > java.lang.NullPointerException > Serialization trace: > chidren (org.apache.hadoop.hive.ql.plan.ExprNodeGenericFuncDesc) > chidren (org.apache.hadoop.hive.ql.plan.ExprNodeGenericFuncDesc) > chidren (org.apache.hadoop.hive.ql.plan.ExprNodeGenericFuncDesc) > chidren (org.apache.hadoop.hive.ql.plan.ExprNodeGenericFuncDesc) > chidren (org.apache.hadoop.hive.ql.plan.ExprNodeGenericFuncDesc) > chidren (org.apache.hadoop.hive.ql.plan.ExprNodeGenericFuncDesc) > chidren (org.apache.hadoop.hive.ql.plan.ExprNodeGenericFuncDesc) > chidren (org.apache.hadoop.hive.ql.plan.ExprNodeGenericFuncDesc) > chidren (org.apache.hadoop.hive.ql.plan.ExprNodeGenericFuncDesc) > chidren (org.apache.hadoop.hive.ql.plan.ExprNodeGenericFuncDesc) > chidren (org.apache.hadoop.hive.ql.plan.ExprNodeGenericFuncDesc) > chidren (org.apache.hadoop.hive.ql.plan.ExprNodeGenericFuncDesc) > chidren (org.apache.hadoop.hive.ql.plan.ExprNodeGenericFuncDesc) > colExprMap (org.apache.hadoop.hive.ql.exec.SelectOperator) > childOperators (org.apache.hadoop.hive.ql.exec.JoinOperator) > reducer (org.apache.hadoop.hive.ql.plan.ReduceWork) > at > org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:125) > at > org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:507) > at > org.apache.hive.com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:776) > at > org.apache.hive.com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:112) > at > org.apache.hive.com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:18) > at > org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:694) > at > org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106) > at > org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:507) > at > org.apache.hive.com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:776) > at > org.apache.hive.com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:112) > at > org.apache.hive.com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:18) > at > org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:694) > at > org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106) > at > org.apache.hive.com.esotericsoftware.kryo.serializers.FieldS
[jira] [Commented] (HIVE-11291) Avoid allocation storm while doing rule matching on operator/expression trees
[ https://issues.apache.org/jira/browse/HIVE-11291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14631675#comment-14631675 ] Hari Sankar Sivarama Subramaniyan commented on HIVE-11291: -- [~ashutoshc] The patch looks good to me, +1 pending tests. I believe this idea has to be incorporated as part of HIVE-11141 Thanks Hari > Avoid allocation storm while doing rule matching on operator/expression trees > - > > Key: HIVE-11291 > URL: https://issues.apache.org/jira/browse/HIVE-11291 > Project: Hive > Issue Type: Improvement >Reporter: Ashutosh Chauhan >Assignee: Ashutosh Chauhan > Attachments: HIVE-11291.patch > > > RuleRegExMatch repeatedly allocates string while trying to find a matching > pattern. This results in huge GC churn for large trees. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11291) Avoid allocation storm while doing rule matching on operator/expression trees
[ https://issues.apache.org/jira/browse/HIVE-11291?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-11291: Attachment: HIVE-11291.patch In most cases, we are trying to do exact matches, regex matching is not required. [~hsubramaniyan] Can you take a look? > Avoid allocation storm while doing rule matching on operator/expression trees > - > > Key: HIVE-11291 > URL: https://issues.apache.org/jira/browse/HIVE-11291 > Project: Hive > Issue Type: Improvement >Reporter: Ashutosh Chauhan >Assignee: Ashutosh Chauhan > Attachments: HIVE-11291.patch > > > RuleRegExMatch repeatedly allocates string while trying to find a matching > pattern. This results in huge GC churn for large trees. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11284) Fix cbo_rp_join0 failure on master
[ https://issues.apache.org/jira/browse/HIVE-11284?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-11284: --- Attachment: HIVE-11284.patch > Fix cbo_rp_join0 failure on master > -- > > Key: HIVE-11284 > URL: https://issues.apache.org/jira/browse/HIVE-11284 > Project: Hive > Issue Type: Bug > Components: CBO, Query Planning >Affects Versions: 2.0.0 >Reporter: Ashutosh Chauhan >Assignee: Jesus Camacho Rodriguez > Attachments: HIVE-11284.patch > > > It first failed in this build : > http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/4621/ -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11269) Intermittent bug with kryo serializers
[ https://issues.apache.org/jira/browse/HIVE-11269?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14631644#comment-14631644 ] Xuefu Zhang commented on HIVE-11269: You seemed hitting a new problem. In either case it's best to provide a repro case. Otherwise, it's hard to find the problem and so the fix. > Intermittent bug with kryo serializers > -- > > Key: HIVE-11269 > URL: https://issues.apache.org/jira/browse/HIVE-11269 > Project: Hive > Issue Type: Bug >Affects Versions: 1.3.0 >Reporter: Soundararajan Velu > > getting kryo serialization exception when running large queries, this works > fine on hive 0.14, > and works fine on 1.3.0 with the following flags, > set hive.plan.serialization.format=kryo; > set hive.exec.parallel=false; > set hive.limit.optimize.enable=false; > set hive.optimize.metadataonly=false; > set hive.optimize.reducededuplication=false; > set hive.optimize.sort.dynamic.partition=false; > set hive.stats.fetch.partition.stats=false; > set hive.vectorized.execution.enabled=false; > set hive.vectorized.execution.reduce.enabled=false; > set hive.cbo.enable=false; > set hive.compute.query.using.stats=false; > set hive.multigroupby.singlereducer=false; > set hive.optimize.ppd=false; > set hive.optimize.skewjoin.compiletime=false; > set hive.optimize.skewjoin=false; > set hive.optimize.union.remove=false; > set hive.mapred.mode=nonstrict; > set hive.auto.convert.join.noconditionaltask=false; > set hive.optimize.sort.dynamic.partition=false; > set hive.rpc.query.plan=true; > StackTrace > colExprMap (org.apache.hadoop.hive.ql.exec.SelectOperator) > childOperators (org.apache.hadoop.hive.ql.exec.JoinOperator) > reducer (org.apache.hadoop.hive.ql.plan.ReduceWork) > at > org.apache.hadoop.hive.ql.exec.Utilities.getBaseWork(Utilities.java:462) > at > org.apache.hadoop.hive.ql.exec.Utilities.getReduceWork(Utilities.java:309) > at > org.apache.hadoop.hive.ql.exec.mr.ExecReducer.configure(ExecReducer.java:112) > ... 14 more > Caused by: org.apache.hive.com.esotericsoftware.kryo.KryoException: > java.lang.NullPointerException > Serialization trace: > chidren (org.apache.hadoop.hive.ql.plan.ExprNodeGenericFuncDesc) > chidren (org.apache.hadoop.hive.ql.plan.ExprNodeGenericFuncDesc) > chidren (org.apache.hadoop.hive.ql.plan.ExprNodeGenericFuncDesc) > chidren (org.apache.hadoop.hive.ql.plan.ExprNodeGenericFuncDesc) > chidren (org.apache.hadoop.hive.ql.plan.ExprNodeGenericFuncDesc) > chidren (org.apache.hadoop.hive.ql.plan.ExprNodeGenericFuncDesc) > chidren (org.apache.hadoop.hive.ql.plan.ExprNodeGenericFuncDesc) > chidren (org.apache.hadoop.hive.ql.plan.ExprNodeGenericFuncDesc) > chidren (org.apache.hadoop.hive.ql.plan.ExprNodeGenericFuncDesc) > chidren (org.apache.hadoop.hive.ql.plan.ExprNodeGenericFuncDesc) > chidren (org.apache.hadoop.hive.ql.plan.ExprNodeGenericFuncDesc) > chidren (org.apache.hadoop.hive.ql.plan.ExprNodeGenericFuncDesc) > chidren (org.apache.hadoop.hive.ql.plan.ExprNodeGenericFuncDesc) > colExprMap (org.apache.hadoop.hive.ql.exec.SelectOperator) > childOperators (org.apache.hadoop.hive.ql.exec.JoinOperator) > reducer (org.apache.hadoop.hive.ql.plan.ReduceWork) > at > org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:125) > at > org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:507) > at > org.apache.hive.com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:776) > at > org.apache.hive.com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:112) > at > org.apache.hive.com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:18) > at > org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:694) > at > org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106) > at > org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:507) > at > org.apache.hive.com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:776) > at > org.apache.hive.com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:112) > at > org.apache.hive.com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:18) > at > org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:694) > at > org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106) > at > org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:507) > at > org.apache.hive.com.esotericsoftware.kryo.Kryo.readCla
[jira] [Updated] (HIVE-11210) Remove dependency on HiveConf from Orc reader & writer
[ https://issues.apache.org/jira/browse/HIVE-11210?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Owen O'Malley updated HIVE-11210: - Attachment: HIVE-11210.patch Removed the use of Configuration.getDouble, which messes up the jenkins build. > Remove dependency on HiveConf from Orc reader & writer > -- > > Key: HIVE-11210 > URL: https://issues.apache.org/jira/browse/HIVE-11210 > Project: Hive > Issue Type: Sub-task >Reporter: Owen O'Malley >Assignee: Owen O'Malley > Attachments: HIVE-11210.patch, HIVE-11210.patch > > > Currently the ORC reader and writer get their default values from HiveConf. I > propose that we make the reader and writer have their own programatic > defaults and the OrcInputFormat and OrcOutputFormat can use the version in > HiveConf. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11210) Remove dependency on HiveConf from Orc reader & writer
[ https://issues.apache.org/jira/browse/HIVE-11210?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Owen O'Malley updated HIVE-11210: - Attachment: (was: HIVE-11210.patch) > Remove dependency on HiveConf from Orc reader & writer > -- > > Key: HIVE-11210 > URL: https://issues.apache.org/jira/browse/HIVE-11210 > Project: Hive > Issue Type: Sub-task >Reporter: Owen O'Malley >Assignee: Owen O'Malley > Attachments: HIVE-11210.patch > > > Currently the ORC reader and writer get their default values from HiveConf. I > propose that we make the reader and writer have their own programatic > defaults and the OrcInputFormat and OrcOutputFormat can use the version in > HiveConf. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11210) Remove dependency on HiveConf from Orc reader & writer
[ https://issues.apache.org/jira/browse/HIVE-11210?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Owen O'Malley updated HIVE-11210: - Attachment: (was: HIVE-11210.patch) > Remove dependency on HiveConf from Orc reader & writer > -- > > Key: HIVE-11210 > URL: https://issues.apache.org/jira/browse/HIVE-11210 > Project: Hive > Issue Type: Sub-task >Reporter: Owen O'Malley >Assignee: Owen O'Malley > Attachments: HIVE-11210.patch, HIVE-11210.patch > > > Currently the ORC reader and writer get their default values from HiveConf. I > propose that we make the reader and writer have their own programatic > defaults and the OrcInputFormat and OrcOutputFormat can use the version in > HiveConf. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11280) Support executing script file from hdfs in new CLI [Beeline-CLI branch]
[ https://issues.apache.org/jira/browse/HIVE-11280?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14631637#comment-14631637 ] Xuefu Zhang commented on HIVE-11280: +1 > Support executing script file from hdfs in new CLI [Beeline-CLI branch] > --- > > Key: HIVE-11280 > URL: https://issues.apache.org/jira/browse/HIVE-11280 > Project: Hive > Issue Type: Sub-task > Components: Beeline, CLI >Reporter: Ferdinand Xu >Assignee: Ferdinand Xu > Attachments: HIVE-11280.1-beeline-cli.patch > > > In HIVE-7136, old CLI is able to read hive scripts from any of the supported > file systems in hadoop eco-system. We need to support it in new CLI as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-10171) Create a storage-api module
[ https://issues.apache.org/jira/browse/HIVE-10171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Owen O'Malley updated HIVE-10171: - Fix Version/s: 2.0.0 > Create a storage-api module > --- > > Key: HIVE-10171 > URL: https://issues.apache.org/jira/browse/HIVE-10171 > Project: Hive > Issue Type: Bug >Reporter: Owen O'Malley >Assignee: Owen O'Malley > Fix For: 2.0.0 > > > To support high performance file formats, I'd like to propose that we move > the minimal set of classes that are required to integrate with Hive in to a > new module named "storage-api". This module will include VectorizedRowBatch, > the various ColumnVector classes, and the SARG classes. It will form the > start of an API that high performance storage formats can use to integrate > with Hive. Both ORC and Parquet can use the new API to support vectorization > and SARGs without performance destroying shims. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HIVE-11253) Move SearchArgument and VectorizedRowBatch classes to storage-api.
[ https://issues.apache.org/jira/browse/HIVE-11253?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Owen O'Malley reassigned HIVE-11253: Assignee: Owen O'Malley > Move SearchArgument and VectorizedRowBatch classes to storage-api. > -- > > Key: HIVE-11253 > URL: https://issues.apache.org/jira/browse/HIVE-11253 > Project: Hive > Issue Type: Sub-task >Reporter: Owen O'Malley >Assignee: Owen O'Malley > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11269) Intermittent bug with kryo serializers
[ https://issues.apache.org/jira/browse/HIVE-11269?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14631576#comment-14631576 ] Soundararajan Velu commented on HIVE-11269: --- during runtime I get the below exception when I run the same query on MR, on Tez I get it on the console itself as described in the previous commment, Launching Job 10 out of 11 Number of reduce tasks not specified. Estimated from input data size: 300 In order to change the average load for a reducer (in bytes): set hive.exec.reducers.bytes.per.reducer= In order to limit the maximum number of reducers: set hive.exec.reducers.max= In order to set a constant number of reducers: set mapreduce.job.reduces= java.lang.RuntimeException: Error caching reduce.xml: org.apache.hive.com.esotericsoftware.kryo.KryoException: java.util.ConcurrentModificationException Serialization trace: classes (sun.misc.Launcher$AppClassLoader) classloader (java.security.ProtectionDomain) context (java.security.AccessControlContext) acc (java.net.URLClassLoader) contextClassLoader (java.lang.Thread) thread (java.util.concurrent.ThreadPoolExecutor$Worker) workers (java.util.concurrent.ThreadPoolExecutor) heartBeatThreadPool (net.rubyeye.xmemcached.impl.MemcachedHandler) handler (net.rubyeye.xmemcached.impl.MemcachedConnector) connector (net.rubyeye.xmemcached.XMemcachedClient) memClient (org.apache.hadoop.hive.ql.udf.generic.MecGet) genericUDF (org.apache.hadoop.hive.ql.plan.ExprNodeGenericFuncDesc) colExprMap (org.apache.hadoop.hive.ql.exec.SelectOperator) childOperators (org.apache.hadoop.hive.ql.exec.JoinOperator) reducer (org.apache.hadoop.hive.ql.plan.ReduceWork) at org.apache.hadoop.hive.ql.exec.Utilities.setBaseWork(Utilities.java:748) at org.apache.hadoop.hive.ql.exec.Utilities.setReduceWork(Utilities.java:687) at org.apache.hadoop.hive.ql.exec.Utilities.setMapRedWork(Utilities.java:678) at org.apache.hadoop.hive.ql.exec.mr.ExecDriver.execute(ExecDriver.java:375) at org.apache.hadoop.hive.ql.exec.mr.MapRedTask.execute(MapRedTask.java:137) at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:160) at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:89) at org.apache.hadoop.hive.ql.exec.TaskRunner.run(TaskRunner.java:75) Caused by: org.apache.hive.com.esotericsoftware.kryo.KryoException: java.util.ConcurrentModificationException Serialization trace: classes (sun.misc.Launcher$AppClassLoader) classloader (java.security.ProtectionDomain) context (java.security.AccessControlContext) acc (java.net.URLClassLoader) contextClassLoader (java.lang.Thread) thread (java.util.concurrent.ThreadPoolExecutor$Worker) workers (java.util.concurrent.ThreadPoolExecutor) heartBeatThreadPool (net.rubyeye.xmemcached.impl.MemcachedHandler) handler (net.rubyeye.xmemcached.impl.MemcachedConnector) connector (net.rubyeye.xmemcached.XMemcachedClient) memClient (org.apache.hadoop.hive.ql.udf.generic.MecGet) genericUDF (org.apache.hadoop.hive.ql.plan.ExprNodeGenericFuncDesc) colExprMap (org.apache.hadoop.hive.ql.exec.SelectOperator) childOperators (org.apache.hadoop.hive.ql.exec.JoinOperator) reducer (org.apache.hadoop.hive.ql.plan.ReduceWork) at org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.write(ObjectField.java:82) at org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:474) at org.apache.hive.com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:538) at org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.write(ObjectField.java:61) at org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:474) at org.apache.hive.com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:614) at org.apache.hive.com.esotericsoftware.kryo.serializers.DefaultArraySerializers$ObjectArraySerializer.write(DefaultArraySerializers.java:337) at org.apache.hive.com.esotericsoftware.kryo.serializers.DefaultArraySerializers$ObjectArraySerializer.write(DefaultArraySerializers.java:276) at org.apache.hive.com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:538) at org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.write(ObjectField.java:61) at org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:474) at org.apache.hive.com.esotericsoftware.kryo.Kryo.writeObjectOrNull(Kryo.java:592) at org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.write(ObjectField.java:68) at org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:474) at org.apache.hive.com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:538) at org.apache.hive.com.esotericsoftware.kryo.ser
[jira] [Commented] (HIVE-11254) Process result sets returned by a stored procedure
[
https://issues.apache.org/jira/browse/HIVE-11254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14631546#comment-14631546
]
Hive QA commented on HIVE-11254:
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12745815/HIVE-11254.2.patch
{color:red}ERROR:{color} -1 due to 8 failed/errored test(s), 9227 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cbo_rp_join0
org.apache.hive.hcatalog.streaming.TestStreaming.testTransactionBatchCommit_Json
org.apache.hive.hplsql.TestHplsqlDb.testCreateProcedureReturnCursor
org.apache.hive.hplsql.TestHplsqlDb.testCreateProcedureReturnCursor2
org.apache.hive.hplsql.TestHplsqlLocal.testException2
org.apache.hive.hplsql.TestHplsqlLocal.testException3
org.apache.hive.hplsql.TestHplsqlLocal.testException4
org.apache.hive.hplsql.TestHplsqlLocal.testException5
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/4637/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/4637/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-4637/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 8 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12745815 - PreCommit-HIVE-TRUNK-Build
> Process result sets returned by a stored procedure
> --
>
> Key: HIVE-11254
> URL: https://issues.apache.org/jira/browse/HIVE-11254
> Project: Hive
> Issue Type: Improvement
> Components: hpl/sql
>Reporter: Dmitry Tolpeko
>Assignee: Dmitry Tolpeko
> Attachments: HIVE-11254.1.patch, HIVE-11254.2.patch
>
>
> Stored procedure can return one or more result sets. A caller should be able
> to process them.
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-11269) Intermittent bug with kryo serializers
[ https://issues.apache.org/jira/browse/HIVE-11269?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14631534#comment-14631534 ] Soundararajan Velu commented on HIVE-11269: --- Thanks Xuefu, I cloned 1.3.0 release 2 weeks back, was this fixed post that, if possible can you please provide the issue id that fixes this bug, Now I am getting the below exception while using custom udaf and udfs, Caused by: java.lang.Exception: XMLEncoder: discarding statement ArrayList.add(FileSinkOperator); ... 67 more Caused by: java.lang.RuntimeException: Cannot serialize object at org.apache.hadoop.hive.ql.exec.Utilities$2.exceptionThrown(Utilities.java:1034) at java.beans.DefaultPersistenceDelegate.initBean(DefaultPersistenceDelegate.java:259) at java.beans.DefaultPersistenceDelegate.initialize(DefaultPersistenceDelegate.java:403) at java.beans.PersistenceDelegate.writeObject(PersistenceDelegate.java:118) at java.beans.Encoder.writeObject(Encoder.java:74) at java.beans.XMLEncoder.writeObject(XMLEncoder.java:327) at java.beans.Encoder.writeExpression(Encoder.java:330) at java.beans.XMLEncoder.writeExpression(XMLEncoder.java:454) at java.beans.PersistenceDelegate.writeObject(PersistenceDelegate.java:115) at java.beans.Encoder.writeObject(Encoder.java:74) at java.beans.XMLEncoder.writeObject(XMLEncoder.java:327) at java.beans.Encoder.writeObject1(Encoder.java:258) at java.beans.Encoder.cloneStatement(Encoder.java:271) at java.beans.Encoder.writeStatement(Encoder.java:301) at java.beans.XMLEncoder.writeStatement(XMLEncoder.java:400) ... 66 more Caused by: java.lang.RuntimeException: Cannot serialize object at org.apache.hadoop.hive.ql.exec.Utilities$2.exceptionThrown(Utilities.java:1034) at java.beans.DefaultPersistenceDelegate.initBean(DefaultPersistenceDelegate.java:259) at java.beans.DefaultPersistenceDelegate.initialize(DefaultPersistenceDelegate.java:403) at java.beans.PersistenceDelegate.writeObject(PersistenceDelegate.java:118) at java.beans.Encoder.writeObject(Encoder.java:74) at java.beans.XMLEncoder.writeObject(XMLEncoder.java:327) at java.beans.Encoder.writeExpression(Encoder.java:330) at java.beans.XMLEncoder.writeExpression(XMLEncoder.java:454) at java.beans.PersistenceDelegate.writeObject(PersistenceDelegate.java:115) at java.beans.Encoder.writeObject(Encoder.java:74) at java.beans.XMLEncoder.writeObject(XMLEncoder.java:327) at java.beans.Encoder.writeExpression(Encoder.java:330) at java.beans.XMLEncoder.writeExpression(XMLEncoder.java:454) at java.beans.DefaultPersistenceDelegate.doProperty(DefaultPersistenceDelegate.java:194) at java.beans.DefaultPersistenceDelegate.initBean(DefaultPersistenceDelegate.java:256) ... 79 more Caused by: java.lang.RuntimeException: Cannot serialize object at org.apache.hadoop.hive.ql.exec.Utilities$2.exceptionThrown(Utilities.java:1034) at java.beans.Encoder.getValue(Encoder.java:108) at java.beans.Encoder.get(Encoder.java:252) at java.beans.PersistenceDelegate.writeObject(PersistenceDelegate.java:112) at java.beans.Encoder.writeObject(Encoder.java:74) at java.beans.XMLEncoder.writeObject(XMLEncoder.java:327) at java.beans.Encoder.writeExpression(Encoder.java:330) at java.beans.XMLEncoder.writeExpression(XMLEncoder.java:454) at java.beans.PersistenceDelegate.writeObject(PersistenceDelegate.java:115) at java.beans.Encoder.writeObject(Encoder.java:74) at java.beans.XMLEncoder.writeObject(XMLEncoder.java:327) at java.beans.Encoder.writeExpression(Encoder.java:330) at java.beans.XMLEncoder.writeExpression(XMLEncoder.java:454) at java.beans.DefaultPersistenceDelegate.doProperty(DefaultPersistenceDelegate.java:194) at java.beans.DefaultPersistenceDelegate.initBean(DefaultPersistenceDelegate.java:256) ... 92 more Caused by: java.lang.InstantiationException: org.apache.hadoop.hive.ql.plan.OpTraits at java.lang.Class.newInstance(Class.java:359) at sun.reflect.GeneratedMethodAccessor41.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at sun.reflect.misc.Trampoline.invoke(MethodUtil.java:75) at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at sun.reflect.misc.MethodUtil.invoke(MethodUtil.java:279) at java.beans.Statement.invokeInternal(Statement.java:292)
[jira] [Commented] (HIVE-11288) Avro SerDe InstanceCache returns incorrect schema
[
https://issues.apache.org/jira/browse/HIVE-11288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14631508#comment-14631508
]
Chaoyu Tang commented on HIVE-11288:
[~gphillips] Could you take a look at the failed test
org.apache.hadoop.hive.serde2.avro.TestInstanceCache.instanceCachesOnlyCreateOneInstance?
is it related to patch?
> Avro SerDe InstanceCache returns incorrect schema
> -
>
> Key: HIVE-11288
> URL: https://issues.apache.org/jira/browse/HIVE-11288
> Project: Hive
> Issue Type: Bug
>Reporter: Greg Phillips
> Attachments: HIVE-11288.patch
>
>
> To reproduce this error, take two fields in an avro schema document matching
> the following:
> "type" : { "type": "array", "items": [ "null", { "type": "map", "values": [
> "null", "string" ] } ] }
> "type" : { "type": "map", "values": [ "null" , { "type": "array", "items": [
> "null" , "string"] } ] }
> After creating two tables in hive with these schemas, the describe statement
> on each of them will only return the schema for the first one loaded. This
> is due to a hashCode() collision in the InstanceCache.
> A patch will be included in this ticket shortly which removes the hashCode
> call from the InstanceCache's internal HashMap, and instead provides the
> entire schema object.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (HIVE-11273) LLAP: Register for finishable state change notifications when adding a task instead of when scheduling it
[ https://issues.apache.org/jira/browse/HIVE-11273?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth resolved HIVE-11273. --- Resolution: Fixed > LLAP: Register for finishable state change notifications when adding a task > instead of when scheduling it > - > > Key: HIVE-11273 > URL: https://issues.apache.org/jira/browse/HIVE-11273 > Project: Hive > Issue Type: Sub-task >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Fix For: llap > > Attachments: HIVE-11273.1.txt > > > Registering when trying to execute is far too late. The task won't be > considered for execution (queue may not be re-oredered) without the > notification coming in. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HIVE-11263) LLAP: TaskExecutorService state is not cleaned up
[
https://issues.apache.org/jira/browse/HIVE-11263?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siddharth Seth resolved HIVE-11263.
---
Resolution: Fixed
Fix Version/s: llap
> LLAP: TaskExecutorService state is not cleaned up
> -
>
> Key: HIVE-11263
> URL: https://issues.apache.org/jira/browse/HIVE-11263
> Project: Hive
> Issue Type: Sub-task
>Reporter: Sergey Shelukhin
>Assignee: Siddharth Seth
> Fix For: llap
>
> Attachments: HIVE-11263.1.txt, HIVE-11263.2.txt
>
>
> See TaskExecutorService::getExecutorsStatus, this is used to report on
> queue/etc. status in JMX.
> Currently, it reports 100s of bogus tasks in queue:
> {noformat}
> "ExecutorsStatus" : [ "attempt_1435700346116_1888_1_04_000205_22
> (sershe_20150714174105_0d013941-1f0e-4f74-9387-a2f29279a185:3/Map 1, in
> queue)", "attempt_1435700346116_1889_1_05_000101_1
> (sershe_20150714174104_b0b9f300-667e-4370-bb01-f9cb7da331e0:4/Map 1, in
> queue)", "attempt_1435700346116_1889_1_05_000191_3
> (sershe_20150714174104_b0b9f300-667e-4370-bb01-f9cb7da331e0:4/Map 1, in
> queue)", "attempt_1435700346116_1887_7_00_000202_3
> (sershe_20150714174737_bea682d1-fa0f-4281-a1cb-439d85bb2016:22/Map 5, in
> queue)", "attempt_1435700346116_1886_1_04_73_15
> (sershe_20150714174108_f9483d76-8fd9-4f82-96ee-17231b6f9b2c:1/Reducer 2, in
> queue)", "attempt_1435700346116_1887_8_04_000166_15
> (sershe_20150714174900_710d7d69-3d66-45e9-865b-cd0f87bb0d98:27/Map 1, in
> queue)", "attempt_1435700346116_1888_11_05_000140_3
> (sershe_20150714174903_50359459-5342-4d1b-852c-622a3fa92a27:28/Map 3, in
> queue)", "attempt_1435700346116_1886_1_04_42_29
> (sershe_20150714174108_f9483d76-8fd9-4f82-96ee-17231b6f9b2c:1/Reducer 2, in
> queue)", "attempt_1435700346116_1888_2_03_000169_12
> (sershe_20150714174310_97ce1d4b-8029-4ef6-a823-46e29f09718a:5/Map 1, in
> queue)", "attempt_1435700346116_1887_1_04_000197_18
> (sershe_20150714174107_8fcfe954-4eeb-46e5-bad5-42a47327b26c:2/Map 1, in
> queue)", "attempt_1435700346116_1887_1_04_000218_21
> (sershe_20150714174107_8fcfe954-4eeb-46e5-bad5-42a47327b26c:2/Map 1, in
> queue)", "attempt_1435700346116_1886_7_09_84_1
> (sershe_20150714174841_462b9bdb-c017-47c2-9fa7-7edfbfc09e60:24/Map 1, in
> queue)", "attempt_1435700346116_1887_5_04_78_0
> (sershe_20150714174509_9a5cd476-b3c8-4679-af8e-1188922713a2:14/Map 3, in
> queue)", "attempt_1435700346116_1887_7_04_000162_6
> (sershe_20150714174737_bea682d1-fa0f-4281-a1cb-439d85bb2016:22/Map 3, in
> queue)", "attempt_1435700346116_1887_7_04_000180_0
> (sershe_20150714174737_bea682d1-fa0f-4281-a1cb-439d85bb2016:22/Map 3, in
> queue)", "attempt_1435700346116_1886_3_04_000144_0
> (sershe_20150714174435_fe3077dd-a97f-4582-995b-5f723170b02f:12/Reducer 2, in
> queue)", "attempt_1435700346116_1887_5_00_000153_1
> (sershe_20150714174509_9a5cd476-b3c8-4679-af8e-1188922713a2:14/Map 5, in
> queue)", "attempt_1435700346116_1887_7_04_000141_7
> (sershe_20150714174737_bea682d1-fa0f-4281-a1cb-439d85bb2016:22/Map 3, in
> queue)", "attempt_1435700346116_1887_1_04_24_7
> (sershe_20150714174107_8fcfe954-4eeb-46e5-bad5-42a47327b26c:2/Map 1, in
> queue)", "attempt_1435700346116_1887_5_04_000130_1
> (sershe_20150714174509_9a5cd476-b3c8-4679-af8e-1188922713a2:14/Map 3, in
> queue)", "attempt_1435700346116_1888_1_04_000200_1
> (sershe_20150714174105_0d013941-1f0e-4f74-9387-a2f29279a185:3/Map 1, in
> queue)", "attempt_1435700346116_1886_15_04_000180_0
> (sershe_20150714175411_bda950b7-8aa5-417f-84f6-dd646247dca8:43/Map 1, in
> queue)", "attempt_1435700346116_1887_7_00_000205_1
> (sershe_20150714174737_bea682d1-fa0f-4281-a1cb-439d85bb2016:22/Map 5, in
> queue)", "attempt_1435700346116_1888_4_04_000183_4
> (sershe_20150714174407_f0924540-f69f-45c2-831a-9d2d1f66a124:10/Map 1, in
> queue)", "attempt_1435700346116_1887_1_04_81_6
> (sershe_20150714174107_8fcfe954-4eeb-46e5-bad5-42a47327b26c:2/Map 1, in
> queue)", "attempt_1435700346116_1888_1_04_80_4
> (sershe_20150714174105_0d013941-1f0e-4f74-9387-a2f29279a185:3/Map 1, in
> queue)", "attempt_1435700346116_1887_7_04_05_3
> (sershe_20150714174737_bea682d1-fa0f-4281-a1cb-439d85bb2016:22/Map 3, in
> queue)", "attempt_1435700346116_1887_7_00_000169_2
> (sershe_20150714174737_bea682d1-fa0f-4281-a1cb-439d85bb2016:22/Map 5, in
> queue)", "attempt_1435700346116_1888_8_04_37_2
> (sershe_20150714174731_261f2d52-8c47-4db6-8f17-8098efe144a2:20/Reducer 3, in
> queue)", "attempt_1435700346116_1887_9_00_96_6
> (sershe_20150714175015_cc1b6647-8479-4c5f-918c-00935bff7232:30/Map 5, in
> queue)", "attempt_1435700346116_1888_11_01_01_2
> (sershe_20150714174903_50359459-5342-4d1b-852c-622a3fa92a27:28/Map 7, in
> queue)", "attempt_1435700346116_1889_1_05_000206_8
> (sershe_20150714174104_b0
[jira] [Commented] (HIVE-11273) LLAP: Register for finishable state change notifications when adding a task instead of when scheduling it
[ https://issues.apache.org/jira/browse/HIVE-11273?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14631487#comment-14631487 ] Siddharth Seth commented on HIVE-11273: --- Thanks for the reviews on this and 11263. 11263 actually depends upon (and applies on) this patch, where most of the test plumbing has been added. I'll go ahead and commit both of them separately since they're separate issues and have been reviewed. > LLAP: Register for finishable state change notifications when adding a task > instead of when scheduling it > - > > Key: HIVE-11273 > URL: https://issues.apache.org/jira/browse/HIVE-11273 > Project: Hive > Issue Type: Sub-task >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Fix For: llap > > Attachments: HIVE-11273.1.txt > > > Registering when trying to execute is far too late. The task won't be > considered for execution (queue may not be re-oredered) without the > notification coming in. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11280) Support executing script file from hdfs in new CLI [Beeline-CLI branch]
[ https://issues.apache.org/jira/browse/HIVE-11280?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14631452#comment-14631452 ] Ferdinand Xu commented on HIVE-11280: - Hi [~xuefuz], do you have interest reviewing this? > Support executing script file from hdfs in new CLI [Beeline-CLI branch] > --- > > Key: HIVE-11280 > URL: https://issues.apache.org/jira/browse/HIVE-11280 > Project: Hive > Issue Type: Sub-task > Components: Beeline, CLI >Reporter: Ferdinand Xu >Assignee: Ferdinand Xu > Attachments: HIVE-11280.1-beeline-cli.patch > > > In HIVE-7136, old CLI is able to read hive scripts from any of the supported > file systems in hadoop eco-system. We need to support it in new CLI as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11288) Avro SerDe InstanceCache returns incorrect schema
[
https://issues.apache.org/jira/browse/HIVE-11288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14631433#comment-14631433
]
Hive QA commented on HIVE-11288:
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12745809/HIVE-11288.patch
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 9225 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cbo_rp_join0
org.apache.hadoop.hive.serde2.avro.TestInstanceCache.instanceCachesOnlyCreateOneInstance
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/4636/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/4636/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-4636/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12745809 - PreCommit-HIVE-TRUNK-Build
> Avro SerDe InstanceCache returns incorrect schema
> -
>
> Key: HIVE-11288
> URL: https://issues.apache.org/jira/browse/HIVE-11288
> Project: Hive
> Issue Type: Bug
>Reporter: Greg Phillips
> Attachments: HIVE-11288.patch
>
>
> To reproduce this error, take two fields in an avro schema document matching
> the following:
> "type" : { "type": "array", "items": [ "null", { "type": "map", "values": [
> "null", "string" ] } ] }
> "type" : { "type": "map", "values": [ "null" , { "type": "array", "items": [
> "null" , "string"] } ] }
> After creating two tables in hive with these schemas, the describe statement
> on each of them will only return the schema for the first one loaded. This
> is due to a hashCode() collision in the InstanceCache.
> A patch will be included in this ticket shortly which removes the hashCode
> call from the InstanceCache's internal HashMap, and instead provides the
> entire schema object.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-11254) Process result sets returned by a stored procedure
[ https://issues.apache.org/jira/browse/HIVE-11254?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dmitry Tolpeko updated HIVE-11254: -- Attachment: HIVE-11254.2.patch Patch 2 created > Process result sets returned by a stored procedure > -- > > Key: HIVE-11254 > URL: https://issues.apache.org/jira/browse/HIVE-11254 > Project: Hive > Issue Type: Improvement > Components: hpl/sql >Reporter: Dmitry Tolpeko >Assignee: Dmitry Tolpeko > Attachments: HIVE-11254.1.patch, HIVE-11254.2.patch > > > Stored procedure can return one or more result sets. A caller should be able > to process them. > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11276) Optimization around job submission and adding jars [Spark Branch]
[ https://issues.apache.org/jira/browse/HIVE-11276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14631311#comment-14631311 ] Xuefu Zhang commented on HIVE-11276: Thanks. I will create a JIRA for that. > Optimization around job submission and adding jars [Spark Branch] > - > > Key: HIVE-11276 > URL: https://issues.apache.org/jira/browse/HIVE-11276 > Project: Hive > Issue Type: Sub-task > Components: Spark >Affects Versions: 1.1.0 >Reporter: Xuefu Zhang >Assignee: Chengxiang Li > > It seems that Hive on Spark has some room for performance improvement on job > submission. Specifically, we are calling refreshLocalResources() for every > job submission despite there is are no changes in the jar list. Since Hive on > Spark is reusing the containers in the whole user session, we might be able > to optimize that. > We do need to take into consideration the case of dynamic allocation, in > which new executors might be added. > This task is some R&D in this area. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11254) Process result sets returned by a stored procedure
[ https://issues.apache.org/jira/browse/HIVE-11254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14631310#comment-14631310 ] Dmitry Tolpeko commented on HIVE-11254: --- Got an error in tests: java.sql.SQLException: Could not open client transport with JDBC Uri: jdbc:hive2://localhost:1: java.net.ConnectException: Connection refused I will modify the connection URL to use embedded URL jdbc:hive2:// > Process result sets returned by a stored procedure > -- > > Key: HIVE-11254 > URL: https://issues.apache.org/jira/browse/HIVE-11254 > Project: Hive > Issue Type: Improvement > Components: hpl/sql >Reporter: Dmitry Tolpeko >Assignee: Dmitry Tolpeko > Attachments: HIVE-11254.1.patch > > > Stored procedure can return one or more result sets. A caller should be able > to process them. > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HIVE-11276) Optimization around job submission and adding jars [Spark Branch]
[ https://issues.apache.org/jira/browse/HIVE-11276?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuefu Zhang resolved HIVE-11276. Resolution: Not A Problem > Optimization around job submission and adding jars [Spark Branch] > - > > Key: HIVE-11276 > URL: https://issues.apache.org/jira/browse/HIVE-11276 > Project: Hive > Issue Type: Sub-task > Components: Spark >Affects Versions: 1.1.0 >Reporter: Xuefu Zhang >Assignee: Chengxiang Li > > It seems that Hive on Spark has some room for performance improvement on job > submission. Specifically, we are calling refreshLocalResources() for every > job submission despite there is are no changes in the jar list. Since Hive on > Spark is reusing the containers in the whole user session, we might be able > to optimize that. > We do need to take into consideration the case of dynamic allocation, in > which new executors might be added. > This task is some R&D in this area. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11288) Avro SerDe InstanceCache returns incorrect schema
[
https://issues.apache.org/jira/browse/HIVE-11288?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Greg Phillips updated HIVE-11288:
-
Attachment: HIVE-11288.patch
> Avro SerDe InstanceCache returns incorrect schema
> -
>
> Key: HIVE-11288
> URL: https://issues.apache.org/jira/browse/HIVE-11288
> Project: Hive
> Issue Type: Bug
>Reporter: Greg Phillips
> Attachments: HIVE-11288.patch
>
>
> To reproduce this error, take two fields in an avro schema document matching
> the following:
> "type" : { "type": "array", "items": [ "null", { "type": "map", "values": [
> "null", "string" ] } ] }
> "type" : { "type": "map", "values": [ "null" , { "type": "array", "items": [
> "null" , "string"] } ] }
> After creating two tables in hive with these schemas, the describe statement
> on each of them will only return the schema for the first one loaded. This
> is due to a hashCode() collision in the InstanceCache.
> A patch will be included in this ticket shortly which removes the hashCode
> call from the InstanceCache's internal HashMap, and instead provides the
> entire schema object.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-11254) Process result sets returned by a stored procedure
[
https://issues.apache.org/jira/browse/HIVE-11254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14631227#comment-14631227
]
Hive QA commented on HIVE-11254:
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12745797/HIVE-11254.1.patch
{color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 9227 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cbo_rp_join0
org.apache.hive.hplsql.TestHplsqlDb.testCreateProcedureReturnCursor
org.apache.hive.hplsql.TestHplsqlDb.testCreateProcedureReturnCursor2
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/4635/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/4635/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-4635/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 3 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12745797 - PreCommit-HIVE-TRUNK-Build
> Process result sets returned by a stored procedure
> --
>
> Key: HIVE-11254
> URL: https://issues.apache.org/jira/browse/HIVE-11254
> Project: Hive
> Issue Type: Improvement
> Components: hpl/sql
>Reporter: Dmitry Tolpeko
>Assignee: Dmitry Tolpeko
> Attachments: HIVE-11254.1.patch
>
>
> Stored procedure can return one or more result sets. A caller should be able
> to process them.
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-11254) Process result sets returned by a stored procedure
[ https://issues.apache.org/jira/browse/HIVE-11254?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dmitry Tolpeko updated HIVE-11254: -- Attachment: HIVE-11254.1.patch > Process result sets returned by a stored procedure > -- > > Key: HIVE-11254 > URL: https://issues.apache.org/jira/browse/HIVE-11254 > Project: Hive > Issue Type: Improvement > Components: hpl/sql >Reporter: Dmitry Tolpeko >Assignee: Dmitry Tolpeko > Attachments: HIVE-11254.1.patch > > > Stored procedure can return one or more result sets. A caller should be able > to process them. > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11280) Support executing script file from hdfs in new CLI [Beeline-CLI branch]
[
https://issues.apache.org/jira/browse/HIVE-11280?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14631107#comment-14631107
]
Hive QA commented on HIVE-11280:
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12745777/HIVE-11280.1-beeline-cli.patch
{color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 9233 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cbo_rp_join0
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_sortmerge_join_2
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_sortmerge_join_8
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-BEELINE-Build/10/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-BEELINE-Build/10/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-BEELINE-Build-10/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 3 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12745777 - PreCommit-HIVE-BEELINE-Build
> Support executing script file from hdfs in new CLI [Beeline-CLI branch]
> ---
>
> Key: HIVE-11280
> URL: https://issues.apache.org/jira/browse/HIVE-11280
> Project: Hive
> Issue Type: Sub-task
> Components: Beeline, CLI
>Reporter: Ferdinand Xu
>Assignee: Ferdinand Xu
> Attachments: HIVE-11280.1-beeline-cli.patch
>
>
> In HIVE-7136, old CLI is able to read hive scripts from any of the supported
> file systems in hadoop eco-system. We need to support it in new CLI as well.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-7193) Hive should support additional LDAP authentication parameters
[
https://issues.apache.org/jira/browse/HIVE-7193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14630992#comment-14630992
]
Lefty Leverenz commented on HIVE-7193:
--
Doc note: The configuration parameters are documented in the HiveServer2
section of Configuration Properties, so I removed the TODOC1.3 label.
* [Configuration Properties -- hive.server2.authentication.ldap.groupDNPattern
|
https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-hive.server2.authentication.ldap.groupDNPattern]
* [Configuration Properties -- hive.server2.authentication.ldap.groupFilter |
https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-hive.server2.authentication.ldap.groupFilter]
* [Configuration Properties -- hive.server2.authentication.ldap.userDNPattern |
https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-hive.server2.authentication.ldap.userDNPattern]
* [Configuration Properties -- hive.server2.authentication.ldap.userFilter |
https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-hive.server2.authentication.ldap.userFilter]
* [Configuration Properties -- hive.server2.authentication.ldap.customLDAPQuery
|
https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-hive.server2.authentication.ldap.customLDAPQuery]
Setting Up HiveServer2 has a link to User and Group Filter Support ... (see
link in last comment).
> Hive should support additional LDAP authentication parameters
> -
>
> Key: HIVE-7193
> URL: https://issues.apache.org/jira/browse/HIVE-7193
> Project: Hive
> Issue Type: Bug
>Affects Versions: 0.10.0
>Reporter: Mala Chikka Kempanna
>Assignee: Naveen Gangam
> Fix For: 1.3.0, 2.0.0
>
> Attachments: HIVE-7193.2.patch, HIVE-7193.3.patch, HIVE-7193.4.patch,
> HIVE-7193.5.patch, HIVE-7193.6.patch, HIVE-7193.patch,
> LDAPAuthentication_Design_Doc.docx, LDAPAuthentication_Design_Doc_V2.docx
>
>
> Currently hive has only following authenticator parameters for LDAP
> authentication for hiveserver2:
> {code:xml}
>
> hive.server2.authentication
> LDAP
>
>
> hive.server2.authentication.ldap.url
> ldap://our_ldap_address
>
> {code}
> We need to include other LDAP properties as part of hive-LDAP authentication
> like below:
> {noformat}
> a group search base -> dc=domain,dc=com
> a group search filter -> member={0}
> a user search base -> dc=domain,dc=com
> a user search filter -> sAMAAccountName={0}
> a list of valid user groups -> group1,group2,group3
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-7193) Hive should support additional LDAP authentication parameters
[
https://issues.apache.org/jira/browse/HIVE-7193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lefty Leverenz updated HIVE-7193:
-
Labels: (was: TODOC1.3)
> Hive should support additional LDAP authentication parameters
> -
>
> Key: HIVE-7193
> URL: https://issues.apache.org/jira/browse/HIVE-7193
> Project: Hive
> Issue Type: Bug
>Affects Versions: 0.10.0
>Reporter: Mala Chikka Kempanna
>Assignee: Naveen Gangam
> Fix For: 1.3.0, 2.0.0
>
> Attachments: HIVE-7193.2.patch, HIVE-7193.3.patch, HIVE-7193.4.patch,
> HIVE-7193.5.patch, HIVE-7193.6.patch, HIVE-7193.patch,
> LDAPAuthentication_Design_Doc.docx, LDAPAuthentication_Design_Doc_V2.docx
>
>
> Currently hive has only following authenticator parameters for LDAP
> authentication for hiveserver2:
> {code:xml}
>
> hive.server2.authentication
> LDAP
>
>
> hive.server2.authentication.ldap.url
> ldap://our_ldap_address
>
> {code}
> We need to include other LDAP properties as part of hive-LDAP authentication
> like below:
> {noformat}
> a group search base -> dc=domain,dc=com
> a group search filter -> member={0}
> a user search base -> dc=domain,dc=com
> a user search filter -> sAMAAccountName={0}
> a list of valid user groups -> group1,group2,group3
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-11285) ObjectInspector for partition columns in FetchOperator in SMBJoin causes exception
[
https://issues.apache.org/jira/browse/HIVE-11285?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Damien Carol updated HIVE-11285:
Description:
STEPS TO REPRODUCE:
{noformat}
*$ cat data.out
1|One
2|Two
{noformat}
{code:sql}
hql>
CREATE TABLE data_table (key INT, value STRING) ROW FORMAT DELIMITED FIELDS
TERMINATED BY '|';
LOAD DATA LOCAL INPATH '${system:user.dir}/data.out' INTO TABLE data_table;
CREATE TABLE smb_table (key INT, value STRING)
CLUSTERED BY (key)
SORTED BY (key) INTO 1 BUCKETS
STORED AS ORC;
CREATE TABLE smb_table_part (key INT, value STRING)
PARTITIONED BY (p1 DECIMAL)
CLUSTERED BY (key)
SORTED BY (key) INTO 1 BUCKETS
STORED AS ORC;
INSERT OVERWRITE TABLE smb_table SELECT * FROM data_table;
INSERT OVERWRITE TABLE smb_table_part PARTITION (p1) SELECT key, value, 100 as
p1 FROM data_table;
SET hive.execution.engine=mr;
SET hive.enforce.sortmergebucketmapjoin=false;
SET hive.auto.convert.sortmerge.join=true;
SET hive.optimize.bucketmapjoin = true;
SET hive.optimize.bucketmapjoin.sortedmerge = true;
SET hive.input.format = org.apache.hadoop.hive.ql.io.BucketizedHiveInputFormat;
SELECT s1.key, s2.p1
FROM smb_table s1
INNER JOIN smb_table_part s2
ON s1.key = s2.key
ORDER BY s1.key;
{code}
ERROR:
{noformat}
2015-07-15 13:39:04,333 WARN main org.apache.hadoop.mapred.YarnChild: Exception
running child : java.lang.RuntimeException:
org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while
processing row
{"key":1,"value":"One"}
at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:185)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:450)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error
while processing row
{"key":1,"value":"One"}
at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:503)
at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:176)
... 8 more
Caused by: java.lang.RuntimeException: Map local work failed
at
org.apache.hadoop.hive.ql.exec.SMBMapJoinOperator.fetchOneRow(SMBMapJoinOperator.java:569)
at
org.apache.hadoop.hive.ql.exec.SMBMapJoinOperator.fetchNextGroup(SMBMapJoinOperator.java:429)
at
org.apache.hadoop.hive.ql.exec.SMBMapJoinOperator.processOp(SMBMapJoinOperator.java:260)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:815)
at
org.apache.hadoop.hive.ql.exec.FilterOperator.processOp(FilterOperator.java:120)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:815)
at
org.apache.hadoop.hive.ql.exec.TableScanOperator.processOp(TableScanOperator.java:95)
at
org.apache.hadoop.hive.ql.exec.MapOperator$MapOpCtx.forward(MapOperator.java:157)
at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:493)
... 9 more
Caused by: java.lang.ClassCastException: org.apache.hadoop.io.IntWritable
cannot be cast to java.lang.Integer
at
org.apache.hadoop.hive.serde2.objectinspector.primitive.JavaIntObjectInspector.getPrimitiveWritableObject(JavaIntObjectInspector.java:35)
at
org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.copyToStandardObject(ObjectInspectorUtils.java:305)
at org.apache.hadoop.hive.ql.exec.JoinUtil.computeValues(JoinUtil.java:193)
at
org.apache.hadoop.hive.ql.exec.CommonJoinOperator.getFilteredValue(CommonJoinOperator.java:408)
at
org.apache.hadoop.hive.ql.exec.SMBMapJoinOperator.processOp(SMBMapJoinOperator.java:270)
at
org.apache.hadoop.hive.ql.exec.SMBMapJoinOperator.fetchOneRow(SMBMapJoinOperator.java:558)
... 17 more
{noformat}
was:
{code}
STEPS TO REPRODUCE:
*$ cat data.out
1|One
2|Two
hql>
CREATE TABLE data_table (key INT, value STRING) ROW FORMAT DELIMITED FIELDS
TERMINATED BY '|';
LOAD DATA LOCAL INPATH '$
{system:user.dir}
/data.out' INTO TABLE data_table;
CREATE TABLE smb_table (key INT, value STRING)
CLUSTERED BY (key)
SORTED BY (key) INTO 1 BUCKETS
STORED AS ORC;
CREATE TABLE smb_table_part (key INT, value STRING)
PARTITIONED BY (p1 DECIMAL)
CLUSTERED BY (key)
SORTED BY (key) INTO 1 BUCKETS
STORED AS ORC;
INSERT OVERWRITE TABLE smb_table SELECT * FROM data_table;
INSERT OVERWRITE TABLE smb_table_part PARTITION (p1) SELECT key, value, 100 as
p1 FROM data_table;
SET hive.execution.engine=mr;
SET hive.enforce.sortmergebucketmapjoin=false;
SET hive.auto.convert.sortmerge.join=true;
SET hive.optimize.bucketmapjoin = true;
SET hive.optimize.bucketmapjoin.sortedmerge = true;
SET hive.input.format =
