[jira] [Commented] (HIVE-6363) IllegalArgumentException is thrown instead of SQLException
[ https://issues.apache.org/jira/browse/HIVE-6363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14164827#comment-14164827 ] Rémy DUBOIS commented on HIVE-6363: --- Hello everyone, I don't understand clearly the status of this issue. I understand a patch has been created, but I also see that HIVE-4194 should fix this issue, but has not been closed after it's been reopened. Can you please clarify? Thanks in advance, > IllegalArgumentException is thrown instead of SQLException > -- > > Key: HIVE-6363 > URL: https://issues.apache.org/jira/browse/HIVE-6363 > Project: Hive > Issue Type: Bug > Components: JDBC >Affects Versions: 0.10.0, 0.11.0, 0.12.0 >Reporter: chandra sekhar gunturi >Priority: Critical > Labels: JDBC, unit-test > Attachments: HIVE-6363.patch.1.txt, HIVE-6363.patch.2.txt > > Original Estimate: 24h > Remaining Estimate: 24h > > parseURL in the following code is throwing IllegalArgumentException > http://svn.apache.org/viewvc/hive/trunk/jdbc/src/java/org/apache/hive/jdbc/Utils.java?view=markup > > This is going to break other JDBC based connectors because > java.sql.DriverManager doesn’t catch IllegalArgumentException while probing > for correct Driver for a given URL. > A simple test case can have class.forName(org.apache.hive.jdbc.HiveDriver) > (Loading hiveserver2 JDBC driver) followed by > class.forName(org.apache.hadoop.hive.jdbc.HiveDriver)(Loading hiveserver JDBC > driver). > In this case hiveserver connection will fail with BAD URL format for > hiveserver. If you reverse the driver loading to hiveserver followed by > hiveserver2, both the connections will be successful. > Following code in java.sql.DriverManager is causing the issue > [[ > // Worker method called by the public getConnection() methods. > private static Connection getConnection( > // Walk through the loaded registeredDrivers attempting to make a connection. > // Remember the first exception that gets raised so we can reraise it. > for(DriverInfo aDriver : registeredDrivers) { > // If the caller does not have permission to load the driver then > // skip it. > if(isDriverAllowed(aDriver.driver, callerCL)) { > try { > Connection con = aDriver.driver.connect(url, info); > if (con != null) { > // Success! > println("getConnection returning " + aDriver.driver.getClass().getName()); > return (con); > } > } catch (SQLException ex) { > if (reason == null) { > reason = ex; > } > } > } else { > println(" skipping: " + aDriver.getClass().getName()); > } > } > } > ]] > Marking it as critical because this is going to restrict consuming JDBC > driver in production environment where many drivers are loaded on requirement > rather than statically loading all drivers. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Review Request 26483: Correctly handle invalid distinct queries.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/26483/ --- Review request for hive and John Pullokkaran. Bugs: HIVE-8413 https://issues.apache.org/jira/browse/HIVE-8413 Repository: hive-git Description --- Correctly handle invalid distinct queries. Diffs - ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 19110ce Diff: https://reviews.apache.org/r/26483/diff/ Testing --- NegativeCliDriver::invalid_distinct1.q,invalid_distinct3.q Thanks, Ashutosh Chauhan
[jira] [Updated] (HIVE-8413) [CBO] Handle ill-formed queries which have distinct in incorrect context
[ https://issues.apache.org/jira/browse/HIVE-8413?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-8413: --- Status: Patch Available (was: Open) > [CBO] Handle ill-formed queries which have distinct in incorrect context > - > > Key: HIVE-8413 > URL: https://issues.apache.org/jira/browse/HIVE-8413 > Project: Hive > Issue Type: Bug > Components: CBO >Affects Versions: 0.14.0 >Reporter: Ashutosh Chauhan >Assignee: Ashutosh Chauhan > Attachments: HIVE-8413.patch > > > e.g., select hash (distinct key) from src; -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8413) [CBO] Handle ill-formed queries which have distinct in incorrect context
[ https://issues.apache.org/jira/browse/HIVE-8413?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-8413: --- Attachment: HIVE-8413.patch Correctly fail to compile in such erroneous case. > [CBO] Handle ill-formed queries which have distinct in incorrect context > - > > Key: HIVE-8413 > URL: https://issues.apache.org/jira/browse/HIVE-8413 > Project: Hive > Issue Type: Bug > Components: CBO >Affects Versions: 0.14.0 >Reporter: Ashutosh Chauhan >Assignee: Ashutosh Chauhan > Attachments: HIVE-8413.patch > > > e.g., select hash (distinct key) from src; -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-8413) [CBO] Handle ill-formed queries which have distinct in incorrect context
Ashutosh Chauhan created HIVE-8413: -- Summary: [CBO] Handle ill-formed queries which have distinct in incorrect context Key: HIVE-8413 URL: https://issues.apache.org/jira/browse/HIVE-8413 Project: Hive Issue Type: Bug Components: CBO Affects Versions: 0.14.0 Reporter: Ashutosh Chauhan Assignee: Ashutosh Chauhan e.g., select hash (distinct key) from src; -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8376) Umbrella Jira for HiveServer2 dynamic service discovery
[ https://issues.apache.org/jira/browse/HIVE-8376?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vaibhav Gumashta updated HIVE-8376: --- Attachment: HiveServer2DynamicServiceDiscovery.pdf Attaching documentation of current work, with steps to achieve rolling upgrade for HiveServer2 using this feature. cc [~thejas] [~leftylev] > Umbrella Jira for HiveServer2 dynamic service discovery > --- > > Key: HIVE-8376 > URL: https://issues.apache.org/jira/browse/HIVE-8376 > Project: Hive > Issue Type: New Feature > Components: HiveServer2, JDBC >Affects Versions: 0.14.0 >Reporter: Vaibhav Gumashta >Assignee: Vaibhav Gumashta > Labels: TODOC14 > Fix For: 0.14.0 > > Attachments: HiveServer2DynamicServiceDiscovery.pdf > > > Creating an ☂ Jira for documentation purpose. I'll add a detailed doc for the > implementation & usage here. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8288) HiveServer2 dynamic discovery should create znodes organized by version number & add support for removing server uri's of a particular version from the server script.
[ https://issues.apache.org/jira/browse/HIVE-8288?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vaibhav Gumashta updated HIVE-8288: --- Resolution: Fixed Status: Resolved (was: Patch Available) Patch committed to trunk and 14. Thanks for reviewing [~thejas]! > HiveServer2 dynamic discovery should create znodes organized by version > number & add support for removing server uri's of a particular version from > the server script. > -- > > Key: HIVE-8288 > URL: https://issues.apache.org/jira/browse/HIVE-8288 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2 >Affects Versions: 0.14.0 >Reporter: Vaibhav Gumashta >Assignee: Vaibhav Gumashta >Priority: Critical > Labels: TODOC14 > Fix For: 0.14.0 > > Attachments: HIVE-8288.1.patch, HIVE-8288.2.patch, HIVE-8288.3.patch, > HIVE-8288.4.patch, HIVE-8288.5.patch > > > Something like: > /hiveserver2/serverUri=host:port;version=versionInfo;sequence=sequenceNumber > would be better to support admin actions like removing all znodes for a > particular version. Also need to add an option to deregister all HiveServer2 > instances from dynamic service discovery via command line. Something like: > hive --service hiveserver2 --deregister 0.14.0-SNAPSHOT -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-8412) Make reduce side join work for all join queries [Spark Branch]
Xuefu Zhang created HIVE-8412: - Summary: Make reduce side join work for all join queries [Spark Branch] Key: HIVE-8412 URL: https://issues.apache.org/jira/browse/HIVE-8412 Project: Hive Issue Type: Sub-task Components: Spark Reporter: Xuefu Zhang Regardless all these join related optimizations such as map join, bucket join, skewed join, etc, reduce side join is the fallback. That means, if a join query wasn't taken care of by any of the optimization, it should work with reduce side join (might in a less optimal fashion). It's found that this isn't case at the moment. For instance, auto_sortmerge_join_1.q failed to execute on Spark. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-8288) HiveServer2 dynamic discovery should create znodes organized by version number & add support for removing server uri's of a particular version from the server script.
[ https://issues.apache.org/jira/browse/HIVE-8288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14164709#comment-14164709 ] Hive QA commented on HIVE-8288: --- {color:green}Overall{color}: +1 all checks pass Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12673671/HIVE-8288.5.patch {color:green}SUCCESS:{color} +1 6530 tests passed Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1183/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1183/console Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-1183/ Messages: {noformat} Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase {noformat} This message is automatically generated. ATTACHMENT ID: 12673671 > HiveServer2 dynamic discovery should create znodes organized by version > number & add support for removing server uri's of a particular version from > the server script. > -- > > Key: HIVE-8288 > URL: https://issues.apache.org/jira/browse/HIVE-8288 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2 >Affects Versions: 0.14.0 >Reporter: Vaibhav Gumashta >Assignee: Vaibhav Gumashta >Priority: Critical > Labels: TODOC14 > Fix For: 0.14.0 > > Attachments: HIVE-8288.1.patch, HIVE-8288.2.patch, HIVE-8288.3.patch, > HIVE-8288.4.patch, HIVE-8288.5.patch > > > Something like: > /hiveserver2/serverUri=host:port;version=versionInfo;sequence=sequenceNumber > would be better to support admin actions like removing all znodes for a > particular version. Also need to add an option to deregister all HiveServer2 > instances from dynamic service discovery via command line. Something like: > hive --service hiveserver2 --deregister 0.14.0-SNAPSHOT -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-8222) CBO Trunk Merge: Fix Check Style issues
[ https://issues.apache.org/jira/browse/HIVE-8222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14164698#comment-14164698 ] Brock Noland commented on HIVE-8222: [~jpullokkaran] - we agreed to let this wait because the merge patch was large, however I felt the consensus was on that JIRA that this would be resolved quickly after merge. I think should be addressed before the 0.14 release. > CBO Trunk Merge: Fix Check Style issues > --- > > Key: HIVE-8222 > URL: https://issues.apache.org/jira/browse/HIVE-8222 > Project: Hive > Issue Type: Sub-task > Components: CBO >Reporter: Laljo John Pullokkaran >Assignee: Laljo John Pullokkaran > Attachments: HIVE-8222.1.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8222) CBO Trunk Merge: Fix Check Style issues
[ https://issues.apache.org/jira/browse/HIVE-8222?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brock Noland updated HIVE-8222: --- Status: Patch Available (was: Open) > CBO Trunk Merge: Fix Check Style issues > --- > > Key: HIVE-8222 > URL: https://issues.apache.org/jira/browse/HIVE-8222 > Project: Hive > Issue Type: Sub-task > Components: CBO >Reporter: Laljo John Pullokkaran >Assignee: Laljo John Pullokkaran > Attachments: HIVE-8222.1.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-8341) Transaction information in config file can grow excessively large
[ https://issues.apache.org/jira/browse/HIVE-8341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14164696#comment-14164696 ] Brock Noland commented on HIVE-8341: I am not sure what the distribution of the numbers are in this list but I assume encoding would help more than compression? > Transaction information in config file can grow excessively large > - > > Key: HIVE-8341 > URL: https://issues.apache.org/jira/browse/HIVE-8341 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 0.14.0 >Reporter: Alan Gates >Assignee: Alan Gates >Priority: Critical > Attachments: HIVE-8341.patch > > > In our testing we have seen cases where the transaction list grows very > large. We need a more efficient way of communicating the list. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HIVE-8406) Research on skewed join [Spark Branch]
[ https://issues.apache.org/jira/browse/HIVE-8406?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li reassigned HIVE-8406: Assignee: Rui Li > Research on skewed join [Spark Branch] > -- > > Key: HIVE-8406 > URL: https://issues.apache.org/jira/browse/HIVE-8406 > Project: Hive > Issue Type: Sub-task > Components: Spark >Reporter: Xuefu Zhang >Assignee: Rui Li > > Research on how to handle skewed join for hive on spark. Here is original > hive's design doc for skewed join, > https://cwiki.apache.org/confluence/display/Hive/Skewed+Join+Optimization. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Review Request 26481: HIVE-8411: Support partial partition spec for certain ALTER PARTITION statements
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/26481/ --- Review request for hive, Gunther Hagleitner and Thejas Nair. Bugs: HIVE-8411 https://issues.apache.org/jira/browse/HIVE-8411 Repository: hive-git Description --- Allow partial partition spec for some alter table statements, such as: alter table alter_partition_change_col1 partition (p1) change column c2 c2 decimal(10,0); Diffs - ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java d5374bc ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java 4e58ad8 ql/src/test/queries/clientpositive/alter_partition_change_col.q baabb9f ql/src/test/results/clientpositive/alter_partition_change_col.q.out 7123e40 Diff: https://reviews.apache.org/r/26481/diff/ Testing --- updated qfile test Thanks, Jason Dere
[jira] [Updated] (HIVE-8411) Support partial partition spec for certain ALTER PARTITION statements
[ https://issues.apache.org/jira/browse/HIVE-8411?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Dere updated HIVE-8411: - Attachment: HIVE-8411.1.patch patch v1. Supports partial partition spec for the following atler table types: change column add/replace columns add/drop properties add/drop serde properties file format > Support partial partition spec for certain ALTER PARTITION statements > - > > Key: HIVE-8411 > URL: https://issues.apache.org/jira/browse/HIVE-8411 > Project: Hive > Issue Type: Bug >Reporter: Jason Dere >Assignee: Jason Dere > Attachments: HIVE-8411.1.patch > > > To help address concerns [~hagleitn] had about having to update many > partitions > [here|https://issues.apache.org/jira/browse/HIVE-7971?focusedCommentId=14131948&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14131948] -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8411) Support partial partition spec for certain ALTER PARTITION statements
[ https://issues.apache.org/jira/browse/HIVE-8411?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Dere updated HIVE-8411: - Status: Patch Available (was: Open) > Support partial partition spec for certain ALTER PARTITION statements > - > > Key: HIVE-8411 > URL: https://issues.apache.org/jira/browse/HIVE-8411 > Project: Hive > Issue Type: Bug >Reporter: Jason Dere >Assignee: Jason Dere > Attachments: HIVE-8411.1.patch > > > To help address concerns [~hagleitn] had about having to update many > partitions > [here|https://issues.apache.org/jira/browse/HIVE-7971?focusedCommentId=14131948&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14131948] -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-8411) Support partial partition spec for certain ALTER PARTITION statements
Jason Dere created HIVE-8411: Summary: Support partial partition spec for certain ALTER PARTITION statements Key: HIVE-8411 URL: https://issues.apache.org/jira/browse/HIVE-8411 Project: Hive Issue Type: Bug Reporter: Jason Dere Assignee: Jason Dere To help address concerns [~hagleitn] had about having to update many partitions [here|https://issues.apache.org/jira/browse/HIVE-7971?focusedCommentId=14131948&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14131948] -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-7868) AvroSerDe error handling could be improved
[ https://issues.apache.org/jira/browse/HIVE-7868?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ferdinand Xu updated HIVE-7868: --- Attachment: HIVE-7868.2.patch > AvroSerDe error handling could be improved > -- > > Key: HIVE-7868 > URL: https://issues.apache.org/jira/browse/HIVE-7868 > Project: Hive > Issue Type: Improvement >Reporter: Brock Noland >Assignee: Ferdinand Xu > Attachments: HIVE-7868.1.patch, HIVE-7868.2.patch > > > When an Avro schema is invalid, AvroSerDe returns an error message instead of > throwing an exception. This is described in > {{AvroSerdeUtils.determineSchemaOrReturnErrorSchema}}: > {noformat} > /** >* Attempt to determine the schema via the usual means, but do not throw >* an exception if we fail. Instead, signal failure via a special >* schema. This is used because Hive calls init on the serde during >* any call, including calls to update the serde properties, meaning >* if the serde is in a bad state, there is no way to update that state. >*/ > {noformat} > I believe we should find a way to provide a better experience to our users. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-8398) ExprNodeColumnDesc cannot be cast to ExprNodeConstantDesc
[ https://issues.apache.org/jira/browse/HIVE-8398?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14164627#comment-14164627 ] Zhichun Wu commented on HIVE-8398: -- attach HIVE-8398.2.patch for trunk. > ExprNodeColumnDesc cannot be cast to ExprNodeConstantDesc > - > > Key: HIVE-8398 > URL: https://issues.apache.org/jira/browse/HIVE-8398 > Project: Hive > Issue Type: Bug >Affects Versions: 0.13.0, 0.14.0 >Reporter: Zhichun Wu >Assignee: Zhichun Wu > Attachments: HIVE-8398.2.patch, HIVE-8398.patch > > > The following explain statement would fail in hive 0.13 and trunk > with "ExprNodeColumnDesc cannot be cast to ExprNodeConstantDesc" exception: > {code} > create table test.t2( key string, value int); > explain select >sum(u.value) value > from test.t2 u > group by u.key > having sum(u.value) > 30; > {code} > The full stack trace: > {code} > java.lang.ClassCastException: > org.apache.hadoop.hive.ql.plan.ExprNodeColumnDesc cannot be cast to > org.apache.hadoop.hive.ql.plan.ExprNodeConstantDesc > at > org.apache.hadoop.hive.ql.parse.TypeCheckProcFactory$DefaultExprProcessor.process(TypeCheckProcFactory.java:1067) > at > org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher.dispatch(DefaultRuleDispatcher.java:90) > at > org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatchAndReturn(DefaultGraphWalker.java:94) > at > org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatch(DefaultGraphWalker.java:78) > at > org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.walk(DefaultGraphWalker.java:132) > at > org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.startWalking(DefaultGraphWalker.java:109) > at > org.apache.hadoop.hive.ql.parse.TypeCheckProcFactory.genExprNode(TypeCheckProcFactory.java:184) > at > org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genAllExprNodeDesc(SemanticAnalyzer.java:9561) > at > org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genExprNodeDesc(SemanticAnalyzer.java:9517) > at > org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genExprNodeDesc(SemanticAnalyzer.java:9488) > at > org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genFilterPlan(SemanticAnalyzer.java:2314) > at > org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genFilterPlan(SemanticAnalyzer.java:2295) > at > org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genHavingPlan(SemanticAnalyzer.java:2139) > at > org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPostGroupByBodyPlan(SemanticAnalyzer.java:8170) > at > org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genBodyPlan(SemanticAnalyzer.java:8133) > at > org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:8963) > at > org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:9216) > at > org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:327) > at > org.apache.hadoop.hive.ql.parse.ExplainSemanticAnalyzer.analyzeInternal(ExplainSemanticAnalyzer.java:64) > at > org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:327) > at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:422) > at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:322) > at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:975) > at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1040) > at org.apache.hadoop.hive.ql.Driver.run(Driver.java:911) > at org.apache.hadoop.hive.ql.Driver.run(Driver.java:901) > at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:268) > at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:220) > at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:423) > at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:792) > at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:686) > at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:625) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at org.apache.hadoop.util.RunJar.main(RunJar.java:212) > {code} > I think it's due to HIVE-3107. HIVE-3107 introduces alternate mapping for a > column in RowResolver. While mapping the having clause in > TypeCheckProcFactory, it first maps "value" to col_1(output of groupby > clause) which has type of ExprNodeColumnDesc (Before HIVE-3107, "value" is > not recognized). When it comes to "u.value", it finds that "u" is a table > alias but fails to cast nodeOutputs\[1\] to ExprNodeConstantDesc. > Here I think we can use the text attribute in the expr node as colAlias > instead. -- This message wa
[jira] [Updated] (HIVE-8398) ExprNodeColumnDesc cannot be cast to ExprNodeConstantDesc
[ https://issues.apache.org/jira/browse/HIVE-8398?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhichun Wu updated HIVE-8398: - Status: Open (was: Patch Available) > ExprNodeColumnDesc cannot be cast to ExprNodeConstantDesc > - > > Key: HIVE-8398 > URL: https://issues.apache.org/jira/browse/HIVE-8398 > Project: Hive > Issue Type: Bug >Affects Versions: 0.13.0, 0.14.0 >Reporter: Zhichun Wu >Assignee: Zhichun Wu > Attachments: HIVE-8398.2.patch, HIVE-8398.patch > > > The following explain statement would fail in hive 0.13 and trunk > with "ExprNodeColumnDesc cannot be cast to ExprNodeConstantDesc" exception: > {code} > create table test.t2( key string, value int); > explain select >sum(u.value) value > from test.t2 u > group by u.key > having sum(u.value) > 30; > {code} > The full stack trace: > {code} > java.lang.ClassCastException: > org.apache.hadoop.hive.ql.plan.ExprNodeColumnDesc cannot be cast to > org.apache.hadoop.hive.ql.plan.ExprNodeConstantDesc > at > org.apache.hadoop.hive.ql.parse.TypeCheckProcFactory$DefaultExprProcessor.process(TypeCheckProcFactory.java:1067) > at > org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher.dispatch(DefaultRuleDispatcher.java:90) > at > org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatchAndReturn(DefaultGraphWalker.java:94) > at > org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatch(DefaultGraphWalker.java:78) > at > org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.walk(DefaultGraphWalker.java:132) > at > org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.startWalking(DefaultGraphWalker.java:109) > at > org.apache.hadoop.hive.ql.parse.TypeCheckProcFactory.genExprNode(TypeCheckProcFactory.java:184) > at > org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genAllExprNodeDesc(SemanticAnalyzer.java:9561) > at > org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genExprNodeDesc(SemanticAnalyzer.java:9517) > at > org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genExprNodeDesc(SemanticAnalyzer.java:9488) > at > org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genFilterPlan(SemanticAnalyzer.java:2314) > at > org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genFilterPlan(SemanticAnalyzer.java:2295) > at > org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genHavingPlan(SemanticAnalyzer.java:2139) > at > org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPostGroupByBodyPlan(SemanticAnalyzer.java:8170) > at > org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genBodyPlan(SemanticAnalyzer.java:8133) > at > org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:8963) > at > org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:9216) > at > org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:327) > at > org.apache.hadoop.hive.ql.parse.ExplainSemanticAnalyzer.analyzeInternal(ExplainSemanticAnalyzer.java:64) > at > org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:327) > at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:422) > at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:322) > at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:975) > at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1040) > at org.apache.hadoop.hive.ql.Driver.run(Driver.java:911) > at org.apache.hadoop.hive.ql.Driver.run(Driver.java:901) > at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:268) > at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:220) > at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:423) > at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:792) > at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:686) > at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:625) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at org.apache.hadoop.util.RunJar.main(RunJar.java:212) > {code} > I think it's due to HIVE-3107. HIVE-3107 introduces alternate mapping for a > column in RowResolver. While mapping the having clause in > TypeCheckProcFactory, it first maps "value" to col_1(output of groupby > clause) which has type of ExprNodeColumnDesc (Before HIVE-3107, "value" is > not recognized). When it comes to "u.value", it finds that "u" is a table > alias but fails to cast nodeOutputs\[1\] to ExprNodeConstantDesc. > Here I think we can use the text attribute in the expr node as colAlias > instead. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8398) ExprNodeColumnDesc cannot be cast to ExprNodeConstantDesc
[ https://issues.apache.org/jira/browse/HIVE-8398?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhichun Wu updated HIVE-8398: - Status: Patch Available (was: Open) > ExprNodeColumnDesc cannot be cast to ExprNodeConstantDesc > - > > Key: HIVE-8398 > URL: https://issues.apache.org/jira/browse/HIVE-8398 > Project: Hive > Issue Type: Bug >Affects Versions: 0.13.0, 0.14.0 >Reporter: Zhichun Wu >Assignee: Zhichun Wu > Attachments: HIVE-8398.2.patch, HIVE-8398.patch > > > The following explain statement would fail in hive 0.13 and trunk > with "ExprNodeColumnDesc cannot be cast to ExprNodeConstantDesc" exception: > {code} > create table test.t2( key string, value int); > explain select >sum(u.value) value > from test.t2 u > group by u.key > having sum(u.value) > 30; > {code} > The full stack trace: > {code} > java.lang.ClassCastException: > org.apache.hadoop.hive.ql.plan.ExprNodeColumnDesc cannot be cast to > org.apache.hadoop.hive.ql.plan.ExprNodeConstantDesc > at > org.apache.hadoop.hive.ql.parse.TypeCheckProcFactory$DefaultExprProcessor.process(TypeCheckProcFactory.java:1067) > at > org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher.dispatch(DefaultRuleDispatcher.java:90) > at > org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatchAndReturn(DefaultGraphWalker.java:94) > at > org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatch(DefaultGraphWalker.java:78) > at > org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.walk(DefaultGraphWalker.java:132) > at > org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.startWalking(DefaultGraphWalker.java:109) > at > org.apache.hadoop.hive.ql.parse.TypeCheckProcFactory.genExprNode(TypeCheckProcFactory.java:184) > at > org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genAllExprNodeDesc(SemanticAnalyzer.java:9561) > at > org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genExprNodeDesc(SemanticAnalyzer.java:9517) > at > org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genExprNodeDesc(SemanticAnalyzer.java:9488) > at > org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genFilterPlan(SemanticAnalyzer.java:2314) > at > org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genFilterPlan(SemanticAnalyzer.java:2295) > at > org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genHavingPlan(SemanticAnalyzer.java:2139) > at > org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPostGroupByBodyPlan(SemanticAnalyzer.java:8170) > at > org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genBodyPlan(SemanticAnalyzer.java:8133) > at > org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:8963) > at > org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:9216) > at > org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:327) > at > org.apache.hadoop.hive.ql.parse.ExplainSemanticAnalyzer.analyzeInternal(ExplainSemanticAnalyzer.java:64) > at > org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:327) > at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:422) > at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:322) > at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:975) > at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1040) > at org.apache.hadoop.hive.ql.Driver.run(Driver.java:911) > at org.apache.hadoop.hive.ql.Driver.run(Driver.java:901) > at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:268) > at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:220) > at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:423) > at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:792) > at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:686) > at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:625) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at org.apache.hadoop.util.RunJar.main(RunJar.java:212) > {code} > I think it's due to HIVE-3107. HIVE-3107 introduces alternate mapping for a > column in RowResolver. While mapping the having clause in > TypeCheckProcFactory, it first maps "value" to col_1(output of groupby > clause) which has type of ExprNodeColumnDesc (Before HIVE-3107, "value" is > not recognized). When it comes to "u.value", it finds that "u" is a table > alias but fails to cast nodeOutputs\[1\] to ExprNodeConstantDesc. > Here I think we can use the text attribute in the expr node as colAlias > instead. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8398) ExprNodeColumnDesc cannot be cast to ExprNodeConstantDesc
[ https://issues.apache.org/jira/browse/HIVE-8398?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhichun Wu updated HIVE-8398: - Attachment: HIVE-8398.2.patch > ExprNodeColumnDesc cannot be cast to ExprNodeConstantDesc > - > > Key: HIVE-8398 > URL: https://issues.apache.org/jira/browse/HIVE-8398 > Project: Hive > Issue Type: Bug >Affects Versions: 0.13.0, 0.14.0 >Reporter: Zhichun Wu >Assignee: Zhichun Wu > Attachments: HIVE-8398.2.patch, HIVE-8398.patch > > > The following explain statement would fail in hive 0.13 and trunk > with "ExprNodeColumnDesc cannot be cast to ExprNodeConstantDesc" exception: > {code} > create table test.t2( key string, value int); > explain select >sum(u.value) value > from test.t2 u > group by u.key > having sum(u.value) > 30; > {code} > The full stack trace: > {code} > java.lang.ClassCastException: > org.apache.hadoop.hive.ql.plan.ExprNodeColumnDesc cannot be cast to > org.apache.hadoop.hive.ql.plan.ExprNodeConstantDesc > at > org.apache.hadoop.hive.ql.parse.TypeCheckProcFactory$DefaultExprProcessor.process(TypeCheckProcFactory.java:1067) > at > org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher.dispatch(DefaultRuleDispatcher.java:90) > at > org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatchAndReturn(DefaultGraphWalker.java:94) > at > org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatch(DefaultGraphWalker.java:78) > at > org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.walk(DefaultGraphWalker.java:132) > at > org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.startWalking(DefaultGraphWalker.java:109) > at > org.apache.hadoop.hive.ql.parse.TypeCheckProcFactory.genExprNode(TypeCheckProcFactory.java:184) > at > org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genAllExprNodeDesc(SemanticAnalyzer.java:9561) > at > org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genExprNodeDesc(SemanticAnalyzer.java:9517) > at > org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genExprNodeDesc(SemanticAnalyzer.java:9488) > at > org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genFilterPlan(SemanticAnalyzer.java:2314) > at > org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genFilterPlan(SemanticAnalyzer.java:2295) > at > org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genHavingPlan(SemanticAnalyzer.java:2139) > at > org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPostGroupByBodyPlan(SemanticAnalyzer.java:8170) > at > org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genBodyPlan(SemanticAnalyzer.java:8133) > at > org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:8963) > at > org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:9216) > at > org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:327) > at > org.apache.hadoop.hive.ql.parse.ExplainSemanticAnalyzer.analyzeInternal(ExplainSemanticAnalyzer.java:64) > at > org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:327) > at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:422) > at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:322) > at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:975) > at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1040) > at org.apache.hadoop.hive.ql.Driver.run(Driver.java:911) > at org.apache.hadoop.hive.ql.Driver.run(Driver.java:901) > at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:268) > at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:220) > at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:423) > at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:792) > at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:686) > at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:625) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at org.apache.hadoop.util.RunJar.main(RunJar.java:212) > {code} > I think it's due to HIVE-3107. HIVE-3107 introduces alternate mapping for a > column in RowResolver. While mapping the having clause in > TypeCheckProcFactory, it first maps "value" to col_1(output of groupby > clause) which has type of ExprNodeColumnDesc (Before HIVE-3107, "value" is > not recognized). When it comes to "u.value", it finds that "u" is a table > alias but fails to cast nodeOutputs\[1\] to ExprNodeConstantDesc. > Here I think we can use the text attribute in the expr node as colAlias > instead. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-8408) hcat cli throws NPE when authorizer using new api is enabled
[ https://issues.apache.org/jira/browse/HIVE-8408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14164623#comment-14164623 ] Thejas M Nair commented on HIVE-8408: - Note that with V2 authorizer, it is only the additional hcat cli checks that get disabled. The regular checks that would happen with hive cli would still happen. > hcat cli throws NPE when authorizer using new api is enabled > > > Key: HIVE-8408 > URL: https://issues.apache.org/jira/browse/HIVE-8408 > Project: Hive > Issue Type: Bug > Components: Authorization, HCatalog >Reporter: Jagruti Varia >Assignee: Thejas M Nair >Priority: Critical > Fix For: 0.14.0 > > Attachments: HIVE-8408.1.patch > > > Hcat cli tries to authorize actions using the authorizer configured for > hive-cli. When authorizer based on new authorization interface is used, it > throws a NPE as it checks only for the old api based authorization > implementation. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8408) hcat cli throws NPE when authorizer using new api is enabled
[ https://issues.apache.org/jira/browse/HIVE-8408?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thejas M Nair updated HIVE-8408: Component/s: HCatalog Authorization > hcat cli throws NPE when authorizer using new api is enabled > > > Key: HIVE-8408 > URL: https://issues.apache.org/jira/browse/HIVE-8408 > Project: Hive > Issue Type: Bug > Components: Authorization, HCatalog >Reporter: Jagruti Varia >Assignee: Thejas M Nair >Priority: Critical > Fix For: 0.14.0 > > Attachments: HIVE-8408.1.patch > > > Hcat cli tries to authorize actions using the authorizer configured for > hive-cli. When authorizer based on new authorization interface is used, it > throws a NPE as it checks only for the old api based authorization > implementation. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8408) hcat cli throws NPE when authorizer using new api is enabled
[ https://issues.apache.org/jira/browse/HIVE-8408?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thejas M Nair updated HIVE-8408: Status: Patch Available (was: Open) > hcat cli throws NPE when authorizer using new api is enabled > > > Key: HIVE-8408 > URL: https://issues.apache.org/jira/browse/HIVE-8408 > Project: Hive > Issue Type: Bug >Reporter: Jagruti Varia >Assignee: Thejas M Nair >Priority: Critical > Fix For: 0.14.0 > > Attachments: HIVE-8408.1.patch > > > Hcat cli tries to authorize actions using the authorizer configured for > hive-cli. When authorizer based on new authorization interface is used, it > throws a NPE as it checks only for the old api based authorization > implementation. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Review Request 26480: HIVE-8408 : hcat cli throws NPE when authorizer using new api is enabled
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/26480/ --- Review request for hive and Sushanth Sowmyan. Repository: hive-git Description --- https://issues.apache.org/jira/browse/HIVE-8408 Diffs - hcatalog/core/src/main/java/org/apache/hive/hcatalog/cli/SemanticAnalysis/CreateTableHook.java 172ff01 hcatalog/core/src/main/java/org/apache/hive/hcatalog/cli/SemanticAnalysis/HCatAuthUtil.java PRE-CREATION hcatalog/core/src/main/java/org/apache/hive/hcatalog/cli/SemanticAnalysis/HCatSemanticAnalyzerBase.java 0184dc0 hcatalog/core/src/test/java/org/apache/hive/hcatalog/cli/SemanticAnalysis/TestHCatAuthUtil.java PRE-CREATION hcatalog/pom.xml 4b75ef5 Diff: https://reviews.apache.org/r/26480/diff/ Testing --- Unit tests, manual tests Thanks, Thejas Nair
[jira] [Updated] (HIVE-8408) hcat cli throws NPE when authorizer using new api is enabled
[ https://issues.apache.org/jira/browse/HIVE-8408?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thejas M Nair updated HIVE-8408: Attachment: HIVE-8408.1.patch Hcat does an additional layer of authorization checks using the current authorizer configured for hive cli. It is designed keeping storage based authorization provider in client side in mind. It does not make sense for it to try and use V2 api based authorizers (storage based authorization in client side uses the v1 api). > hcat cli throws NPE when authorizer using new api is enabled > > > Key: HIVE-8408 > URL: https://issues.apache.org/jira/browse/HIVE-8408 > Project: Hive > Issue Type: Bug >Reporter: Jagruti Varia >Assignee: Thejas M Nair >Priority: Critical > Fix For: 0.14.0 > > Attachments: HIVE-8408.1.patch > > > Hcat cli tries to authorize actions using the authorizer configured for > hive-cli. When authorizer based on new authorization interface is used, it > throws a NPE as it checks only for the old api based authorization > implementation. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8364) We're not waiting for all inputs in MapRecordProcessor on Tez
[ https://issues.apache.org/jira/browse/HIVE-8364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vikram Dixit K updated HIVE-8364: - Resolution: Duplicate Status: Resolved (was: Patch Available) > We're not waiting for all inputs in MapRecordProcessor on Tez > - > > Key: HIVE-8364 > URL: https://issues.apache.org/jira/browse/HIVE-8364 > Project: Hive > Issue Type: Bug >Reporter: Gunther Hagleitner >Assignee: Vikram Dixit K > Fix For: 0.14.0 > > Attachments: HIVE-8364.1.patch, HIVE-8364.2.patch > > > Seems like this could be a race condition: We're blocking for some inputs to > become available, but the main MR input is just assumed ready... -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-7953) Investigate query failures (2)
[ https://issues.apache.org/jira/browse/HIVE-7953?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14164603#comment-14164603 ] Thomas Friedrich commented on HIVE-7953: Thanks, Xuefu. I will create the JIRAs once I ran all the test cases of this JIRA. > Investigate query failures (2) > -- > > Key: HIVE-7953 > URL: https://issues.apache.org/jira/browse/HIVE-7953 > Project: Hive > Issue Type: Sub-task > Components: Spark >Reporter: Brock Noland >Assignee: Thomas Friedrich > > I ran all q-file tests and the following failed with an exception: > http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/HIVE-SPARK-ALL-TESTS-Build/lastCompletedBuild/testReport/ > we don't necessary want to run all these tests as part of the spark tests, > but we should understand why they failed with an exception. This JIRA is to > look into these failures and document them with one of: > * New JIRA > * Covered under existing JIRA > * More investigation required > Tests: > {noformat} > > org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_temp_table_external > 0.33 sec2 > > org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_bucket_num_reducers > 4.3 sec 2 > > org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_sortmerge_join_2 > 11 sec 2 > > org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_load_hdfs_file_with_space_in_the_name > 0.65 sec2 > > org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_bucketsortoptimize_insert_4 > 4.7 sec 2 > > org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_bucketsortoptimize_insert_7 > 2.8 sec 2 > > org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_bucketsortoptimize_insert_2 > 5.5 sec 2 > org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_groupby_position > 1.5 sec 2 > > org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_exim_18_part_external > 2.4 sec 2 > > org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_sortmerge_join_6 > 11 sec 2 > org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_smb_mapjoin_11 > 5.1 sec 2 > > org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_bucketsortoptimize_insert_8 > 10 sec 2 > org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_parquet_join > 5.4 sec 2 > > org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_stats_empty_dyn_part > 0.81 sec2 > > org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_dbtxnmgr_compact1 >0.31 sec2 > org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_dbtxnmgr_ddl1 > 0.26 sec2 > org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_dbtxnmgr_query2 > 0.73 sec2 > > org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_sortmerge_join_3 > 8.5 sec 2 > org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_dbtxnmgr_query5 > 0.34 sec2 > org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_rcfile_bigdata > 0.93 sec2 > > org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_groupby_multi_single_reducer > 6.3 sec 2 > > org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_dbtxnmgr_compact3 >2.4 sec 2 > > org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_dbtxnmgr_compact2 >0.56 sec2 > > org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_stats_partscan_1_23 > 3.1 sec 2 > > org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_list_bucket_dml_2 >4.3 sec 2 > > org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_exim_15_external_part > 3.2 sec 2 > > org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_exim_16_part_external > 2.8 sec 2 > > org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_exim_17_part_managed > 3.4 sec 2 > > org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_exim_20_part_managed_location > 3.3 sec 2 > > org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_exim_19_00_part_external_location > 6.9 sec 2 > > org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_external_table_with_space_in_location_path > {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-7953) Investigate query failures (2)
[ https://issues.apache.org/jira/browse/HIVE-7953?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14164597#comment-14164597 ] Thomas Friedrich commented on HIVE-7953: The 4 tests bucketsortoptimize_insert_2 bucketsortoptimize_insert_4 bucketsortoptimize_insert_7 bucketsortoptimize_insert_8 all fail with the same NPE related to SMB joins: order object is null in SMBMapJoinOperator: // fetch the first group for all small table aliases for (byte pos = 0; pos < order.length; pos++) { if (pos != posBigTable) { fetchNextGroup(pos); } Daemon Thread [Executor task launch worker-3] (Suspended (exception NullPointerException)) SMBMapJoinOperator.processOp(Object, int) line: 258 FilterOperator(Operator).forward(Object, ObjectInspector) line: 799 FilterOperator.processOp(Object, int) line: 137 TableScanOperator(Operator).forward(Object, ObjectInspector) line: 799 TableScanOperator.processOp(Object, int) line: 95 MapOperator(Operator).forward(Object, ObjectInspector) line: 799 MapOperator.process(Writable) line: 536 SparkMapRecordHandler.processRow(Object, Object) line: 139 HiveMapFunctionResultList.processNextRecord(Tuple2) line: 47 HiveMapFunctionResultList.processNextRecord(Object) line: 28 HiveBaseFunctionResultList$ResultIterator.hasNext() line: 108 Wrappers$JIteratorWrapper.hasNext() line: 41 Iterator$class.foreach(Iterator, Function1) line: 727 Wrappers$JIteratorWrapper(AbstractIterator).foreach(Function1) line: 1157 RDD$$anonfun$foreach$1.apply(Iterator) line: 760 RDD$$anonfun$foreach$1.apply(Object) line: 760 SparkContext$$anonfun$runJob$3.apply(TaskContext, Iterator) line: 1118 SparkContext$$anonfun$runJob$3.apply(Object, Object) line: 1118 ResultTask.runTask(TaskContext) line: 61 ResultTask(Task).run(long) line: 56 Executor$TaskRunner.run() line: 182 ThreadPoolExecutor.runWorker(ThreadPoolExecutor$Worker) line: 1145 ThreadPoolExecutor$Worker.run() line: 615 Thread.run() line: 745 There is also a NPE in the FileSinkOperator: the FileSystem object fs is null: // in recent hadoop versions, use deleteOnExit to clean tmp files. if (isNativeTable) { autoDelete = fs.deleteOnExit(fsp.outPaths[0]); Daemon Thread [Executor task launch worker-1] (Suspended (exception NullPointerException)) FileSinkOperator.createBucketFiles(FileSinkOperator$FSPaths) line: 495 FileSinkOperator.closeOp(boolean) line: 925 FileSinkOperator(Operator).close(boolean) line: 582 SelectOperator(Operator).close(boolean) line: 594 SMBMapJoinOperator(Operator).close(boolean) line: 594 DummyStoreOperator(Operator).close(boolean) line: 594 FilterOperator(Operator).close(boolean) line: 594 TableScanOperator(Operator).close(boolean) line: 594 MapOperator(Operator).close(boolean) line: 594 SparkMapRecordHandler.close() line: 175 HiveMapFunctionResultList.closeRecordProcessor() line: 57 HiveBaseFunctionResultList$ResultIterator.hasNext() line: 122 Wrappers$JIteratorWrapper.hasNext() line: 41 Iterator$class.foreach(Iterator, Function1) line: 727 Wrappers$JIteratorWrapper(AbstractIterator).foreach(Function1) line: 1157 RDD$$anonfun$foreach$1.apply(Iterator) line: 760 RDD$$anonfun$foreach$1.apply(Object) line: 760 SparkContext$$anonfun$runJob$3.apply(TaskContext, Iterator) line: 1118 SparkContext$$anonfun$runJob$3.apply(Object, Object) line: 1118 ResultTask.runTask(TaskContext) line: 61 ResultTask(Task).run(long) line: 56 Executor$TaskRunner.run() line: 182 ThreadPoolExecutor.runWorker(ThreadPoolExecutor$Worker) line: 1145 ThreadPoolExecutor$Worker.run() line: 615 Thread.run() line: 745 > Investigate query failures (2) > -- > > Key: HIVE-7953 > URL: https://issues.apache.org/jira/browse/HIVE-7953 > Project: Hive > Issue Type: Sub-task > Components: Spark >Reporter: Brock Noland >Assignee: Thomas Friedrich > > I ran all q-file tests and the following failed with an exception: > http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/HIVE-SPARK-ALL-TESTS-Build/lastCompletedBuild/testReport/ > we don't necessary want to run all these tests as part of the spark tests, > but we should understand why they failed with an exception. This JIRA is to > look into these failures and document
[jira] [Commented] (HIVE-8364) We're not waiting for all inputs in MapRecordProcessor on Tez
[ https://issues.apache.org/jira/browse/HIVE-8364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14164596#comment-14164596 ] Vikram Dixit K commented on HIVE-8364: -- Superseded by HIVE-8409. > We're not waiting for all inputs in MapRecordProcessor on Tez > - > > Key: HIVE-8364 > URL: https://issues.apache.org/jira/browse/HIVE-8364 > Project: Hive > Issue Type: Bug >Reporter: Gunther Hagleitner >Assignee: Vikram Dixit K > Fix For: 0.14.0 > > Attachments: HIVE-8364.1.patch, HIVE-8364.2.patch > > > Seems like this could be a race condition: We're blocking for some inputs to > become available, but the main MR input is just assumed ready... -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8409) SMB joins fail intermittently on tez
[ https://issues.apache.org/jira/browse/HIVE-8409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vikram Dixit K updated HIVE-8409: - Status: Patch Available (was: Open) > SMB joins fail intermittently on tez > > > Key: HIVE-8409 > URL: https://issues.apache.org/jira/browse/HIVE-8409 > Project: Hive > Issue Type: Bug > Components: Tez >Affects Versions: 0.14.0 >Reporter: Vikram Dixit K >Assignee: Vikram Dixit K >Priority: Critical > Fix For: 0.14.0 > > Attachments: HIVE-8409.1.patch > > > Flakiness with regard to SMB joins in tez. TEZ-1647 is required to complete > the fix. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-8021) CBO: support CTAS and insert ... select
[ https://issues.apache.org/jira/browse/HIVE-8021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14164590#comment-14164590 ] Sergey Shelukhin commented on HIVE-8021: at least 3 tests fail with CTAS on top of existing CBO failures/out file changes. One of them is caused by general select issue that is not uncovered by other q files; I haven't yet looked at others, will do so soon > CBO: support CTAS and insert ... select > --- > > Key: HIVE-8021 > URL: https://issues.apache.org/jira/browse/HIVE-8021 > Project: Hive > Issue Type: Sub-task > Components: CBO >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-8021.01.patch, HIVE-8021.01.patch, > HIVE-8021.02.patch, HIVE-8021.03.patch, HIVE-8021.patch, > HIVE-8021.preliminary.patch > > > Need to send only the select part to CBO for now -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-8410) Typo in DOAP - incorrect category URL
Sebb created HIVE-8410: -- Summary: Typo in DOAP - incorrect category URL Key: HIVE-8410 URL: https://issues.apache.org/jira/browse/HIVE-8410 Project: Hive Issue Type: Bug Environment: http://svn.apache.org/repos/asf/hive/trunk/doap_Hive.rdf Reporter: Sebb The DOAP contains the following: {code} http://www.apache.org/category/database"; /> {code} However, the URL is incorrect; it must be {code} http://projects.apache.org/category/database"; /> {code} Please fix this -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8409) SMB joins fail intermittently on tez
[ https://issues.apache.org/jira/browse/HIVE-8409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vikram Dixit K updated HIVE-8409: - Attachment: (was: HIVE-8409.1.patch) > SMB joins fail intermittently on tez > > > Key: HIVE-8409 > URL: https://issues.apache.org/jira/browse/HIVE-8409 > Project: Hive > Issue Type: Bug > Components: Tez >Affects Versions: 0.14.0 >Reporter: Vikram Dixit K >Assignee: Vikram Dixit K >Priority: Critical > Fix For: 0.14.0 > > Attachments: HIVE-8409.1.patch > > > Flakiness with regard to SMB joins in tez. TEZ-1647 is required to complete > the fix. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8409) SMB joins fail intermittently on tez
[ https://issues.apache.org/jira/browse/HIVE-8409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vikram Dixit K updated HIVE-8409: - Attachment: HIVE-8409.1.patch > SMB joins fail intermittently on tez > > > Key: HIVE-8409 > URL: https://issues.apache.org/jira/browse/HIVE-8409 > Project: Hive > Issue Type: Bug > Components: Tez >Affects Versions: 0.14.0 >Reporter: Vikram Dixit K >Assignee: Vikram Dixit K >Priority: Critical > Fix For: 0.14.0 > > Attachments: HIVE-8409.1.patch > > > Flakiness with regard to SMB joins in tez. TEZ-1647 is required to complete > the fix. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8409) SMB joins fail intermittently on tez
[ https://issues.apache.org/jira/browse/HIVE-8409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vikram Dixit K updated HIVE-8409: - Status: Open (was: Patch Available) > SMB joins fail intermittently on tez > > > Key: HIVE-8409 > URL: https://issues.apache.org/jira/browse/HIVE-8409 > Project: Hive > Issue Type: Bug > Components: Tez >Affects Versions: 0.14.0 >Reporter: Vikram Dixit K >Assignee: Vikram Dixit K >Priority: Critical > Fix For: 0.14.0 > > Attachments: HIVE-8409.1.patch > > > Flakiness with regard to SMB joins in tez. TEZ-1647 is required to complete > the fix. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8409) SMB joins fail intermittently on tez
[ https://issues.apache.org/jira/browse/HIVE-8409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vikram Dixit K updated HIVE-8409: - Attachment: HIVE-8409.1.patch > SMB joins fail intermittently on tez > > > Key: HIVE-8409 > URL: https://issues.apache.org/jira/browse/HIVE-8409 > Project: Hive > Issue Type: Bug > Components: Tez >Affects Versions: 0.14.0 >Reporter: Vikram Dixit K >Assignee: Vikram Dixit K >Priority: Critical > Fix For: 0.14.0 > > Attachments: HIVE-8409.1.patch > > > Flakiness with regard to SMB joins in tez. TEZ-1647 is required to complete > the fix. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8409) SMB joins fail intermittently on tez
[ https://issues.apache.org/jira/browse/HIVE-8409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vikram Dixit K updated HIVE-8409: - Status: Patch Available (was: Open) > SMB joins fail intermittently on tez > > > Key: HIVE-8409 > URL: https://issues.apache.org/jira/browse/HIVE-8409 > Project: Hive > Issue Type: Bug > Components: Tez >Affects Versions: 0.14.0 >Reporter: Vikram Dixit K >Assignee: Vikram Dixit K >Priority: Critical > Fix For: 0.14.0 > > Attachments: HIVE-8409.1.patch > > > Flakiness with regard to SMB joins in tez. TEZ-1647 is required to complete > the fix. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-8409) SMB joins fail intermittently on tez
Vikram Dixit K created HIVE-8409: Summary: SMB joins fail intermittently on tez Key: HIVE-8409 URL: https://issues.apache.org/jira/browse/HIVE-8409 Project: Hive Issue Type: Bug Components: Tez Affects Versions: 0.14.0 Reporter: Vikram Dixit K Assignee: Vikram Dixit K Priority: Critical Fix For: 0.14.0 Flakiness with regard to SMB joins in tez. TEZ-1647 is required to complete the fix. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-4329) HCatalog should use getHiveRecordWriter rather than getRecordWriter
[ https://issues.apache.org/jira/browse/HIVE-4329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14164575#comment-14164575 ] Brock Noland commented on HIVE-4329: Could you open a follow-on ticket for the parquet static partitioning issue and link it to HIVE-8120? > HCatalog should use getHiveRecordWriter rather than getRecordWriter > --- > > Key: HIVE-4329 > URL: https://issues.apache.org/jira/browse/HIVE-4329 > Project: Hive > Issue Type: Bug > Components: HCatalog, Serializers/Deserializers >Affects Versions: 0.14.0 > Environment: discovered in Pig, but it looks like the root cause > impacts all non-Hive users >Reporter: Sean Busbey >Assignee: David Chen > Attachments: HIVE-4329.0.patch, HIVE-4329.1.patch, HIVE-4329.2.patch, > HIVE-4329.3.patch, HIVE-4329.4.patch > > > Attempting to write to a HCatalog defined table backed by the AvroSerde fails > with the following stacktrace: > {code} > java.lang.ClassCastException: org.apache.hadoop.io.NullWritable cannot be > cast to org.apache.hadoop.io.LongWritable > at > org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat$1.write(AvroContainerOutputFormat.java:84) > at > org.apache.hcatalog.mapreduce.FileRecordWriterContainer.write(FileRecordWriterContainer.java:253) > at > org.apache.hcatalog.mapreduce.FileRecordWriterContainer.write(FileRecordWriterContainer.java:53) > at > org.apache.hcatalog.pig.HCatBaseStorer.putNext(HCatBaseStorer.java:242) > at org.apache.hcatalog.pig.HCatStorer.putNext(HCatStorer.java:52) > at > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat$PigRecordWriter.write(PigOutputFormat.java:139) > at > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat$PigRecordWriter.write(PigOutputFormat.java:98) > at > org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.write(MapTask.java:559) > at > org.apache.hadoop.mapreduce.task.TaskInputOutputContextImpl.write(TaskInputOutputContextImpl.java:85) > {code} > The proximal cause of this failure is that the AvroContainerOutputFormat's > signature mandates a LongWritable key and HCat's FileRecordWriterContainer > forces a NullWritable. I'm not sure of a general fix, other than redefining > HiveOutputFormat to mandate a WritableComparable. > It looks like accepting WritableComparable is what's done in the other Hive > OutputFormats, and there's no reason AvroContainerOutputFormat couldn't also > be changed, since it's ignoring the key. That way fixing things so > FileRecordWriterContainer can always use NullWritable could get spun into a > different issue? > The underlying cause for failure to write to AvroSerde tables is that > AvroContainerOutputFormat doesn't meaningfully implement getRecordWriter, so > fixing the above will just push the failure into the placeholder RecordWriter. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8292) Reading from partitioned bucketed tables has high overhead in MapOperator.cleanUpInputFileChangedOp
[ https://issues.apache.org/jira/browse/HIVE-8292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gopal V updated HIVE-8292: -- Assignee: Gopal V (was: Vikram Dixit K) Status: Patch Available (was: Open) > Reading from partitioned bucketed tables has high overhead in > MapOperator.cleanUpInputFileChangedOp > --- > > Key: HIVE-8292 > URL: https://issues.apache.org/jira/browse/HIVE-8292 > Project: Hive > Issue Type: Bug > Components: Query Processor >Affects Versions: 0.14.0 > Environment: cn105 >Reporter: Mostafa Mokhtar >Assignee: Gopal V > Fix For: 0.14.0 > > Attachments: 2014_09_29_14_46_04.jfr, HIVE-8292.1.patch, > HIVE-8292.2.patch > > > Reading from bucketed partitioned tables has significantly higher overhead > compared to non-bucketed non-partitioned files. > 50% of the profile is spent in MapOperator.cleanUpInputFileChangedOp > 5% the CPU in > {code} > Path onepath = normalizePath(onefile); > {code} > And > 45% the CPU in > {code} > onepath.toUri().relativize(fpath.toUri()).equals(fpath.toUri()); > {code} > From the profiler > {code} > Stack Trace Sample CountPercentage(%) > hive.ql.exec.tez.MapRecordSource.processRow(Object) 5,327 62.348 >hive.ql.exec.vector.VectorMapOperator.process(Writable)5,326 62.336 > hive.ql.exec.Operator.cleanUpInputFileChanged() 4,851 56.777 > hive.ql.exec.MapOperator.cleanUpInputFileChangedOp() 4,849 56.753 > java.net.URI.relativize(URI) 3,903 45.681 > java.net.URI.relativize(URI, URI) 3,903 > 45.681 >java.net.URI.normalize(String) 2,169 > 25.386 >java.net.URI.equal(String, String) > 526 6.156 >java.net.URI.equalIgnoringCase(String, > String) 1 0.012 >java.lang.String.substring(int) > 1 0.012 > hive.ql.exec.MapOperator.normalizePath(String)506 5.922 > org.apache.commons.logging.impl.Log4JLogger.info(Object) 32 > 0.375 > java.net.URI.equals(Object) 12 0.14 > java.util.HashMap$KeySet.iterator() 5 > 0.059 > java.util.HashMap.get(Object)4 > 0.047 > java.util.LinkedHashMap.get(Object) 3 > 0.035 > hive.ql.exec.Operator.cleanUpInputFileChanged() 1 0.012 > hive.ql.exec.Operator.forward(Object, ObjectInspector) 473 5.536 > hive.ql.exec.mr.ExecMapperContext.inputFileChanged()1 0.012 > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HIVE-8120) Umbrella JIRA tracking Parquet improvements
[ https://issues.apache.org/jira/browse/HIVE-8120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brock Noland reassigned HIVE-8120: -- Assignee: Brock Noland > Umbrella JIRA tracking Parquet improvements > --- > > Key: HIVE-8120 > URL: https://issues.apache.org/jira/browse/HIVE-8120 > Project: Hive > Issue Type: Improvement >Reporter: Brock Noland >Assignee: Brock Noland > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-6914) parquet-hive cannot write nested map (map value is map)
[ https://issues.apache.org/jira/browse/HIVE-6914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14164563#comment-14164563 ] Brock Noland commented on HIVE-6914: Linking to HIVE-8120. FYI [~spena] who is looking into some parquet hive schema stuff as well. > parquet-hive cannot write nested map (map value is map) > --- > > Key: HIVE-6914 > URL: https://issues.apache.org/jira/browse/HIVE-6914 > Project: Hive > Issue Type: Bug > Components: File Formats >Affects Versions: 0.12.0, 0.13.0 >Reporter: Tongjie Chen > Labels: parquet, serialization > Attachments: HIVE-6914.1.patch > > > // table schema (identical for both plain text version and parquet version) > desc hive> desc text_mmap; > m map> > // sample nested map entry > {"level1":{"level2_key1":"value1","level2_key2":"value2"}} > The following query will fail, > insert overwrite table parquet_mmap select * from text_mmap; > Caused by: parquet.io.ParquetEncodingException: This should be an > ArrayWritable or MapWritable: > org.apache.hadoop.hive.ql.io.parquet.writable.BinaryWritable@f2f8106 > at > org.apache.hadoop.hive.ql.io.parquet.write.DataWritableWriter.writeData(DataWritableWriter.java:85) > at > org.apache.hadoop.hive.ql.io.parquet.write.DataWritableWriter.writeArray(DataWritableWriter.java:118) > at > org.apache.hadoop.hive.ql.io.parquet.write.DataWritableWriter.writeData(DataWritableWriter.java:80) > at > org.apache.hadoop.hive.ql.io.parquet.write.DataWritableWriter.writeData(DataWritableWriter.java:82) > at > org.apache.hadoop.hive.ql.io.parquet.write.DataWritableWriter.write(DataWritableWriter.java:55) > at > org.apache.hadoop.hive.ql.io.parquet.write.DataWritableWriteSupport.write(DataWritableWriteSupport.java:59) > at > org.apache.hadoop.hive.ql.io.parquet.write.DataWritableWriteSupport.write(DataWritableWriteSupport.java:31) > at > parquet.hadoop.InternalParquetRecordWriter.write(InternalParquetRecordWriter.java:115) > at parquet.hadoop.ParquetRecordWriter.write(ParquetRecordWriter.java:81) > at parquet.hadoop.ParquetRecordWriter.write(ParquetRecordWriter.java:37) > at > org.apache.hadoop.hive.ql.io.parquet.write.ParquetRecordWriterWrapper.write(ParquetRecordWriterWrapper.java:77) > at > org.apache.hadoop.hive.ql.io.parquet.write.ParquetRecordWriterWrapper.write(ParquetRecordWriterWrapper.java:90) > at > org.apache.hadoop.hive.ql.exec.FileSinkOperator.processOp(FileSinkOperator.java:622) > at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:793) > at > org.apache.hadoop.hive.ql.exec.SelectOperator.processOp(SelectOperator.java:87) > at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:793) > at > org.apache.hadoop.hive.ql.exec.TableScanOperator.processOp(TableScanOperator.java:92) > at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:793) > at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:540) > ... 9 more -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-8395) CBO: enable by default
[ https://issues.apache.org/jira/browse/HIVE-8395?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14164561#comment-14164561 ] Hive QA commented on HIVE-8395: --- {color:red}Overall{color}: -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12673670/HIVE-8395.01.patch {color:red}ERROR:{color} -1 due to 356 failed/errored test(s), 6542 tests executed *Failed tests:* {noformat} org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver_accumulo_predicate_pushdown org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_allcolref_in_udf org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_alter_partition_coltype org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ambiguous_col org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_annotate_stats_filter org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_annotate_stats_groupby org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_annotate_stats_groupby2 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_annotate_stats_join org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_annotate_stats_limit org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_annotate_stats_part org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_annotate_stats_select org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_annotate_stats_table org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_annotate_stats_union org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ansi_sql_arithmetic org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_authorization_explain org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join10 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join11 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join12 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join13 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join16 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join18 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join18_multi_distinct org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join22 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join27 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join_filters org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join_nulls org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join_reordering_values org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join_without_localtask org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_binarysortable_1 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucket_groupby org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_combine2 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_constantPropagateForSubQuery org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_constprog2 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer1 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer10 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer12 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer15 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer2 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer3 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer6 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer7 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer8 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_count org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_create_genericudaf org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_create_view org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cross_join org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cross_product_check_1 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cross_product_check_2 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_decimal_precision org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_decimal_udf org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_decimal_udf2 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_distinct_stats org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_explain_dependency org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_explain_logical org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_fetch_aggregation org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_filter_join_breaktask org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_filter_join_breaktask2 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby2_limit org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_distinct_samekey org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_position org.apache.hadoop.hive.cli.Tes
[jira] [Commented] (HIVE-8341) Transaction information in config file can grow excessively large
[ https://issues.apache.org/jira/browse/HIVE-8341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14164553#comment-14164553 ] Gunther Hagleitner commented on HIVE-8341: -- I use this script: {noformat} #!/usr/bin/python import sys import random for line in sys.stdin: print random.random() {noformat} and this query: {noformat} add FILE /home/gunther/rand.py; select transform () using 'rand.py' as (foo float) from ; {noformat} Does that help? > Transaction information in config file can grow excessively large > - > > Key: HIVE-8341 > URL: https://issues.apache.org/jira/browse/HIVE-8341 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 0.14.0 >Reporter: Alan Gates >Assignee: Alan Gates >Priority: Critical > Attachments: HIVE-8341.patch > > > In our testing we have seen cases where the transaction list grows very > large. We need a more efficient way of communicating the list. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-7932) It may cause NP exception when add accessed columns to ReadEntity
[ https://issues.apache.org/jira/browse/HIVE-7932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14164544#comment-14164544 ] Prasad Mujumdar commented on HIVE-7932: --- [~vikram.dixit] Requesting backport to 0.14. It's a followup to HIVE-7730 which is already in 0.14. Thanks! > It may cause NP exception when add accessed columns to ReadEntity > - > > Key: HIVE-7932 > URL: https://issues.apache.org/jira/browse/HIVE-7932 > Project: Hive > Issue Type: Bug >Reporter: Xiaomeng Huang >Assignee: Xiaomeng Huang > Fix For: 0.15.0 > > Attachments: HIVE-7932.001.patch, HIVE-7932.002.patch > > > {code} > case TABLE: >entity.getAccessedColumns().addAll( > tableToColumnAccessMap.get(entity.getTable().getCompleteName())); > {code} > if tableToColumnAccessMap.get(entity.getTable().getCompleteName()) is null, > addAll(null) will throw null pointer exception. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8083) Authorization DDLs should not enforce hive identifier syntax for user or group
[ https://issues.apache.org/jira/browse/HIVE-8083?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasad Mujumdar updated HIVE-8083: -- Release Note: The user name in the grant and revoke statements may be optionally surrounded by backtick characters (`) irrespective of the hive.support.quoted.identifiers setting. > Authorization DDLs should not enforce hive identifier syntax for user or group > -- > > Key: HIVE-8083 > URL: https://issues.apache.org/jira/browse/HIVE-8083 > Project: Hive > Issue Type: Bug > Components: SQL, SQLStandardAuthorization >Affects Versions: 0.13.0, 0.13.1 >Reporter: Prasad Mujumdar >Assignee: Prasad Mujumdar > Labels: TODOC14 > Fix For: 0.14.0 > > Attachments: HIVE-8083.1.patch, HIVE-8083.2.patch, HIVE-8083.3.patch > > > The compiler expects principals (user, group and role) as hive identifiers > for authorization DDLs. The user and group are entities that belong to > external namespace and we can't expect those to follow hive identifier syntax > rules. For example, a userid or group can contain '-' which is not allowed by > compiler. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-8083) Authorization DDLs should not enforce hive identifier syntax for user or group
[ https://issues.apache.org/jira/browse/HIVE-8083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14164534#comment-14164534 ] Prasad Mujumdar commented on HIVE-8083: --- [~leftylev] Thanks for pointing that out. Just updated the wiki. > Authorization DDLs should not enforce hive identifier syntax for user or group > -- > > Key: HIVE-8083 > URL: https://issues.apache.org/jira/browse/HIVE-8083 > Project: Hive > Issue Type: Bug > Components: SQL, SQLStandardAuthorization >Affects Versions: 0.13.0, 0.13.1 >Reporter: Prasad Mujumdar >Assignee: Prasad Mujumdar > Labels: TODOC14 > Fix For: 0.14.0 > > Attachments: HIVE-8083.1.patch, HIVE-8083.2.patch, HIVE-8083.3.patch > > > The compiler expects principals (user, group and role) as hive identifiers > for authorization DDLs. The user and group are entities that belong to > external namespace and we can't expect those to follow hive identifier syntax > rules. For example, a userid or group can contain '-' which is not allowed by > compiler. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-7932) It may cause NP exception when add accessed columns to ReadEntity
[ https://issues.apache.org/jira/browse/HIVE-7932?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasad Mujumdar updated HIVE-7932: -- Resolution: Fixed Fix Version/s: 0.15.0 Status: Resolved (was: Patch Available) Patch committed to trunk. Thanks [~Huang Xiaomeng]! > It may cause NP exception when add accessed columns to ReadEntity > - > > Key: HIVE-7932 > URL: https://issues.apache.org/jira/browse/HIVE-7932 > Project: Hive > Issue Type: Bug >Reporter: Xiaomeng Huang >Assignee: Xiaomeng Huang > Fix For: 0.15.0 > > Attachments: HIVE-7932.001.patch, HIVE-7932.002.patch > > > {code} > case TABLE: >entity.getAccessedColumns().addAll( > tableToColumnAccessMap.get(entity.getTable().getCompleteName())); > {code} > if tableToColumnAccessMap.get(entity.getTable().getCompleteName()) is null, > addAll(null) will throw null pointer exception. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-8374) schematool fails on Postgres versions < 9.2
[ https://issues.apache.org/jira/browse/HIVE-8374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14164500#comment-14164500 ] Mohit Sabharwal commented on HIVE-8374: --- Instead of having the user supply an option dependent on Postgres version, changed to the patch to automatically run a "pre-upgrade" script that creates the language, but any error is ignored. Kept the dbOpts option as that is useful when we add any db specific options. > schematool fails on Postgres versions < 9.2 > --- > > Key: HIVE-8374 > URL: https://issues.apache.org/jira/browse/HIVE-8374 > Project: Hive > Issue Type: Bug > Components: Database/Schema >Reporter: Mohit Sabharwal >Assignee: Mohit Sabharwal > Attachments: HIVE-8374.1.patch, HIVE-8374.patch > > > The upgrade script for HIVE-5700 creates an UDF with language 'plpgsql', > which is available by default only for Postgres 9.2+. > For older Postgres versions, the language must be explicitly created, > otherwise schematool fails with the error: > {code} > Error: ERROR: language "plpgsql" does not exist > Hint: Use CREATE LANGUAGE to load the language into the database. > (state=42704,code=0) > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8374) schematool fails on Postgres versions < 9.2
[ https://issues.apache.org/jira/browse/HIVE-8374?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mohit Sabharwal updated HIVE-8374: -- Attachment: HIVE-8374.1.patch > schematool fails on Postgres versions < 9.2 > --- > > Key: HIVE-8374 > URL: https://issues.apache.org/jira/browse/HIVE-8374 > Project: Hive > Issue Type: Bug > Components: Database/Schema >Reporter: Mohit Sabharwal >Assignee: Mohit Sabharwal > Attachments: HIVE-8374.1.patch, HIVE-8374.patch > > > The upgrade script for HIVE-5700 creates an UDF with language 'plpgsql', > which is available by default only for Postgres 9.2+. > For older Postgres versions, the language must be explicitly created, > otherwise schematool fails with the error: > {code} > Error: ERROR: language "plpgsql" does not exist > Hint: Use CREATE LANGUAGE to load the language into the database. > (state=42704,code=0) > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: Review Request 26461: [CBO] Handle filters with non-boolean return type
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/26461/#review55923 --- Ship it! Ship It! - John Pullokkaran On Oct. 8, 2014, 10:23 p.m., Ashutosh Chauhan wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/26461/ > --- > > (Updated Oct. 8, 2014, 10:23 p.m.) > > > Review request for hive and John Pullokkaran. > > > Bugs: HIVE-8407 > https://issues.apache.org/jira/browse/HIVE-8407 > > > Repository: hive-git > > > Description > --- > > [CBO] Handle filters with non-boolean return type > > > Diffs > - > > ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 19110ce > > Diff: https://reviews.apache.org/r/26461/diff/ > > > Testing > --- > > annotate_stats_filter.q > > > Thanks, > > Ashutosh Chauhan > >
Re: Review Request 26403: HIVE-8374: schematool fails on Postgres versions < 9.2
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/26403/ --- (Updated Oct. 9, 2014, 12:51 a.m.) Review request for hive. Changes --- Instead of having the user supply an option dependent on Postgres version, changed to the patch to automatically run a "pre-upgrade" script that creates the language, but any error is ignored. Kept the dbOpts option as that is useful when we add any db specific options. Bugs: HIVE-8374 https://issues.apache.org/jira/browse/HIVE-8374 Repository: hive-git Description (updated) --- HIVE-8374: schematool fails on Postgres versions < 9.2 The upgrade script for HIVE-5700 creates an UDF with language 'plpgsql', which is available by default only for Postgres 9.2+. For older Postgres versions, the language must be explicitly created, otherwise schematool fails. This patch allows for running "pre-upgrade" scripts before running a upgrade script. The errors from pre-upgrade scripts are ignored. This enables us to run statements like "create language" as a pre-upgrade step and ignore failures. This patch also adds a command line option to schematool to specify db options. This will be useful for adding future options that can conditionally execute certain statements. Diffs (updated) - beeline/src/java/org/apache/hive/beeline/HiveSchemaHelper.java a21fa652e9c1aa3fd763afa4136f6fcb7c7b7459 beeline/src/java/org/apache/hive/beeline/HiveSchemaTool.java c376687cb47332323912e4c6dbe713b7b37ae834 itests/hive-unit/src/test/java/org/apache/hive/beeline/TestSchemaTool.java 749fb9e86b4f74f768da356cf82f621fdef399cd metastore/scripts/upgrade/postgres/015-HIVE-5700.postgres.sql aedb2b5dbf9b513cb7c124f59e05ea9831455a72 metastore/scripts/upgrade/postgres/pre-0-upgrade-0.12.0-to-0.13.0.postgres.sql PRE-CREATION metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreSchemaInfo.java af56ef6a7c6e27ead312a68d30d94802095f2c60 Diff: https://reviews.apache.org/r/26403/diff/ Testing --- Added unit test Thanks, Mohit Sabharwal
[jira] [Commented] (HIVE-7953) Investigate query failures (2)
[ https://issues.apache.org/jira/browse/HIVE-7953?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14164475#comment-14164475 ] Xuefu Zhang commented on HIVE-7953: --- Hi [~tfriedr], thank you very much for the nice analysis. After you have gone thru all the tests in the set, please create individual JIRAs to track each issue that you find. Thanks. > Investigate query failures (2) > -- > > Key: HIVE-7953 > URL: https://issues.apache.org/jira/browse/HIVE-7953 > Project: Hive > Issue Type: Sub-task > Components: Spark >Reporter: Brock Noland >Assignee: Thomas Friedrich > > I ran all q-file tests and the following failed with an exception: > http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/HIVE-SPARK-ALL-TESTS-Build/lastCompletedBuild/testReport/ > we don't necessary want to run all these tests as part of the spark tests, > but we should understand why they failed with an exception. This JIRA is to > look into these failures and document them with one of: > * New JIRA > * Covered under existing JIRA > * More investigation required > Tests: > {noformat} > > org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_temp_table_external > 0.33 sec2 > > org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_bucket_num_reducers > 4.3 sec 2 > > org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_sortmerge_join_2 > 11 sec 2 > > org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_load_hdfs_file_with_space_in_the_name > 0.65 sec2 > > org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_bucketsortoptimize_insert_4 > 4.7 sec 2 > > org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_bucketsortoptimize_insert_7 > 2.8 sec 2 > > org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_bucketsortoptimize_insert_2 > 5.5 sec 2 > org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_groupby_position > 1.5 sec 2 > > org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_exim_18_part_external > 2.4 sec 2 > > org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_sortmerge_join_6 > 11 sec 2 > org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_smb_mapjoin_11 > 5.1 sec 2 > > org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_bucketsortoptimize_insert_8 > 10 sec 2 > org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_parquet_join > 5.4 sec 2 > > org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_stats_empty_dyn_part > 0.81 sec2 > > org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_dbtxnmgr_compact1 >0.31 sec2 > org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_dbtxnmgr_ddl1 > 0.26 sec2 > org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_dbtxnmgr_query2 > 0.73 sec2 > > org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_sortmerge_join_3 > 8.5 sec 2 > org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_dbtxnmgr_query5 > 0.34 sec2 > org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_rcfile_bigdata > 0.93 sec2 > > org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_groupby_multi_single_reducer > 6.3 sec 2 > > org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_dbtxnmgr_compact3 >2.4 sec 2 > > org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_dbtxnmgr_compact2 >0.56 sec2 > > org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_stats_partscan_1_23 > 3.1 sec 2 > > org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_list_bucket_dml_2 >4.3 sec 2 > > org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_exim_15_external_part > 3.2 sec 2 > > org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_exim_16_part_external > 2.8 sec 2 > > org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_exim_17_part_managed > 3.4 sec 2 > > org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_exim_20_part_managed_location > 3.3 sec 2 > > org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_exim_19_00_part_external_location > 6.9 sec 2 > > org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_external_table_with_space_in_location_path > {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-8341) Transaction information in config file can grow excessively large
[ https://issues.apache.org/jira/browse/HIVE-8341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14164472#comment-14164472 ] Alan Gates commented on HIVE-8341: -- Do you have a simple query with a transform in it that shows the issue with the process builder? > Transaction information in config file can grow excessively large > - > > Key: HIVE-8341 > URL: https://issues.apache.org/jira/browse/HIVE-8341 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 0.14.0 >Reporter: Alan Gates >Assignee: Alan Gates >Priority: Critical > Attachments: HIVE-8341.patch > > > In our testing we have seen cases where the transaction list grows very > large. We need a more efficient way of communicating the list. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-8341) Transaction information in config file can grow excessively large
[ https://issues.apache.org/jira/browse/HIVE-8341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14164458#comment-14164458 ] Gunther Hagleitner commented on HIVE-8341: -- I'm not sure what values ProcEnv supports, but maybe base64 would be better? > Transaction information in config file can grow excessively large > - > > Key: HIVE-8341 > URL: https://issues.apache.org/jira/browse/HIVE-8341 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 0.14.0 >Reporter: Alan Gates >Assignee: Alan Gates >Priority: Critical > Attachments: HIVE-8341.patch > > > In our testing we have seen cases where the transaction list grows very > large. We need a more efficient way of communicating the list. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-8341) Transaction information in config file can grow excessively large
[ https://issues.apache.org/jira/browse/HIVE-8341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14164454#comment-14164454 ] Gunther Hagleitner commented on HIVE-8341: -- This made it worse, the string encoding you're doing doesn't work with the process builder: {noformat} Caused by: java.lang.IllegalArgumentException: Invalid environment variable value: at java.lang.ProcessEnvironment.validateValue(ProcessEnvironment.java:120) at java.lang.ProcessEnvironment.access$400(ProcessEnvironment.java:61) at java.lang.ProcessEnvironment$Value.valueOf(ProcessEnvironment.java:202) at java.lang.ProcessEnvironment$StringEnvironment.put(ProcessEnvironment.java:241) at java.lang.ProcessEnvironment$StringEnvironment.put(ProcessEnvironment.java:220) at org.apache.hadoop.hive.ql.exec.ScriptOperator.addJobConfToEnvironment(ScriptOperator.java:156) at org.apache.hadoop.hive.ql.exec.ScriptOperator.processOp(ScriptOperator.java:317) {noformat} > Transaction information in config file can grow excessively large > - > > Key: HIVE-8341 > URL: https://issues.apache.org/jira/browse/HIVE-8341 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 0.14.0 >Reporter: Alan Gates >Assignee: Alan Gates >Priority: Critical > Attachments: HIVE-8341.patch > > > In our testing we have seen cases where the transaction list grows very > large. We need a more efficient way of communicating the list. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8404) ColumnPruner doesnt prune columns from limit operator
[ https://issues.apache.org/jira/browse/HIVE-8404?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-8404: --- Resolution: Fixed Fix Version/s: 0.15.0 Status: Resolved (was: Patch Available) Failures are not related to patch. Also, ran those tests and cant repro them on top of patch. Committed to trunk. > ColumnPruner doesnt prune columns from limit operator > - > > Key: HIVE-8404 > URL: https://issues.apache.org/jira/browse/HIVE-8404 > Project: Hive > Issue Type: Bug > Components: Logical Optimizer >Affects Versions: 0.12.0, 0.13.0, 0.13.1 >Reporter: Ashutosh Chauhan >Assignee: Ashutosh Chauhan > Fix For: 0.15.0 > > Attachments: HIVE-8404.1.patch, HIVE-8404.patch > > > Discovered while working on HIVE-8358 input26.q failure -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-7880) Support subquery [Spark Branch]
[ https://issues.apache.org/jira/browse/HIVE-7880?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuefu Zhang updated HIVE-7880: -- Resolution: Fixed Fix Version/s: spark-branch Status: Resolved (was: Patch Available) Patch committed to Spark branch. > Support subquery [Spark Branch] > --- > > Key: HIVE-7880 > URL: https://issues.apache.org/jira/browse/HIVE-7880 > Project: Hive > Issue Type: Sub-task > Components: Spark >Reporter: Chengxiang Li >Assignee: Xuefu Zhang > Labels: Spark-M2 > Fix For: spark-branch > > Attachments: HIVE-7880.1-spark.patch > > > While try to enable SubQuery qtests, I found that SubQuery cases return null > value currently, we should enable subquery for Hive on Spark. We should > enable subquery_exists.q and subquery_in.q in this task as Tez does. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8386) HCAT api call is case sensitive on fields in struct column
[ https://issues.apache.org/jira/browse/HIVE-8386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Dere updated HIVE-8386: - Attachment: HIVE-8386.2.patch Re-attaching patch for pre-commit tests > HCAT api call is case sensitive on fields in struct column > -- > > Key: HIVE-8386 > URL: https://issues.apache.org/jira/browse/HIVE-8386 > Project: Hive > Issue Type: Bug > Components: HCatalog, Types >Reporter: Jason Dere >Assignee: Jason Dere > Attachments: HIVE-8386.1.patch, HIVE-8386.2.patch > > > Falcon using hcat api to verify the target table schema and getting the error: > {noformat} > 2014-10-07 00:30:23,255 ERROR - > [1972803970@qtp-1214921164-3:gfoetl:POST//entities/submitAndSchedule/feed > a0c221e3-efa8-4235-a403-b1047f23ec05] ~ Failure reason (FalconWebException:40) > java.lang.RuntimeException: cannot find field givenName(lowercase form: > givenname) in [givenName, surname, middleName, gender, age, > isGivenNameLowerCase, isGivenNameUpperCase, isPrimary, isSurnameLowerCase, > isSurnameUpperCase] > at > org.apache.hadoop.hive.serde2.typeinfo.StructTypeInfo.getStructFieldTypeInfo(StructTypeInfo.java:109) > at > org.apache.hive.hcatalog.data.schema.HCatSchemaUtils.constructHCatSchema(HCatSchemaUtils.java:154) > at > org.apache.hive.hcatalog.data.schema.HCatSchemaUtils.getHCatSchema(HCatSchemaUtils.java:165) > at > org.apache.hive.hcatalog.data.schema.HCatSchemaUtils.getHCatFieldSchema(HCatSchemaUtils.java:127) > at > org.apache.hive.hcatalog.data.schema.HCatSchemaUtils.getHCatFieldSchema(HCatSchemaUtils.java:115) > at org.apache.hive.hcatalog.api.HCatTable.(HCatTable.java:59) > at > org.apache.hive.hcatalog.api.HCatClientHMSImpl.getTable(HCatClientHMSImpl.java:157) > at > org.apache.falcon.catalog.HiveCatalogService.tableExists(HiveCatalogService.java:143) > at > org.apache.falcon.entity.parser.FeedEntityParser.validateStorageExists(FeedEntityParser.java:367) > at > org.apache.falcon.entity.parser.FeedEntityParser.validateFeedStorage(FeedEntityParser.java:309) > at > org.apache.falcon.entity.parser.FeedEntityParser.validate(FeedEntityParser.java:79) > at > org.apache.falcon.entity.parser.FeedEntityParser.validate(FeedEntityParser.java:54) > at > org.apache.falcon.resource.AbstractEntityManager.validate(AbstractEntityManager.java:364) > at > org.apache.falcon.resource.AbstractEntityManager.submitInternal(AbstractEntityManager.java:331) > at > org.apache.falcon.resource.AbstractEntityManager.submit(AbstractEntityManager.java:153) > at > org.apache.falcon.resource.ConfigSyncService.submit(ConfigSyncService.java:44) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at > org.apache.falcon.resource.channel.IPCChannel.invoke(IPCChannel.java:48) > at > org.apache.falcon.resource.proxy.SchedulableEntityManagerProxy$1.doExecute(SchedulableEntityManagerProxy.java:118) > at > org.apache.falcon.resource.proxy.SchedulableEntityManagerProxy$EntityProxy.execute(SchedulableEntityManagerProxy.java:410) > at > org.apache.falcon.resource.proxy.SchedulableEntityManagerProxy.submit_aroundBody0(SchedulableEntityManagerProxy.java:120) > at > org.apache.falcon.resource.proxy.SchedulableEntityManagerProxy$AjcClosure1.run(SchedulableEntityManagerProxy.java:1) > at > org.aspectj.runtime.reflect.JoinPointImpl.proceed(JoinPointImpl.java:149) > at > org.apache.falcon.aspect.AbstractFalconAspect.logAround(AbstractFalconAspect.java:50) > at > org.apache.falcon.resource.proxy.SchedulableEntityManagerProxy.submit(SchedulableEntityManagerProxy.java:107) > at > org.apache.falcon.resource.proxy.SchedulableEntityManagerProxy.submit_aroundBody12(SchedulableEntityManagerProxy.java:341) > at > org.apache.falcon.resource.proxy.SchedulableEntityManagerProxy$AjcClosure13.run(SchedulableEntityManagerProxy.java:1) > at > org.aspectj.runtime.reflect.JoinPointImpl.proceed(JoinPointImpl.java:149) > at > org.apache.falcon.aspect.AbstractFalconAspect.logAround(AbstractFalconAspect.java:50) > at > org.apache.falcon.resource.proxy.SchedulableEntityManagerProxy.submitAndSchedule_aroundBody16(SchedulableEntityManagerProxy.java:341) > at > org.apache.falcon.resource.proxy.SchedulableEntityManagerProxy$AjcClosure17.run(SchedulableEntityManagerProxy.java:1) > at > org.aspectj.runtime.reflect.JoinPointImpl.proceed(JoinPointImpl.java:149) > at > org.apac
[jira] [Commented] (HIVE-8404) ColumnPruner doesnt prune columns from limit operator
[ https://issues.apache.org/jira/browse/HIVE-8404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14164421#comment-14164421 ] Hive QA commented on HIVE-8404: --- {color:red}Overall{color}: -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12673694/HIVE-8404.1.patch {color:red}ERROR:{color} -1 due to 7 failed/errored test(s), 6528 tests executed *Failed tests:* {noformat} org.apache.hadoop.hive.ql.txn.compactor.TestCompactor.testStatsAfterCompactionPartTbl org.apache.hive.jdbc.TestJdbcWithMiniHS2.testConnection org.apache.hive.jdbc.TestJdbcWithMiniHS2.testConnectionSchemaAPIs org.apache.hive.jdbc.TestJdbcWithMiniHS2.testNewConnectionConfiguration org.apache.hive.jdbc.TestJdbcWithMiniHS2.testRootScratchDir org.apache.hive.jdbc.TestJdbcWithMiniHS2.testSessionScratchDirs org.apache.hive.jdbc.TestJdbcWithMiniHS2.testURIDatabaseName {noformat} Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1181/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1181/console Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-1181/ Messages: {noformat} Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 7 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12673694 > ColumnPruner doesnt prune columns from limit operator > - > > Key: HIVE-8404 > URL: https://issues.apache.org/jira/browse/HIVE-8404 > Project: Hive > Issue Type: Bug > Components: Logical Optimizer >Affects Versions: 0.12.0, 0.13.0, 0.13.1 >Reporter: Ashutosh Chauhan >Assignee: Ashutosh Chauhan > Attachments: HIVE-8404.1.patch, HIVE-8404.patch > > > Discovered while working on HIVE-8358 input26.q failure -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-7880) Support subquery [Spark Branch]
[ https://issues.apache.org/jira/browse/HIVE-7880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14164411#comment-14164411 ] Hive QA commented on HIVE-7880: --- {color:red}Overall{color}: -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12673728/HIVE-7880.1-spark.patch {color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 6755 tests executed *Failed tests:* {noformat} org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_sample_islocalmode_hook org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_parallel {noformat} Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/197/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/197/console Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-SPARK-Build-197/ Messages: {noformat} Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 2 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12673728 > Support subquery [Spark Branch] > --- > > Key: HIVE-7880 > URL: https://issues.apache.org/jira/browse/HIVE-7880 > Project: Hive > Issue Type: Sub-task > Components: Spark >Reporter: Chengxiang Li >Assignee: Xuefu Zhang > Labels: Spark-M2 > Attachments: HIVE-7880.1-spark.patch > > > While try to enable SubQuery qtests, I found that SubQuery cases return null > value currently, we should enable subquery for Hive on Spark. We should > enable subquery_exists.q and subquery_in.q in this task as Tez does. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8363) AccumuloStorageHandler compile failure hadoop-1
[ https://issues.apache.org/jira/browse/HIVE-8363?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szehon Ho updated HIVE-8363: Resolution: Fixed Status: Resolved (was: Patch Available) Committed to 0.14 branch, thanks guys. > AccumuloStorageHandler compile failure hadoop-1 > --- > > Key: HIVE-8363 > URL: https://issues.apache.org/jira/browse/HIVE-8363 > Project: Hive > Issue Type: Bug > Components: StorageHandler >Reporter: Szehon Ho >Assignee: Josh Elser >Priority: Blocker > Fix For: 0.14.0 > > Attachments: HIVE-8363.1.patch > > > There's an error about AccumuloStorageHandler compiling on hadoop-1. It > seems the signature of split() is not the same. Looks like we can should use > another utils to fix this. > {code} > [ERROR] Failed to execute goal > org.apache.maven.plugins:maven-compiler-plugin:3.1:compile (default-compile) > on project hive-accumulo-handler: Compilation failure > [ERROR] > /data/hive-ptest/working/apache-svn-trunk-source/accumulo-handler/src/java/org/apache/hadoop/hive/accumulo/columns/ColumnMapper.java:[57,52] > no suitable method found for split(java.lang.String,char) > [ERROR] method > org.apache.hadoop.util.StringUtils.split(java.lang.String,char,char) is not > applicable > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-7953) Investigate query failures (2)
[ https://issues.apache.org/jira/browse/HIVE-7953?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14164391#comment-14164391 ] Thomas Friedrich commented on HIVE-7953: More details for auto_sortmerge_join_2 and auto_sortmerge_join_3: They fail with NPE in SMBJoinDesc: filterMap is null: filterMap = compactFilter(filterMap); Map result = new LinkedHashMap(); for (int i = 0 ; i < filterMap.length; i++) { if (filterMap[i] == null) { continue; } Thread [main] (Suspended (exception NullPointerException)) SMBJoinDesc(JoinDesc).toCompactString(int[][]) line: 482 SMBJoinDesc(JoinDesc).getFilterMapString() line: 473 NativeMethodAccessorImpl.invoke0(Method, Object, Object[]) line: not available [native method] NativeMethodAccessorImpl.invoke(Object, Object[]) line: 57 DelegatingMethodAccessorImpl.invoke(Object, Object[]) line: 43 Method.invoke(Object, Object...) line: 606 ExplainTask.outputPlan(Serializable, PrintStream, boolean, boolean, int, String) line: 622 > Investigate query failures (2) > -- > > Key: HIVE-7953 > URL: https://issues.apache.org/jira/browse/HIVE-7953 > Project: Hive > Issue Type: Sub-task > Components: Spark >Reporter: Brock Noland >Assignee: Thomas Friedrich > > I ran all q-file tests and the following failed with an exception: > http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/HIVE-SPARK-ALL-TESTS-Build/lastCompletedBuild/testReport/ > we don't necessary want to run all these tests as part of the spark tests, > but we should understand why they failed with an exception. This JIRA is to > look into these failures and document them with one of: > * New JIRA > * Covered under existing JIRA > * More investigation required > Tests: > {noformat} > > org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_temp_table_external > 0.33 sec2 > > org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_bucket_num_reducers > 4.3 sec 2 > > org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_sortmerge_join_2 > 11 sec 2 > > org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_load_hdfs_file_with_space_in_the_name > 0.65 sec2 > > org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_bucketsortoptimize_insert_4 > 4.7 sec 2 > > org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_bucketsortoptimize_insert_7 > 2.8 sec 2 > > org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_bucketsortoptimize_insert_2 > 5.5 sec 2 > org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_groupby_position > 1.5 sec 2 > > org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_exim_18_part_external > 2.4 sec 2 > > org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_sortmerge_join_6 > 11 sec 2 > org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_smb_mapjoin_11 > 5.1 sec 2 > > org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_bucketsortoptimize_insert_8 > 10 sec 2 > org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_parquet_join > 5.4 sec 2 > > org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_stats_empty_dyn_part > 0.81 sec2 > > org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_dbtxnmgr_compact1 >0.31 sec2 > org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_dbtxnmgr_ddl1 > 0.26 sec2 > org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_dbtxnmgr_query2 > 0.73 sec2 > > org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_sortmerge_join_3 > 8.5 sec 2 > org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_dbtxnmgr_query5 > 0.34 sec2 > org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_rcfile_bigdata > 0.93 sec2 > > org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_groupby_multi_single_reducer > 6.3 sec 2 > > org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_dbtxnmgr_compact3 >2.4 sec 2 > > org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_dbtxnmgr_compact2 >0.56 sec2 > > org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_stats_partscan_1_23 > 3.1 sec 2 > > org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_list_bucket_dml_2 >4.3 sec 2 > > org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_exim_15_external_part > 3.2 sec 2 > > org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_exim_16_part_external > 2.8 sec 2 > > org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_exim_17_part_managed > 3.4 sec 2 >
[jira] [Updated] (HIVE-6669) sourcing txn-script from schema script results in failure for mysql & oracle
[ https://issues.apache.org/jira/browse/HIVE-6669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alan Gates updated HIVE-6669: - Attachment: HIVE-6669.2.patch A new version of the patch that quotes postgres table and field names. > sourcing txn-script from schema script results in failure for mysql & oracle > > > Key: HIVE-6669 > URL: https://issues.apache.org/jira/browse/HIVE-6669 > Project: Hive > Issue Type: Bug > Components: Metastore >Affects Versions: 0.14.0 >Reporter: Prasad Mujumdar >Assignee: Alan Gates >Priority: Blocker > Attachments: HIVE-6669.2.patch, HIVE-6669.patch > > > This issues is addressed in 0.13 by in-lining the the transaction schema > statements in the schema initialization script (HIVE-6559) > The 0.14 schema initialization is not fixed. This is the followup ticket for > to address the problem in 0.14. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8344) Hive on Tez sets mapreduce.framework.name to yarn-tez
[ https://issues.apache.org/jira/browse/HIVE-8344?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gunther Hagleitner updated HIVE-8344: - Resolution: Fixed Fix Version/s: 0.14.0 Status: Resolved (was: Patch Available) Committed to trunk and hive .14 > Hive on Tez sets mapreduce.framework.name to yarn-tez > - > > Key: HIVE-8344 > URL: https://issues.apache.org/jira/browse/HIVE-8344 > Project: Hive > Issue Type: Bug >Reporter: Gunther Hagleitner >Assignee: Gunther Hagleitner > Fix For: 0.14.0 > > Attachments: HIVE-8344.1.patch, HIVE-8344.2.patch, HIVE-8344.3.patch > > > This was done to run MR jobs when in Tez mode (emulate MR on Tez). However, > we don't switch back when the user specifies MR as exec engine. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8258) Compactor cleaners can be starved on a busy table or partition.
[ https://issues.apache.org/jira/browse/HIVE-8258?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alan Gates updated HIVE-8258: - Attachment: HIVE-8258.5.patch Rebased patch. > Compactor cleaners can be starved on a busy table or partition. > --- > > Key: HIVE-8258 > URL: https://issues.apache.org/jira/browse/HIVE-8258 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 0.13.1 >Reporter: Alan Gates >Assignee: Alan Gates >Priority: Critical > Fix For: 0.14.0 > > Attachments: HIVE-8258.2.patch, HIVE-8258.3.patch, HIVE-8258.4.patch, > HIVE-8258.5.patch, HIVE-8258.patch > > > Currently the cleaning thread in the compactor does not run on a table or > partition while any locks are held on this partition. This leaves it open to > starvation in the case of a busy table or partition. It only needs to wait > until all locks on the table/partition at the time of the compaction have > expired. Any jobs initiated after that (and thus any locks obtained) will be > for the new versions of the files. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8258) Compactor cleaners can be starved on a busy table or partition.
[ https://issues.apache.org/jira/browse/HIVE-8258?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alan Gates updated HIVE-8258: - Status: Patch Available (was: Open) > Compactor cleaners can be starved on a busy table or partition. > --- > > Key: HIVE-8258 > URL: https://issues.apache.org/jira/browse/HIVE-8258 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 0.13.1 >Reporter: Alan Gates >Assignee: Alan Gates >Priority: Critical > Fix For: 0.14.0 > > Attachments: HIVE-8258.2.patch, HIVE-8258.3.patch, HIVE-8258.4.patch, > HIVE-8258.5.patch, HIVE-8258.patch > > > Currently the cleaning thread in the compactor does not run on a table or > partition while any locks are held on this partition. This leaves it open to > starvation in the case of a busy table or partition. It only needs to wait > until all locks on the table/partition at the time of the compaction have > expired. Any jobs initiated after that (and thus any locks obtained) will be > for the new versions of the files. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8258) Compactor cleaners can be starved on a busy table or partition.
[ https://issues.apache.org/jira/browse/HIVE-8258?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alan Gates updated HIVE-8258: - Status: Open (was: Patch Available) > Compactor cleaners can be starved on a busy table or partition. > --- > > Key: HIVE-8258 > URL: https://issues.apache.org/jira/browse/HIVE-8258 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 0.13.1 >Reporter: Alan Gates >Assignee: Alan Gates >Priority: Critical > Fix For: 0.14.0 > > Attachments: HIVE-8258.2.patch, HIVE-8258.3.patch, HIVE-8258.4.patch, > HIVE-8258.patch > > > Currently the cleaning thread in the compactor does not run on a table or > partition while any locks are held on this partition. This leaves it open to > starvation in the case of a busy table or partition. It only needs to wait > until all locks on the table/partition at the time of the compaction have > expired. Any jobs initiated after that (and thus any locks obtained) will be > for the new versions of the files. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8367) delete writes records in wrong order in some cases
[ https://issues.apache.org/jira/browse/HIVE-8367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alan Gates updated HIVE-8367: - Attachment: HIVE-8367.2.patch Rebased version of the patch. > delete writes records in wrong order in some cases > -- > > Key: HIVE-8367 > URL: https://issues.apache.org/jira/browse/HIVE-8367 > Project: Hive > Issue Type: Bug > Components: Query Processor >Affects Versions: 0.14.0 >Reporter: Alan Gates >Assignee: Alan Gates >Priority: Blocker > Fix For: 0.14.0 > > Attachments: HIVE-8367.2.patch, HIVE-8367.patch > > > I have found one query with 10k records where you do: > create table > insert into table -- 10k records > delete from table -- just some records > The records in the delete delta are not ordered properly by rowid. > I assume this applies to updates as well, but I haven't tested it yet. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-8408) hcat cli throws NPE when authorizer using new api is enabled
Thejas M Nair created HIVE-8408: --- Summary: hcat cli throws NPE when authorizer using new api is enabled Key: HIVE-8408 URL: https://issues.apache.org/jira/browse/HIVE-8408 Project: Hive Issue Type: Bug Reporter: Thejas M Nair Assignee: Thejas M Nair Priority: Critical Fix For: 0.14.0 Hcat cli tries to authorize actions using the authorizer configured for hive-cli. When authorizer based on new authorization interface is used, it throws a NPE as it checks only for the old api based authorization implementation. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-8408) hcat cli throws NPE when authorizer using new api is enabled
[ https://issues.apache.org/jira/browse/HIVE-8408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14164362#comment-14164362 ] Thejas M Nair commented on HIVE-8408: - The stack trace - {noformat} ERROR [main]: ql.Driver (SessionState.java:printError(825)) - FAILED: SemanticException org.apache.hadoop.hive.ql.parse.SemanticException: java.lang.NullPointerException org.apache.hadoop.hive.ql.parse.SemanticException: org.apache.hadoop.hive.ql.parse.SemanticException: java.lang.NullPointerException at org.apache.hive.hcatalog.cli.SemanticAnalysis.HCatSemanticAnalyzer.postAnalyze(HCatSemanticAnalyzer.java:266) at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:418) at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:303) at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1067) at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1129) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1004) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:994) at org.apache.hive.hcatalog.cli.HCatDriver.run(HCatDriver.java:43) at org.apache.hive.hcatalog.cli.HCatCli.processCmd(HCatCli.java:291) at org.apache.hive.hcatalog.cli.HCatCli.processLine(HCatCli.java:245) at org.apache.hive.hcatalog.cli.HCatCli.main(HCatCli.java:183) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.util.RunJar.run(RunJar.java:221) at org.apache.hadoop.util.RunJar.main(RunJar.java:136) Caused by: org.apache.hadoop.hive.ql.parse.SemanticException: java.lang.NullPointerException at org.apache.hive.hcatalog.cli.SemanticAnalysis.HCatSemanticAnalyzerBase.authorizeDDL(HCatSemanticAnalyzerBase.java:114) at org.apache.hive.hcatalog.cli.SemanticAnalysis.HCatSemanticAnalyzer.postAnalyze(HCatSemanticAnalyzer.java:261) ... 16 more Caused by: java.lang.NullPointerException at org.apache.hive.hcatalog.cli.SemanticAnalysis.HCatSemanticAnalyzerBase.authorize(HCatSemanticAnalyzerBase.java:139) at org.apache.hive.hcatalog.cli.SemanticAnalysis.HCatSemanticAnalyzer.authorizeDDLWork(HCatSemanticAnalyzer.java:324) at org.apache.hive.hcatalog.cli.SemanticAnalysis.HCatSemanticAnalyzerBase.authorizeDDL(HCatSemanticAnalyzerBase.java:105) ... 17 more {noformat} > hcat cli throws NPE when authorizer using new api is enabled > > > Key: HIVE-8408 > URL: https://issues.apache.org/jira/browse/HIVE-8408 > Project: Hive > Issue Type: Bug >Reporter: Thejas M Nair >Assignee: Thejas M Nair >Priority: Critical > Fix For: 0.14.0 > > > Hcat cli tries to authorize actions using the authorizer configured for > hive-cli. When authorizer based on new authorization interface is used, it > throws a NPE as it checks only for the old api based authorization > implementation. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8408) hcat cli throws NPE when authorizer using new api is enabled
[ https://issues.apache.org/jira/browse/HIVE-8408?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thejas M Nair updated HIVE-8408: Reporter: Jagruti Varia (was: Thejas M Nair) > hcat cli throws NPE when authorizer using new api is enabled > > > Key: HIVE-8408 > URL: https://issues.apache.org/jira/browse/HIVE-8408 > Project: Hive > Issue Type: Bug >Reporter: Jagruti Varia >Assignee: Thejas M Nair >Priority: Critical > Fix For: 0.14.0 > > > Hcat cli tries to authorize actions using the authorizer configured for > hive-cli. When authorizer based on new authorization interface is used, it > throws a NPE as it checks only for the old api based authorization > implementation. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8367) delete writes records in wrong order in some cases
[ https://issues.apache.org/jira/browse/HIVE-8367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alan Gates updated HIVE-8367: - Status: Patch Available (was: Open) > delete writes records in wrong order in some cases > -- > > Key: HIVE-8367 > URL: https://issues.apache.org/jira/browse/HIVE-8367 > Project: Hive > Issue Type: Bug > Components: Query Processor >Affects Versions: 0.14.0 >Reporter: Alan Gates >Assignee: Alan Gates >Priority: Blocker > Fix For: 0.14.0 > > Attachments: HIVE-8367.2.patch, HIVE-8367.patch > > > I have found one query with 10k records where you do: > create table > insert into table -- 10k records > delete from table -- just some records > The records in the delete delta are not ordered properly by rowid. > I assume this applies to updates as well, but I haven't tested it yet. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8367) delete writes records in wrong order in some cases
[ https://issues.apache.org/jira/browse/HIVE-8367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alan Gates updated HIVE-8367: - Status: Open (was: Patch Available) > delete writes records in wrong order in some cases > -- > > Key: HIVE-8367 > URL: https://issues.apache.org/jira/browse/HIVE-8367 > Project: Hive > Issue Type: Bug > Components: Query Processor >Affects Versions: 0.14.0 >Reporter: Alan Gates >Assignee: Alan Gates >Priority: Blocker > Fix For: 0.14.0 > > Attachments: HIVE-8367.patch > > > I have found one query with 10k records where you do: > create table > insert into table -- 10k records > delete from table -- just some records > The records in the delete delta are not ordered properly by rowid. > I assume this applies to updates as well, but I haven't tested it yet. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8368) compactor is improperly writing delete records in base file
[ https://issues.apache.org/jira/browse/HIVE-8368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alan Gates updated HIVE-8368: - Status: Patch Available (was: Open) > compactor is improperly writing delete records in base file > --- > > Key: HIVE-8368 > URL: https://issues.apache.org/jira/browse/HIVE-8368 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 0.14.0 >Reporter: Alan Gates >Assignee: Alan Gates >Priority: Critical > Fix For: 0.14.0 > > Attachments: HIVE-8368.2.patch, HIVE-8368.patch > > > When the compactor reads records from the base and deltas, it is not properly > dropping delete records. This leads to oversized base files, and possibly to > wrong query results. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8368) compactor is improperly writing delete records in base file
[ https://issues.apache.org/jira/browse/HIVE-8368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alan Gates updated HIVE-8368: - Attachment: HIVE-8368.2.patch Rebased version of the patch. > compactor is improperly writing delete records in base file > --- > > Key: HIVE-8368 > URL: https://issues.apache.org/jira/browse/HIVE-8368 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 0.14.0 >Reporter: Alan Gates >Assignee: Alan Gates >Priority: Critical > Fix For: 0.14.0 > > Attachments: HIVE-8368.2.patch, HIVE-8368.patch > > > When the compactor reads records from the base and deltas, it is not properly > dropping delete records. This leads to oversized base files, and possibly to > wrong query results. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8368) compactor is improperly writing delete records in base file
[ https://issues.apache.org/jira/browse/HIVE-8368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alan Gates updated HIVE-8368: - Status: Open (was: Patch Available) > compactor is improperly writing delete records in base file > --- > > Key: HIVE-8368 > URL: https://issues.apache.org/jira/browse/HIVE-8368 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 0.14.0 >Reporter: Alan Gates >Assignee: Alan Gates >Priority: Critical > Fix For: 0.14.0 > > Attachments: HIVE-8368.patch > > > When the compactor reads records from the base and deltas, it is not properly > dropping delete records. This leads to oversized base files, and possibly to > wrong query results. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8402) Orc pushing SARGs into delta files causing ArrayOutOfBoundsExceptions
[ https://issues.apache.org/jira/browse/HIVE-8402?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alan Gates updated HIVE-8402: - Status: Patch Available (was: Open) > Orc pushing SARGs into delta files causing ArrayOutOfBoundsExceptions > - > > Key: HIVE-8402 > URL: https://issues.apache.org/jira/browse/HIVE-8402 > Project: Hive > Issue Type: Bug > Components: File Formats, Transactions >Affects Versions: 0.14.0 >Reporter: Alan Gates >Assignee: Alan Gates >Priority: Blocker > Fix For: 0.14.0 > > Attachments: HIVE-8402.patch > > > ORC is in some instances pushing SARGs into delta files. This is wrong > behavior in general as it may result in failing to pull the most recent > version of a row. When the SARG is applied to a row that is deleted it > causes an ArrayOutOfBoundsException because there is no data in the row. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8402) Orc pushing SARGs into delta files causing ArrayOutOfBoundsExceptions
[ https://issues.apache.org/jira/browse/HIVE-8402?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alan Gates updated HIVE-8402: - Attachment: HIVE-8402.patch A patch to change orc to not push sargs into the deltas. And to answer my earlier unknown, this did only happen when a base was also present. When there was no base file the sarg was not being written into the options passed to OrcRawRecordMerge (see OrcInputFormat.getReader, around line 1121). > Orc pushing SARGs into delta files causing ArrayOutOfBoundsExceptions > - > > Key: HIVE-8402 > URL: https://issues.apache.org/jira/browse/HIVE-8402 > Project: Hive > Issue Type: Bug > Components: File Formats, Transactions >Affects Versions: 0.14.0 >Reporter: Alan Gates >Assignee: Alan Gates >Priority: Blocker > Fix For: 0.14.0 > > Attachments: HIVE-8402.patch > > > ORC is in some instances pushing SARGs into delta files. This is wrong > behavior in general as it may result in failing to pull the most recent > version of a row. When the SARG is applied to a row that is deleted it > causes an ArrayOutOfBoundsException because there is no data in the row. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8393) Handle SIGINT on Tez
[ https://issues.apache.org/jira/browse/HIVE-8393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gunther Hagleitner updated HIVE-8393: - Resolution: Fixed Status: Resolved (was: Patch Available) Committed to trunk and hive .14 > Handle SIGINT on Tez > > > Key: HIVE-8393 > URL: https://issues.apache.org/jira/browse/HIVE-8393 > Project: Hive > Issue Type: Bug >Reporter: Gunther Hagleitner >Assignee: Gunther Hagleitner > Attachments: HIVE-8393.1.patch, HIVE-8393.2.patch > > > The signal handler in hive currently only interrupts MR jobs. The same logic > can be used to interrupt Tez dags as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8393) Handle SIGINT on Tez
[ https://issues.apache.org/jira/browse/HIVE-8393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gunther Hagleitner updated HIVE-8393: - Fix Version/s: 0.14.0 > Handle SIGINT on Tez > > > Key: HIVE-8393 > URL: https://issues.apache.org/jira/browse/HIVE-8393 > Project: Hive > Issue Type: Bug >Reporter: Gunther Hagleitner >Assignee: Gunther Hagleitner > Fix For: 0.14.0 > > Attachments: HIVE-8393.1.patch, HIVE-8393.2.patch > > > The signal handler in hive currently only interrupts MR jobs. The same logic > can be used to interrupt Tez dags as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-8367) delete writes records in wrong order in some cases
[ https://issues.apache.org/jira/browse/HIVE-8367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14164332#comment-14164332 ] Alan Gates commented on HIVE-8367: -- bq. What was the original query where the issue showed up? {code} create table concur_orc_tab(name varchar(50), age int, gpa decimal(3, 2)) clustered by (age) into 2 buckets stored as orc TBLPROPERTIES ('transactional'='true'); insert into table concur_orc_tab select * from texttab; -- loads 10k records into the table delete from concur_orc_tab where age >= 20 and age < 30; {code} This resulted in only some rows being deleted (~300 of the 1700 that should have been deleted) What precisely was the problem and how does the RS deduplication change help? The problem was that because the code was turning off the RS deduplication it was getting a plan with two MR jobs. The sort by ROW__ID was done in job one, and the bucketing was done in job two. This meant that the bucketing in job 2 partially undid the sorting of job 1, resulting in only some of the records showing up as deleted (since the records have to be written in the delta file in proper order). The minimum number of reducers on which to apply the RS deduplication is pushed to 1 so that this optimization is used for even small queries. How is the changes to sort order of ROW__ID related? That should never have been set to descending in the first place. ROW__ID needs to be stored ascending to work properly. I suspect it was a fluke of most of the qfile tests that they worked with this on. (Actually Thejas asked at the time why this was necessary and rather than fixing it (which I should have done) I just said I didn't know. Oops.) bq. ReduceSinkDeDuplication.java change is not needed What change? I don't see any changes to that file in the patch. > delete writes records in wrong order in some cases > -- > > Key: HIVE-8367 > URL: https://issues.apache.org/jira/browse/HIVE-8367 > Project: Hive > Issue Type: Bug > Components: Query Processor >Affects Versions: 0.14.0 >Reporter: Alan Gates >Assignee: Alan Gates >Priority: Blocker > Fix For: 0.14.0 > > Attachments: HIVE-8367.patch > > > I have found one query with 10k records where you do: > create table > insert into table -- 10k records > delete from table -- just some records > The records in the delete delta are not ordered properly by rowid. > I assume this applies to updates as well, but I haven't tested it yet. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8225) CBO trunk merge: union11 test fails due to incorrect plan
[ https://issues.apache.org/jira/browse/HIVE-8225?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-8225: -- Attachment: HIVE-8225.7.patch resubmit patch > CBO trunk merge: union11 test fails due to incorrect plan > - > > Key: HIVE-8225 > URL: https://issues.apache.org/jira/browse/HIVE-8225 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Pengcheng Xiong >Priority: Critical > Fix For: 0.14.0 > > Attachments: HIVE-8225.1.patch, HIVE-8225.2.patch, HIVE-8225.3.patch, > HIVE-8225.4.patch, HIVE-8225.5.patch, HIVE-8225.6.patch, HIVE-8225.7.patch, > HIVE-8225.7.patch, HIVE-8225.inprogress.patch, HIVE-8225.inprogress.patch, > HIVE-8225.patch > > > The result changes to as if the union didn't have count() inside. The issue > can be fixed by using srcunion.value outside the subquery in count (replace > count(1) with count(srcunion.value)). Otherwise, it looks like count(1) node > from union-ed queries is not present in AST at all, which might cause this > result. > -Interestingly, adding group by to each query in a union produces completely > weird result (count(1) is 309 for each key, whereas it should be 1 and the > "logical" incorrect value if internal count is lost is 500)- Nm, that groups > by table column called key, which is weird but is what Hive does -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8225) CBO trunk merge: union11 test fails due to incorrect plan
[ https://issues.apache.org/jira/browse/HIVE-8225?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-8225: -- Status: Open (was: Patch Available) > CBO trunk merge: union11 test fails due to incorrect plan > - > > Key: HIVE-8225 > URL: https://issues.apache.org/jira/browse/HIVE-8225 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Pengcheng Xiong >Priority: Critical > Fix For: 0.14.0 > > Attachments: HIVE-8225.1.patch, HIVE-8225.2.patch, HIVE-8225.3.patch, > HIVE-8225.4.patch, HIVE-8225.5.patch, HIVE-8225.6.patch, HIVE-8225.7.patch, > HIVE-8225.inprogress.patch, HIVE-8225.inprogress.patch, HIVE-8225.patch > > > The result changes to as if the union didn't have count() inside. The issue > can be fixed by using srcunion.value outside the subquery in count (replace > count(1) with count(srcunion.value)). Otherwise, it looks like count(1) node > from union-ed queries is not present in AST at all, which might cause this > result. > -Interestingly, adding group by to each query in a union produces completely > weird result (count(1) is 309 for each key, whereas it should be 1 and the > "logical" incorrect value if internal count is lost is 500)- Nm, that groups > by table column called key, which is weird but is what Hive does -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-7932) It may cause NP exception when add accessed columns to ReadEntity
[ https://issues.apache.org/jira/browse/HIVE-7932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14164320#comment-14164320 ] Prasad Mujumdar commented on HIVE-7932: --- +1 Looks fine to me. Thanks for adding the testcase. > It may cause NP exception when add accessed columns to ReadEntity > - > > Key: HIVE-7932 > URL: https://issues.apache.org/jira/browse/HIVE-7932 > Project: Hive > Issue Type: Bug >Reporter: Xiaomeng Huang >Assignee: Xiaomeng Huang > Attachments: HIVE-7932.001.patch, HIVE-7932.002.patch > > > {code} > case TABLE: >entity.getAccessedColumns().addAll( > tableToColumnAccessMap.get(entity.getTable().getCompleteName())); > {code} > if tableToColumnAccessMap.get(entity.getTable().getCompleteName()) is null, > addAll(null) will throw null pointer exception. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-8225) CBO trunk merge: union11 test fails due to incorrect plan
[ https://issues.apache.org/jira/browse/HIVE-8225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14164316#comment-14164316 ] Sergey Shelukhin commented on HIVE-8225: is it possible to resubmit patch for HiveQA? > CBO trunk merge: union11 test fails due to incorrect plan > - > > Key: HIVE-8225 > URL: https://issues.apache.org/jira/browse/HIVE-8225 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Critical > Fix For: 0.14.0 > > Attachments: HIVE-8225.1.patch, HIVE-8225.2.patch, HIVE-8225.3.patch, > HIVE-8225.4.patch, HIVE-8225.5.patch, HIVE-8225.6.patch, HIVE-8225.7.patch, > HIVE-8225.inprogress.patch, HIVE-8225.inprogress.patch, HIVE-8225.patch > > > The result changes to as if the union didn't have count() inside. The issue > can be fixed by using srcunion.value outside the subquery in count (replace > count(1) with count(srcunion.value)). Otherwise, it looks like count(1) node > from union-ed queries is not present in AST at all, which might cause this > result. > -Interestingly, adding group by to each query in a union produces completely > weird result (count(1) is 309 for each key, whereas it should be 1 and the > "logical" incorrect value if internal count is lost is 500)- Nm, that groups > by table column called key, which is weird but is what Hive does -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8225) CBO trunk merge: union11 test fails due to incorrect plan
[ https://issues.apache.org/jira/browse/HIVE-8225?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-8225: --- Assignee: Pengcheng Xiong (was: Sergey Shelukhin) > CBO trunk merge: union11 test fails due to incorrect plan > - > > Key: HIVE-8225 > URL: https://issues.apache.org/jira/browse/HIVE-8225 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Pengcheng Xiong >Priority: Critical > Fix For: 0.14.0 > > Attachments: HIVE-8225.1.patch, HIVE-8225.2.patch, HIVE-8225.3.patch, > HIVE-8225.4.patch, HIVE-8225.5.patch, HIVE-8225.6.patch, HIVE-8225.7.patch, > HIVE-8225.inprogress.patch, HIVE-8225.inprogress.patch, HIVE-8225.patch > > > The result changes to as if the union didn't have count() inside. The issue > can be fixed by using srcunion.value outside the subquery in count (replace > count(1) with count(srcunion.value)). Otherwise, it looks like count(1) node > from union-ed queries is not present in AST at all, which might cause this > result. > -Interestingly, adding group by to each query in a union produces completely > weird result (count(1) is 309 for each key, whereas it should be 1 and the > "logical" incorrect value if internal count is lost is 500)- Nm, that groups > by table column called key, which is weird but is what Hive does -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8391) Comparion between TIMESTAMP and Integer types goes to STRING as "common comparison denominator" instead of a numeric type
[ https://issues.apache.org/jira/browse/HIVE-8391?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-8391: --- Attachment: (was: vectorization_7.q) > Comparion between TIMESTAMP and Integer types goes to STRING as "common > comparison denominator" instead of a numeric type > - > > Key: HIVE-8391 > URL: https://issues.apache.org/jira/browse/HIVE-8391 > Project: Hive > Issue Type: Bug > Components: CBO, Logical Optimizer >Affects Versions: 0.14.0 >Reporter: Matt McCline >Assignee: Jason Dere >Priority: Critical > Fix For: 0.14.0 > > Attachments: HIVE-8391.1.patch, HIVE-8391.2.patch > > > Discovered while investigating why vectorization_7.q has different results > for non-vectorized [+CBO] and vectorized [+CBO]. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HIVE-7831) Research commented out unset in Utiltities [Spark Branch]
[ https://issues.apache.org/jira/browse/HIVE-7831?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuefu Zhang resolved HIVE-7831. --- Resolution: Not a Problem Fix Version/s: spark-branch > Research commented out unset in Utiltities [Spark Branch] > - > > Key: HIVE-7831 > URL: https://issues.apache.org/jira/browse/HIVE-7831 > Project: Hive > Issue Type: Sub-task > Components: Spark >Reporter: Brock Noland > Fix For: spark-branch > > > We did the following in HIVE-7370 > {noformat} > // TODO HIVE-7831 > // conf.unset(FsPermission.UMASK_LABEL); > {noformat} > We should understand that. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-7953) Investigate query failures (2)
[ https://issues.apache.org/jira/browse/HIVE-7953?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14164301#comment-14164301 ] Thomas Friedrich commented on HIVE-7953: The two test cases auto_sortmerge_join_2 auto_sortmerge_join_3 are from the minitez.query.files.shared list. Both fail with the same exception when running the statement select count(*) FROM bucket_big a JOIN bucket_small b ON a.key = b.key I debugged the code and they fail in Daemon Thread [Executor task launch worker-1] (Suspended (exception HiveException)) MapOperator.process(Writable) line: 546 SparkMapRecordHandler.processRow(Object, Object) line: 139 HiveMapFunctionResultList.processNextRecord(Tuple2) line: 47 HiveMapFunctionResultList.processNextRecord(Object) line: 28 HiveBaseFunctionResultList$ResultIterator.hasNext() line: 108 Wrappers$JIteratorWrapper.hasNext() line: 41 Iterator$class.foreach(Iterator, Function1) line: 727 Wrappers$JIteratorWrapper(AbstractIterator).foreach(Function1) line: 1157 RDD$$anonfun$foreach$1.apply(Iterator) line: 760 RDD$$anonfun$foreach$1.apply(Object) line: 760 SparkContext$$anonfun$runJob$3.apply(TaskContext, Iterator) line: 1118 SparkContext$$anonfun$runJob$3.apply(Object, Object) line: 1118 ResultTask.runTask(TaskContext) line: 61 ResultTask(Task).run(long) line: 56 Executor$TaskRunner.run() line: 182 ThreadPoolExecutor.runWorker(ThreadPoolExecutor$Worker) line: 1145 ThreadPoolExecutor$Worker.run() line: 615 Thread.run() line: 745 The SparkException is: org.apache.spark.SparkException: Job aborted due to stage failure: Task 1 in stage 26.0 failed 1 times, most recent failure: Lost task 1.0 in stage 26.0 (TID 27, localhost): java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row {"key":"100","value":"val_100","ds":"2008-04-08"} > Investigate query failures (2) > -- > > Key: HIVE-7953 > URL: https://issues.apache.org/jira/browse/HIVE-7953 > Project: Hive > Issue Type: Sub-task > Components: Spark >Reporter: Brock Noland >Assignee: Thomas Friedrich > > I ran all q-file tests and the following failed with an exception: > http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/HIVE-SPARK-ALL-TESTS-Build/lastCompletedBuild/testReport/ > we don't necessary want to run all these tests as part of the spark tests, > but we should understand why they failed with an exception. This JIRA is to > look into these failures and document them with one of: > * New JIRA > * Covered under existing JIRA > * More investigation required > Tests: > {noformat} > > org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_temp_table_external > 0.33 sec2 > > org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_bucket_num_reducers > 4.3 sec 2 > > org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_sortmerge_join_2 > 11 sec 2 > > org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_load_hdfs_file_with_space_in_the_name > 0.65 sec2 > > org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_bucketsortoptimize_insert_4 > 4.7 sec 2 > > org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_bucketsortoptimize_insert_7 > 2.8 sec 2 > > org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_bucketsortoptimize_insert_2 > 5.5 sec 2 > org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_groupby_position > 1.5 sec 2 > > org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_exim_18_part_external > 2.4 sec 2 > > org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_sortmerge_join_6 > 11 sec 2 > org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_smb_mapjoin_11 > 5.1 sec 2 > > org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_bucketsortoptimize_insert_8 > 10 sec 2 > org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_parquet_join > 5.4 sec 2 > > org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_stats_empty_dyn_part > 0.81 sec2 > > org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_dbtxnmgr_compact1 >0.31 sec2 > org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_dbtxnmgr_ddl1 > 0.26 sec2 > org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_dbtxnmgr_query2 > 0.73 sec2 > > org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_sortmerge_join_3 > 8.5 sec 2 > org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_dbtxnmgr_query5 > 0
[jira] [Commented] (HIVE-7831) Research commented out unset in Utiltities [Spark Branch]
[ https://issues.apache.org/jira/browse/HIVE-7831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14164302#comment-14164302 ] Xuefu Zhang commented on HIVE-7831: --- I don't see any commented out code any more. Since tests are passing w/o this change, I assume this change is not necessary. Closing this JIRA, therefore. > Research commented out unset in Utiltities [Spark Branch] > - > > Key: HIVE-7831 > URL: https://issues.apache.org/jira/browse/HIVE-7831 > Project: Hive > Issue Type: Sub-task > Components: Spark >Reporter: Brock Noland > Fix For: spark-branch > > > We did the following in HIVE-7370 > {noformat} > // TODO HIVE-7831 > // conf.unset(FsPermission.UMASK_LABEL); > {noformat} > We should understand that. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-8367) delete writes records in wrong order in some cases
[ https://issues.apache.org/jira/browse/HIVE-8367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14164303#comment-14164303 ] Eugene Koifman commented on HIVE-8367: -- also, ReduceSinkDeDuplication.java change is not needed > delete writes records in wrong order in some cases > -- > > Key: HIVE-8367 > URL: https://issues.apache.org/jira/browse/HIVE-8367 > Project: Hive > Issue Type: Bug > Components: Query Processor >Affects Versions: 0.14.0 >Reporter: Alan Gates >Assignee: Alan Gates >Priority: Blocker > Fix For: 0.14.0 > > Attachments: HIVE-8367.patch > > > I have found one query with 10k records where you do: > create table > insert into table -- 10k records > delete from table -- just some records > The records in the delete delta are not ordered properly by rowid. > I assume this applies to updates as well, but I haven't tested it yet. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-7953) Investigate query failures (2)
[ https://issues.apache.org/jira/browse/HIVE-7953?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14164299#comment-14164299 ] Thomas Friedrich commented on HIVE-7953: The following 4 tests are from the minimr.query.files list: temp_table_external load_hdfs_file_with_space_in_the_name external_table_with_space_in_location_path bucket_num_reducers The 3 tests temp_table_external load_hdfs_file_with_space_in_the_name external_table_with_space_in_location_path create dirs in HDFS, but TestSparkCliDriver is not using HDFS. That's why the tests fail when creating a dir, for example mkdir -p hdfs:///tmp/temp_table_external In the non-spark case, these tests are executed with the TestMinimrCliDriver which initializes Hadoop and created a mini-cluster (see configuration in itests/qtest/pom.xml). We may need to configure a TestSparkMinimrCliDriver in itests/qtest-spark/pom.xml if we want to run these test cases. The test bucket_num_reducers doesn't use HDFS, and fails with a different error: Exception: Number of MapReduce jobs is incorrect expected:<1> but was:<0> junit.framework.AssertionFailedError: Number of MapReduce jobs is incorrect expected:<1> but was:<0> at org.apache.hadoop.hive.ql.hooks.VerifyNumReducersHook.run(VerifyNumReducersHook.java:46) > Investigate query failures (2) > -- > > Key: HIVE-7953 > URL: https://issues.apache.org/jira/browse/HIVE-7953 > Project: Hive > Issue Type: Sub-task > Components: Spark >Reporter: Brock Noland >Assignee: Thomas Friedrich > > I ran all q-file tests and the following failed with an exception: > http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/HIVE-SPARK-ALL-TESTS-Build/lastCompletedBuild/testReport/ > we don't necessary want to run all these tests as part of the spark tests, > but we should understand why they failed with an exception. This JIRA is to > look into these failures and document them with one of: > * New JIRA > * Covered under existing JIRA > * More investigation required > Tests: > {noformat} > > org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_temp_table_external > 0.33 sec2 > > org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_bucket_num_reducers > 4.3 sec 2 > > org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_sortmerge_join_2 > 11 sec 2 > > org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_load_hdfs_file_with_space_in_the_name > 0.65 sec2 > > org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_bucketsortoptimize_insert_4 > 4.7 sec 2 > > org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_bucketsortoptimize_insert_7 > 2.8 sec 2 > > org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_bucketsortoptimize_insert_2 > 5.5 sec 2 > org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_groupby_position > 1.5 sec 2 > > org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_exim_18_part_external > 2.4 sec 2 > > org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_sortmerge_join_6 > 11 sec 2 > org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_smb_mapjoin_11 > 5.1 sec 2 > > org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_bucketsortoptimize_insert_8 > 10 sec 2 > org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_parquet_join > 5.4 sec 2 > > org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_stats_empty_dyn_part > 0.81 sec2 > > org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_dbtxnmgr_compact1 >0.31 sec2 > org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_dbtxnmgr_ddl1 > 0.26 sec2 > org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_dbtxnmgr_query2 > 0.73 sec2 > > org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_sortmerge_join_3 > 8.5 sec 2 > org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_dbtxnmgr_query5 > 0.34 sec2 > org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_rcfile_bigdata > 0.93 sec2 > > org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_groupby_multi_single_reducer > 6.3 sec 2 > > org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_dbtxnmgr_compact3 >2.4 sec 2 > > org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_dbtxnmgr_compact2 >0.56 sec2 > > org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_stats_partscan_1_23 > 3.1 sec 2 > > org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_list_bucket_dml_2 >4.3 sec 2 > > org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_exim_15_external_part > 3.2 sec 2 > > org.apache.hadoop.hiv
[jira] [Commented] (HIVE-8367) delete writes records in wrong order in some cases
[ https://issues.apache.org/jira/browse/HIVE-8367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14164298#comment-14164298 ] Eugene Koifman commented on HIVE-8367: -- I think this needs more info. What was the original query where the issue showed up? What precisely was the problem and how does the RS deduplication change help? The explanation for the latter would be useful to add to the code where this setting is set. How is the changes to sort order of ROW__ID related? > delete writes records in wrong order in some cases > -- > > Key: HIVE-8367 > URL: https://issues.apache.org/jira/browse/HIVE-8367 > Project: Hive > Issue Type: Bug > Components: Query Processor >Affects Versions: 0.14.0 >Reporter: Alan Gates >Assignee: Alan Gates >Priority: Blocker > Fix For: 0.14.0 > > Attachments: HIVE-8367.patch > > > I have found one query with 10k records where you do: > create table > insert into table -- 10k records > delete from table -- just some records > The records in the delete delta are not ordered properly by rowid. > I assume this applies to updates as well, but I haven't tested it yet. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-7831) Research commented out unset in Utiltities [Spark Branch]
[ https://issues.apache.org/jira/browse/HIVE-7831?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuefu Zhang updated HIVE-7831: -- Summary: Research commented out unset in Utiltities [Spark Branch] (was: Research commented out unset in Utiltities) > Research commented out unset in Utiltities [Spark Branch] > - > > Key: HIVE-7831 > URL: https://issues.apache.org/jira/browse/HIVE-7831 > Project: Hive > Issue Type: Sub-task > Components: Spark >Reporter: Brock Noland > > We did the following in HIVE-7370 > {noformat} > // TODO HIVE-7831 > // conf.unset(FsPermission.UMASK_LABEL); > {noformat} > We should understand that. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8391) Comparion between TIMESTAMP and Integer types goes to STRING as "common comparison denominator" instead of a numeric type
[ https://issues.apache.org/jira/browse/HIVE-8391?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Dere updated HIVE-8391: - Status: Patch Available (was: Open) > Comparion between TIMESTAMP and Integer types goes to STRING as "common > comparison denominator" instead of a numeric type > - > > Key: HIVE-8391 > URL: https://issues.apache.org/jira/browse/HIVE-8391 > Project: Hive > Issue Type: Bug > Components: CBO, Logical Optimizer >Affects Versions: 0.14.0 >Reporter: Matt McCline >Assignee: Jason Dere >Priority: Critical > Fix For: 0.14.0 > > Attachments: HIVE-8391.1.patch, HIVE-8391.2.patch, vectorization_7.q > > > Discovered while investigating why vectorization_7.q has different results > for non-vectorized [+CBO] and vectorized [+CBO]. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8391) Comparion between TIMESTAMP and Integer types goes to STRING as "common comparison denominator" instead of a numeric type
[ https://issues.apache.org/jira/browse/HIVE-8391?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Dere updated HIVE-8391: - Attachment: HIVE-8391.2.patch Patch v2 allows timestamp and numeric types to be compared as double, without having to allow implicit conversions between timestamp and numeric. Also add q file test. > Comparion between TIMESTAMP and Integer types goes to STRING as "common > comparison denominator" instead of a numeric type > - > > Key: HIVE-8391 > URL: https://issues.apache.org/jira/browse/HIVE-8391 > Project: Hive > Issue Type: Bug > Components: CBO, Logical Optimizer >Affects Versions: 0.14.0 >Reporter: Matt McCline >Assignee: Jason Dere >Priority: Critical > Fix For: 0.14.0 > > Attachments: HIVE-8391.1.patch, HIVE-8391.2.patch, vectorization_7.q > > > Discovered while investigating why vectorization_7.q has different results > for non-vectorized [+CBO] and vectorized [+CBO]. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-7776) enable sample10.q.[Spark Branch]
[ https://issues.apache.org/jira/browse/HIVE-7776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14164294#comment-14164294 ] Xuefu Zhang commented on HIVE-7776: --- Hi [~chengxiang li], Could you take a look at the above test result and see if the patch is ready to be committed? Thanks. > enable sample10.q.[Spark Branch] > > > Key: HIVE-7776 > URL: https://issues.apache.org/jira/browse/HIVE-7776 > Project: Hive > Issue Type: Sub-task > Components: Spark >Reporter: Chengxiang Li >Assignee: Chengxiang Li > Attachments: HIVE-7776.1-spark.patch, HIVE-7776.2-spark.patch, > HIVE-7776.3-spark.patch, HIVE-7776.3-spark.patch > > > sample10.q contain dynamic partition operation, should enable this qtest > after hive on spark support dynamic partition. -- This message was sent by Atlassian JIRA (v6.3.4#6332)