[jira] [Commented] (HIVE-12202) NPE thrown when reading legacy ACID delta files
[ https://issues.apache.org/jira/browse/HIVE-12202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14985892#comment-14985892 ] Eugene Koifman commented on HIVE-12202: --- +1 > NPE thrown when reading legacy ACID delta files > --- > > Key: HIVE-12202 > URL: https://issues.apache.org/jira/browse/HIVE-12202 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 1.3.0 >Reporter: Elliot West >Assignee: Elliot West > Labels: transactions > Attachments: HIVE-12202.0.patch > > > When reading legacy ACID deltas of the form {{delta_$startTxnId_$endTxnId}} a > {{NullPointerException}} is thrown on: > {code:title=org.apache.hadoop.hive.ql.io.AcidUtils.deserializeDeltas#371} > if(dmd.getStmtIds().isEmpty()) { > {code} > The older ACID data format (pre-Hive 1.3.0) which does not include the > statement ID, and code written for that format should still be supported. > Therefore the above condition should also include a null check or > alternatively {{AcidInputFormat.DeltaMetaData}} should never return null, and > return an empty list in this specific scenario. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12202) NPE thrown when reading legacy ACID delta files
[ https://issues.apache.org/jira/browse/HIVE-12202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14983536#comment-14983536 ] Thejas M Nair commented on HIVE-12202: -- cc [~ekoifman] > NPE thrown when reading legacy ACID delta files > --- > > Key: HIVE-12202 > URL: https://issues.apache.org/jira/browse/HIVE-12202 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 1.3.0 >Reporter: Elliot West >Assignee: Elliot West > Labels: transactions > Attachments: HIVE-12202.0.patch > > > When reading legacy ACID deltas of the form {{delta_$startTxnId_$endTxnId}} a > {{NullPointerException}} is thrown on: > {code:title=org.apache.hadoop.hive.ql.io.AcidUtils.deserializeDeltas#371} > if(dmd.getStmtIds().isEmpty()) { > {code} > The older ACID data format (pre-Hive 1.3.0) which does not include the > statement ID, and code written for that format should still be supported. > Therefore the above condition should also include a null check or > alternatively {{AcidInputFormat.DeltaMetaData}} should never return null, and > return an empty list in this specific scenario. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12202) NPE thrown when reading legacy ACID delta files
[ https://issues.apache.org/jira/browse/HIVE-12202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14968800#comment-14968800 ] Elliot West commented on HIVE-12202: Test failures do not seem to be related to this patch. > NPE thrown when reading legacy ACID delta files > --- > > Key: HIVE-12202 > URL: https://issues.apache.org/jira/browse/HIVE-12202 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 1.3.0 >Reporter: Elliot West >Assignee: Elliot West > Labels: transactions > Attachments: HIVE-12202.0.patch > > > When reading legacy ACID deltas of the form {{delta_$startTxnId_$endTxnId}} a > {{NullPointerException}} is thrown on: > {code:title=org.apache.hadoop.hive.ql.io.AcidUtils.deserializeDeltas#371} > if(dmd.getStmtIds().isEmpty()) { > {code} > The older ACID data format (pre-Hive 1.3.0) which does not include the > statement ID, and code written for that format should still be supported. > Therefore the above condition should also include a null check or > alternatively {{AcidInputFormat.DeltaMetaData}} should never return null, and > return an empty list in this specific scenario. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12202) NPE thrown when reading legacy ACID delta files
[ https://issues.apache.org/jira/browse/HIVE-12202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14967376#comment-14967376 ] Elliot West commented on HIVE-12202: I've checked to see that {{AcidUtils.serializeDeltas}} is being used correctly in conjunction with {{AcidUtils.deserializeDeltas}}. It appears that {{serializeDeltas}} does indeed create {{DeltaMetaData}} instances with an empty list for the statement IDs for delta paths containing only {{$startTxnId}} and {{$endTxnId}}. However, the deserialization process in {{AcidInputFormat.DeltaMetaData.readFields(DataInput)}} incorrectly sets {{stmtIds}} to {{null}} at line 152 if no statement count was serialized. Hence {{AcidUtils.deserializeDeltas}} then gets tripped up by an NPE at line 371. > NPE thrown when reading legacy ACID delta files > --- > > Key: HIVE-12202 > URL: https://issues.apache.org/jira/browse/HIVE-12202 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 1.3.0 >Reporter: Elliot West >Assignee: Elliot West > Labels: transactions > > When reading legacy ACID deltas of the form {{delta_$startTxnId_$endTxnId}} a > {{NullPointerException}} is thrown on: > {code:title=org.apache.hadoop.hive.ql.io.AcidUtils.deserializeDeltas#371} > if(dmd.getStmtIds().isEmpty()) { > {code} > The older ACID data format (pre-Hive 1.3.0) which does not include the > statement ID, and code written for that format should still be supported. > Therefore the above condition should also include a null check or > alternatively {{AcidInputFormat.DeltaMetaData}} should never return null, and > return an empty list in this specific scenario. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12202) NPE thrown when reading legacy ACID delta files
[ https://issues.apache.org/jira/browse/HIVE-12202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14967402#comment-14967402 ] Elliot West commented on HIVE-12202: Some clarification: in my earlier comment, by 'no statement count' I mean a statement count < 1. Also, I believe this is a bug and a lack of resilience to incorrect usage of the API. > NPE thrown when reading legacy ACID delta files > --- > > Key: HIVE-12202 > URL: https://issues.apache.org/jira/browse/HIVE-12202 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 1.3.0 >Reporter: Elliot West >Assignee: Elliot West > Labels: transactions > > When reading legacy ACID deltas of the form {{delta_$startTxnId_$endTxnId}} a > {{NullPointerException}} is thrown on: > {code:title=org.apache.hadoop.hive.ql.io.AcidUtils.deserializeDeltas#371} > if(dmd.getStmtIds().isEmpty()) { > {code} > The older ACID data format (pre-Hive 1.3.0) which does not include the > statement ID, and code written for that format should still be supported. > Therefore the above condition should also include a null check or > alternatively {{AcidInputFormat.DeltaMetaData}} should never return null, and > return an empty list in this specific scenario. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12202) NPE thrown when reading legacy ACID delta files
[ https://issues.apache.org/jira/browse/HIVE-12202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14967524#comment-14967524 ] Elliot West commented on HIVE-12202: And another clarification to an earlier comment of mine: when I said 'a lack of resilience', I meant 'and not a lack of resilience'. Apologies, not quite with it today! > NPE thrown when reading legacy ACID delta files > --- > > Key: HIVE-12202 > URL: https://issues.apache.org/jira/browse/HIVE-12202 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 1.3.0 >Reporter: Elliot West >Assignee: Elliot West > Labels: transactions > Attachments: HIVE-12202.0.patch > > > When reading legacy ACID deltas of the form {{delta_$startTxnId_$endTxnId}} a > {{NullPointerException}} is thrown on: > {code:title=org.apache.hadoop.hive.ql.io.AcidUtils.deserializeDeltas#371} > if(dmd.getStmtIds().isEmpty()) { > {code} > The older ACID data format (pre-Hive 1.3.0) which does not include the > statement ID, and code written for that format should still be supported. > Therefore the above condition should also include a null check or > alternatively {{AcidInputFormat.DeltaMetaData}} should never return null, and > return an empty list in this specific scenario. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12202) NPE thrown when reading legacy ACID delta files
[ https://issues.apache.org/jira/browse/HIVE-12202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14967652#comment-14967652 ] Hive QA commented on HIVE-12202: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12767804/HIVE-12202.0.patch {color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 4 failed/errored test(s), 9685 tests executed *Failed tests:* {noformat} org.apache.hadoop.hive.cli.TestMiniTezCliDriver.initializationError org.apache.hive.hcatalog.api.TestHCatClient.testTableSchemaPropagation org.apache.hive.hcatalog.hbase.TestPigHBaseStorageHandler.org.apache.hive.hcatalog.hbase.TestPigHBaseStorageHandler org.apache.hive.jdbc.TestSSL.testSSLVersion {noformat} Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/5723/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/5723/console Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-5723/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 4 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12767804 - PreCommit-HIVE-TRUNK-Build > NPE thrown when reading legacy ACID delta files > --- > > Key: HIVE-12202 > URL: https://issues.apache.org/jira/browse/HIVE-12202 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 1.3.0 >Reporter: Elliot West >Assignee: Elliot West > Labels: transactions > Attachments: HIVE-12202.0.patch > > > When reading legacy ACID deltas of the form {{delta_$startTxnId_$endTxnId}} a > {{NullPointerException}} is thrown on: > {code:title=org.apache.hadoop.hive.ql.io.AcidUtils.deserializeDeltas#371} > if(dmd.getStmtIds().isEmpty()) { > {code} > The older ACID data format (pre-Hive 1.3.0) which does not include the > statement ID, and code written for that format should still be supported. > Therefore the above condition should also include a null check or > alternatively {{AcidInputFormat.DeltaMetaData}} should never return null, and > return an empty list in this specific scenario. -- This message was sent by Atlassian JIRA (v6.3.4#6332)