[jira] [Commented] (HIVE-12990) LLAP: ORC cache NPE without FileID support
[
https://issues.apache.org/jira/browse/HIVE-12990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15134687#comment-15134687
]
Prasanth Jayachandran commented on HIVE-12990:
--
LGTM, +1
> LLAP: ORC cache NPE without FileID support
> --
>
> Key: HIVE-12990
> URL: https://issues.apache.org/jira/browse/HIVE-12990
> Project: Hive
> Issue Type: Bug
> Components: llap
>Affects Versions: 2.1.0
>Reporter: Gopal V
>Assignee: Sergey Shelukhin
> Attachments: HIVE-12990.01.patch, HIVE-12990.patch
>
>
> {code}
>OrcBatchKey stripeKey = hasFileId ? new OrcBatchKey(fileId, -1, 0) : null;
>...
> if (hasFileId && metadataCache != null) {
> stripeKey.stripeIx = stripeIx;
> stripeMetadata = metadataCache.getStripeMetadata(stripeKey);
> }
> ...
> public void setStripeMetadata(OrcStripeMetadata m) {
> assert stripes != null;
> stripes[m.getStripeIx()] = m;
> }
> {code}
> {code}
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.llap.io.metadata.OrcStripeMetadata.getStripeIx(OrcStripeMetadata.java:106)
> at
> org.apache.hadoop.hive.llap.io.decode.OrcEncodedDataConsumer.setStripeMetadata(OrcEncodedDataConsumer.java:70)
> at
> org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader.readStripesMetadata(OrcEncodedDataReader.java:685)
> at
> org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader.performDataRead(OrcEncodedDataReader.java:283)
> at
> org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader$4.run(OrcEncodedDataReader.java:215)
> at
> org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader$4.run(OrcEncodedDataReader.java:212)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
> at
> org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader.callInternal(OrcEncodedDataReader.java:212)
> at
> org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader.callInternal(OrcEncodedDataReader.java:93)
> ... 5 more
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-12990) LLAP: ORC cache NPE without FileID support
[
https://issues.apache.org/jira/browse/HIVE-12990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15133919#comment-15133919
]
Hive QA commented on HIVE-12990:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12786316/HIVE-12990.01.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 10051 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_import
org.apache.hive.jdbc.TestSSL.testSSLVersion
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6871/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6871/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-6871/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12786316 - PreCommit-HIVE-TRUNK-Build
> LLAP: ORC cache NPE without FileID support
> --
>
> Key: HIVE-12990
> URL: https://issues.apache.org/jira/browse/HIVE-12990
> Project: Hive
> Issue Type: Bug
> Components: llap
>Affects Versions: 2.1.0
>Reporter: Gopal V
>Assignee: Sergey Shelukhin
> Attachments: HIVE-12990.01.patch, HIVE-12990.patch
>
>
> {code}
>OrcBatchKey stripeKey = hasFileId ? new OrcBatchKey(fileId, -1, 0) : null;
>...
> if (hasFileId && metadataCache != null) {
> stripeKey.stripeIx = stripeIx;
> stripeMetadata = metadataCache.getStripeMetadata(stripeKey);
> }
> ...
> public void setStripeMetadata(OrcStripeMetadata m) {
> assert stripes != null;
> stripes[m.getStripeIx()] = m;
> }
> {code}
> {code}
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.llap.io.metadata.OrcStripeMetadata.getStripeIx(OrcStripeMetadata.java:106)
> at
> org.apache.hadoop.hive.llap.io.decode.OrcEncodedDataConsumer.setStripeMetadata(OrcEncodedDataConsumer.java:70)
> at
> org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader.readStripesMetadata(OrcEncodedDataReader.java:685)
> at
> org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader.performDataRead(OrcEncodedDataReader.java:283)
> at
> org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader$4.run(OrcEncodedDataReader.java:215)
> at
> org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader$4.run(OrcEncodedDataReader.java:212)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
> at
> org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader.callInternal(OrcEncodedDataReader.java:212)
> at
> org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader.callInternal(OrcEncodedDataReader.java:93)
> ... 5 more
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-12990) LLAP: ORC cache NPE without FileID support
[
https://issues.apache.org/jira/browse/HIVE-12990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15131662#comment-15131662
]
Gopal V commented on HIVE-12990:
Looks like this still sends through a NULL fileId through - the auto-unbox here
throws an NPE
{code}
@Override
public EncodedReader encodedReader(
Long fileId, DataCache dataCache, DataReader dataReader, PoolFactory pf)
throws IOException {
return new EncodedReaderImpl(fileId, types,
codec, bufferSize, rowIndexStride, dataCache, dataReader, pf);
}
{code}
{code}
Caused by: java.lang.NullPointerException
at
org.apache.hadoop.hive.ql.io.orc.encoded.ReaderImpl.encodedReader(ReaderImpl.java:38)
at
org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader.performDataRead(OrcEncodedDataReader.java:331)
at
org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader$4.run(OrcEncodedDataReader.java:215)
at
org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader$4.run(OrcEncodedDataReader.java:212)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
at
org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader.callInternal(OrcEncodedDataReader.java:212)
at
org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader.callInternal(OrcEncodedDataReader.java:93)
{code}
> LLAP: ORC cache NPE without FileID support
> --
>
> Key: HIVE-12990
> URL: https://issues.apache.org/jira/browse/HIVE-12990
> Project: Hive
> Issue Type: Bug
> Components: llap
>Affects Versions: 2.1.0
>Reporter: Gopal V
>Assignee: Sergey Shelukhin
> Attachments: HIVE-12990.patch
>
>
> {code}
>OrcBatchKey stripeKey = hasFileId ? new OrcBatchKey(fileId, -1, 0) : null;
>...
> if (hasFileId && metadataCache != null) {
> stripeKey.stripeIx = stripeIx;
> stripeMetadata = metadataCache.getStripeMetadata(stripeKey);
> }
> ...
> public void setStripeMetadata(OrcStripeMetadata m) {
> assert stripes != null;
> stripes[m.getStripeIx()] = m;
> }
> {code}
> {code}
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.llap.io.metadata.OrcStripeMetadata.getStripeIx(OrcStripeMetadata.java:106)
> at
> org.apache.hadoop.hive.llap.io.decode.OrcEncodedDataConsumer.setStripeMetadata(OrcEncodedDataConsumer.java:70)
> at
> org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader.readStripesMetadata(OrcEncodedDataReader.java:685)
> at
> org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader.performDataRead(OrcEncodedDataReader.java:283)
> at
> org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader$4.run(OrcEncodedDataReader.java:215)
> at
> org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader$4.run(OrcEncodedDataReader.java:212)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
> at
> org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader.callInternal(OrcEncodedDataReader.java:212)
> at
> org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader.callInternal(OrcEncodedDataReader.java:93)
> ... 5 more
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-12990) LLAP: ORC cache NPE without FileID support
[
https://issues.apache.org/jira/browse/HIVE-12990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15131722#comment-15131722
]
Gopal V commented on HIVE-12990:
Finally figure out what's happening there.
The CLI parameters set aren't passed to the IO Elevator, so the params have to
be setup in llap-daemon-site.xml for the system to pick it up.
Which brings a very different problem, FS configs (like auth tokens) are passed
via the jobconf today.
> LLAP: ORC cache NPE without FileID support
> --
>
> Key: HIVE-12990
> URL: https://issues.apache.org/jira/browse/HIVE-12990
> Project: Hive
> Issue Type: Bug
> Components: llap
>Affects Versions: 2.1.0
>Reporter: Gopal V
>Assignee: Sergey Shelukhin
> Attachments: HIVE-12990.patch
>
>
> {code}
>OrcBatchKey stripeKey = hasFileId ? new OrcBatchKey(fileId, -1, 0) : null;
>...
> if (hasFileId && metadataCache != null) {
> stripeKey.stripeIx = stripeIx;
> stripeMetadata = metadataCache.getStripeMetadata(stripeKey);
> }
> ...
> public void setStripeMetadata(OrcStripeMetadata m) {
> assert stripes != null;
> stripes[m.getStripeIx()] = m;
> }
> {code}
> {code}
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.llap.io.metadata.OrcStripeMetadata.getStripeIx(OrcStripeMetadata.java:106)
> at
> org.apache.hadoop.hive.llap.io.decode.OrcEncodedDataConsumer.setStripeMetadata(OrcEncodedDataConsumer.java:70)
> at
> org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader.readStripesMetadata(OrcEncodedDataReader.java:685)
> at
> org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader.performDataRead(OrcEncodedDataReader.java:283)
> at
> org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader$4.run(OrcEncodedDataReader.java:215)
> at
> org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader$4.run(OrcEncodedDataReader.java:212)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
> at
> org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader.callInternal(OrcEncodedDataReader.java:212)
> at
> org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader.callInternal(OrcEncodedDataReader.java:93)
> ... 5 more
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
