[
https://issues.apache.org/jira/browse/TRAFODION-3126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16530663#comment-16530663
]
ASF GitHub Bot commented on TRAFODION-3126:
-------------------------------------------
Github user selvaganesang commented on a diff in the pull request:
https://github.com/apache/trafodion/pull/1626#discussion_r199663083
--- Diff: core/sql/src/main/java/org/trafodion/sql/HDFSClient.java ---
@@ -142,21 +154,32 @@
HDFSRead()
{
}
-
+
public Object call() throws IOException
{
int bytesRead;
int totalBytesRead = 0;
if (compressed_) {
bufArray_ = new byte[ioByteArraySizeInKB_ * 1024];
- } else
- if (! buf_.hasArray()) {
- try {
- fsdis_.seek(pos_);
- } catch (EOFException e) {
- isEOF_ = 1;
- return new Integer(totalBytesRead);
- }
+ }
+ else {
+ // alluxio doesn't support direct ByteBuffer reads
+ // Hence, create a non-direct ByteBuffer, read into
+ // byteArray backing up this ByteBuffer and
+ // then copy the data read to direct ByteBuffer for the
+ // native layer to process the data
+ if ((! alluxioNotInstalled_) && fs_ instanceof
alluxio.hadoop.FileSystem) {
+ savedBuf_ = buf_;
+ buf_ = ByteBuffer.allocate(savedBuf_.capacity());
--- End diff --
HdfsClient object is constructed for every range or every HdfsScanBuf size
within a range in HdfsScan. The call method is called only once. So, it is ok
to allocate.
However, It is possible to optimize on this, if you push the detection of
file system before HdfsClient object is constructed to HdfsScan, if the
allocation becomes a bottleneck.
> Refactored HDFS client implementation should also support Alluxio file system
> -----------------------------------------------------------------------------
>
> Key: TRAFODION-3126
> URL: https://issues.apache.org/jira/browse/TRAFODION-3126
> Project: Apache Trafodion
> Issue Type: Sub-task
> Components: sql-exe
> Reporter: Selvaganesan Govindarajan
> Assignee: Selvaganesan Govindarajan
> Priority: Major
>
> New implementation returns the following error when Alluxio file system is
> involved.
> *** ERROR[8447] An error occurred during hdfs access. Error Detail:
> SETUP_HDFS_SCAN java.util.concurrent.ExecutionException:
> java.lang.UnsupportedOperationException: Byte-buffer read unsupported by
> input stream
> java.util.concurrent.FutureTask.report(FutureTask.java:122)
> java.util.concurrent.FutureTask.get(FutureTask.java:192)
> org.trafodion.sql.HDFSClient.trafHdfsReadBuffer(HDFSClient.java:278)
> org.trafodion.sql.HdfsScan.trafHdfsRead(HdfsScan.java:201) Caused by
> java.lang.UnsupportedOperationException: Byte-buffer read unsupported by
> input stream
> org.apache.hadoop.fs.FSDataInputStream.read(FSDataInputStream.java:150)
> org.trafodion.sql.HDFSClient$HDFSRead.call(HDFSClient.java:172)
> java.util.concurrent.FutureTask.run(FutureTask.java:266)
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> java.lang.Thread.run(Thread.java:745)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
