huozhanfeng opened a new issue #6307:
URL: https://github.com/apache/incubator-doris/issues/6307
**Describe the bug**
Currently, Doris supports loading OSS/S3A files by using params like
`fs.s3a.access.key`, but there is a bug when using it to load such type files.
Here is the error log when loading OSS files
<pre>
java.lang.UnsupportedOperationException: Byte-buffer read unsupported by
input stream
at
org.apache.hadoop.fs.FSDataInputStream.read(FSDataInputStream.java:146)
at
org.apache.doris.broker.hdfs.FileSystemManager.readByteBuffer(FileSystemManager.java:678)
at
org.apache.doris.broker.hdfs.FileSystemManager.readByteBufferFully(FileSystemManager.java:689)
at
org.apache.doris.broker.hdfs.FileSystemManager.pread(FileSystemManager.java:570)
at
org.apache.doris.broker.hdfs.HDFSBrokerServiceImpl.pread(HDFSBrokerServiceImpl.java:157)
at
org.apache.doris.thrift.TPaloBrokerService$Processor$pread.getResult(TPaloBrokerService.java:937)
at
org.apache.doris.thrift.TPaloBrokerService$Processor$pread.getResult(TPaloBrokerService.java:922)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
at
org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
</pre>
**To Reproduce**
We can simply use such load configuration to reproduce the bug
<pre>
WITH BROKER hdfs_broker (
"fs.s3a.access.key"="...",
"fs.s3a.secret.key"="...",
"fs.s3a.endpoint"="..."
)
</pre>
**Additional context**
It's a bug related to this issue.
S3A input stream to support ByteBufferReadable
https://issues.apache.org/jira/browse/HADOOP-14603
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
