[
https://issues.apache.org/jira/browse/HADOOP-19211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17862752#comment-17862752
]
ASF GitHub Bot commented on HADOOP-19211:
-----------------------------------------
wujinhu commented on PR #6904:
URL: https://github.com/apache/hadoop/pull/6904#issuecomment-2205923493
> reviewed, only some minor comments, as you have based this on the most
recent s3a work
>
> In ITestS3AContractVectoredRead we added a test to check what happens if
you call openFile(), pass in a length shorter than that of the object and see
what breaks. Do you need to worry about this?
AliyunOSSFileSystem does not override the `FileSystem::openFileWithOptions`
interface, so it does not read the length passed by openFile() call. And I got
errors below:
```
[INFO] Running
org.apache.hadoop.fs.aliyun.oss.contract.TestAliyunOSSContractVectoredRead
[ERROR] Tests run: 48, Failures: 0, Errors: 2, Skipped: 0, Time elapsed:
171.493 s <<< FAILURE! - in
org.apache.hadoop.fs.aliyun.oss.contract.TestAliyunOSSContractVectoredRead
[ERROR] testEOFRanges416Handling[Buffer type :
direct](org.apache.hadoop.fs.aliyun.oss.contract.TestAliyunOSSContractVectoredRead)
Time elapsed: 3.76 s <<< ERROR!
java.io.EOFException: Range starts beyond the file length (65536): range
[65536-65636], length=100, reference=null
at
org.apache.hadoop.fs.VectoredReadUtils.validateAndSortRanges(VectoredReadUtils.java:330)
at
org.apache.hadoop.fs.aliyun.oss.AliyunOSSInputStream.readVectored(AliyunOSSInputStream.java:566)
at
org.apache.hadoop.fs.FSDataInputStream.readVectored(FSDataInputStream.java:299)
at
org.apache.hadoop.fs.aliyun.oss.contract.TestAliyunOSSContractVectoredRead.testEOFRanges416Handling(TestAliyunOSSContractVectoredRead.java:85)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:61)
at
org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299)
at
org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.lang.Thread.run(Thread.java:748)
[ERROR] testEOFRanges416Handling[Buffer type :
array](org.apache.hadoop.fs.aliyun.oss.contract.TestAliyunOSSContractVectoredRead)
Time elapsed: 3.686 s <<< ERROR!
java.io.EOFException: Range starts beyond the file length (65536): range
[65536-65636], length=100, reference=null
at
org.apache.hadoop.fs.VectoredReadUtils.validateAndSortRanges(VectoredReadUtils.java:330)
at
org.apache.hadoop.fs.aliyun.oss.AliyunOSSInputStream.readVectored(AliyunOSSInputStream.java:566)
at
org.apache.hadoop.fs.FSDataInputStream.readVectored(FSDataInputStream.java:299)
at
org.apache.hadoop.fs.aliyun.oss.contract.TestAliyunOSSContractVectoredRead.testEOFRanges416Handling(TestAliyunOSSContractVectoredRead.java:85)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:61)
at
org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299)
at
org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.lang.Thread.run(Thread.java:748)
[INFO]
[INFO] Results:
[INFO]
[ERROR] Errors:
[ERROR] TestAliyunOSSContractVectoredRead.testEOFRanges416Handling:85 »
EOF Range star...
[ERROR] TestAliyunOSSContractVectoredRead.testEOFRanges416Handling:85 »
EOF Range star...
[INFO]
[ERROR] Tests run: 48, Failures: 0, Errors: 2, Skipped: 0
```
I'm not sure whether I should implement the interface just like s3a does in
this patch or next patch. Could you please give some suggestions. Thanks. :)
> AliyunOSS: Support vectored read API
> ------------------------------------
>
> Key: HADOOP-19211
> URL: https://issues.apache.org/jira/browse/HADOOP-19211
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs/oss
> Affects Versions: 3.2.4, 3.3.6
> Reporter: wujinhu
> Assignee: wujinhu
> Priority: Major
> Labels: pull-request-available
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
