[
https://issues.apache.org/jira/browse/HDDS-6356?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
cchenaxchen updated HDDS-6356:
------------------------------
Description:
I use the lastest code
create bucket command is:
bin/ozone sh volume create vol1
bin/ozone sh bucket create vol1/defaultbucket --layout=FILE_SYSTEM_OPTIMIZED -t
EC -r rs-3-2-1024k
I run the mapreduce
bin/hadoop jar
share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-3.2.1-tests.jar
TestDFSIO -Dtest.build.data=o3fs://defaultbucket.vol1/dfsio/test2 -write
-nrFiles 3 -size 10000MB
Error: java.lang.IllegalArgumentException: Expected writeOffset= 1616000000
Expected offset=1613758464
at
org.apache.hadoop.ozone.shaded.com.google.common.base.Preconditions.checkArgument(Preconditions.java:144)
at
org.apache.hadoop.ozone.client.io.ECKeyOutputStream.close(ECKeyOutputStream.java:539)
at
org.apache.hadoop.fs.ozone.OzoneFSOutputStream.close(OzoneFSOutputStream.java:56)
at
org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:72)
at
org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:101)
at org.apache.hadoop.fs.IOMapperBase.map(IOMapperBase.java:136)
at org.apache.hadoop.fs.IOMapperBase.map(IOMapperBase.java:37)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:465)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:349)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:168)
when I add some logs
so the offset is less than writeOffset
and the stripeSize is equal numDataBlks * ecChunkSize, it is a problem
2022-02-23 16:16:01,347 INFO [main]
org.apache.hadoop.ozone.client.io.KeyOutputStream: name = main lastStripeSize
is 3145728 writeOffset is 4838000000 offset is 4834983936
2022-02-23 16:16:01,349 WARN [main] org.apache.hadoop.mapred.YarnChild:
Exception running child : java.lang.IllegalArgumentException
at
org.apache.hadoop.ozone.shaded.com.google.common.base.Preconditions.checkArgument(Preconditions.java:130)
at
org.apache.hadoop.ozone.client.io.ECKeyOutputStream.close(ECKeyOutputStream.java:550)
at
org.apache.hadoop.fs.ozone.OzoneFSOutputStream.close(OzoneFSOutputStream.java:56)
at
org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:72)
at
org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:101)
at org.apache.hadoop.fs.IOMapperBase.map(IOMapperBase.java:136)
at org.apache.hadoop.fs.IOMapperBase.map(IOMapperBase.java:37)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:465)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:349)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:168)
was:
I use the lastest code
create bucket command is:
bin/ozone sh volume create vol1
bin/ozone sh bucket create vol1/defaultbucket --layout=FILE_SYSTEM_OPTIMIZED -t
EC -r rs-3-2-1024k
I run the mapreduce
bin/hadoop jar
share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-3.2.1-tests.jar
TestDFSIO -Dtest.build.data=o3fs://defaultbucket.vol1/dfsio/test2 -write
-nrFiles 3 -size 10000MB
Error: java.lang.IllegalArgumentException
at
org.apache.hadoop.ozone.shaded.com.google.common.base.Preconditions.checkArgument(Preconditions.java:130)
at
org.apache.hadoop.ozone.client.io.ECKeyOutputStream.close(ECKeyOutputStream.java:532)
at
org.apache.hadoop.fs.ozone.OzoneFSOutputStream.close(OzoneFSOutputStream.java:56)
at
org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:72)
at
org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:101)
at org.apache.hadoop.fs.IOMapperBase.map(IOMapperBase.java:136)
at org.apache.hadoop.fs.IOMapperBase.map(IOMapperBase.java:37)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:465)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:349)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:168)
when I add some logs
so the offset is less than writeOffset
and the stripeSize is equal numDataBlks * ecChunkSize, it is a problem
2022-02-23 16:16:01,347 INFO [main]
org.apache.hadoop.ozone.client.io.KeyOutputStream: name = main lastStripeSize
is 3145728 writeOffset is 4838000000 offset is 4834983936
2022-02-23 16:16:01,349 WARN [main] org.apache.hadoop.mapred.YarnChild:
Exception running child : java.lang.IllegalArgumentException
at
org.apache.hadoop.ozone.shaded.com.google.common.base.Preconditions.checkArgument(Preconditions.java:130)
at
org.apache.hadoop.ozone.client.io.ECKeyOutputStream.close(ECKeyOutputStream.java:550)
at
org.apache.hadoop.fs.ozone.OzoneFSOutputStream.close(OzoneFSOutputStream.java:56)
at
org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:72)
at
org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:101)
at org.apache.hadoop.fs.IOMapperBase.map(IOMapperBase.java:136)
at org.apache.hadoop.fs.IOMapperBase.map(IOMapperBase.java:37)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:465)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:349)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:168)
> EC: the offset is less than writeoffset
> ---------------------------------------
>
> Key: HDDS-6356
> URL: https://issues.apache.org/jira/browse/HDDS-6356
> Project: Apache Ozone
> Issue Type: Sub-task
> Components: EC
> Reporter: chen chao
> Assignee: cchenaxchen
> Priority: Major
>
> I use the lastest code
> create bucket command is:
> bin/ozone sh volume create vol1
> bin/ozone sh bucket create vol1/defaultbucket --layout=FILE_SYSTEM_OPTIMIZED
> -t EC -r rs-3-2-1024k
>
> I run the mapreduce
> bin/hadoop jar
> share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-3.2.1-tests.jar
> TestDFSIO -Dtest.build.data=o3fs://defaultbucket.vol1/dfsio/test2 -write
> -nrFiles 3 -size 10000MB
>
> Error: java.lang.IllegalArgumentException: Expected writeOffset= 1616000000
> Expected offset=1613758464
> at
> org.apache.hadoop.ozone.shaded.com.google.common.base.Preconditions.checkArgument(Preconditions.java:144)
> at
> org.apache.hadoop.ozone.client.io.ECKeyOutputStream.close(ECKeyOutputStream.java:539)
> at
> org.apache.hadoop.fs.ozone.OzoneFSOutputStream.close(OzoneFSOutputStream.java:56)
> at
> org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:72)
> at
> org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:101)
> at org.apache.hadoop.fs.IOMapperBase.map(IOMapperBase.java:136)
> at org.apache.hadoop.fs.IOMapperBase.map(IOMapperBase.java:37)
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:465)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:349)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:168)
>
> when I add some logs
> so the offset is less than writeOffset
> and the stripeSize is equal numDataBlks * ecChunkSize, it is a problem
> 2022-02-23 16:16:01,347 INFO [main]
> org.apache.hadoop.ozone.client.io.KeyOutputStream: name = main lastStripeSize
> is 3145728 writeOffset is 4838000000 offset is 4834983936
> 2022-02-23 16:16:01,349 WARN [main] org.apache.hadoop.mapred.YarnChild:
> Exception running child : java.lang.IllegalArgumentException
> at
> org.apache.hadoop.ozone.shaded.com.google.common.base.Preconditions.checkArgument(Preconditions.java:130)
> at
> org.apache.hadoop.ozone.client.io.ECKeyOutputStream.close(ECKeyOutputStream.java:550)
> at
> org.apache.hadoop.fs.ozone.OzoneFSOutputStream.close(OzoneFSOutputStream.java:56)
> at
> org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:72)
> at
> org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:101)
> at org.apache.hadoop.fs.IOMapperBase.map(IOMapperBase.java:136)
> at org.apache.hadoop.fs.IOMapperBase.map(IOMapperBase.java:37)
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:465)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:349)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:168)
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
