[
https://issues.apache.org/jira/browse/HDDS-2356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16962583#comment-16962583
]
Bharat Viswanadham edited comment on HDDS-2356 at 10/30/19 1:23 AM:
--------------------------------------------------------------------
# When uploaded 2 parts, and when complete upload 1 part no error
# During complete multipart upload name/part number not matching with uploaded
part and part number then InvalidPart error
# When parts are not specified in sorted order InvalidPartOrder
# During complete multipart upload when no uploaded parts, and we specify some
parts then also InvalidPart
# Uploaded parts 1,2,3 and during complete we can do upload 1,3 (No error)
# When part 3 uploaded, complete with part 3 can be done
Found few scenarios where we behave differently need to handled in Ozone S3.
And also if you can enable audit logs and see what is the parts we are getting
during complete MPU and for that key what are parts available, then also we can
see where the error occurs. But we should handle the above scenarios in Ozone.
But the question is why goofys when uploading is uploading like this or we
missed any other cases still.
was (Author: bharatviswa):
# When uploaded 2 parts, and when complete upload 1 part no error
# During complete multipart upload name/part number not matching with uploaded
part and part number then InvalidPart error
# When parts are not specified in sorted order InvalidPartOrder
# During complete multipart upload when no uploaded parts, and we specify some
parts then also InvalidPart
# Uploaded parts 1,2,3 and during complete we can do upload 1,3 (No error)
# When part 3 uploaded, complete with part 3 can be done
Found few scenarios where we behave differently need to handled in Ozone S3.
And also if you can enable audit logs and see what is the parts we are getting
during complete MPU and for that key what are parts available, then also we can
see where the error occurs. But we should handle above scenarios in Ozone.
> Multipart upload report errors while writing to ozone Ratis pipeline
> --------------------------------------------------------------------
>
> Key: HDDS-2356
> URL: https://issues.apache.org/jira/browse/HDDS-2356
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: Ozone Manager
> Affects Versions: 0.4.1
> Environment: Env: 4 VMs in total: 3 Datanodes on 3 VMs, 1 OM & 1 SCM
> on a separate VM
> Reporter: Li Cheng
> Assignee: Bharat Viswanadham
> Priority: Blocker
> Fix For: 0.5.0
>
>
> Env: 4 VMs in total: 3 Datanodes on 3 VMs, 1 OM & 1 SCM on a separate VM, say
> it's VM0.
> I use goofys as a fuse and enable ozone S3 gateway to mount ozone to a path
> on VM0, while reading data from VM0 local disk and write to mount path. The
> dataset has various sizes of files from 0 byte to GB-level and it has a
> number of ~50,000 files.
> The writing is slow (1GB for ~10 mins) and it stops after around 4GB. As I
> look at hadoop-root-om-VM_50_210_centos.out log, I see OM throwing errors
> related with Multipart upload. This error eventually causes the writing to
> terminate and OM to be closed.
>
> 2019-10-24 16:01:59,527 [OMDoubleBufferFlushThread] ERROR - Terminating with
> exit status 2: OMDoubleBuffer flush
> threadOMDoubleBufferFlushThreadencountered Throwable error
> java.util.ConcurrentModificationException
> at java.util.TreeMap.forEach(TreeMap.java:1004)
> at
> org.apache.hadoop.ozone.om.helpers.OmMultipartKeyInfo.getProto(OmMultipartKeyInfo.java:111)
> at
> org.apache.hadoop.ozone.om.codec.OmMultipartKeyInfoCodec.toPersistedFormat(OmMultipartKeyInfoCodec.java:38)
> at
> org.apache.hadoop.ozone.om.codec.OmMultipartKeyInfoCodec.toPersistedFormat(OmMultipartKeyInfoCodec.java:31)
> at
> org.apache.hadoop.hdds.utils.db.CodecRegistry.asRawData(CodecRegistry.java:68)
> at
> org.apache.hadoop.hdds.utils.db.TypedTable.putWithBatch(TypedTable.java:125)
> at
> org.apache.hadoop.ozone.om.response.s3.multipart.S3MultipartUploadCommitPartResponse.addToDBBatch(S3MultipartUploadCommitPartResponse.java:112)
> at
> org.apache.hadoop.ozone.om.ratis.OzoneManagerDoubleBuffer.lambda$flushTransactions$0(OzoneManagerDoubleBuffer.java:137)
> at java.util.Iterator.forEachRemaining(Iterator.java:116)
> at
> org.apache.hadoop.ozone.om.ratis.OzoneManagerDoubleBuffer.flushTransactions(OzoneManagerDoubleBuffer.java:135)
> at java.lang.Thread.run(Thread.java:745)
> 2019-10-24 16:01:59,629 [shutdown-hook-0] INFO - SHUTDOWN_MSG:
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
