[
https://issues.apache.org/jira/browse/HDDS-2356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16961568#comment-16961568
]
Bharat Viswanadham edited comment on HDDS-2356 at 10/29/19 12:10 AM:
---------------------------------------------------------------------
{quote}[~bharat] In term of reproduction, I have a dataset which includes small
files as well as big files and I'm using s3 gateway from ozone and mount ozone
cluster to a local path by goofys. All the data are recursively written to the
mount path, which essentially leads to ozone cluster. The ozone cluster is
deployed on a 3-node VMs env and each VM has only 1 disk for ozone data
writing. I think it's a pretty simple scenario to reproduce. The solely
operation is writing to ozone cluster thru fuse.
{quote}
I have tried with a test to run parallel MPU for a key, and it still passes.
{quote}All the data are recursively written to the mount path, which
essentially leads to ozone cluster.
{quote}
Mean here using cp to move the files to mount path?.
If possible, could you give some steps/exact commands to repro this, which will
help in debugging this issue? I have tried mount on docker, but after a few
large files cp, I get OutofMemory from docker.
was (Author: bharatviswa):
[~bharat] In term of reproduction, I have a dataset which includes small files
as well as big files and I'm using s3 gateway from ozone and mount ozone
cluster to a local path by goofys. All the data are recursively written to the
mount path, which essentially leads to ozone cluster. The ozone cluster is
deployed on a 3-node VMs env and each VM has only 1 disk for ozone data
writing. I think it's a pretty simple scenario to reproduce. The solely
operation is writing to ozone cluster thru fuse.
I have tried with a test to run parallel MPU for a key, and it still passes.
All the data are recursively written to the mount path, which essentially leads
to ozone cluster.
Mean here using cp to move the files to mount path?.
If possible, could you give some steps/commands to repro this, which will help
in debug this issue. I have tried mount on docker, but after few large files
cp, I get OutofMemory from docker.
> Multipart upload report errors while writing to ozone Ratis pipeline
> --------------------------------------------------------------------
>
> Key: HDDS-2356
> URL: https://issues.apache.org/jira/browse/HDDS-2356
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: Ozone Manager
> Affects Versions: 0.4.1
> Environment: Env: 4 VMs in total: 3 Datanodes on 3 VMs, 1 OM & 1 SCM
> on a separate VM
> Reporter: Li Cheng
> Priority: Blocker
> Fix For: 0.5.0
>
>
> Env: 4 VMs in total: 3 Datanodes on 3 VMs, 1 OM & 1 SCM on a separate VM, say
> it's VM0.
> I use goofys as a fuse and enable ozone S3 gateway to mount ozone to a path
> on VM0, while reading data from VM0 local disk and write to mount path. The
> dataset has various sizes of files from 0 byte to GB-level and it has a
> number of ~50,000 files.
> The writing is slow (1GB for ~10 mins) and it stops after around 4GB. As I
> look at hadoop-root-om-VM_50_210_centos.out log, I see OM throwing errors
> related with Multipart upload. This error eventually causes the writing to
> terminate and OM to be closed.
>
> 2019-10-24 16:01:59,527 [OMDoubleBufferFlushThread] ERROR - Terminating with
> exit status 2: OMDoubleBuffer flush
> threadOMDoubleBufferFlushThreadencountered Throwable error
> java.util.ConcurrentModificationException
> at java.util.TreeMap.forEach(TreeMap.java:1004)
> at
> org.apache.hadoop.ozone.om.helpers.OmMultipartKeyInfo.getProto(OmMultipartKeyInfo.java:111)
> at
> org.apache.hadoop.ozone.om.codec.OmMultipartKeyInfoCodec.toPersistedFormat(OmMultipartKeyInfoCodec.java:38)
> at
> org.apache.hadoop.ozone.om.codec.OmMultipartKeyInfoCodec.toPersistedFormat(OmMultipartKeyInfoCodec.java:31)
> at
> org.apache.hadoop.hdds.utils.db.CodecRegistry.asRawData(CodecRegistry.java:68)
> at
> org.apache.hadoop.hdds.utils.db.TypedTable.putWithBatch(TypedTable.java:125)
> at
> org.apache.hadoop.ozone.om.response.s3.multipart.S3MultipartUploadCommitPartResponse.addToDBBatch(S3MultipartUploadCommitPartResponse.java:112)
> at
> org.apache.hadoop.ozone.om.ratis.OzoneManagerDoubleBuffer.lambda$flushTransactions$0(OzoneManagerDoubleBuffer.java:137)
> at java.util.Iterator.forEachRemaining(Iterator.java:116)
> at
> org.apache.hadoop.ozone.om.ratis.OzoneManagerDoubleBuffer.flushTransactions(OzoneManagerDoubleBuffer.java:135)
> at java.lang.Thread.run(Thread.java:745)
> 2019-10-24 16:01:59,629 [shutdown-hook-0] INFO - SHUTDOWN_MSG:
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
