[jira] [Updated] (HDDS-1572) Implement a Pipeline scrubber to maintain healthy number of pipelines in a cluster

2019-11-18 Thread Li Cheng (Jira)
[ https://issues.apache.org/jira/browse/HDDS-1572?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Li Cheng updated HDDS-1572: --- Description: The design document talks about initial requirements for the pipeline scrubber. - Scan the

[jira] [Assigned] (HDDS-1572) Implement a Pipeline scrubber to maintain healthy number of pipelines in a cluster

2019-11-18 Thread Li Cheng (Jira)
[ https://issues.apache.org/jira/browse/HDDS-1572?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Li Cheng reassigned HDDS-1572: -- Assignee: Li Cheng > Implement a Pipeline scrubber to maintain healthy number of pipelines in a >

[jira] [Assigned] (HDDS-1573) Add scrubber metrics and pipeline metrics

2019-11-17 Thread Li Cheng (Jira)
[ https://issues.apache.org/jira/browse/HDDS-1573?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Li Cheng reassigned HDDS-1573: -- Assignee: Li Cheng > Add scrubber metrics and pipeline metrics >

[jira] [Resolved] (HDDS-2396) OM rocksdb core dump during writing

2019-11-17 Thread Li Cheng (Jira)
[ https://issues.apache.org/jira/browse/HDDS-2396?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Li Cheng resolved HDDS-2396. Assignee: Li Cheng Resolution: Fixed > OM rocksdb core dump during writing >

[jira] [Commented] (HDDS-2396) OM rocksdb core dump during writing

2019-11-17 Thread Li Cheng (Jira)
[ https://issues.apache.org/jira/browse/HDDS-2396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16976315#comment-16976315 ] Li Cheng commented on HDDS-2396: This is resolved in [https://github.com/apache/hadoop-ozone/pull/100]/.

[jira] [Commented] (HDDS-2356) Multipart upload report errors while writing to ozone Ratis pipeline

2019-11-17 Thread Li Cheng (Jira)
[ https://issues.apache.org/jira/browse/HDDS-2356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16976258#comment-16976258 ] Li Cheng commented on HDDS-2356: I checked out [https://github.com/apache/hadoop-ozone/pull/163] and

[jira] [Commented] (HDDS-2356) Multipart upload report errors while writing to ozone Ratis pipeline

2019-11-17 Thread Li Cheng (Jira)
[ https://issues.apache.org/jira/browse/HDDS-2356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16976255#comment-16976255 ] Li Cheng commented on HDDS-2356: [~bharat] Make sense. I failed to find Key:plc_1570869510243_5542 related

[jira] [Commented] (HDDS-2356) Multipart upload report errors while writing to ozone Ratis pipeline

2019-11-17 Thread Li Cheng (Jira)
[ https://issues.apache.org/jira/browse/HDDS-2356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16976237#comment-16976237 ] Li Cheng commented on HDDS-2356: Tried [https://github.com/apache/hadoop-ozone/pull/163] to run. It did

[jira] [Updated] (HDDS-2356) Multipart upload report errors while writing to ozone Ratis pipeline

2019-11-17 Thread Li Cheng (Jira)
[ https://issues.apache.org/jira/browse/HDDS-2356?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Li Cheng updated HDDS-2356: --- Attachment: 2018-11-15-OM-logs.txt > Multipart upload report errors while writing to ozone Ratis pipeline >

[jira] [Resolved] (HDDS-2492) Fix test clean up issue in TestSCMPipelineManager

2019-11-15 Thread Li Cheng (Jira)
[ https://issues.apache.org/jira/browse/HDDS-2492?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Li Cheng resolved HDDS-2492. Fix Version/s: 0.4.1 Resolution: Fixed > Fix test clean up issue in TestSCMPipelineManager >

[jira] [Work started] (HDDS-2492) Fix test clean up issue in TestSCMPipelineManager

2019-11-14 Thread Li Cheng (Jira)
[ https://issues.apache.org/jira/browse/HDDS-2492?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HDDS-2492 started by Li Cheng. -- > Fix test clean up issue in TestSCMPipelineManager >

[jira] [Commented] (HDDS-2492) Fix test clean up issue in TestSCMPipelineManager

2019-11-14 Thread Li Cheng (Jira)
[ https://issues.apache.org/jira/browse/HDDS-2492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16974837#comment-16974837 ] Li Cheng commented on HDDS-2492: [https://github.com/apache/hadoop-ozone/pull/179] > Fix test clean up

[jira] [Updated] (HDDS-2356) Multipart upload report errors while writing to ozone Ratis pipeline

2019-11-13 Thread Li Cheng (Jira)
[ https://issues.apache.org/jira/browse/HDDS-2356?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Li Cheng updated HDDS-2356: --- Attachment: om-audit-VM_50_210_centos.log > Multipart upload report errors while writing to ozone Ratis

[jira] [Commented] (HDDS-2356) Multipart upload report errors while writing to ozone Ratis pipeline

2019-11-13 Thread Li Cheng (Jira)
[ https://issues.apache.org/jira/browse/HDDS-2356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16973103#comment-16973103 ] Li Cheng commented on HDDS-2356: [~bharat] I run the test again. And now see the error in audit log: ]} |

[jira] [Comment Edited] (HDDS-2356) Multipart upload report errors while writing to ozone Ratis pipeline

2019-11-10 Thread Li Cheng (Jira)
[ https://issues.apache.org/jira/browse/HDDS-2356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16971290#comment-16971290 ] Li Cheng edited comment on HDDS-2356 at 11/11/19 7:21 AM: -- [~bharat] In terms of

[jira] [Updated] (HDDS-2356) Multipart upload report errors while writing to ozone Ratis pipeline

2019-11-10 Thread Li Cheng (Jira)
[ https://issues.apache.org/jira/browse/HDDS-2356?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Li Cheng updated HDDS-2356: --- Attachment: om_audit_log_plc_1570863541668_9278.txt > Multipart upload report errors while writing to ozone

[jira] [Commented] (HDDS-2356) Multipart upload report errors while writing to ozone Ratis pipeline

2019-11-10 Thread Li Cheng (Jira)
[ https://issues.apache.org/jira/browse/HDDS-2356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16971290#comment-16971290 ] Li Cheng commented on HDDS-2356: [~bharat] In terms of the key in the last stacktrace: 2019-11-08

[jira] [Comment Edited] (HDDS-2356) Multipart upload report errors while writing to ozone Ratis pipeline

2019-11-10 Thread Li Cheng (Jira)
[ https://issues.apache.org/jira/browse/HDDS-2356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16971289#comment-16971289 ] Li Cheng edited comment on HDDS-2356 at 11/11/19 3:14 AM: -- [~bharat] As I said,

[jira] [Commented] (HDDS-2356) Multipart upload report errors while writing to ozone Ratis pipeline

2019-11-10 Thread Li Cheng (Jira)
[ https://issues.apache.org/jira/browse/HDDS-2356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16971289#comment-16971289 ] Li Cheng commented on HDDS-2356: [~bharat] As I said, debug logs for Goofys doesn't work since goofys

[jira] [Commented] (HDDS-2356) Multipart upload report errors while writing to ozone Ratis pipeline

2019-11-08 Thread Li Cheng (Jira)
[ https://issues.apache.org/jira/browse/HDDS-2356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16970119#comment-16970119 ] Li Cheng commented on HDDS-2356: [~bharat] New error shows up using today's master branch.   2019-11-08

[jira] [Updated] (HDDS-2443) Python client/interface for Ozone

2019-11-08 Thread Li Cheng (Jira)
[ https://issues.apache.org/jira/browse/HDDS-2443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Li Cheng updated HDDS-2443: --- Attachment: (was: OzoneS3.py) > Python client/interface for Ozone > - > >

[jira] [Updated] (HDDS-2443) Python client/interface for Ozone

2019-11-08 Thread Li Cheng (Jira)
[ https://issues.apache.org/jira/browse/HDDS-2443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Li Cheng updated HDDS-2443: --- Attachment: OzoneS3.py > Python client/interface for Ozone > - > >

[jira] [Updated] (HDDS-2443) Python client/interface for Ozone

2019-11-08 Thread Li Cheng (Jira)
[ https://issues.apache.org/jira/browse/HDDS-2443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Li Cheng updated HDDS-2443: --- Attachment: OzoneS3.py > Python client/interface for Ozone > - > >

[jira] [Commented] (HDDS-2443) Python client/interface for Ozone

2019-11-08 Thread Li Cheng (Jira)
[ https://issues.apache.org/jira/browse/HDDS-2443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16969974#comment-16969974 ] Li Cheng commented on HDDS-2443: [~elek] That would be great! I've uploaded my naive version of interface.

[jira] [Updated] (HDDS-2356) Multipart upload report errors while writing to ozone Ratis pipeline

2019-11-07 Thread Li Cheng (Jira)
[ https://issues.apache.org/jira/browse/HDDS-2356?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Li Cheng updated HDDS-2356: --- Attachment: 2019-11-06_18_13_57_422_ERROR > Multipart upload report errors while writing to ozone Ratis

[jira] [Commented] (HDDS-2356) Multipart upload report errors while writing to ozone Ratis pipeline

2019-11-07 Thread Li Cheng (Jira)
[ https://issues.apache.org/jira/browse/HDDS-2356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16969888#comment-16969888 ] Li Cheng commented on HDDS-2356: [~bharat] Please check the attachment one more time. I re-upload the

[jira] [Commented] (HDDS-2443) Python client/interface for Ozone

2019-11-07 Thread Li Cheng (Jira)
[ https://issues.apache.org/jira/browse/HDDS-2443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16969771#comment-16969771 ] Li Cheng commented on HDDS-2443: Prototyping with S3 gateway + boto3 now. Reads, Writes and Deletes can be

[jira] [Created] (HDDS-2443) Python client/interface for Ozone

2019-11-07 Thread Li Cheng (Jira)
Li Cheng created HDDS-2443: -- Summary: Python client/interface for Ozone Key: HDDS-2443 URL: https://issues.apache.org/jira/browse/HDDS-2443 Project: Hadoop Distributed Data Store Issue Type: New

[jira] [Commented] (HDDS-2356) Multipart upload report errors while writing to ozone Ratis pipeline

2019-11-06 Thread Li Cheng (Jira)
[ https://issues.apache.org/jira/browse/HDDS-2356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16968978#comment-16968978 ] Li Cheng commented on HDDS-2356: [~bharat] My program won't abort, but I see some errors in s3g logs with

[jira] [Updated] (HDDS-2356) Multipart upload report errors while writing to ozone Ratis pipeline

2019-11-06 Thread Li Cheng (Jira)
[ https://issues.apache.org/jira/browse/HDDS-2356?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Li Cheng updated HDDS-2356: --- Description: Env: 4 VMs in total: 3 Datanodes on 3 VMs, 1 OM & 1 SCM on a separate VM, say it's VM0. I use

[jira] [Updated] (HDDS-2356) Multipart upload report errors while writing to ozone Ratis pipeline

2019-11-06 Thread Li Cheng (Jira)
[ https://issues.apache.org/jira/browse/HDDS-2356?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Li Cheng updated HDDS-2356: --- Description: Env: 4 VMs in total: 3 Datanodes on 3 VMs, 1 OM & 1 SCM on a separate VM, say it's VM0. I use

[jira] [Commented] (HDDS-2356) Multipart upload report errors while writing to ozone Ratis pipeline

2019-11-06 Thread Li Cheng (Jira)
[ https://issues.apache.org/jira/browse/HDDS-2356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16968957#comment-16968957 ] Li Cheng commented on HDDS-2356: See new Multipart upload error in yesterday's master. [~bharat] Check

[jira] [Commented] (HDDS-2396) OM rocksdb core dump during writing

2019-11-03 Thread Li Cheng (Jira)
[ https://issues.apache.org/jira/browse/HDDS-2396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16966372#comment-16966372 ] Li Cheng commented on HDDS-2396: [~bharat] Probably not up to date if HDDS 2379 is resolved in recent a

[jira] [Comment Edited] (HDDS-2356) Multipart upload report errors while writing to ozone Ratis pipeline

2019-10-31 Thread Li Cheng (Jira)
[ https://issues.apache.org/jira/browse/HDDS-2356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16964554#comment-16964554 ] Li Cheng edited comment on HDDS-2356 at 11/1/19 3:45 AM: - Also see a core dump in

[jira] [Commented] (HDDS-2396) OM rocksdb core dump during writing

2019-10-31 Thread Li Cheng (Jira)
[ https://issues.apache.org/jira/browse/HDDS-2396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16964578#comment-16964578 ] Li Cheng commented on HDDS-2396: Attached the entire log for the core dump. Will try to turn on ulimit and

[jira] [Updated] (HDDS-2396) OM rocksdb core dump during writing

2019-10-31 Thread Li Cheng (Jira)
[ https://issues.apache.org/jira/browse/HDDS-2396?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Li Cheng updated HDDS-2396: --- Attachment: hs_err_pid9340.log > OM rocksdb core dump during writing > --- >

[jira] [Created] (HDDS-2396) OM rocksdb core dump during writing

2019-10-31 Thread Li Cheng (Jira)
Li Cheng created HDDS-2396: -- Summary: OM rocksdb core dump during writing Key: HDDS-2396 URL: https://issues.apache.org/jira/browse/HDDS-2396 Project: Hadoop Distributed Data Store Issue Type: Bug

[jira] [Commented] (HDDS-2395) Handle Ozone S3 completeMPU to match with aws s3 behavior.

2019-10-31 Thread Li Cheng (Jira)
[ https://issues.apache.org/jira/browse/HDDS-2395?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16964573#comment-16964573 ] Li Cheng commented on HDDS-2395: Also please note the exclude list issue.   2019-11-01 11:25:24,047

[jira] [Commented] (HDDS-2356) Multipart upload report errors while writing to ozone Ratis pipeline

2019-10-31 Thread Li Cheng (Jira)
[ https://issues.apache.org/jira/browse/HDDS-2356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16964554#comment-16964554 ] Li Cheng commented on HDDS-2356: Also see a core dump in rocksdb during last night's testing. Please check

[jira] [Updated] (HDDS-2356) Multipart upload report errors while writing to ozone Ratis pipeline

2019-10-31 Thread Li Cheng (Jira)
[ https://issues.apache.org/jira/browse/HDDS-2356?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Li Cheng updated HDDS-2356: --- Attachment: hs_err_pid9340.log > Multipart upload report errors while writing to ozone Ratis pipeline >

[jira] [Commented] (HDDS-2356) Multipart upload report errors while writing to ozone Ratis pipeline

2019-10-31 Thread Li Cheng (Jira)
[ https://issues.apache.org/jira/browse/HDDS-2356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16964138#comment-16964138 ] Li Cheng commented on HDDS-2356: [~msingh] I check OM audit log and it looks like this one:

[jira] [Commented] (HDDS-2356) Multipart upload report errors while writing to ozone Ratis pipeline

2019-10-31 Thread Li Cheng (Jira)
[ https://issues.apache.org/jira/browse/HDDS-2356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16963875#comment-16963875 ] Li Cheng commented on HDDS-2356: I tried multiprocessing in python to fetch data locally and write to

[jira] [Updated] (HDDS-2356) Multipart upload report errors while writing to ozone Ratis pipeline

2019-10-31 Thread Li Cheng (Jira)
[ https://issues.apache.org/jira/browse/HDDS-2356?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Li Cheng updated HDDS-2356: --- Attachment: image-2019-10-31-18-56-56-177.png > Multipart upload report errors while writing to ozone Ratis

[jira] [Updated] (HDDS-2356) Multipart upload report errors while writing to ozone Ratis pipeline

2019-10-31 Thread Li Cheng (Jira)
[ https://issues.apache.org/jira/browse/HDDS-2356?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Li Cheng updated HDDS-2356: --- Description: Env: 4 VMs in total: 3 Datanodes on 3 VMs, 1 OM & 1 SCM on a separate VM, say it's VM0. I use

[jira] [Updated] (HDDS-2356) Multipart upload report errors while writing to ozone Ratis pipeline

2019-10-31 Thread Li Cheng (Jira)
[ https://issues.apache.org/jira/browse/HDDS-2356?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Li Cheng updated HDDS-2356: --- Description: Env: 4 VMs in total: 3 Datanodes on 3 VMs, 1 OM & 1 SCM on a separate VM, say it's VM0. I use

[jira] [Commented] (HDDS-2356) Multipart upload report errors while writing to ozone Ratis pipeline

2019-10-31 Thread Li Cheng (Jira)
[ https://issues.apache.org/jira/browse/HDDS-2356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16963684#comment-16963684 ] Li Cheng commented on HDDS-2356: [~bharat] I tried debug_s3 and debug_fuse in goofys, but I believe it

[jira] [Issue Comment Deleted] (HDDS-2186) Fix tests using MiniOzoneCluster for its memory related exceptions

2019-10-30 Thread Li Cheng (Jira)
[ https://issues.apache.org/jira/browse/HDDS-2186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Li Cheng updated HDDS-2186: --- Comment: was deleted (was: https://github.com/apache/hadoop-ozone/pull/28) > Fix tests using

[jira] [Commented] (HDDS-2186) Fix tests using MiniOzoneCluster for its memory related exceptions

2019-10-30 Thread Li Cheng (Jira)
[ https://issues.apache.org/jira/browse/HDDS-2186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16962740#comment-16962740 ] Li Cheng commented on HDDS-2186: The OOM issue resolved in https://github.com/apache/hadoop-ozone/pull/28.

[jira] [Resolved] (HDDS-2186) Fix tests using MiniOzoneCluster for its memory related exceptions

2019-10-30 Thread Li Cheng (Jira)
[ https://issues.apache.org/jira/browse/HDDS-2186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Li Cheng resolved HDDS-2186. Fix Version/s: (was: 0.5.0) HDDS-1564 Target Version/s: HDDS-1564

[jira] [Comment Edited] (HDDS-2356) Multipart upload report errors while writing to ozone Ratis pipeline

2019-10-29 Thread Li Cheng (Jira)
[ https://issues.apache.org/jira/browse/HDDS-2356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16961604#comment-16961604 ] Li Cheng edited comment on HDDS-2356 at 10/29/19 6:53 AM: -- [~bharat] I'm using

[jira] [Assigned] (HDDS-2356) Multipart upload report errors while writing to ozone Ratis pipeline

2019-10-28 Thread Li Cheng (Jira)
[ https://issues.apache.org/jira/browse/HDDS-2356?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Li Cheng reassigned HDDS-2356: -- Assignee: Bharat Viswanadham > Multipart upload report errors while writing to ozone Ratis pipeline >

[jira] [Commented] (HDDS-2356) Multipart upload report errors while writing to ozone Ratis pipeline

2019-10-28 Thread Li Cheng (Jira)
[ https://issues.apache.org/jira/browse/HDDS-2356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16961607#comment-16961607 ] Li Cheng commented on HDDS-2356: [~bharat] The long printing logs happen only in your branch tho. The full

[jira] [Comment Edited] (HDDS-2356) Multipart upload report errors while writing to ozone Ratis pipeline

2019-10-28 Thread Li Cheng (Jira)
[ https://issues.apache.org/jira/browse/HDDS-2356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16961604#comment-16961604 ] Li Cheng edited comment on HDDS-2356 at 10/29/19 2:29 AM: -- [~bharat] I'm using

[jira] [Comment Edited] (HDDS-2356) Multipart upload report errors while writing to ozone Ratis pipeline

2019-10-28 Thread Li Cheng (Jira)
[ https://issues.apache.org/jira/browse/HDDS-2356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16961604#comment-16961604 ] Li Cheng edited comment on HDDS-2356 at 10/29/19 2:25 AM: -- [~bharat] I'm using

[jira] [Comment Edited] (HDDS-2356) Multipart upload report errors while writing to ozone Ratis pipeline

2019-10-28 Thread Li Cheng (Jira)
[ https://issues.apache.org/jira/browse/HDDS-2356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16961604#comment-16961604 ] Li Cheng edited comment on HDDS-2356 at 10/29/19 2:24 AM: -- [~bharat] I'm using

[jira] [Commented] (HDDS-2356) Multipart upload report errors while writing to ozone Ratis pipeline

2019-10-28 Thread Li Cheng (Jira)
[ https://issues.apache.org/jira/browse/HDDS-2356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16961604#comment-16961604 ] Li Cheng commented on HDDS-2356: [~bharat] I'm using python to write to a OS path. sth like: for sub_file

[jira] [Reopened] (HDDS-2356) Multipart upload report errors while writing to ozone Ratis pipeline

2019-10-28 Thread Li Cheng (Jira)
[ https://issues.apache.org/jira/browse/HDDS-2356?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Li Cheng reopened HDDS-2356: Assignee: (was: Bharat Viswanadham) > Multipart upload report errors while writing to ozone Ratis

[jira] [Commented] (HDDS-2356) Multipart upload report errors while writing to ozone Ratis pipeline

2019-10-28 Thread Li Cheng (Jira)
[ https://issues.apache.org/jira/browse/HDDS-2356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16960870#comment-16960870 ] Li Cheng commented on HDDS-2356: I take this Jira to track issues that seem to be related with Multipart

[jira] [Commented] (HDDS-2356) Multipart upload report errors while writing to ozone Ratis pipeline

2019-10-28 Thread Li Cheng (Jira)
[ https://issues.apache.org/jira/browse/HDDS-2356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16960866#comment-16960866 ] Li Cheng commented on HDDS-2356: MISMATCH_MULTIPART_LIST seems to be a recurring error. Never be able to

[jira] [Commented] (HDDS-2322) DoubleBuffer flush termination and OM shutdown's after that.

2019-10-28 Thread Li Cheng (Jira)
[ https://issues.apache.org/jira/browse/HDDS-2322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16960802#comment-16960802 ] Li Cheng commented on HDDS-2322: https://issues.apache.org/jira/browse/HDDS-2356 is still having issues.

[jira] [Commented] (HDDS-2356) Multipart upload report errors while writing to ozone Ratis pipeline

2019-10-28 Thread Li Cheng (Jira)
[ https://issues.apache.org/jira/browse/HDDS-2356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16960799#comment-16960799 ] Li Cheng commented on HDDS-2356: [~bharat] In term of reproduction, I have a dataset which includes small

[jira] [Commented] (HDDS-2356) Multipart upload report errors while writing to ozone Ratis pipeline

2019-10-28 Thread Li Cheng (Jira)
[ https://issues.apache.org/jira/browse/HDDS-2356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16960796#comment-16960796 ] Li Cheng commented on HDDS-2356: Also it's print the same pipeline id in s3g logs like crazy. Wonder if

[jira] [Commented] (HDDS-2356) Multipart upload report errors while writing to ozone Ratis pipeline

2019-10-27 Thread Li Cheng (Jira)
[ https://issues.apache.org/jira/browse/HDDS-2356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16960744#comment-16960744 ] Li Cheng commented on HDDS-2356: [~bharat] shows up another error. See the stack:   2019-10-28

[jira] [Commented] (HDDS-2356) Multipart upload report errors while writing to ozone Ratis pipeline

2019-10-25 Thread Li Cheng (Jira)
[ https://issues.apache.org/jira/browse/HDDS-2356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16959468#comment-16959468 ] Li Cheng commented on HDDS-2356: [~bharat]Yea, you are right. It does happen randomly. Seeing it again.

[jira] [Commented] (HDDS-2356) Multipart upload report errors while writing to ozone Ratis pipeline

2019-10-24 Thread Li Cheng (Jira)
[ https://issues.apache.org/jira/browse/HDDS-2356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16959407#comment-16959407 ] Li Cheng commented on HDDS-2356: Quick update. I tried to make ozone have more handlers (like 10+ times

[jira] [Created] (HDDS-2356) Multipart upload report errors while writing to ozone Ratis pipeline

2019-10-24 Thread Li Cheng (Jira)
Li Cheng created HDDS-2356: -- Summary: Multipart upload report errors while writing to ozone Ratis pipeline Key: HDDS-2356 URL: https://issues.apache.org/jira/browse/HDDS-2356 Project: Hadoop Distributed

[jira] [Assigned] (HDDS-1576) Ensure constraint of one raft log per disk is met unless fast media

2019-10-21 Thread Li Cheng (Jira)
[ https://issues.apache.org/jira/browse/HDDS-1576?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Li Cheng reassigned HDDS-1576: -- Assignee: Li Cheng > Ensure constraint of one raft log per disk is met unless fast media >

[jira] [Commented] (HDDS-1574) Ensure same datanodes are not a part of multiple pipelines

2019-10-21 Thread Li Cheng (Jira)
[ https://issues.apache.org/jira/browse/HDDS-1574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16955811#comment-16955811 ] Li Cheng commented on HDDS-1574: [~swagle] Cloud you elaborate more to help understand this task? Do you

[jira] [Commented] (HDDS-1570) Refactor heartbeat reports to report all the pipelines that are open

2019-10-17 Thread Li Cheng (Jira)
[ https://issues.apache.org/jira/browse/HDDS-1570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16953672#comment-16953672 ] Li Cheng commented on HDDS-1570: As I look into this JIRA, I would like discuss some bout this JIRA: 1.

[jira] [Commented] (HDDS-1569) Add ability to SCM for creating multiple pipelines with same datanode

2019-10-17 Thread Li Cheng (Jira)
[ https://issues.apache.org/jira/browse/HDDS-1569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16953570#comment-16953570 ] Li Cheng commented on HDDS-1569: PR [https://github.com/apache/hadoop-ozone/pull/28] is under review. >

[jira] [Issue Comment Deleted] (HDDS-2186) Fix tests using MiniOzoneCluster for its memory related exceptions

2019-10-17 Thread Li Cheng (Jira)
[ https://issues.apache.org/jira/browse/HDDS-2186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Li Cheng updated HDDS-2186: --- Comment: was deleted (was: [https://github.com/apache/hadoop/pull/1431]) > Fix tests using MiniOzoneCluster

[jira] [Issue Comment Deleted] (HDDS-2186) Fix tests using MiniOzoneCluster for its memory related exceptions

2019-10-17 Thread Li Cheng (Jira)
[ https://issues.apache.org/jira/browse/HDDS-2186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Li Cheng updated HDDS-2186: --- Comment: was deleted (was: [https://github.com/apache/hadoop/pull/1431]) > Fix tests using MiniOzoneCluster

[jira] [Commented] (HDDS-2186) Fix tests using MiniOzoneCluster for its memory related exceptions

2019-10-17 Thread Li Cheng (Jira)
[ https://issues.apache.org/jira/browse/HDDS-2186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16953568#comment-16953568 ] Li Cheng commented on HDDS-2186: Try to resolve this in [https://github.com/apache/hadoop-ozone/pull/28]

[jira] [Work started] (HDDS-2186) Fix tests using MiniOzoneCluster for its memory related exceptions

2019-10-12 Thread Li Cheng (Jira)
[ https://issues.apache.org/jira/browse/HDDS-2186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HDDS-2186 started by Li Cheng. -- > Fix tests using MiniOzoneCluster for its memory related exceptions >

[jira] [Commented] (HDDS-2186) Fix tests using MiniOzoneCluster for its memory related exceptions

2019-10-12 Thread Li Cheng (Jira)
[ https://issues.apache.org/jira/browse/HDDS-2186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16949933#comment-16949933 ] Li Cheng commented on HDDS-2186: [https://github.com/apache/hadoop/pull/1431] > Fix tests using

[jira] [Commented] (HDDS-2186) Fix tests using MiniOzoneCluster for its memory related exceptions

2019-10-12 Thread Li Cheng (Jira)
[ https://issues.apache.org/jira/browse/HDDS-2186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16949932#comment-16949932 ] Li Cheng commented on HDDS-2186: [https://github.com/apache/hadoop/pull/1431] > Fix tests using

[jira] [Updated] (HDDS-2186) Fix tests using MiniOzoneCluster for its memory related exceptions

2019-10-12 Thread Li Cheng (Jira)
[ https://issues.apache.org/jira/browse/HDDS-2186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Li Cheng updated HDDS-2186: --- Fix Version/s: HDDS-1564 > Fix tests using MiniOzoneCluster for its memory related exceptions >

[jira] [Updated] (HDDS-2186) Fix tests using MiniOzoneCluster for its memory related exceptions

2019-10-12 Thread Li Cheng (Jira)
[ https://issues.apache.org/jira/browse/HDDS-2186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Li Cheng updated HDDS-2186: --- Component/s: test > Fix tests using MiniOzoneCluster for its memory related exceptions >

[jira] [Commented] (HDDS-2186) Fix tests using MiniOzoneCluster for its memory related exceptions

2019-10-12 Thread Li Cheng (Jira)
[ https://issues.apache.org/jira/browse/HDDS-2186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16949930#comment-16949930 ] Li Cheng commented on HDDS-2186: It turns out the miniOzoneCluster running out of memory is triggered by

[jira] [Assigned] (HDDS-2186) Fix tests using MiniOzoneCluster for its memory related exceptions

2019-10-11 Thread Li Cheng (Jira)
[ https://issues.apache.org/jira/browse/HDDS-2186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Li Cheng reassigned HDDS-2186: -- Assignee: Li Cheng > Fix tests using MiniOzoneCluster for its memory related exceptions >

[jira] [Commented] (HDDS-2186) Fix tests using MiniOzoneCluster for its memory related exceptions

2019-09-27 Thread Li Cheng (Jira)
[ https://issues.apache.org/jira/browse/HDDS-2186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16939296#comment-16939296 ] Li Cheng commented on HDDS-2186: [~ljain] After some investigation, it turned out MiniOzoneCluster is

[jira] [Commented] (HDDS-2186) Fix tests using MiniOzoneCluster for its memory related exceptions

2019-09-26 Thread Li Cheng (Jira)
[ https://issues.apache.org/jira/browse/HDDS-2186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16938670#comment-16938670 ] Li Cheng commented on HDDS-2186: Note that CI is also affected and it cannot print out the correct output

[jira] [Updated] (HDDS-2186) Fix tests using MiniOzoneCluster for its memory related exceptions

2019-09-26 Thread Li Cheng (Jira)
[ https://issues.apache.org/jira/browse/HDDS-2186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Li Cheng updated HDDS-2186: --- Labels: flaky-test (was: ) > Fix tests using MiniOzoneCluster for its memory related exceptions >

[jira] [Updated] (HDDS-2186) Fix tests using MiniOzoneCluster for its memory related exceptions

2019-09-26 Thread Li Cheng (Jira)
[ https://issues.apache.org/jira/browse/HDDS-2186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Li Cheng updated HDDS-2186: --- Affects Version/s: HDDS-1564 > Fix tests using MiniOzoneCluster for its memory related exceptions >

[jira] [Created] (HDDS-2186) Fix tests using MiniOzoneCluster for its memory related exceptions

2019-09-26 Thread Li Cheng (Jira)
Li Cheng created HDDS-2186: -- Summary: Fix tests using MiniOzoneCluster for its memory related exceptions Key: HDDS-2186 URL: https://issues.apache.org/jira/browse/HDDS-2186 Project: Hadoop Distributed Data

[jira] [Assigned] (HDDS-1574) Ensure same datanodes are not a part of multiple pipelines

2019-09-19 Thread Li Cheng (Jira)
[ https://issues.apache.org/jira/browse/HDDS-1574?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Li Cheng reassigned HDDS-1574: -- Assignee: Li Cheng (was: Siddharth Wagle) > Ensure same datanodes are not a part of multiple

[jira] [Assigned] (HDDS-1570) Refactor heartbeat reports to report all the pipelines that are open

2019-09-19 Thread Li Cheng (Jira)
[ https://issues.apache.org/jira/browse/HDDS-1570?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Li Cheng reassigned HDDS-1570: -- Assignee: Li Cheng (was: Siddharth Wagle) > Refactor heartbeat reports to report all the pipelines

[jira] [Commented] (HDDS-1933) Datanode should use hostname in place of ip addresses to allow DN's to work when ipaddress change

2019-09-19 Thread Li Cheng (Jira)
[ https://issues.apache.org/jira/browse/HDDS-1933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16933136#comment-16933136 ] Li Cheng commented on HDDS-1933: After discussion, we suppose the change could remain as it is now. And

[jira] [Commented] (HDDS-1933) Datanode should use hostname in place of ip addresses to allow DN's to work when ipaddress change

2019-09-18 Thread Li Cheng (Jira)
[ https://issues.apache.org/jira/browse/HDDS-1933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16932172#comment-16932172 ] Li Cheng commented on HDDS-1933: [~swagle] I was talking with [~Sammi] about this. How do we decide

[jira] [Commented] (HDDS-2116) Create SCMPipelineAllocationManager as background thread for pipeline creation

2019-09-18 Thread Li Cheng (Jira)
[ https://issues.apache.org/jira/browse/HDDS-2116?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16932146#comment-16932146 ] Li Cheng commented on HDDS-2116: [~swagle] I think for now the role is taken by PipelinePlacementPolicy,

[jira] [Commented] (HDDS-2116) Create SCMAllocationManager as background thread for pipeline creation

2019-09-17 Thread Li Cheng (Jira)
[ https://issues.apache.org/jira/browse/HDDS-2116?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16931329#comment-16931329 ] Li Cheng commented on HDDS-2116: I agree on the multi-thread part. [~swagle] what else function do you

[jira] [Commented] (HDDS-1569) Add ability to SCM for creating multiple pipelines with same datanode

2019-09-12 Thread Li Cheng (Jira)
[ https://issues.apache.org/jira/browse/HDDS-1569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16928362#comment-16928362 ] Li Cheng commented on HDDS-1569: I separate SCMAllocationManager to another Jira 

[jira] [Created] (HDDS-2116) Create SCMAllocationManager as background thread for pipeline creation

2019-09-12 Thread Li Cheng (Jira)
Li Cheng created HDDS-2116: -- Summary: Create SCMAllocationManager as background thread for pipeline creation Key: HDDS-2116 URL: https://issues.apache.org/jira/browse/HDDS-2116 Project: Hadoop Distributed

[jira] [Assigned] (HDDS-2115) Add smoke/acceptance test for Pipeline related CLI

2019-09-12 Thread Li Cheng (Jira)
[ https://issues.apache.org/jira/browse/HDDS-2115?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Li Cheng reassigned HDDS-2115: -- Assignee: Li Cheng > Add smoke/acceptance test for Pipeline related CLI >

[jira] [Created] (HDDS-2115) Add smoke/acceptance test for Pipeline related CLI

2019-09-12 Thread Li Cheng (Jira)
Li Cheng created HDDS-2115: -- Summary: Add smoke/acceptance test for Pipeline related CLI Key: HDDS-2115 URL: https://issues.apache.org/jira/browse/HDDS-2115 Project: Hadoop Distributed Data Store

[jira] [Resolved] (HDDS-1571) Create an interface for pipeline placement policy to support network topologies

2019-09-10 Thread Li Cheng (Jira)
[ https://issues.apache.org/jira/browse/HDDS-1571?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Li Cheng resolved HDDS-1571. Fix Version/s: 0.5.0 Resolution: Fixed > Create an interface for pipeline placement policy to

[jira] [Assigned] (HDDS-1564) Ozone multi-raft support

2019-09-09 Thread Li Cheng (Jira)
[ https://issues.apache.org/jira/browse/HDDS-1564?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Li Cheng reassigned HDDS-1564: -- Assignee: Li Cheng (was: Siddharth Wagle) > Ozone multi-raft support > > >

[jira] [Created] (HDDS-2089) Add CLI createPipeline

2019-09-05 Thread Li Cheng (Jira)
Li Cheng created HDDS-2089: -- Summary: Add CLI createPipeline Key: HDDS-2089 URL: https://issues.apache.org/jira/browse/HDDS-2089 Project: Hadoop Distributed Data Store Issue Type: Sub-task

[jira] [Commented] (HDDS-1571) Create an interface for pipeline placement policy to support network topologies

2019-09-05 Thread Li Cheng (Jira)
[ https://issues.apache.org/jira/browse/HDDS-1571?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16923227#comment-16923227 ] Li Cheng commented on HDDS-1571: Put up a PlacementPolicy as basic interface and ScmCommonPlacementPolicy

[jira] [Resolved] (HDDS-1577) Add default pipeline placement policy implementation

2019-09-04 Thread Li Cheng (Jira)
[ https://issues.apache.org/jira/browse/HDDS-1577?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Li Cheng resolved HDDS-1577. Fix Version/s: 0.5.0 Resolution: Fixed > Add default pipeline placement policy implementation >

  1   2   >