[jira] [Created] (HDDS-2458) Avoid list copy in ChecksumData
Attila Doroszlai created HDDS-2458:
--
Summary: Avoid list copy in ChecksumData
Key: HDDS-2458
URL: https://issues.apache.org/jira/browse/HDDS-2458
Project: Hadoop Distributed Data Store
Issue Type: Improvement
Components: Ozone Datanode
Reporter: Attila Doroszlai
Assignee: Attila Doroszlai
{{ChecksumData}} is initially created with empty list of checksums, then it is
updated with computed checksums, copying the list. The computed list can be
set directly.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-2356) Multipart upload report errors while writing to ozone Ratis pipeline
[ https://issues.apache.org/jira/browse/HDDS-2356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16971357#comment-16971357 ] Sammi Chen commented on HDDS-2356: -- Hi [~bharat], thanks for helping fix the multi-upload issues. Li and I am working on enable Ozone in Tencent's production environment. Currenlty we have two main blocking issues, one is this multi-upload, another is performance. Mukul and Shashi are helping us with the performance improvement. This multi-upload issue is consistenly happen in our environement with big file with size, say 5GB. It will be more efficient if you would try to reproduce the case locally. We would love to assit if you need any reproduce help. > Multipart upload report errors while writing to ozone Ratis pipeline > > > Key: HDDS-2356 > URL: https://issues.apache.org/jira/browse/HDDS-2356 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: Ozone Manager >Affects Versions: 0.4.1 > Environment: Env: 4 VMs in total: 3 Datanodes on 3 VMs, 1 OM & 1 SCM > on a separate VM >Reporter: Li Cheng >Assignee: Bharat Viswanadham >Priority: Blocker > Fix For: 0.5.0 > > Attachments: 2019-11-06_18_13_57_422_ERROR, hs_err_pid9340.log, > image-2019-10-31-18-56-56-177.png, om_audit_log_plc_1570863541668_9278.txt > > > Env: 4 VMs in total: 3 Datanodes on 3 VMs, 1 OM & 1 SCM on a separate VM, say > it's VM0. > I use goofys as a fuse and enable ozone S3 gateway to mount ozone to a path > on VM0, while reading data from VM0 local disk and write to mount path. The > dataset has various sizes of files from 0 byte to GB-level and it has a > number of ~50,000 files. > The writing is slow (1GB for ~10 mins) and it stops after around 4GB. As I > look at hadoop-root-om-VM_50_210_centos.out log, I see OM throwing errors > related with Multipart upload. This error eventually causes the writing to > terminate and OM to be closed. > > Updated on 11/06/2019: > See new multipart upload error NO_SUCH_MULTIPART_UPLOAD_ERROR and full logs > are in the attachment. > 2019-11-05 18:12:37,766 ERROR > org.apache.hadoop.ozone.om.request.s3.multipart.S3MultipartUploadCommitPartRequest: > MultipartUpload Commit is failed for Key:./2 > 0191012/plc_1570863541668_9278 in Volume/Bucket > s325d55ad283aa400af464c76d713c07ad/ozone-test > NO_SUCH_MULTIPART_UPLOAD_ERROR > org.apache.hadoop.ozone.om.exceptions.OMException: No such Multipart upload > is with specified uploadId fcda8608-b431-48b7-8386- > 0a332f1a709a-103084683261641950 > at > org.apache.hadoop.ozone.om.request.s3.multipart.S3MultipartUploadCommitPartRequest.validateAndUpdateCache(S3MultipartUploadCommitPartRequest.java:1 > 56) > at > org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.submitRequestDirectlyToOM(OzoneManagerProtocolServerSideTranslatorPB. > java:217) > at > org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.processRequest(OzoneManagerProtocolServerSideTranslatorPB.java:132) > at > org.apache.hadoop.hdds.server.OzoneProtocolMessageDispatcher.processRequest(OzoneProtocolMessageDispatcher.java:72) > at > org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.submitRequest(OzoneManagerProtocolServerSideTranslatorPB.java:100) > at > org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos$OzoneManagerService$2.callBlockingMethod(OzoneManagerProtocolProtos.java) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:524) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1025) > at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:876) > at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:822) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2682) > > Updated on 10/28/2019: > See MISMATCH_MULTIPART_LIST error. > > 2019-10-28 11:44:34,079 [qtp1383524016-70] ERROR - Error in Complete > Multipart Upload Request for bucket: ozone-test, key: > 20191012/plc_1570863541668_927 > 8 > MISMATCH_MULTIPART_LIST org.apache.hadoop.ozone.om.exceptions.OMException: > Complete Multipart Upload Failed: volume: > s3c89e813c80ffcea9543004d57b2a1239bucket: > ozone-testkey: 20191012/plc_1570863541668_9278 > at > org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.handleError(OzoneManagerProtocolClientSideTranslatorPB.java:732) > at >
[jira] [Comment Edited] (HDDS-2356) Multipart upload report errors while writing to ozone Ratis pipeline
[
https://issues.apache.org/jira/browse/HDDS-2356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16971290#comment-16971290
]
Li Cheng edited comment on HDDS-2356 at 11/11/19 7:21 AM:
--
[~bharat] In terms of the key in the last stacktrace:
2019-11-08 20:08:24,832 ERROR
org.apache.hadoop.ozone.om.request.s3.multipart.S3MultipartUploadCompleteRequest:
MultipartUpload Complete request failed for Key: plc_1570863541668_9278 in
Volume/Bucket s325d55ad283aa400af464c76d713c07ad/ozone-test
INVALID_PART org.apache.hadoop.ozone.om.exceptions.OMException: Complete
Multipart Upload Failed: volume: s325d55ad283aa400af464c76d713c07adbucket:
ozone-testkey: plc_1570863541668_9278
OM audit logs show a bunch of entries for key plc_1570863541668_9278 with
different clientID, for instance:
2019-11-08 20:19:56,241 | INFO | OMAudit | user=root | ip=9.134.50.210 |
op=ALLOCATE_BLOCK \{volume=s325d55ad283aa400af464c76d713c07ad,
bucket=ozone-test, key=plc_1570863541668_9278, dataSize=5242880,
replicationType=RATIS, replicationFactor=THREE, keyLocationInfo=[],
clientID=103102209394803336} | ret=SUCCESS |
I'm uploading the entire OM audit logs about this key plc_1570863541668_9278.
This file size:
[root@VM_50_210_centos /data/idex_data/zip]# ls -altr -h
./20191012/plc_1570863541668_9278
-rw-r--r-- 1 1003 users 1.4G 10月 22 10:33 ./20191012/plc_1570863541668_9278
You can try creating a similar size of file on you own and see it reproduces
the issue. Please refer to the description for env details.
was (Author: timmylicheng):
[~bharat] In terms of the key in the last stacktrace:
2019-11-08 20:08:24,832 ERROR
org.apache.hadoop.ozone.om.request.s3.multipart.S3MultipartUploadCompleteRequest:
MultipartUpload Complete request failed for Key: plc_1570863541668_9278 in
Volume/Bucket s325d55ad283aa400af464c76d713c07ad/ozone-test
INVALID_PART org.apache.hadoop.ozone.om.exceptions.OMException: Complete
Multipart Upload Failed: volume: s325d55ad283aa400af464c76d713c07adbucket:
ozone-testkey: plc_1570863541668_9278
OM audit logs show a bunch of entries for key plc_1570863541668_9278 with
different clientID, for instance:
2019-11-08 20:19:56,241 | INFO | OMAudit | user=root | ip=9.134.50.210 |
op=ALLOCATE_BLOCK \{volume=s325d55ad283aa400af464c76d713c07ad,
bucket=ozone-test, key=plc_1570863541668_9278, dataSize=5242880,
replicationType=RATIS, replicationFactor=THREE, keyLocationInfo=[],
clientID=103102209394803336} | ret=SUCCESS |
I'm uploading the entire OM audit logs about this key plc_1570863541668_9278.
> Multipart upload report errors while writing to ozone Ratis pipeline
>
>
> Key: HDDS-2356
> URL: https://issues.apache.org/jira/browse/HDDS-2356
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: Ozone Manager
>Affects Versions: 0.4.1
> Environment: Env: 4 VMs in total: 3 Datanodes on 3 VMs, 1 OM & 1 SCM
> on a separate VM
>Reporter: Li Cheng
>Assignee: Bharat Viswanadham
>Priority: Blocker
> Fix For: 0.5.0
>
> Attachments: 2019-11-06_18_13_57_422_ERROR, hs_err_pid9340.log,
> image-2019-10-31-18-56-56-177.png, om_audit_log_plc_1570863541668_9278.txt
>
>
> Env: 4 VMs in total: 3 Datanodes on 3 VMs, 1 OM & 1 SCM on a separate VM, say
> it's VM0.
> I use goofys as a fuse and enable ozone S3 gateway to mount ozone to a path
> on VM0, while reading data from VM0 local disk and write to mount path. The
> dataset has various sizes of files from 0 byte to GB-level and it has a
> number of ~50,000 files.
> The writing is slow (1GB for ~10 mins) and it stops after around 4GB. As I
> look at hadoop-root-om-VM_50_210_centos.out log, I see OM throwing errors
> related with Multipart upload. This error eventually causes the writing to
> terminate and OM to be closed.
>
> Updated on 11/06/2019:
> See new multipart upload error NO_SUCH_MULTIPART_UPLOAD_ERROR and full logs
> are in the attachment.
> 2019-11-05 18:12:37,766 ERROR
> org.apache.hadoop.ozone.om.request.s3.multipart.S3MultipartUploadCommitPartRequest:
> MultipartUpload Commit is failed for Key:./2
> 0191012/plc_1570863541668_9278 in Volume/Bucket
> s325d55ad283aa400af464c76d713c07ad/ozone-test
> NO_SUCH_MULTIPART_UPLOAD_ERROR
> org.apache.hadoop.ozone.om.exceptions.OMException: No such Multipart upload
> is with specified uploadId fcda8608-b431-48b7-8386-
> 0a332f1a709a-103084683261641950
> at
> org.apache.hadoop.ozone.om.request.s3.multipart.S3MultipartUploadCommitPartRequest.validateAndUpdateCache(S3MultipartUploadCommitPartRequest.java:1
> 56)
> at
> org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.submitRequestDirectlyToOM(OzoneManagerProtocolServerSideTranslatorPB.
> java:217)
> at
>
[jira] [Commented] (HDFS-14971) HDFS : help info of fsck "-list-corruptfileblocks" command needs to be rectified
[
https://issues.apache.org/jira/browse/HDFS-14971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16971343#comment-16971343
]
Hadoop QA commented on HDFS-14971:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 1m
16s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 24m
26s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
17s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
54s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
25s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
16m 37s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
36s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
25s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
15m 47s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
31s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}104m 29s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
34s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}179m 48s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.tools.TestDFSZKFailoverController |
| | hadoop.hdfs.server.balancer.TestBalancer |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.4 Server=19.03.4 Image:yetus/hadoop:104ccca9169 |
| JIRA Issue | HDFS-14971 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12985489/HDFS-14971.001.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 026d9502e262 4.15.0-66-generic #75-Ubuntu SMP Tue Oct 1
05:24:09 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 320008b |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_222 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/28287/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/28287/testReport/ |
| Max. process+thread count | 2886 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
[jira] [Updated] (HDFS-14975) Add LF for SetECPolicyCommand usage
[
https://issues.apache.org/jira/browse/HDFS-14975?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Fei Hui updated HDFS-14975:
---
Summary: Add LF for SetECPolicyCommand usage (was: Add CR for
SetECPolicyCommand usage)
> Add LF for SetECPolicyCommand usage
> ---
>
> Key: HDFS-14975
> URL: https://issues.apache.org/jira/browse/HDFS-14975
> Project: Hadoop HDFS
> Issue Type: Improvement
>Affects Versions: 3.3.0
>Reporter: Fei Hui
>Assignee: Fei Hui
>Priority: Trivial
> Fix For: 3.3.0
>
> Attachments: HDFS-14975.001.patch
>
>
> *bin/hdfs ec -help* output the following message
> {quote}
> [-listPolicies]
> Get the list of all erasure coding policies.
> [-addPolicies -policyFile ]
> Add a list of user defined erasure coding policies.
> The path of the xml file which defines the EC policies to add
> [-getPolicy -path ]
> Get the erasure coding policy of a file/directory.
> The path of the file/directory for getting the erasure coding policy
> [-removePolicy -policy ]
> Remove an user defined erasure coding policy.
> The name of the erasure coding policy
> [-setPolicy -path [-policy ] [-replicate]]
> Set the erasure coding policy for a file/directory.
> The path of the file/directory to set the erasure coding policy
> The name of the erasure coding policy
> -replicate force 3x replication scheme on the directory
> -replicate and -policy are optional arguments. They cannot been used at the
> same time
> [-unsetPolicy -path ]
> Unset the erasure coding policy for a directory.
> The path of the directory from which the erasure coding policy will
> be
> unset.
>
> [-listCodecs]
> Get the list of supported erasure coding codecs and coders.
> A coder is an implementation of a codec. A codec can have different
> implementations, thus different coders.
> The coders for a codec are listed in a fall back order.
> [-enablePolicy -policy ]
> Enable the erasure coding policy.
> The name of the erasure coding policy
> [-disablePolicy -policy ]
> Disable the erasure coding policy.
> The name of the erasure coding policy
> [-verifyClusterSetup [-policy ...]]
> Verify if the cluster setup can support all enabled erasure coding policies.
> If optional parameter -policy is specified, verify if the cluster setup can
> support the given policy.
> {quote}
> The output format is not so good to users
> We should add CR between SetECPolicyCommand and UnsetECPolicyCommand like
> other commands
> {quote}
> [-setPolicy -path [-policy ] [-replicate]]
> Set the erasure coding policy for a file/directory.
> The path of the file/directory to set the erasure coding policy
> The name of the erasure coding policy
> -replicate force 3x replication scheme on the directory
> -replicate and -policy are optional arguments. They cannot been used at the
> same time
> -here-
> [-unsetPolicy -path ]
> Unset the erasure coding policy for a directory.
> The path of the directory from which the erasure coding policy will be
> unset.
> {quote}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14975) Add CR for SetECPolicyCommand usage
[
https://issues.apache.org/jira/browse/HDFS-14975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16971340#comment-16971340
]
Fei Hui commented on HDFS-14975:
[~aajisaka] Thanks
Commit message should be changed to LF
> Add CR for SetECPolicyCommand usage
> ---
>
> Key: HDFS-14975
> URL: https://issues.apache.org/jira/browse/HDFS-14975
> Project: Hadoop HDFS
> Issue Type: Improvement
>Affects Versions: 3.3.0
>Reporter: Fei Hui
>Assignee: Fei Hui
>Priority: Trivial
> Fix For: 3.3.0
>
> Attachments: HDFS-14975.001.patch
>
>
> *bin/hdfs ec -help* output the following message
> {quote}
> [-listPolicies]
> Get the list of all erasure coding policies.
> [-addPolicies -policyFile ]
> Add a list of user defined erasure coding policies.
> The path of the xml file which defines the EC policies to add
> [-getPolicy -path ]
> Get the erasure coding policy of a file/directory.
> The path of the file/directory for getting the erasure coding policy
> [-removePolicy -policy ]
> Remove an user defined erasure coding policy.
> The name of the erasure coding policy
> [-setPolicy -path [-policy ] [-replicate]]
> Set the erasure coding policy for a file/directory.
> The path of the file/directory to set the erasure coding policy
> The name of the erasure coding policy
> -replicate force 3x replication scheme on the directory
> -replicate and -policy are optional arguments. They cannot been used at the
> same time
> [-unsetPolicy -path ]
> Unset the erasure coding policy for a directory.
> The path of the directory from which the erasure coding policy will
> be
> unset.
>
> [-listCodecs]
> Get the list of supported erasure coding codecs and coders.
> A coder is an implementation of a codec. A codec can have different
> implementations, thus different coders.
> The coders for a codec are listed in a fall back order.
> [-enablePolicy -policy ]
> Enable the erasure coding policy.
> The name of the erasure coding policy
> [-disablePolicy -policy ]
> Disable the erasure coding policy.
> The name of the erasure coding policy
> [-verifyClusterSetup [-policy ...]]
> Verify if the cluster setup can support all enabled erasure coding policies.
> If optional parameter -policy is specified, verify if the cluster setup can
> support the given policy.
> {quote}
> The output format is not so good to users
> We should add CR between SetECPolicyCommand and UnsetECPolicyCommand like
> other commands
> {quote}
> [-setPolicy -path [-policy ] [-replicate]]
> Set the erasure coding policy for a file/directory.
> The path of the file/directory to set the erasure coding policy
> The name of the erasure coding policy
> -replicate force 3x replication scheme on the directory
> -replicate and -policy are optional arguments. They cannot been used at the
> same time
> -here-
> [-unsetPolicy -path ]
> Unset the erasure coding policy for a directory.
> The path of the directory from which the erasure coding policy will be
> unset.
> {quote}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14975) Add CR for SetECPolicyCommand usage
[
https://issues.apache.org/jira/browse/HDFS-14975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16971331#comment-16971331
]
Akira Ajisaka commented on HDFS-14975:
--
Thanks [~ferhui] and [~ayushtkn] for the fix.
Note that the commit log is confusing. Actually LF ('\n') has been added
instead of CR.
> Add CR for SetECPolicyCommand usage
> ---
>
> Key: HDFS-14975
> URL: https://issues.apache.org/jira/browse/HDFS-14975
> Project: Hadoop HDFS
> Issue Type: Improvement
>Affects Versions: 3.3.0
>Reporter: Fei Hui
>Assignee: Fei Hui
>Priority: Trivial
> Fix For: 3.3.0
>
> Attachments: HDFS-14975.001.patch
>
>
> *bin/hdfs ec -help* output the following message
> {quote}
> [-listPolicies]
> Get the list of all erasure coding policies.
> [-addPolicies -policyFile ]
> Add a list of user defined erasure coding policies.
> The path of the xml file which defines the EC policies to add
> [-getPolicy -path ]
> Get the erasure coding policy of a file/directory.
> The path of the file/directory for getting the erasure coding policy
> [-removePolicy -policy ]
> Remove an user defined erasure coding policy.
> The name of the erasure coding policy
> [-setPolicy -path [-policy ] [-replicate]]
> Set the erasure coding policy for a file/directory.
> The path of the file/directory to set the erasure coding policy
> The name of the erasure coding policy
> -replicate force 3x replication scheme on the directory
> -replicate and -policy are optional arguments. They cannot been used at the
> same time
> [-unsetPolicy -path ]
> Unset the erasure coding policy for a directory.
> The path of the directory from which the erasure coding policy will
> be
> unset.
>
> [-listCodecs]
> Get the list of supported erasure coding codecs and coders.
> A coder is an implementation of a codec. A codec can have different
> implementations, thus different coders.
> The coders for a codec are listed in a fall back order.
> [-enablePolicy -policy ]
> Enable the erasure coding policy.
> The name of the erasure coding policy
> [-disablePolicy -policy ]
> Disable the erasure coding policy.
> The name of the erasure coding policy
> [-verifyClusterSetup [-policy ...]]
> Verify if the cluster setup can support all enabled erasure coding policies.
> If optional parameter -policy is specified, verify if the cluster setup can
> support the given policy.
> {quote}
> The output format is not so good to users
> We should add CR between SetECPolicyCommand and UnsetECPolicyCommand like
> other commands
> {quote}
> [-setPolicy -path [-policy ] [-replicate]]
> Set the erasure coding policy for a file/directory.
> The path of the file/directory to set the erasure coding policy
> The name of the erasure coding policy
> -replicate force 3x replication scheme on the directory
> -replicate and -policy are optional arguments. They cannot been used at the
> same time
> -here-
> [-unsetPolicy -path ]
> Unset the erasure coding policy for a directory.
> The path of the directory from which the erasure coding policy will be
> unset.
> {quote}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14975) Add CR for SetECPolicyCommand usage
[
https://issues.apache.org/jira/browse/HDFS-14975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16971327#comment-16971327
]
Hudson commented on HDFS-14975:
---
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #17625 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/17625/])
HDFS-14975. Add CR for SetECPolicyCommand usage. Contributed by Fei Hui.
(ayushsaxena: rev 77934bc07b9bef3e129826e93e1c1e8a47c00c95)
* (edit)
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/ECAdmin.java
> Add CR for SetECPolicyCommand usage
> ---
>
> Key: HDFS-14975
> URL: https://issues.apache.org/jira/browse/HDFS-14975
> Project: Hadoop HDFS
> Issue Type: Improvement
>Affects Versions: 3.3.0
>Reporter: Fei Hui
>Assignee: Fei Hui
>Priority: Trivial
> Fix For: 3.3.0
>
> Attachments: HDFS-14975.001.patch
>
>
> *bin/hdfs ec -help* output the following message
> {quote}
> [-listPolicies]
> Get the list of all erasure coding policies.
> [-addPolicies -policyFile ]
> Add a list of user defined erasure coding policies.
> The path of the xml file which defines the EC policies to add
> [-getPolicy -path ]
> Get the erasure coding policy of a file/directory.
> The path of the file/directory for getting the erasure coding policy
> [-removePolicy -policy ]
> Remove an user defined erasure coding policy.
> The name of the erasure coding policy
> [-setPolicy -path [-policy ] [-replicate]]
> Set the erasure coding policy for a file/directory.
> The path of the file/directory to set the erasure coding policy
> The name of the erasure coding policy
> -replicate force 3x replication scheme on the directory
> -replicate and -policy are optional arguments. They cannot been used at the
> same time
> [-unsetPolicy -path ]
> Unset the erasure coding policy for a directory.
> The path of the directory from which the erasure coding policy will
> be
> unset.
>
> [-listCodecs]
> Get the list of supported erasure coding codecs and coders.
> A coder is an implementation of a codec. A codec can have different
> implementations, thus different coders.
> The coders for a codec are listed in a fall back order.
> [-enablePolicy -policy ]
> Enable the erasure coding policy.
> The name of the erasure coding policy
> [-disablePolicy -policy ]
> Disable the erasure coding policy.
> The name of the erasure coding policy
> [-verifyClusterSetup [-policy ...]]
> Verify if the cluster setup can support all enabled erasure coding policies.
> If optional parameter -policy is specified, verify if the cluster setup can
> support the given policy.
> {quote}
> The output format is not so good to users
> We should add CR between SetECPolicyCommand and UnsetECPolicyCommand like
> other commands
> {quote}
> [-setPolicy -path [-policy ] [-replicate]]
> Set the erasure coding policy for a file/directory.
> The path of the file/directory to set the erasure coding policy
> The name of the erasure coding policy
> -replicate force 3x replication scheme on the directory
> -replicate and -policy are optional arguments. They cannot been used at the
> same time
> -here-
> [-unsetPolicy -path ]
> Unset the erasure coding policy for a directory.
> The path of the directory from which the erasure coding policy will be
> unset.
> {quote}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14648) DeadNodeDetector basic model
[
https://issues.apache.org/jira/browse/HDFS-14648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16971324#comment-16971324
]
Hadoop QA commented on HDFS-14648:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
47s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
19s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m
6s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
53s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
51s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
15m 30s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
15s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
41s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
11s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 3m
9s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 50s{color} | {color:orange} hadoop-hdfs-project: The patch generated 2 new +
112 unchanged - 1 fixed = 114 total (was 113) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 30s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
38s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
52s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}108m 18s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
46s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}184m 58s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.hdfs.server.namenode.TestAddOverReplicatedStripedBlocks |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.4 Server=19.03.4 Image:yetus/hadoop:104ccca9169 |
| JIRA Issue | HDFS-14648 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12985481/HDFS-14648.007.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle xml |
| uname | Linux 338d8e78a090 4.15.0-66-generic #75-Ubuntu SMP Tue Oct 1
05:24:09 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
[jira] [Updated] (HDFS-14975) Add CR for SetECPolicyCommand usage
[
https://issues.apache.org/jira/browse/HDFS-14975?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ayush Saxena updated HDFS-14975:
Fix Version/s: 3.3.0
Hadoop Flags: Reviewed
Resolution: Fixed
Status: Resolved (was: Patch Available)
Committed to trunk!!!
> Add CR for SetECPolicyCommand usage
> ---
>
> Key: HDFS-14975
> URL: https://issues.apache.org/jira/browse/HDFS-14975
> Project: Hadoop HDFS
> Issue Type: Improvement
>Affects Versions: 3.3.0
>Reporter: Fei Hui
>Assignee: Fei Hui
>Priority: Trivial
> Fix For: 3.3.0
>
> Attachments: HDFS-14975.001.patch
>
>
> *bin/hdfs ec -help* output the following message
> {quote}
> [-listPolicies]
> Get the list of all erasure coding policies.
> [-addPolicies -policyFile ]
> Add a list of user defined erasure coding policies.
> The path of the xml file which defines the EC policies to add
> [-getPolicy -path ]
> Get the erasure coding policy of a file/directory.
> The path of the file/directory for getting the erasure coding policy
> [-removePolicy -policy ]
> Remove an user defined erasure coding policy.
> The name of the erasure coding policy
> [-setPolicy -path [-policy ] [-replicate]]
> Set the erasure coding policy for a file/directory.
> The path of the file/directory to set the erasure coding policy
> The name of the erasure coding policy
> -replicate force 3x replication scheme on the directory
> -replicate and -policy are optional arguments. They cannot been used at the
> same time
> [-unsetPolicy -path ]
> Unset the erasure coding policy for a directory.
> The path of the directory from which the erasure coding policy will
> be
> unset.
>
> [-listCodecs]
> Get the list of supported erasure coding codecs and coders.
> A coder is an implementation of a codec. A codec can have different
> implementations, thus different coders.
> The coders for a codec are listed in a fall back order.
> [-enablePolicy -policy ]
> Enable the erasure coding policy.
> The name of the erasure coding policy
> [-disablePolicy -policy ]
> Disable the erasure coding policy.
> The name of the erasure coding policy
> [-verifyClusterSetup [-policy ...]]
> Verify if the cluster setup can support all enabled erasure coding policies.
> If optional parameter -policy is specified, verify if the cluster setup can
> support the given policy.
> {quote}
> The output format is not so good to users
> We should add CR between SetECPolicyCommand and UnsetECPolicyCommand like
> other commands
> {quote}
> [-setPolicy -path [-policy ] [-replicate]]
> Set the erasure coding policy for a file/directory.
> The path of the file/directory to set the erasure coding policy
> The name of the erasure coding policy
> -replicate force 3x replication scheme on the directory
> -replicate and -policy are optional arguments. They cannot been used at the
> same time
> -here-
> [-unsetPolicy -path ]
> Unset the erasure coding policy for a directory.
> The path of the directory from which the erasure coding policy will be
> unset.
> {quote}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14962) RBF: ConnectionPool#newConnection() error log wrong protocol class
[
https://issues.apache.org/jira/browse/HDFS-14962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16971321#comment-16971321
]
Hudson commented on HDFS-14962:
---
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #17624 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/17624/])
HDFS-14962. RBF: ConnectionPool#newConnection() error log wrong protocol
(ayushsaxena: rev b25a37c3229e1a66699d649f6caf80ffc71db5b8)
* (edit)
hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/ConnectionPool.java
* (edit)
hadoop-hdfs-project/hadoop-hdfs-rbf/src/test/java/org/apache/hadoop/hdfs/server/federation/router/TestConnectionManager.java
> RBF: ConnectionPool#newConnection() error log wrong protocol class
> --
>
> Key: HDFS-14962
> URL: https://issues.apache.org/jira/browse/HDFS-14962
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: rbf
>Affects Versions: 3.3.0
>Reporter: Yuxuan Wang
>Assignee: Yuxuan Wang
>Priority: Minor
> Labels: RBF
> Fix For: 3.3.0
>
>
> ConnectionPool#newConnection() has following code:
> {code:java}
> String msg = "Unsupported protocol for connection to NameNode: "
> + ((proto != null) ? proto.getClass().getName() : "null");
> {code}
> *proto.getClass().getName()* should be *proto.getName()*
> My IDE can figure out the issue.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14975) Add CR for SetECPolicyCommand usage
[
https://issues.apache.org/jira/browse/HDFS-14975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16971320#comment-16971320
]
Ayush Saxena commented on HDFS-14975:
-
+1
> Add CR for SetECPolicyCommand usage
> ---
>
> Key: HDFS-14975
> URL: https://issues.apache.org/jira/browse/HDFS-14975
> Project: Hadoop HDFS
> Issue Type: Improvement
>Affects Versions: 3.3.0
>Reporter: Fei Hui
>Assignee: Fei Hui
>Priority: Trivial
> Attachments: HDFS-14975.001.patch
>
>
> *bin/hdfs ec -help* output the following message
> {quote}
> [-listPolicies]
> Get the list of all erasure coding policies.
> [-addPolicies -policyFile ]
> Add a list of user defined erasure coding policies.
> The path of the xml file which defines the EC policies to add
> [-getPolicy -path ]
> Get the erasure coding policy of a file/directory.
> The path of the file/directory for getting the erasure coding policy
> [-removePolicy -policy ]
> Remove an user defined erasure coding policy.
> The name of the erasure coding policy
> [-setPolicy -path [-policy ] [-replicate]]
> Set the erasure coding policy for a file/directory.
> The path of the file/directory to set the erasure coding policy
> The name of the erasure coding policy
> -replicate force 3x replication scheme on the directory
> -replicate and -policy are optional arguments. They cannot been used at the
> same time
> [-unsetPolicy -path ]
> Unset the erasure coding policy for a directory.
> The path of the directory from which the erasure coding policy will
> be
> unset.
>
> [-listCodecs]
> Get the list of supported erasure coding codecs and coders.
> A coder is an implementation of a codec. A codec can have different
> implementations, thus different coders.
> The coders for a codec are listed in a fall back order.
> [-enablePolicy -policy ]
> Enable the erasure coding policy.
> The name of the erasure coding policy
> [-disablePolicy -policy ]
> Disable the erasure coding policy.
> The name of the erasure coding policy
> [-verifyClusterSetup [-policy ...]]
> Verify if the cluster setup can support all enabled erasure coding policies.
> If optional parameter -policy is specified, verify if the cluster setup can
> support the given policy.
> {quote}
> The output format is not so good to users
> We should add CR between SetECPolicyCommand and UnsetECPolicyCommand like
> other commands
> {quote}
> [-setPolicy -path [-policy ] [-replicate]]
> Set the erasure coding policy for a file/directory.
> The path of the file/directory to set the erasure coding policy
> The name of the erasure coding policy
> -replicate force 3x replication scheme on the directory
> -replicate and -policy are optional arguments. They cannot been used at the
> same time
> -here-
> [-unsetPolicy -path ]
> Unset the erasure coding policy for a directory.
> The path of the directory from which the erasure coding policy will be
> unset.
> {quote}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14928) UI: unifying the WebUI across different components.
[ https://issues.apache.org/jira/browse/HDFS-14928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16971319#comment-16971319 ] Xieming Li commented on HDFS-14928: --- [~elgoiri] [~tasanuma] Thank you for the review. > UI: unifying the WebUI across different components. > --- > > Key: HDFS-14928 > URL: https://issues.apache.org/jira/browse/HDFS-14928 > Project: Hadoop HDFS > Issue Type: Improvement > Components: ui >Reporter: Xieming Li >Assignee: Xieming Li >Priority: Trivial > Fix For: 3.3.0 > > Attachments: DN_orig.png, DN_with_legend.png.png, DN_wo_legend.png, > HDFS-14892-2.jpg, HDFS-14928.001.patch, HDFS-14928.002.patch, > HDFS-14928.003.patch, HDFS-14928.004.patch, HDFS-14928.jpg, NN_orig.png, > NN_with_legend.png, NN_wo_legend.png, RBF_orig.png, RBF_with_legend.png, > RBF_wo_legend.png > > > The WebUI of different components could be unified. > *Router:* > |Current| !RBF_orig.png|width=500! | > |Proposed 1 (With Icon) | !RBF_wo_legend.png|width=500! | > |Proposed 2 (With Icon and Legend)|!RBF_with_legend.png|width=500! | > *NameNode:* > |Current| !NN_orig.png|width=500! | > |Proposed 1 (With Icon) | !NN_wo_legend.png|width=500! | > |Proposed 2 (With Icon and Legend)| !NN_with_legend.png|width=500! | > *DataNode:* > |Current| !DN_orig.png|width=500! | > |Proposed 1 (With Icon) | !DN_wo_legend.png|width=500! | > |Proposed 2 (With Icon and Legend)| !DN_with_legend.png.png|width=500! | -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14928) UI: unifying the WebUI across different components.
[ https://issues.apache.org/jira/browse/HDFS-14928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16971318#comment-16971318 ] Hudson commented on HDFS-14928: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #17623 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/17623/]) HDFS-14928. UI: unifying the WebUI across different components. (tasanuma: rev 6663d6a5c2d0e01523c81e0719fd305ee279ea55) * (edit) hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/webapps/router/federationhealth.html * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/hdfs/dfshealth.html * (edit) hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/webapps/static/rbf.css * (edit) hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/webapps/router/federationhealth.js * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/static/hadoop.css > UI: unifying the WebUI across different components. > --- > > Key: HDFS-14928 > URL: https://issues.apache.org/jira/browse/HDFS-14928 > Project: Hadoop HDFS > Issue Type: Improvement > Components: ui >Reporter: Xieming Li >Assignee: Xieming Li >Priority: Trivial > Fix For: 3.3.0 > > Attachments: DN_orig.png, DN_with_legend.png.png, DN_wo_legend.png, > HDFS-14892-2.jpg, HDFS-14928.001.patch, HDFS-14928.002.patch, > HDFS-14928.003.patch, HDFS-14928.004.patch, HDFS-14928.jpg, NN_orig.png, > NN_with_legend.png, NN_wo_legend.png, RBF_orig.png, RBF_with_legend.png, > RBF_wo_legend.png > > > The WebUI of different components could be unified. > *Router:* > |Current| !RBF_orig.png|width=500! | > |Proposed 1 (With Icon) | !RBF_wo_legend.png|width=500! | > |Proposed 2 (With Icon and Legend)|!RBF_with_legend.png|width=500! | > *NameNode:* > |Current| !NN_orig.png|width=500! | > |Proposed 1 (With Icon) | !NN_wo_legend.png|width=500! | > |Proposed 2 (With Icon and Legend)| !NN_with_legend.png|width=500! | > *DataNode:* > |Current| !DN_orig.png|width=500! | > |Proposed 1 (With Icon) | !DN_wo_legend.png|width=500! | > |Proposed 2 (With Icon and Legend)| !DN_with_legend.png.png|width=500! | -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14962) RBF: ConnectionPool#newConnection() error log wrong protocol class
[
https://issues.apache.org/jira/browse/HDFS-14962?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ayush Saxena updated HDFS-14962:
Hadoop Flags: Reviewed
Resolution: Fixed
Status: Resolved (was: Patch Available)
Merged.
Thanx [~John Smith] for the contribution and [~elgoiri] for the review!!!
> RBF: ConnectionPool#newConnection() error log wrong protocol class
> --
>
> Key: HDFS-14962
> URL: https://issues.apache.org/jira/browse/HDFS-14962
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: rbf
>Affects Versions: 3.3.0
>Reporter: Yuxuan Wang
>Assignee: Yuxuan Wang
>Priority: Minor
> Labels: RBF
>
> ConnectionPool#newConnection() has following code:
> {code:java}
> String msg = "Unsupported protocol for connection to NameNode: "
> + ((proto != null) ? proto.getClass().getName() : "null");
> {code}
> *proto.getClass().getName()* should be *proto.getName()*
> My IDE can figure out the issue.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14962) RBF: ConnectionPool#newConnection() error log wrong protocol class
[
https://issues.apache.org/jira/browse/HDFS-14962?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ayush Saxena updated HDFS-14962:
Fix Version/s: 3.3.0
> RBF: ConnectionPool#newConnection() error log wrong protocol class
> --
>
> Key: HDFS-14962
> URL: https://issues.apache.org/jira/browse/HDFS-14962
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: rbf
>Affects Versions: 3.3.0
>Reporter: Yuxuan Wang
>Assignee: Yuxuan Wang
>Priority: Minor
> Labels: RBF
> Fix For: 3.3.0
>
>
> ConnectionPool#newConnection() has following code:
> {code:java}
> String msg = "Unsupported protocol for connection to NameNode: "
> + ((proto != null) ? proto.getClass().getName() : "null");
> {code}
> *proto.getClass().getName()* should be *proto.getName()*
> My IDE can figure out the issue.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14928) UI: unifying the WebUI across different components.
[ https://issues.apache.org/jira/browse/HDFS-14928?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takanobu Asanuma updated HDFS-14928: Fix Version/s: 3.3.0 Resolution: Fixed Status: Resolved (was: Patch Available) Committed to trunk. Thanks for your contribution, [~risyomei]. Thanks for you review and discussion, [~elgoiri] and [~hemanthboyina]. > UI: unifying the WebUI across different components. > --- > > Key: HDFS-14928 > URL: https://issues.apache.org/jira/browse/HDFS-14928 > Project: Hadoop HDFS > Issue Type: Improvement > Components: ui >Reporter: Xieming Li >Assignee: Xieming Li >Priority: Trivial > Fix For: 3.3.0 > > Attachments: DN_orig.png, DN_with_legend.png.png, DN_wo_legend.png, > HDFS-14892-2.jpg, HDFS-14928.001.patch, HDFS-14928.002.patch, > HDFS-14928.003.patch, HDFS-14928.004.patch, HDFS-14928.jpg, NN_orig.png, > NN_with_legend.png, NN_wo_legend.png, RBF_orig.png, RBF_with_legend.png, > RBF_wo_legend.png > > > The WebUI of different components could be unified. > *Router:* > |Current| !RBF_orig.png|width=500! | > |Proposed 1 (With Icon) | !RBF_wo_legend.png|width=500! | > |Proposed 2 (With Icon and Legend)|!RBF_with_legend.png|width=500! | > *NameNode:* > |Current| !NN_orig.png|width=500! | > |Proposed 1 (With Icon) | !NN_wo_legend.png|width=500! | > |Proposed 2 (With Icon and Legend)| !NN_with_legend.png|width=500! | > *DataNode:* > |Current| !DN_orig.png|width=500! | > |Proposed 1 (With Icon) | !DN_wo_legend.png|width=500! | > |Proposed 2 (With Icon and Legend)| !DN_with_legend.png.png|width=500! | -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14928) UI: unifying the WebUI across different components.
[ https://issues.apache.org/jira/browse/HDFS-14928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16971312#comment-16971312 ] Takanobu Asanuma commented on HDFS-14928: - +1 on 004.patch. Will merge it soon. > UI: unifying the WebUI across different components. > --- > > Key: HDFS-14928 > URL: https://issues.apache.org/jira/browse/HDFS-14928 > Project: Hadoop HDFS > Issue Type: Improvement > Components: ui >Reporter: Xieming Li >Assignee: Xieming Li >Priority: Trivial > Attachments: DN_orig.png, DN_with_legend.png.png, DN_wo_legend.png, > HDFS-14892-2.jpg, HDFS-14928.001.patch, HDFS-14928.002.patch, > HDFS-14928.003.patch, HDFS-14928.004.patch, HDFS-14928.jpg, NN_orig.png, > NN_with_legend.png, NN_wo_legend.png, RBF_orig.png, RBF_with_legend.png, > RBF_wo_legend.png > > > The WebUI of different components could be unified. > *Router:* > |Current| !RBF_orig.png|width=500! | > |Proposed 1 (With Icon) | !RBF_wo_legend.png|width=500! | > |Proposed 2 (With Icon and Legend)|!RBF_with_legend.png|width=500! | > *NameNode:* > |Current| !NN_orig.png|width=500! | > |Proposed 1 (With Icon) | !NN_wo_legend.png|width=500! | > |Proposed 2 (With Icon and Legend)| !NN_with_legend.png|width=500! | > *DataNode:* > |Current| !DN_orig.png|width=500! | > |Proposed 1 (With Icon) | !DN_wo_legend.png|width=500! | > |Proposed 2 (With Icon and Legend)| !DN_with_legend.png.png|width=500! | -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-14966) In NameNode Web UI, Some values are shown as negative
[ https://issues.apache.org/jira/browse/HDFS-14966?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] bright.zhou reassigned HDFS-14966: -- Assignee: (was: bright.zhou) > In NameNode Web UI, Some values are shown as negative > -- > > Key: HDFS-14966 > URL: https://issues.apache.org/jira/browse/HDFS-14966 > Project: Hadoop HDFS > Issue Type: Bug > Components: ui >Affects Versions: 3.1.1 >Reporter: bright.zhou >Priority: Minor > Attachments: image1.png, image2.png > > > Simulate 10k large cluster test HDFS, NameNode web ui show wrongly nagtive > value. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14971) HDFS : help info of fsck "-list-corruptfileblocks" command needs to be rectified
[ https://issues.apache.org/jira/browse/HDFS-14971?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] bright.zhou updated HDFS-14971: --- Attachment: HDFS-14971.001.patch Status: Patch Available (was: Open) > HDFS : help info of fsck "-list-corruptfileblocks" command needs to be > rectified > > > Key: HDFS-14971 > URL: https://issues.apache.org/jira/browse/HDFS-14971 > Project: Hadoop HDFS > Issue Type: Bug > Components: tools >Affects Versions: 3.1.2 > Environment: HA Cluster >Reporter: Souryakanta Dwivedy >Assignee: bright.zhou >Priority: Minor > Attachments: HDFS-14971.001.patch, image-2019-11-08-18-58-41-220.png > > > HDFS : help info of fsck "-list-corruptfileblocks" command needs to be > rectified > Check the help info of fsck -list-corruptfileblocks it is specified as > "-list-corruptfileblocks print out list of missing blocks and files they > belong to" > It should be rectified as corrupted blocks and files as it is going provide > information about corrupted blocks and files not missing blocks and files > Expected output :- > "-list-corruptfileblocks print out list of corrupted blocks and files they > belong to" > > !image-2019-11-08-18-58-41-220.png! > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-14966) In NameNode Web UI, Some values are shown as negative
[ https://issues.apache.org/jira/browse/HDFS-14966?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] bright.zhou reassigned HDFS-14966: -- Assignee: bright.zhou > In NameNode Web UI, Some values are shown as negative > -- > > Key: HDFS-14966 > URL: https://issues.apache.org/jira/browse/HDFS-14966 > Project: Hadoop HDFS > Issue Type: Bug > Components: ui >Affects Versions: 3.1.1 >Reporter: bright.zhou >Assignee: bright.zhou >Priority: Minor > Attachments: image1.png, image2.png > > > Simulate 10k large cluster test HDFS, NameNode web ui show wrongly nagtive > value. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-2356) Multipart upload report errors while writing to ozone Ratis pipeline
[ https://issues.apache.org/jira/browse/HDDS-2356?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Li Cheng updated HDDS-2356: --- Attachment: om_audit_log_plc_1570863541668_9278.txt > Multipart upload report errors while writing to ozone Ratis pipeline > > > Key: HDDS-2356 > URL: https://issues.apache.org/jira/browse/HDDS-2356 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: Ozone Manager >Affects Versions: 0.4.1 > Environment: Env: 4 VMs in total: 3 Datanodes on 3 VMs, 1 OM & 1 SCM > on a separate VM >Reporter: Li Cheng >Assignee: Bharat Viswanadham >Priority: Blocker > Fix For: 0.5.0 > > Attachments: 2019-11-06_18_13_57_422_ERROR, hs_err_pid9340.log, > image-2019-10-31-18-56-56-177.png, om_audit_log_plc_1570863541668_9278.txt > > > Env: 4 VMs in total: 3 Datanodes on 3 VMs, 1 OM & 1 SCM on a separate VM, say > it's VM0. > I use goofys as a fuse and enable ozone S3 gateway to mount ozone to a path > on VM0, while reading data from VM0 local disk and write to mount path. The > dataset has various sizes of files from 0 byte to GB-level and it has a > number of ~50,000 files. > The writing is slow (1GB for ~10 mins) and it stops after around 4GB. As I > look at hadoop-root-om-VM_50_210_centos.out log, I see OM throwing errors > related with Multipart upload. This error eventually causes the writing to > terminate and OM to be closed. > > Updated on 11/06/2019: > See new multipart upload error NO_SUCH_MULTIPART_UPLOAD_ERROR and full logs > are in the attachment. > 2019-11-05 18:12:37,766 ERROR > org.apache.hadoop.ozone.om.request.s3.multipart.S3MultipartUploadCommitPartRequest: > MultipartUpload Commit is failed for Key:./2 > 0191012/plc_1570863541668_9278 in Volume/Bucket > s325d55ad283aa400af464c76d713c07ad/ozone-test > NO_SUCH_MULTIPART_UPLOAD_ERROR > org.apache.hadoop.ozone.om.exceptions.OMException: No such Multipart upload > is with specified uploadId fcda8608-b431-48b7-8386- > 0a332f1a709a-103084683261641950 > at > org.apache.hadoop.ozone.om.request.s3.multipart.S3MultipartUploadCommitPartRequest.validateAndUpdateCache(S3MultipartUploadCommitPartRequest.java:1 > 56) > at > org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.submitRequestDirectlyToOM(OzoneManagerProtocolServerSideTranslatorPB. > java:217) > at > org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.processRequest(OzoneManagerProtocolServerSideTranslatorPB.java:132) > at > org.apache.hadoop.hdds.server.OzoneProtocolMessageDispatcher.processRequest(OzoneProtocolMessageDispatcher.java:72) > at > org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.submitRequest(OzoneManagerProtocolServerSideTranslatorPB.java:100) > at > org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos$OzoneManagerService$2.callBlockingMethod(OzoneManagerProtocolProtos.java) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:524) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1025) > at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:876) > at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:822) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2682) > > Updated on 10/28/2019: > See MISMATCH_MULTIPART_LIST error. > > 2019-10-28 11:44:34,079 [qtp1383524016-70] ERROR - Error in Complete > Multipart Upload Request for bucket: ozone-test, key: > 20191012/plc_1570863541668_927 > 8 > MISMATCH_MULTIPART_LIST org.apache.hadoop.ozone.om.exceptions.OMException: > Complete Multipart Upload Failed: volume: > s3c89e813c80ffcea9543004d57b2a1239bucket: > ozone-testkey: 20191012/plc_1570863541668_9278 > at > org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.handleError(OzoneManagerProtocolClientSideTranslatorPB.java:732) > at > org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.completeMultipartUpload(OzoneManagerProtocolClientSideTranslatorPB > .java:1104) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:497) > at > org.apache.hadoop.hdds.tracing.TraceAllMethod.invoke(TraceAllMethod.java:66) > at com.sun.proxy.$Proxy82.completeMultipartUpload(Unknown Source) > at >
[jira] [Commented] (HDDS-2356) Multipart upload report errors while writing to ozone Ratis pipeline
[
https://issues.apache.org/jira/browse/HDDS-2356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16971290#comment-16971290
]
Li Cheng commented on HDDS-2356:
[~bharat] In terms of the key in the last stacktrace:
2019-11-08 20:08:24,832 ERROR
org.apache.hadoop.ozone.om.request.s3.multipart.S3MultipartUploadCompleteRequest:
MultipartUpload Complete request failed for Key: plc_1570863541668_9278 in
Volume/Bucket s325d55ad283aa400af464c76d713c07ad/ozone-test
INVALID_PART org.apache.hadoop.ozone.om.exceptions.OMException: Complete
Multipart Upload Failed: volume: s325d55ad283aa400af464c76d713c07adbucket:
ozone-testkey: plc_1570863541668_9278
OM audit logs show a bunch of entries for key plc_1570863541668_9278 with
different clientID, for instance:
2019-11-08 20:19:56,241 | INFO | OMAudit | user=root | ip=9.134.50.210 |
op=ALLOCATE_BLOCK \{volume=s325d55ad283aa400af464c76d713c07ad,
bucket=ozone-test, key=plc_1570863541668_9278, dataSize=5242880,
replicationType=RATIS, replicationFactor=THREE, keyLocationInfo=[],
clientID=103102209394803336} | ret=SUCCESS |
I'm uploading the entire OM audit logs about this key plc_1570863541668_9278.
> Multipart upload report errors while writing to ozone Ratis pipeline
>
>
> Key: HDDS-2356
> URL: https://issues.apache.org/jira/browse/HDDS-2356
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: Ozone Manager
>Affects Versions: 0.4.1
> Environment: Env: 4 VMs in total: 3 Datanodes on 3 VMs, 1 OM & 1 SCM
> on a separate VM
>Reporter: Li Cheng
>Assignee: Bharat Viswanadham
>Priority: Blocker
> Fix For: 0.5.0
>
> Attachments: 2019-11-06_18_13_57_422_ERROR, hs_err_pid9340.log,
> image-2019-10-31-18-56-56-177.png
>
>
> Env: 4 VMs in total: 3 Datanodes on 3 VMs, 1 OM & 1 SCM on a separate VM, say
> it's VM0.
> I use goofys as a fuse and enable ozone S3 gateway to mount ozone to a path
> on VM0, while reading data from VM0 local disk and write to mount path. The
> dataset has various sizes of files from 0 byte to GB-level and it has a
> number of ~50,000 files.
> The writing is slow (1GB for ~10 mins) and it stops after around 4GB. As I
> look at hadoop-root-om-VM_50_210_centos.out log, I see OM throwing errors
> related with Multipart upload. This error eventually causes the writing to
> terminate and OM to be closed.
>
> Updated on 11/06/2019:
> See new multipart upload error NO_SUCH_MULTIPART_UPLOAD_ERROR and full logs
> are in the attachment.
> 2019-11-05 18:12:37,766 ERROR
> org.apache.hadoop.ozone.om.request.s3.multipart.S3MultipartUploadCommitPartRequest:
> MultipartUpload Commit is failed for Key:./2
> 0191012/plc_1570863541668_9278 in Volume/Bucket
> s325d55ad283aa400af464c76d713c07ad/ozone-test
> NO_SUCH_MULTIPART_UPLOAD_ERROR
> org.apache.hadoop.ozone.om.exceptions.OMException: No such Multipart upload
> is with specified uploadId fcda8608-b431-48b7-8386-
> 0a332f1a709a-103084683261641950
> at
> org.apache.hadoop.ozone.om.request.s3.multipart.S3MultipartUploadCommitPartRequest.validateAndUpdateCache(S3MultipartUploadCommitPartRequest.java:1
> 56)
> at
> org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.submitRequestDirectlyToOM(OzoneManagerProtocolServerSideTranslatorPB.
> java:217)
> at
> org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.processRequest(OzoneManagerProtocolServerSideTranslatorPB.java:132)
> at
> org.apache.hadoop.hdds.server.OzoneProtocolMessageDispatcher.processRequest(OzoneProtocolMessageDispatcher.java:72)
> at
> org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.submitRequest(OzoneManagerProtocolServerSideTranslatorPB.java:100)
> at
> org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos$OzoneManagerService$2.callBlockingMethod(OzoneManagerProtocolProtos.java)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:524)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1025)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:876)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:822)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2682)
>
> Updated on 10/28/2019:
> See MISMATCH_MULTIPART_LIST error.
>
> 2019-10-28 11:44:34,079 [qtp1383524016-70] ERROR - Error in Complete
> Multipart Upload Request for bucket: ozone-test, key:
> 20191012/plc_1570863541668_927
> 8
> MISMATCH_MULTIPART_LIST org.apache.hadoop.ozone.om.exceptions.OMException:
>
[jira] [Comment Edited] (HDDS-2356) Multipart upload report errors while writing to ozone Ratis pipeline
[ https://issues.apache.org/jira/browse/HDDS-2356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16971289#comment-16971289 ] Li Cheng edited comment on HDDS-2356 at 11/11/19 3:14 AM: -- [~bharat] As I said, debug logs for Goofys doesn't work since goofys debug mode is doing everything in single thread. The same error is not reproduced? Are you modeling some sample dataset and env to test the issues? I can try reproducing from my side and upload OM log, s3g log as well audit log here. Does that work? Also, do you see 2019-11-06_18_13_57_422_ERROR in the attachment? Does it help? was (Author: timmylicheng): [~bharat] As I said, debug logs for Goofys doesn't work since goofys debug mode is doing everything in single thread. The same error is not reproduced? Are you modeling some sample dataset and env to test the issues? I can try reproducing from my side and upload OM log, s3g log as well audit log here. Does that work? > Multipart upload report errors while writing to ozone Ratis pipeline > > > Key: HDDS-2356 > URL: https://issues.apache.org/jira/browse/HDDS-2356 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: Ozone Manager >Affects Versions: 0.4.1 > Environment: Env: 4 VMs in total: 3 Datanodes on 3 VMs, 1 OM & 1 SCM > on a separate VM >Reporter: Li Cheng >Assignee: Bharat Viswanadham >Priority: Blocker > Fix For: 0.5.0 > > Attachments: 2019-11-06_18_13_57_422_ERROR, hs_err_pid9340.log, > image-2019-10-31-18-56-56-177.png > > > Env: 4 VMs in total: 3 Datanodes on 3 VMs, 1 OM & 1 SCM on a separate VM, say > it's VM0. > I use goofys as a fuse and enable ozone S3 gateway to mount ozone to a path > on VM0, while reading data from VM0 local disk and write to mount path. The > dataset has various sizes of files from 0 byte to GB-level and it has a > number of ~50,000 files. > The writing is slow (1GB for ~10 mins) and it stops after around 4GB. As I > look at hadoop-root-om-VM_50_210_centos.out log, I see OM throwing errors > related with Multipart upload. This error eventually causes the writing to > terminate and OM to be closed. > > Updated on 11/06/2019: > See new multipart upload error NO_SUCH_MULTIPART_UPLOAD_ERROR and full logs > are in the attachment. > 2019-11-05 18:12:37,766 ERROR > org.apache.hadoop.ozone.om.request.s3.multipart.S3MultipartUploadCommitPartRequest: > MultipartUpload Commit is failed for Key:./2 > 0191012/plc_1570863541668_9278 in Volume/Bucket > s325d55ad283aa400af464c76d713c07ad/ozone-test > NO_SUCH_MULTIPART_UPLOAD_ERROR > org.apache.hadoop.ozone.om.exceptions.OMException: No such Multipart upload > is with specified uploadId fcda8608-b431-48b7-8386- > 0a332f1a709a-103084683261641950 > at > org.apache.hadoop.ozone.om.request.s3.multipart.S3MultipartUploadCommitPartRequest.validateAndUpdateCache(S3MultipartUploadCommitPartRequest.java:1 > 56) > at > org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.submitRequestDirectlyToOM(OzoneManagerProtocolServerSideTranslatorPB. > java:217) > at > org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.processRequest(OzoneManagerProtocolServerSideTranslatorPB.java:132) > at > org.apache.hadoop.hdds.server.OzoneProtocolMessageDispatcher.processRequest(OzoneProtocolMessageDispatcher.java:72) > at > org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.submitRequest(OzoneManagerProtocolServerSideTranslatorPB.java:100) > at > org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos$OzoneManagerService$2.callBlockingMethod(OzoneManagerProtocolProtos.java) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:524) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1025) > at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:876) > at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:822) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2682) > > Updated on 10/28/2019: > See MISMATCH_MULTIPART_LIST error. > > 2019-10-28 11:44:34,079 [qtp1383524016-70] ERROR - Error in Complete > Multipart Upload Request for bucket: ozone-test, key: > 20191012/plc_1570863541668_927 > 8 > MISMATCH_MULTIPART_LIST org.apache.hadoop.ozone.om.exceptions.OMException: > Complete Multipart Upload Failed: volume: > s3c89e813c80ffcea9543004d57b2a1239bucket: > ozone-testkey: 20191012/plc_1570863541668_9278 > at >
[jira] [Commented] (HDDS-2356) Multipart upload report errors while writing to ozone Ratis pipeline
[ https://issues.apache.org/jira/browse/HDDS-2356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16971289#comment-16971289 ] Li Cheng commented on HDDS-2356: [~bharat] As I said, debug logs for Goofys doesn't work since goofys debug mode is doing everything in single thread. The same error is not reproduced? Are you modeling some sample dataset and env to test the issues? I can try reproducing from my side and upload OM log, s3g log as well audit log here. Does that work? > Multipart upload report errors while writing to ozone Ratis pipeline > > > Key: HDDS-2356 > URL: https://issues.apache.org/jira/browse/HDDS-2356 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: Ozone Manager >Affects Versions: 0.4.1 > Environment: Env: 4 VMs in total: 3 Datanodes on 3 VMs, 1 OM & 1 SCM > on a separate VM >Reporter: Li Cheng >Assignee: Bharat Viswanadham >Priority: Blocker > Fix For: 0.5.0 > > Attachments: 2019-11-06_18_13_57_422_ERROR, hs_err_pid9340.log, > image-2019-10-31-18-56-56-177.png > > > Env: 4 VMs in total: 3 Datanodes on 3 VMs, 1 OM & 1 SCM on a separate VM, say > it's VM0. > I use goofys as a fuse and enable ozone S3 gateway to mount ozone to a path > on VM0, while reading data from VM0 local disk and write to mount path. The > dataset has various sizes of files from 0 byte to GB-level and it has a > number of ~50,000 files. > The writing is slow (1GB for ~10 mins) and it stops after around 4GB. As I > look at hadoop-root-om-VM_50_210_centos.out log, I see OM throwing errors > related with Multipart upload. This error eventually causes the writing to > terminate and OM to be closed. > > Updated on 11/06/2019: > See new multipart upload error NO_SUCH_MULTIPART_UPLOAD_ERROR and full logs > are in the attachment. > 2019-11-05 18:12:37,766 ERROR > org.apache.hadoop.ozone.om.request.s3.multipart.S3MultipartUploadCommitPartRequest: > MultipartUpload Commit is failed for Key:./2 > 0191012/plc_1570863541668_9278 in Volume/Bucket > s325d55ad283aa400af464c76d713c07ad/ozone-test > NO_SUCH_MULTIPART_UPLOAD_ERROR > org.apache.hadoop.ozone.om.exceptions.OMException: No such Multipart upload > is with specified uploadId fcda8608-b431-48b7-8386- > 0a332f1a709a-103084683261641950 > at > org.apache.hadoop.ozone.om.request.s3.multipart.S3MultipartUploadCommitPartRequest.validateAndUpdateCache(S3MultipartUploadCommitPartRequest.java:1 > 56) > at > org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.submitRequestDirectlyToOM(OzoneManagerProtocolServerSideTranslatorPB. > java:217) > at > org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.processRequest(OzoneManagerProtocolServerSideTranslatorPB.java:132) > at > org.apache.hadoop.hdds.server.OzoneProtocolMessageDispatcher.processRequest(OzoneProtocolMessageDispatcher.java:72) > at > org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.submitRequest(OzoneManagerProtocolServerSideTranslatorPB.java:100) > at > org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos$OzoneManagerService$2.callBlockingMethod(OzoneManagerProtocolProtos.java) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:524) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1025) > at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:876) > at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:822) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2682) > > Updated on 10/28/2019: > See MISMATCH_MULTIPART_LIST error. > > 2019-10-28 11:44:34,079 [qtp1383524016-70] ERROR - Error in Complete > Multipart Upload Request for bucket: ozone-test, key: > 20191012/plc_1570863541668_927 > 8 > MISMATCH_MULTIPART_LIST org.apache.hadoop.ozone.om.exceptions.OMException: > Complete Multipart Upload Failed: volume: > s3c89e813c80ffcea9543004d57b2a1239bucket: > ozone-testkey: 20191012/plc_1570863541668_9278 > at > org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.handleError(OzoneManagerProtocolClientSideTranslatorPB.java:732) > at > org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.completeMultipartUpload(OzoneManagerProtocolClientSideTranslatorPB > .java:1104) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at >
[jira] [Commented] (HDFS-14648) DeadNodeDetector basic model
[
https://issues.apache.org/jira/browse/HDFS-14648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16971280#comment-16971280
]
Lisheng Sun commented on HDFS-14648:
Thanks [~linyiqun] for deeply review and good commnets.
I updated the patch according to your comemnts and uploaed the v007 patch.
Could you help take a review to it? Thank you a lot.
> DeadNodeDetector basic model
>
>
> Key: HDFS-14648
> URL: https://issues.apache.org/jira/browse/HDFS-14648
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Lisheng Sun
>Assignee: Lisheng Sun
>Priority: Major
> Attachments: HDFS-14648.001.patch, HDFS-14648.002.patch,
> HDFS-14648.003.patch, HDFS-14648.004.patch, HDFS-14648.005.patch,
> HDFS-14648.006.patch, HDFS-14648.007.patch
>
>
> This Jira constructs DeadNodeDetector state machine model. The function it
> implements as follow:
> # When a DFSInputstream is opened, a BlockReader is opened. If some DataNode
> of the block is found to inaccessible, put the DataNode into
> DeadNodeDetector#deadnode.(HDFS-14649) will optimize this part. Because when
> DataNode is not accessible, it is likely that the replica has been removed
> from the DataNode.Therefore, it needs to be confirmed by re-probing and
> requires a higher priority processing.
> # DeadNodeDetector will periodically detect the Node in
> DeadNodeDetector#deadnode, If the access is successful, the Node will be
> moved from DeadNodeDetector#deadnode. Continuous detection of the dead node
> is necessary. The DataNode need rejoin the cluster due to a service
> restart/machine repair. The DataNode may be permanently excluded if there is
> no added probe mechanism.
> # DeadNodeDetector#dfsInputStreamNodes Record the DFSInputstream using
> DataNode. When the DFSInputstream is closed, it will be moved from
> DeadNodeDetector#dfsInputStreamNodes.
> # Every time get the global deanode, update the DeadNodeDetector#deadnode.
> The new DeadNodeDetector#deadnode Equals to the intersection of the old
> DeadNodeDetector#deadnode and the Datanodes are by
> DeadNodeDetector#dfsInputStreamNodes.
> # DeadNodeDetector has a switch that is turned off by default. When it is
> closed, each DFSInputstream still uses its own local deadnode.
> # This feature has been used in the XIAOMI production environment for a long
> time. Reduced hbase read stuck, due to node hangs.
> # Just open the DeadNodeDetector switch and you can use it directly. No
> other restrictions. Don't want to use DeadNodeDetector, just close it.
> {code:java}
> if (sharedDeadNodesEnabled && deadNodeDetector == null) {
> deadNodeDetector = new DeadNodeDetector(name);
> deadNodeDetectorThr = new Daemon(deadNodeDetector);
> deadNodeDetectorThr.start();
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14648) DeadNodeDetector basic model
[
https://issues.apache.org/jira/browse/HDFS-14648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lisheng Sun updated HDFS-14648:
---
Attachment: HDFS-14648.007.patch
> DeadNodeDetector basic model
>
>
> Key: HDFS-14648
> URL: https://issues.apache.org/jira/browse/HDFS-14648
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Lisheng Sun
>Assignee: Lisheng Sun
>Priority: Major
> Attachments: HDFS-14648.001.patch, HDFS-14648.002.patch,
> HDFS-14648.003.patch, HDFS-14648.004.patch, HDFS-14648.005.patch,
> HDFS-14648.006.patch, HDFS-14648.007.patch
>
>
> This Jira constructs DeadNodeDetector state machine model. The function it
> implements as follow:
> # When a DFSInputstream is opened, a BlockReader is opened. If some DataNode
> of the block is found to inaccessible, put the DataNode into
> DeadNodeDetector#deadnode.(HDFS-14649) will optimize this part. Because when
> DataNode is not accessible, it is likely that the replica has been removed
> from the DataNode.Therefore, it needs to be confirmed by re-probing and
> requires a higher priority processing.
> # DeadNodeDetector will periodically detect the Node in
> DeadNodeDetector#deadnode, If the access is successful, the Node will be
> moved from DeadNodeDetector#deadnode. Continuous detection of the dead node
> is necessary. The DataNode need rejoin the cluster due to a service
> restart/machine repair. The DataNode may be permanently excluded if there is
> no added probe mechanism.
> # DeadNodeDetector#dfsInputStreamNodes Record the DFSInputstream using
> DataNode. When the DFSInputstream is closed, it will be moved from
> DeadNodeDetector#dfsInputStreamNodes.
> # Every time get the global deanode, update the DeadNodeDetector#deadnode.
> The new DeadNodeDetector#deadnode Equals to the intersection of the old
> DeadNodeDetector#deadnode and the Datanodes are by
> DeadNodeDetector#dfsInputStreamNodes.
> # DeadNodeDetector has a switch that is turned off by default. When it is
> closed, each DFSInputstream still uses its own local deadnode.
> # This feature has been used in the XIAOMI production environment for a long
> time. Reduced hbase read stuck, due to node hangs.
> # Just open the DeadNodeDetector switch and you can use it directly. No
> other restrictions. Don't want to use DeadNodeDetector, just close it.
> {code:java}
> if (sharedDeadNodesEnabled && deadNodeDetector == null) {
> deadNodeDetector = new DeadNodeDetector(name);
> deadNodeDetectorThr = new Daemon(deadNodeDetector);
> deadNodeDetectorThr.start();
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-2417) Add the list trash command to the client side
[ https://issues.apache.org/jira/browse/HDDS-2417?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dinesh Chitlangia resolved HDDS-2417. - Fix Version/s: 0.6.0 Resolution: Implemented [~MatthewSharp] Thanks for the contribution, [~aengineer] Thanks for the reviews. > Add the list trash command to the client side > - > > Key: HDDS-2417 > URL: https://issues.apache.org/jira/browse/HDDS-2417 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: Ozone Manager >Reporter: Anu Engineer >Assignee: Matthew Sharp >Priority: Major > Labels: pull-request-available > Fix For: 0.6.0 > > Time Spent: 20m > Remaining Estimate: 0h > > Add the list-trash command to the protobuf files and to the client side > translator. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-2417) Add the list trash command to the client side
[ https://issues.apache.org/jira/browse/HDDS-2417?focusedWorklogId=341116=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-341116 ] ASF GitHub Bot logged work on HDDS-2417: Author: ASF GitHub Bot Created on: 11/Nov/19 01:31 Start Date: 11/Nov/19 01:31 Worklog Time Spent: 10m Work Description: dineshchitlangia commented on pull request #138: HDDS-2417 Add the list trash command to the client side URL: https://github.com/apache/hadoop-ozone/pull/138 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 341116) Time Spent: 20m (was: 10m) > Add the list trash command to the client side > - > > Key: HDDS-2417 > URL: https://issues.apache.org/jira/browse/HDDS-2417 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: Ozone Manager >Reporter: Anu Engineer >Assignee: Matthew Sharp >Priority: Major > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > Add the list-trash command to the protobuf files and to the client side > translator. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDDS-2457) Add the ability to get the list of buckets that are deleted
[ https://issues.apache.org/jira/browse/HDDS-2457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthew Sharp reassigned HDDS-2457: --- Assignee: Matthew Sharp > Add the ability to get the list of buckets that are deleted > --- > > Key: HDDS-2457 > URL: https://issues.apache.org/jira/browse/HDDS-2457 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Matthew Sharp >Assignee: Matthew Sharp >Priority: Major > > The initial focus is returning a list of deleted keys, but showing the > deleted buckets should be explored further. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDDS-2440) Add empty-trash to ozone shell.
[ https://issues.apache.org/jira/browse/HDDS-2440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthew Sharp reassigned HDDS-2440: --- Assignee: Matthew Sharp > Add empty-trash to ozone shell. > --- > > Key: HDDS-2440 > URL: https://issues.apache.org/jira/browse/HDDS-2440 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: Ozone CLI >Reporter: Anu Engineer >Assignee: Matthew Sharp >Priority: Major > > Add emptry-trash command to Ozone shell. We should decide if we want to add > this to the admin shell or normal shell. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDDS-2439) Add robot tests for empty-trash as owner.
[ https://issues.apache.org/jira/browse/HDDS-2439?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthew Sharp reassigned HDDS-2439: --- Assignee: Matthew Sharp > Add robot tests for empty-trash as owner. > - > > Key: HDDS-2439 > URL: https://issues.apache.org/jira/browse/HDDS-2439 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: Ozone Manager >Reporter: Anu Engineer >Assignee: Matthew Sharp >Priority: Major > > We need to make sure that only Owner or Admins can execute the empty-trash > command. We need to verify this using end-to-end tests, for example, robot > tests -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDDS-2441) Add documentation for Empty-Trash command.
[ https://issues.apache.org/jira/browse/HDDS-2441?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthew Sharp reassigned HDDS-2441: --- Assignee: Matthew Sharp > Add documentation for Empty-Trash command. > -- > > Key: HDDS-2441 > URL: https://issues.apache.org/jira/browse/HDDS-2441 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: documentation >Reporter: Anu Engineer >Assignee: Matthew Sharp >Priority: Major > > Add documentation for empty-trash command. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDDS-2434) Add server side support for empty-trash command.
[ https://issues.apache.org/jira/browse/HDDS-2434?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthew Sharp reassigned HDDS-2434: --- Assignee: Matthew Sharp > Add server side support for empty-trash command. > > > Key: HDDS-2434 > URL: https://issues.apache.org/jira/browse/HDDS-2434 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: Ozone Manager >Reporter: Anu Engineer >Assignee: Matthew Sharp >Priority: Major > > Add server side support for empty-trash command. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDDS-2438) Add the core logic for empty-trash
[ https://issues.apache.org/jira/browse/HDDS-2438?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthew Sharp reassigned HDDS-2438: --- Assignee: Matthew Sharp > Add the core logic for empty-trash > -- > > Key: HDDS-2438 > URL: https://issues.apache.org/jira/browse/HDDS-2438 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: Ozone Manager >Reporter: Anu Engineer >Assignee: Matthew Sharp >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDDS-2433) Add client side support for the empty-trash command.
[ https://issues.apache.org/jira/browse/HDDS-2433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthew Sharp reassigned HDDS-2433: --- Assignee: Matthew Sharp > Add client side support for the empty-trash command. > > > Key: HDDS-2433 > URL: https://issues.apache.org/jira/browse/HDDS-2433 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: Ozone Manager >Reporter: Anu Engineer >Assignee: Matthew Sharp >Priority: Major > > Add client side support for the empty-trash command. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDDS-2437) Restrict empty-trash to admins and owners only
[ https://issues.apache.org/jira/browse/HDDS-2437?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthew Sharp reassigned HDDS-2437: --- Assignee: Matthew Sharp > Restrict empty-trash to admins and owners only > -- > > Key: HDDS-2437 > URL: https://issues.apache.org/jira/browse/HDDS-2437 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: Ozone Manager >Reporter: Anu Engineer >Assignee: Matthew Sharp >Priority: Major > > Make sure that only the owner of a key/adminstrator can empty-trash. The > delete ACL is not enough to empty-trash. This is becasue a shared bucket can > have deletes but the owner should be able to recover them. Once empty-trash > is executed even the owner will be able to recover the deleted keys -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDDS-2436) Add security profile support for empty-trash command
[ https://issues.apache.org/jira/browse/HDDS-2436?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthew Sharp reassigned HDDS-2436: --- Assignee: Matthew Sharp > Add security profile support for empty-trash command > > > Key: HDDS-2436 > URL: https://issues.apache.org/jira/browse/HDDS-2436 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: Ozone Manager >Reporter: Anu Engineer >Assignee: Matthew Sharp >Priority: Major > > Add support for a certain groups to have the ability to have empty-trash. It > might be the case where we want this command only to be run by admins. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDDS-2435) Add the ability to disable empty-trash command.
[ https://issues.apache.org/jira/browse/HDDS-2435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthew Sharp reassigned HDDS-2435: --- Assignee: Matthew Sharp > Add the ability to disable empty-trash command. > --- > > Key: HDDS-2435 > URL: https://issues.apache.org/jira/browse/HDDS-2435 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Anu Engineer >Assignee: Matthew Sharp >Priority: Major > > Add a configuration key to disable empty-trash command. We can discuss if > this should be a system-wide setting or per bucket. It is easier to do this > system-wide I guess. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-2457) Add the ability to get the list of buckets that are deleted
Matthew Sharp created HDDS-2457: --- Summary: Add the ability to get the list of buckets that are deleted Key: HDDS-2457 URL: https://issues.apache.org/jira/browse/HDDS-2457 Project: Hadoop Distributed Data Store Issue Type: Sub-task Reporter: Matthew Sharp The initial focus is returning a list of deleted keys, but showing the deleted buckets should be explored further. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-14971) HDFS : help info of fsck "-list-corruptfileblocks" command needs to be rectified
[ https://issues.apache.org/jira/browse/HDFS-14971?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Surendra Singh Lilhore reassigned HDFS-14971: - Assignee: bright.zhou > HDFS : help info of fsck "-list-corruptfileblocks" command needs to be > rectified > > > Key: HDFS-14971 > URL: https://issues.apache.org/jira/browse/HDFS-14971 > Project: Hadoop HDFS > Issue Type: Bug > Components: tools >Affects Versions: 3.1.2 > Environment: HA Cluster >Reporter: Souryakanta Dwivedy >Assignee: bright.zhou >Priority: Minor > Attachments: image-2019-11-08-18-58-41-220.png > > > HDFS : help info of fsck "-list-corruptfileblocks" command needs to be > rectified > Check the help info of fsck -list-corruptfileblocks it is specified as > "-list-corruptfileblocks print out list of missing blocks and files they > belong to" > It should be rectified as corrupted blocks and files as it is going provide > information about corrupted blocks and files not missing blocks and files > Expected output :- > "-list-corruptfileblocks print out list of corrupted blocks and files they > belong to" > > !image-2019-11-08-18-58-41-220.png! > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14971) HDFS : help info of fsck "-list-corruptfileblocks" command needs to be rectified
[ https://issues.apache.org/jira/browse/HDFS-14971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16971151#comment-16971151 ] Surendra Singh Lilhore commented on HDFS-14971: --- [~zgw], assigned this Jira to you and added as contributor. > HDFS : help info of fsck "-list-corruptfileblocks" command needs to be > rectified > > > Key: HDFS-14971 > URL: https://issues.apache.org/jira/browse/HDFS-14971 > Project: Hadoop HDFS > Issue Type: Bug > Components: tools >Affects Versions: 3.1.2 > Environment: HA Cluster >Reporter: Souryakanta Dwivedy >Assignee: bright.zhou >Priority: Minor > Attachments: image-2019-11-08-18-58-41-220.png > > > HDFS : help info of fsck "-list-corruptfileblocks" command needs to be > rectified > Check the help info of fsck -list-corruptfileblocks it is specified as > "-list-corruptfileblocks print out list of missing blocks and files they > belong to" > It should be rectified as corrupted blocks and files as it is going provide > information about corrupted blocks and files not missing blocks and files > Expected output :- > "-list-corruptfileblocks print out list of corrupted blocks and files they > belong to" > > !image-2019-11-08-18-58-41-220.png! > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-2456) Add explicit base image version for images derived from ozone-runner
[
https://issues.apache.org/jira/browse/HDDS-2456?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Attila Doroszlai updated HDDS-2456:
---
Status: Patch Available (was: Open)
> Add explicit base image version for images derived from ozone-runner
>
>
> Key: HDDS-2456
> URL: https://issues.apache.org/jira/browse/HDDS-2456
> Project: Hadoop Distributed Data Store
> Issue Type: Improvement
> Components: docker
>Reporter: Attila Doroszlai
>Assignee: Attila Doroszlai
>Priority: Minor
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> {{ozone-om-ha}} and {{ozonescripts}} build images based on
> {{apache/ozone-runner}}.
> Problem: They do not specify base image versions, so it defaults to
> {{latest}}. If a new {{ozone-runner}} image is published on Docker Hub,
> developers needs to manually pull the {{latest}} image for it to take effect
> on these derived images.
> Solution: Use explicit base image version (defined by
> {{OZONE_RUNNER_VERSION}} variable in {{.env}} file.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-2456) Add explicit base image version for images derived from ozone-runner
[
https://issues.apache.org/jira/browse/HDDS-2456?focusedWorklogId=341062=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-341062
]
ASF GitHub Bot logged work on HDDS-2456:
Author: ASF GitHub Bot
Created on: 10/Nov/19 14:39
Start Date: 10/Nov/19 14:39
Worklog Time Spent: 10m
Work Description: adoroszlai commented on pull request #139: HDDS-2456.
Add explicit base image version for images derived from ozone-runner
URL: https://github.com/apache/hadoop-ozone/pull/139
## What changes were proposed in this pull request?
Use explicit base image version (`$OZONE_RUNNER_VERSION`) instead of
implicit `latest` for images derived from `ozone-runner` (in `ozone-om-ha` and
`ozonescripts` compose environments). Also tag the resulting image with the
same version instead of implicit `latest`. The goal is to force rebuild of
these images if a new `ozone-runner` image is released (and
`docker.ozone-runner.version` updated in `hadoop-ozone/dist/pom.xml`).
https://issues.apache.org/jira/browse/HDDS-2456
## How was this patch tested?
```
$ cd hadoop-ozone/dist/target/ozone-0.5.0-SNAPSHOT/compose/ozonescripts
$ docker-compose up -d
Creating network "ozonescripts_default" with the default driver
Building datanode
Step 1/16 : ARG OZONE_RUNNER_VERSION
Step 2/16 : FROM apache/ozone-runner:${OZONE_RUNNER_VERSION}
20191107-1: Pulling from apache/ozone-runner
Digest:
sha256:185e18663394e1976d73a6f7de2aa8ade4d20a0e7314b415cad6f26f446323fe
Status: Downloaded newer image for apache/ozone-runner:20191107-1
...
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 341062)
Remaining Estimate: 0h
Time Spent: 10m
> Add explicit base image version for images derived from ozone-runner
>
>
> Key: HDDS-2456
> URL: https://issues.apache.org/jira/browse/HDDS-2456
> Project: Hadoop Distributed Data Store
> Issue Type: Improvement
> Components: docker
>Reporter: Attila Doroszlai
>Assignee: Attila Doroszlai
>Priority: Minor
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> {{ozone-om-ha}} and {{ozonescripts}} build images based on
> {{apache/ozone-runner}}.
> Problem: They do not specify base image versions, so it defaults to
> {{latest}}. If a new {{ozone-runner}} image is published on Docker Hub,
> developers needs to manually pull the {{latest}} image for it to take effect
> on these derived images.
> Solution: Use explicit base image version (defined by
> {{OZONE_RUNNER_VERSION}} variable in {{.env}} file.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-2456) Add explicit base image version for images derived from ozone-runner
[
https://issues.apache.org/jira/browse/HDDS-2456?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HDDS-2456:
-
Labels: pull-request-available (was: )
> Add explicit base image version for images derived from ozone-runner
>
>
> Key: HDDS-2456
> URL: https://issues.apache.org/jira/browse/HDDS-2456
> Project: Hadoop Distributed Data Store
> Issue Type: Improvement
> Components: docker
>Reporter: Attila Doroszlai
>Assignee: Attila Doroszlai
>Priority: Minor
> Labels: pull-request-available
>
> {{ozone-om-ha}} and {{ozonescripts}} build images based on
> {{apache/ozone-runner}}.
> Problem: They do not specify base image versions, so it defaults to
> {{latest}}. If a new {{ozone-runner}} image is published on Docker Hub,
> developers needs to manually pull the {{latest}} image for it to take effect
> on these derived images.
> Solution: Use explicit base image version (defined by
> {{OZONE_RUNNER_VERSION}} variable in {{.env}} file.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-2456) Add explicit base image version for images derived from ozone-runner
Attila Doroszlai created HDDS-2456:
--
Summary: Add explicit base image version for images derived from
ozone-runner
Key: HDDS-2456
URL: https://issues.apache.org/jira/browse/HDDS-2456
Project: Hadoop Distributed Data Store
Issue Type: Improvement
Components: docker
Reporter: Attila Doroszlai
Assignee: Attila Doroszlai
{{ozone-om-ha}} and {{ozonescripts}} build images based on
{{apache/ozone-runner}}.
Problem: They do not specify base image versions, so it defaults to {{latest}}.
If a new {{ozone-runner}} image is published on Docker Hub, developers needs
to manually pull the {{latest}} image for it to take effect on these derived
images.
Solution: Use explicit base image version (defined by {{OZONE_RUNNER_VERSION}}
variable in {{.env}} file.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14967) TestWebHDFS - Many test cases are failing in Windows
[ https://issues.apache.org/jira/browse/HDFS-14967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16971060#comment-16971060 ] Renukaprasad C commented on HDFS-14967: --- Sure, I will handle and update the patch. Thanks [~ayushtkn] > TestWebHDFS - Many test cases are failing in Windows > - > > Key: HDFS-14967 > URL: https://issues.apache.org/jira/browse/HDFS-14967 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Renukaprasad C >Assignee: Renukaprasad C >Priority: Major > Attachments: HDFS-14967.001.patch > > > In TestWebHDFS test class, few test cases are not closing the MiniDFSCluster, > which results in remaining test failures in Windows. Once cluster status is > open, all consecutive test cases fail to get the lock on Data dir which > results in test case failure. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
