[jira] [Created] (HDDS-250) Cleanup ContainerData
Hanisha Koneru created HDDS-250: --- Summary: Cleanup ContainerData Key: HDDS-250 URL: https://issues.apache.org/jira/browse/HDDS-250 Project: Hadoop Distributed Data Store Issue Type: Improvement Reporter: Hanisha Koneru Assignee: Hanisha Koneru Fix For: 0.2.1 The following functions in ContainerData are redundant. MetadataPath and ChunksPath are specific to KeyValueContainerData. ContainerPath is the common path in ContainerData which points to the base dir of the container. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-249) Fail if multiple SCM IDs on the DataNode and add SCM ID check after version request
Bharat Viswanadham created HDDS-249: --- Summary: Fail if multiple SCM IDs on the DataNode and add SCM ID check after version request Key: HDDS-249 URL: https://issues.apache.org/jira/browse/HDDS-249 Project: Hadoop Distributed Data Store Issue Type: Improvement Reporter: Bharat Viswanadham Assignee: Bharat Viswanadham This Jira take care of following conditions: # If multiple Scm directories exist on datanode, it fails that volume. # validate SCMID response from SCM. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-248) Refactor DatanodeContainerProtocol.proto
Hanisha Koneru created HDDS-248: --- Summary: Refactor DatanodeContainerProtocol.proto Key: HDDS-248 URL: https://issues.apache.org/jira/browse/HDDS-248 Project: Hadoop Distributed Data Store Issue Type: Improvement Reporter: Hanisha Koneru Assignee: Hanisha Koneru Fix For: 0.2.1 This Jira proposes to cleanup the DatanodeContainerProtocol protos and refactor as per the new implementation of StorageIO in HDDS-48. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-247) Handle CLOSED_CONTAINER_IO exception in ozoneClient
Shashikant Banerjee created HDDS-247: Summary: Handle CLOSED_CONTAINER_IO exception in ozoneClient Key: HDDS-247 URL: https://issues.apache.org/jira/browse/HDDS-247 Project: Hadoop Distributed Data Store Issue Type: Bug Components: Ozone Client Reporter: Shashikant Banerjee Assignee: Shashikant Banerjee Fix For: 0.2.1 In case of ongoing writes by Ozone client to a container,the container might get closed on the Datanodes because of node loss, out of space issues etc. In such cases, the operation will fail with CLOSED_CONTAINER_IO exception exception. In cases as such, ozone client should try to get the committed length of the block from the Datanodes, and update the KSM. This Jira aims to address this issue. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
Re: [VOTE] Release Apache Hadoop 3.0.3 (RC0)
Welcome Jonathan. http://hadoop.apache.org/releases.html stated: "Hadoop is released as source code tarballs with corresponding binary tarballs for convenience. " and Andrew Wang said "The binary artifacts (including JARs) are technically just convenience artifacts" and it seems not an uncommon practice to do follow-up builds to release maven artifacts. IIRC, Andrew once shared with me that we started in 3.x to use a single build to to do both release binaries creation and maven artifacts deployment, prior releases are using multiple builds: Referring to https://wiki.apache.org/hadoop/HowToRelease - 3.x: step 4 in "Creating the release candidate (X.Y.Z-RC)" section does both release binaries creation and maven artifacts deployment. - prior to 3.x: step 4 does release binary creation, and step 10 does maven artifacts deployment, *each step does its build so two builds here*. As a matter of fact, I did not run step 10 for 3.0.3. That said, I agree that ideally it's better to do a single build to generate release binaries and deploy maven artifacts from the same build. Hope it helps. Welcome other folks to chime in. Best, --Yongjun On Mon, Jul 9, 2018 at 2:08 PM, Jonathan Eagles wrote: > Thank you, Yongjun Zhang for resolving this issue for me. I have verified > the 3.0.3 build is now working for me for tez to specify as a hadoop > dependency. > > As for release procedure, can someone comment on what to do now that the > artifacts published to maven are different than the voted on artifacts. I > believe the source code is what is voted on and the maven artifacts are > just for convenience, but would like an "official" answer. > > Reference: > https://issues.apache.org/jira/browse/TEZ-3955 > > Regards, > jeagles > > On Mon, Jul 9, 2018 at 12:26 PM, Yongjun Zhang > wrote: > >> HI Jonathan, >> >> I have updated the artifacts, so now >> >> https://repository.apache.org/#nexus-search;gav~org.apache.hadoop~~3 >> .0.2~~ >> https://repository.apache.org/#nexus-search;gav~org.apache.hadoop~~3.0.3 >> ~~ >> >> are more consistent, except that 3.0.3 has an extra entry for rbf. Would >> you please try again? >> >> The propagation to >> https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-project >> will take some time. I did nothing different than last time, so keep >> finger crossed that it will propagate there. >> >> Thanks Sammi Chen and Andrew Wang for info and advice, and sorry for the >> inconvenience again. >> >> Best, >> >> --Yongjun >> >> On Mon, Jul 2, 2018 at 9:30 AM, Jonathan Eagles >> wrote: >> >>> Release 3.0.3 is still broken due to the missing artifacts. Any update >>> on when these artifacts will be published? >>> >>> On Wed, Jun 27, 2018 at 8:25 PM, Chen, Sammi >>> wrote: >>> Hi Yongjun, The artifacts will be pushed to https://mvnrepository.com/arti fact/org.apache.hadoop/hadoop-project after step 6 of Publishing steps. For 2.9.1, I remember I absolutely did the step before. I redo the step 6 today and now 2.9.1 is pushed to the mvn repo. You can double check it. I suspect sometimes Nexus may fail to notify user when this is unexpected failures. Bests, Sammi *From:* Yongjun Zhang [mailto:yzh...@cloudera.com] *Sent:* Sunday, June 17, 2018 12:17 PM *To:* Jonathan Eagles ; Chen, Sammi < sammi.c...@intel.com> *Cc:* Eric Payne ; Hadoop Common < common-...@hadoop.apache.org>; Hdfs-dev ; mapreduce-...@hadoop.apache.org; yarn-...@hadoop.apache.org *Subject:* Re: [VOTE] Release Apache Hadoop 3.0.3 (RC0) + Junping, Sammi Hi Jonathan, Many thanks for reporting the issues and sorry for the inconvenience. 1. Shouldn't the build be looking for artifacts in https://repository.apache.org/content/repositories/releases rather than https://repository.apache.org/content/repositories/snapshots ? 2. Not seeing the artifact published here as well. https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-project Indeed, I did not see 2.9.1 there too. So included Sammi Chen. Hi Junping, would you please share which step in https://wiki.apache.org/hadoop/HowToRelease should have done this? Thanks a lot. --Yongjun On Fri, Jun 15, 2018 at 10:52 PM, Jonathan Eagles wrote: Upgraded Tez dependency to hadoop 3.0.3 and found this issue. Anyone else seeing this issue? [ERROR] Failed to execute goal on project hadoop-shim: Could not resolve dependencies for project org.apache.tez:hadoop-shim:jar:0.10.0-SNAPSHOT: Failed to collect dependencies at
[jira] [Created] (HDDS-246) Ratis leader should throw BlockNotCommittedException for uncommitted blocks to Ozone Client
Shashikant Banerjee created HDDS-246: Summary: Ratis leader should throw BlockNotCommittedException for uncommitted blocks to Ozone Client Key: HDDS-246 URL: https://issues.apache.org/jira/browse/HDDS-246 Project: Hadoop Distributed Data Store Issue Type: Bug Components: Ozone Datanode Reporter: Shashikant Banerjee Assignee: Shashikant Banerjee Fix For: 0.2.1 As a part of closing the container on a datanode, all the open keys(blocks) will be committed.In between if the client calls getCommittedBlockLength for uncommitted block on the container, the leader will throw BlockNotCommitted exception to the Client Back. The client should retry to fetch the committed block length and update the OzoneManager with the length. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-245) Handle ContainerReports in the SCM
Elek, Marton created HDDS-245: - Summary: Handle ContainerReports in the SCM Key: HDDS-245 URL: https://issues.apache.org/jira/browse/HDDS-245 Project: Hadoop Distributed Data Store Issue Type: Improvement Components: SCM Reporter: Elek, Marton Assignee: Elek, Marton Fix For: 0.2.1 HDDS-242 provides a new class ContainerReportHandler which could handle the ContainerReports from the SCMHeartbeatDispatchere. HDDS-228 introduces a new map to store the container -> datanode[] mapping HDDS-199 implements the ReplicationManager which could send commands to the datanodes to copy the datanode. To wire all these components, we need to put implementation to the ContainerReportHandler (created in HDDS-242). The ContainerReportHandler should process the new ContainerReportForDatanode events, update the containerStateMap and node2ContainerMap and calculate the missing/duplicate containers and send the ReplicateCommand to the ReplicateManager. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-244) PutKey should get executed only if all WriteChunk request fior the same block complete in Ratis
Shashikant Banerjee created HDDS-244: Summary: PutKey should get executed only if all WriteChunk request fior the same block complete in Ratis Key: HDDS-244 URL: https://issues.apache.org/jira/browse/HDDS-244 Project: Hadoop Distributed Data Store Issue Type: Bug Components: Ozone Datanode Reporter: Shashikant Banerjee Assignee: Shashikant Banerjee Fix For: 0.2.1 In Ratis, all the WriteChunk Requests are submitted to Ratis with Replication_Majority semantics. That means, the command execution from Ratis completes any 2 of 3 datanodes complete execution of the request. It might happen that on one of the follower, PutKey might start execution while all the WriteChunks requests processing for the same block are still in progress. There needs to be a synchronization enforced between PutKey and Corresponding WriteChunkrequets in the ContainerStateMachine. This Jira aims to address this. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
