[jira] [Created] (HDFS-13992) cross-cluster rack awareness for distcp
Ruslan Dautkhanov created HDFS-13992: Summary: cross-cluster rack awareness for distcp Key: HDFS-13992 URL: https://issues.apache.org/jira/browse/HDFS-13992 Project: Hadoop HDFS Issue Type: New Feature Affects Versions: 2.7.7, 3.0.3, 3.1.1, 2.8.4 Reporter: Ruslan Dautkhanov Would be great if distcp supported cross-cluster rack awareness. For example, we have hdfs cluster1 and hdfs cluster2. Both clusters span three switches, and both have rack awareness enabled. And also both clusters name same switches same way. So when distcp runs data replication job, it could replicate hdfs blocks only to counterpart datanodes on destination cluster that are in the same physical network switch, minimizing latencies and maximizing bandwidth. It could be an option, activate through `distcp` clommand-line switch. We have multiple clusters with default replication of 3 and all those cluster live in same three different "racks" / "top of the rack switches". This could drastically minimize inter-switch network traffic during huge distcp jobs. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
Re: [ANNOUNCE] Apache Hadoop Ozone 0.2.1-alpha release
> Just seeing this, why is this its own website with its own downloads section, look & feel etc instead of being part of our larger site? Ozone site is created before the new hadoop site. It was far more easier to create a separated sub-site. Since that the hadoop site is migrated and the ozone site is moved from a separated subdomain to hadoop.apache.org/ozone and the source of the ozone site is moved to the hadoop-site repository. Now it would be easier to merge the two sites as before. Technically it would mean to put both hadoop + ozone download links to the same download page (https://hadoop.apache.org/releases.html) and link both of the documentation from the documentation menu (now it contains multiple versions of hadoop, but it could contain multiple sections). Personally both work for me. I don't know which one is less confusing or which one is better from UX point of view. I would be happy to work in both directions. Marton On 10/11/18 7:27 PM, Vinod Kumar Vavilapalli wrote: > Congratulations on the release! > > Just seeing this, why is this its own website with its own downloads section, > look & feel etc instead of being part of our larger site? > > +Vinod > > >> On Oct 1, 2018, at 5:24 PM, Elek, Marton wrote: >> >> >> It gives me great pleasure to announce that the Apache Hadoop community has >> voted to release Apache Hadoop Ozone 0.2.1-alpha. >> >> Apache Hadoop Ozone is an object store for Hadoop built using Hadoop >> Distributed Data Store. >> >> For more information and to download, please check >> >> https://hadoop.apache.org/ozone >> >> Note: This release is alpha quality, it's not recommended to use in >> production. >> >> Many thanks to everyone who contributed to the release, and everyone in the >> Apache Hadoop community! The release is a result of work from many >> contributors. Thank you for all of them. >> >> On behalf of the Hadoop community, >> Márton Elek >> >> >> ps: Hadoop Ozone and HDDS are released separately from the main Hadoop >> releases, this release doesn't include new Hadoop Yarn/Mapreduce/Hdfs >> versions. >> >> - >> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org >> For additional commands, e-mail: common-dev-h...@hadoop.apache.org >> > > > - > To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org > For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org > - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-656) Add logic for pipeline report and action processing in new pipeline code
Lokesh Jain created HDDS-656: Summary: Add logic for pipeline report and action processing in new pipeline code Key: HDDS-656 URL: https://issues.apache.org/jira/browse/HDDS-656 Project: Hadoop Distributed Data Store Issue Type: Bug Components: SCM Reporter: Lokesh Jain Assignee: Lokesh Jain As part of pipeline refactoring, new pipeline management classes were added as part of HDDS-587. This Jira adds logic for pipeline report and action processing in the new code. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86
For more details, see https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/926/ [Oct 13, 2018 6:40:49 AM] (elek) HDDS-616. Collect all the robot test outputs and return with the right [Oct 13, 2018 1:55:57 PM] (elek) HDDS-518. Implement PutObject Rest endpoint. Contributed by chencan. [Oct 13, 2018 4:19:45 PM] (arp) HDDS-654. Add ServiceLoader resource for OzoneFileSystem. Contributed by [Oct 13, 2018 5:41:06 PM] (aengineer) HDDS-653. TestMetadataStore#testIterator fails on Windows. Contributed [Oct 13, 2018 7:12:08 PM] (jitendra) HDDS-629. Make ApplyTransaction calls in ContainerStateMachine [Oct 13, 2018 7:14:39 PM] (jitendra) Revert "HDDS-629. Make ApplyTransaction calls in ContainerStateMachine [Oct 13, 2018 7:15:42 PM] (jitendra) HDDS-603. Add BlockCommitSequenceId field per Container and expose it in -1 overall The following subsystems voted -1: asflicense findbugs hadolint pathlen unit xml The following subsystems voted -1 but were configured to be filtered/ignored: cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace The following subsystems are considered long running: (runtime bigger than 1h 0m 0s) unit Specific tests: XML : Parsing Error(s): hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/public/crossdomain.xml Failed CTEST tests : test_test_libhdfs_threaded_hdfs_static test_libhdfs_threaded_hdfspp_test_shim_static Failed junit tests : hadoop.security.TestRaceWhenRelogin hadoop.util.TestBasicDiskValidator hadoop.hdfs.TestEncryptionZones hadoop.hdfs.TestEncryptionZonesWithKMS hadoop.hdfs.client.impl.TestBlockReaderLocal hadoop.hdfs.web.TestWebHdfsTimeouts hadoop.yarn.webapp.TestRMWithXFSFilter hadoop.yarn.server.timelineservice.reader.TestTimelineReaderWebServicesHBaseStorage hadoop.yarn.sls.TestSLSRunner cc: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/926/artifact/out/diff-compile-cc-root.txt [4.0K] javac: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/926/artifact/out/diff-compile-javac-root.txt [300K] checkstyle: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/926/artifact/out/diff-checkstyle-root.txt [17M] hadolint: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/926/artifact/out/diff-patch-hadolint.txt [4.0K] pathlen: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/926/artifact/out/pathlen.txt [12K] pylint: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/926/artifact/out/diff-patch-pylint.txt [40K] shellcheck: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/926/artifact/out/diff-patch-shellcheck.txt [20K] shelldocs: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/926/artifact/out/diff-patch-shelldocs.txt [12K] whitespace: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/926/artifact/out/whitespace-eol.txt [9.3M] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/926/artifact/out/whitespace-tabs.txt [1.1M] xml: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/926/artifact/out/xml.txt [4.0K] findbugs: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/926/artifact/out/branch-findbugs-hadoop-hdds_client.txt [4.0K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/926/artifact/out/branch-findbugs-hadoop-hdds_container-service.txt [4.0K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/926/artifact/out/branch-findbugs-hadoop-hdds_framework.txt [4.0K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/926/artifact/out/branch-findbugs-hadoop-hdds_server-scm.txt [4.0K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/926/artifact/out/branch-findbugs-hadoop-hdds_tools.txt [4.0K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/926/artifact/out/branch-findbugs-hadoop-ozone_client.txt [8.0K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/926/artifact/out/branch-findbugs-hadoop-ozone_common.txt [4.0K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/926/artifact/out/branch-findbugs-hadoop-ozone_objectstore-service.txt [8.0K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/926/artifact/out/branch-findbugs-hadoop-ozone_ozone-manager.txt [4.0K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/926/artifact/out/branch-findbugs-hadoop-ozone_ozonefs.txt [8.0K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/926/artifact/out/branch-findbugs-hadoop-ozone_s3gateway.txt [4.0K]
