[DISCUSS] Move to gitbox
Hi all, Apache Hadoop git repository is in git-wip-us server and it will be decommissioned. If there are no objection, I'll file a JIRA ticket with INFRA to migrate to https://gitbox.apache.org/ and update documentation. According to ASF infra team, the timeframe is as follows: > - December 9th 2018 -> January 9th 2019: Voluntary (coordinated) relocation > - January 9th -> February 6th: Mandated (coordinated) relocation > - February 7th: All remaining repositories are mass migrated. > This timeline may change to accommodate various scenarios. If we got consensus by January 9th, I can file a ticket with INFRA and migrate it. Even if we cannot got consensus, the repository will be migrated by February 7th. Regards, Akira - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-9095) Removed Unused field from Resource: NUM_MANDATORY_RESOURCES
Szilard Nemeth created YARN-9095: Summary: Removed Unused field from Resource: NUM_MANDATORY_RESOURCES Key: YARN-9095 URL: https://issues.apache.org/jira/browse/YARN-9095 Project: Hadoop YARN Issue Type: Improvement Reporter: Szilard Nemeth I suppose this constant remained in the code from historical reasons, but this is not used anymore so it could be removed. This field is especially confusing for new readers, as ResourceInformation now has a field named MANDATORY_RESOURCES and this map contains not only memory and vcores but GPU and FPGA as well. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-9094) Remove unused interface method: NodeResourceUpdaterPlugin#handleUpdatedResourceFromRM
Szilard Nemeth created YARN-9094: Summary: Remove unused interface method: NodeResourceUpdaterPlugin#handleUpdatedResourceFromRM Key: YARN-9094 URL: https://issues.apache.org/jira/browse/YARN-9094 Project: Hadoop YARN Issue Type: Improvement Reporter: Szilard Nemeth Additionally, there's a typo can be fixed in the javadoc of NodeResourceUpdaterPlugin#updateConfiguredResource: look for "mododule" -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-9093) Remove commented code block from the beginning of TestDefaultContainerExecutor
Szilard Nemeth created YARN-9093: Summary: Remove commented code block from the beginning of TestDefaultContainerExecutor Key: YARN-9093 URL: https://issues.apache.org/jira/browse/YARN-9093 Project: Hadoop YARN Issue Type: Improvement Reporter: Szilard Nemeth -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-9092) Create an object for cgroups mount enable and cgroups mount path as they belong together
Szilard Nemeth created YARN-9092: Summary: Create an object for cgroups mount enable and cgroups mount path as they belong together Key: YARN-9092 URL: https://issues.apache.org/jira/browse/YARN-9092 Project: Hadoop YARN Issue Type: Improvement Reporter: Szilard Nemeth Assignee: Szilard Nemeth YarnConfiguration.NM_LINUX_CONTAINER_CGROUPS_MOUNT and YarnConfiguration.NM_LINUX_CONTAINER_CGROUPS_MOUNT_PATH are used in conjunction many places in the code, so for the sake of readabilty and simplicity, it is better to wrap the values of these configs to an object and use it instead of having 2 fields in CGroupsHandlerImpl and in CgroupsLCEResourcesHandler as well. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
Re: [VOTE] Merge HDFS-12943 branch to trunk - Consistent Reads from Standby
Hi Daryn, Wanted to backup Chen's earlier response to your concerns about rotating calls in the call queue. Our design 1. targets directly the livelock problem by rejecting calls on the Observer that are not likely to be responded in timely matter: HDFS-13873. 2. The call queue rotation is only done on Observers, and never on the active NN, so it stays free of attacks like you suggest. If this is a satisfactory mitigation for the problem could you please reconsider your -1, so that people could continue voting on this thread. Thanks, --Konst On Thu, Dec 6, 2018 at 10:38 AM Daryn Sharp wrote: > -1 pending additional info. After a cursory scan, I have serious concerns > regarding the design. This seems like a feature that should have been > purely implemented in hdfs w/o touching the common IPC layer. > > The biggest issue in the alignment context. It's purpose appears to be > for allowing handlers to reinsert calls back into the call queue. That's > completely unacceptable. A buggy or malicious client can easily cause > livelock in the IPC layer with handlers only looping on calls that never > satisfy the condition. Why is this not implemented via RetriableExceptions? > > On Thu, Dec 6, 2018 at 1:24 AM Yongjun Zhang > wrote: > >> Great work guys. >> >> Wonder if we can elaborate what's impact of not having #2 fixed, and why >> #2 >> is not needed for the feature to complete? >> 2. Need to fix automatic failover with ZKFC. Currently it does not doesn't >> know about ObserverNodes trying to convert them to SBNs. >> >> Thanks. >> --Yongjun >> >> >> On Wed, Dec 5, 2018 at 5:27 PM Konstantin Shvachko >> wrote: >> >> > Hi Hadoop developers, >> > >> > I would like to propose to merge to trunk the feature branch HDFS-12943 >> for >> > Consistent Reads from Standby Node. The feature is intended to scale >> read >> > RPC workloads. On large clusters reads comprise 95% of all RPCs to the >> > NameNode. We should be able to accommodate higher overall RPC workloads >> (up >> > to 4x by some estimates) by adding multiple ObserverNodes. >> > >> > The main functionality has been implemented see sub-tasks of HDFS-12943. >> > We followed up with the test plan. Testing was done on two independent >> > clusters (see HDFS-14058 and HDFS-14059) with security enabled. >> > We ran standard HDFS commands, MR jobs, admin commands including manual >> > failover. >> > We know of one cluster running this feature in production. >> > >> > There are a few outstanding issues: >> > 1. Need to provide proper documentation - a user guide for the new >> feature >> > 2. Need to fix automatic failover with ZKFC. Currently it does not >> doesn't >> > know about ObserverNodes trying to convert them to SBNs. >> > 3. Scale testing and performance fine-tuning >> > 4. As testing progresses, we continue fixing non-critical bugs like >> > HDFS-14116. >> > >> > I attached a unified patch to the umbrella jira for the review and >> Jenkins >> > build. >> > Please vote on this thread. The vote will run for 7 days until Wed Dec >> 12. >> > >> > Thanks, >> > --Konstantin >> > >> > > > -- > > Daryn >
[jira] [Created] (YARN-9091) Improve terminal message when connection is refused
Eric Yang created YARN-9091: --- Summary: Improve terminal message when connection is refused Key: YARN-9091 URL: https://issues.apache.org/jira/browse/YARN-9091 Project: Hadoop YARN Issue Type: Sub-task Reporter: Eric Yang If a user does not have proper access to login to a container. UI2 version of Terminal will not display any message. It would be nice to report back the connection has been refused. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[NOTICE] Mandatory relocation of Apache git repositories on git-wip-us.apache.org
[IF YOUR PROJECT DOES NOT HAVE GIT REPOSITORIES ON GIT-WIP-US PLEASE DISREGARD THIS EMAIL; IT WAS MASS-MAILED TO ALL APACHE PROJECTS] Hello Apache projects, I am writing to you because you may have git repositories on the git-wip-us server, which is slated to be decommissioned in the coming months. All repositories will be moved to the new gitbox service which includes direct write access on github as well as the standard ASF commit access via gitbox.apache.org. ## Why this move? ## The move comes as a result of retiring the git-wip service, as the hardware it runs on is longing for retirement. In lieu of this, we have decided to consolidate the two services (git-wip and gitbox), to ease the management of our repository systems and future-proof the underlying hardware. The move is fully automated, and ideally, nothing will change in your workflow other than added features and access to GitHub. ## Timeframe for relocation ## Initially, we are asking that projects voluntarily request to move their repositories to gitbox, hence this email. The voluntary timeframe is between now and January 9th 2019, during which projects are free to either move over to gitbox or stay put on git-wip. After this phase, we will be requiring the remaining projects to move within one month, after which we will move the remaining projects over. To have your project moved in this initial phase, you will need: - Consensus in the project (documented via the mailing list) - File a JIRA ticket with INFRA to voluntarily move your project repos over to gitbox (as stated, this is highly automated and will take between a minute and an hour, depending on the size and number of your repositories) To sum up the preliminary timeline; - December 9th 2018 -> January 9th 2019: Voluntary (coordinated) relocation - January 9th -> February 6th: Mandated (coordinated) relocation - February 7th: All remaining repositories are mass migrated. This timeline may change to accommodate various scenarios. ## Using GitHub with ASF repositories ## When your project has moved, you are free to use either the ASF repository system (gitbox.apache.org) OR GitHub for your development and code pushes. To be able to use GitHub, please follow the primer at: https://reference.apache.org/committer/github We appreciate your understanding of this issue, and hope that your project can coordinate voluntarily moving your repositories in a timely manner. All settings, such as commit mail targets, issue linking, PR notification schemes etc will automatically be migrated to gitbox as well. With regards, Daniel on behalf of ASF Infra. PS:For inquiries, please reply to us...@infra.apache.org, not your project's dev list :-). - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-9090) [Submarine] Adjust the submarine installation script document
Xun Liu created YARN-9090: - Summary: [Submarine] Adjust the submarine installation script document Key: YARN-9090 URL: https://issues.apache.org/jira/browse/YARN-9090 Project: Hadoop YARN Issue Type: Sub-task Reporter: Xun Liu Assignee: Xun Liu Migrate the submarine installation script document from the hadoop-yarn project. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86
For more details, see https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/980/ [Dec 6, 2018 12:50:28 PM] (vinayakumarb) HDFS-14113. EC : Add Configuration to restrict UserDefined Policies. [Dec 6, 2018 7:39:59 PM] (bharat) HDDS-864. Use strongly typed codec implementations for the tables of the [Dec 6, 2018 8:48:17 PM] (jlowe) MAPREDUCE-7159. FrameworkUploader: ensure proper permissions of [Dec 6, 2018 9:27:28 PM] (ajay) HDDS-880. Create api for ACL handling in Ozone. (Contributed by Ajay [Dec 6, 2018 9:33:52 PM] (hanishakoneru) HDDS-858. Start a Standalone Ratis Server on OM [Dec 6, 2018 11:37:34 PM] (bharat) HDDS-892. Parse aws v2 headers without spaces in Ozone s3 gateway. -1 overall The following subsystems voted -1: asflicense findbugs hadolint pathlen unit The following subsystems voted -1 but were configured to be filtered/ignored: cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace The following subsystems are considered long running: (runtime bigger than 1h 0m 0s) unit Specific tests: Failed junit tests : hadoop.registry.secure.TestSecureLogins hadoop.hdfs.server.namenode.TestPersistentStoragePolicySatisfier hadoop.hdfs.web.TestWebHdfsTimeouts hadoop.hdfs.server.datanode.TestDirectoryScanner hadoop.yarn.server.resourcemanager.scheduler.capacity.TestQueueManagementDynamicEditPolicy cc: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/980/artifact/out/diff-compile-cc-root.txt [4.0K] javac: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/980/artifact/out/diff-compile-javac-root.txt [336K] checkstyle: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/980/artifact/out/diff-checkstyle-root.txt [17M] hadolint: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/980/artifact/out/diff-patch-hadolint.txt [4.0K] pathlen: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/980/artifact/out/pathlen.txt [12K] pylint: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/980/artifact/out/diff-patch-pylint.txt [40K] shellcheck: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/980/artifact/out/diff-patch-shellcheck.txt [20K] shelldocs: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/980/artifact/out/diff-patch-shelldocs.txt [12K] whitespace: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/980/artifact/out/whitespace-eol.txt [9.3M] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/980/artifact/out/whitespace-tabs.txt [1.1M] findbugs: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/980/artifact/out/branch-findbugs-hadoop-hdds_client.txt [8.0K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/980/artifact/out/branch-findbugs-hadoop-hdds_container-service.txt [4.0K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/980/artifact/out/branch-findbugs-hadoop-hdds_framework.txt [4.0K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/980/artifact/out/branch-findbugs-hadoop-hdds_server-scm.txt [8.0K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/980/artifact/out/branch-findbugs-hadoop-hdds_tools.txt [4.0K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/980/artifact/out/branch-findbugs-hadoop-ozone_client.txt [8.0K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/980/artifact/out/branch-findbugs-hadoop-ozone_common.txt [4.0K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/980/artifact/out/branch-findbugs-hadoop-ozone_objectstore-service.txt [8.0K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/980/artifact/out/branch-findbugs-hadoop-ozone_ozone-manager.txt [4.0K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/980/artifact/out/branch-findbugs-hadoop-ozone_ozonefs.txt [12K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/980/artifact/out/branch-findbugs-hadoop-ozone_s3gateway.txt [4.0K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/980/artifact/out/branch-findbugs-hadoop-ozone_tools.txt [8.0K] javadoc: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/980/artifact/out/diff-javadoc-javadoc-root.txt [752K] unit: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/980/artifact/out/patch-unit-hadoop-common-project_hadoop-registry.txt [12K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/980/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt [592K]
