Re: [DISCUSS] Use of AssertJ for testing
assertj is already on the test classpath for some modules already, though an older version (3.8.0) from what's current (3.12.0) No problems seem to have surfaced, so I don't see anything wrong with picking it up more broadly. if someone wants to update the assertj version, it's only a test artifact so no worries about that On Mon, Apr 1, 2019 at 5:37 AM Akira Ajisaka wrote: > Hi folks, > > Now I'm going to upgrade the JUnit version from 4 to 5 for Java 11 support. > I wanted to start with the small module, so I uploaded a patch to upgrade > the API in hadoop-yarn-api module at first (YARN-8943), and in this JIRA, > Szilard Nemeth suggested using AssertJ with JUnit 5. (Thanks Szilard > for the suggestion!) > > I think the JUnit upgrade and the use of AssertJ are separated, but > related tasks. > Therefore, I'd like to decide: > - Use AssertJ or not > - If we are going to use AssertJ, when to use AssertJ (before > upgrading JUnit or after?) > > My opinion is: > - JUnit migration is required for Java 11, so upgrading JUnit as soon > as possible. > - After the migration, we may use AssertJ for existing tests. > - We may use AssertJ for new tests. (not must) > > Any thoughts? > > Thanks, > Akira > > - > To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org > For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org > >
[jira] [Created] (YARN-9443) Fast RM Failover using Ratis (Raft protocol)
Prabhu Joseph created YARN-9443: --- Summary: Fast RM Failover using Ratis (Raft protocol) Key: YARN-9443 URL: https://issues.apache.org/jira/browse/YARN-9443 Project: Hadoop YARN Issue Type: New Feature Components: resourcemanager Affects Versions: 3.2.0 Reporter: Prabhu Joseph During Failover, the RM Standby will have a lag as it has to recover from Zookeeper / FileSystem StateStore. RM HA using Ratis (Raft Protocol) can achieve Fast failover as all RMs are in sync already. This is used by Ozone - HDDS-505. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-9442) container working directory has group read permissions
Jim Brennan created YARN-9442: - Summary: container working directory has group read permissions Key: YARN-9442 URL: https://issues.apache.org/jira/browse/YARN-9442 Project: Hadoop YARN Issue Type: Improvement Components: yarn Affects Versions: 3.2.2 Reporter: Jim Brennan Container working directories are currently created with permissions 0750, owned by the user and with the group set to the node manager group. Is there any reason why these directories need group read permissions? I have been testing with group read permissions removed and so far I haven't encountered any problems. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86
For more details, see https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1096/ [Apr 3, 2019 4:16:59 AM] (aajisaka) HADOOP-16226. new Path(String str) does not remove all the trailing [Apr 3, 2019 6:01:30 AM] (yqlin) HDDS-1365. Fix error handling in KeyValueContainerCheck. Contributed by [Apr 3, 2019 10:35:02 AM] (aajisaka) HADOOP-16232. Fix errors in the checkstyle configration xmls. [Apr 3, 2019 1:27:28 PM] (sunilg) YARN-4901. QueueMetrics needs to be cleared before MockRM is [Apr 3, 2019 4:49:10 PM] (shashikant) HDDS-1164. Add New blockade Tests to test Replica Manager. Contributed [Apr 3, 2019 5:56:33 PM] (todd) HDFS-14394: Add -std=c99 / -std=gnu99 to libhdfs compile flags [Apr 3, 2019 6:00:59 PM] (weichiu) HDFS-10477. Stop decommission a rack of DataNodes caused NameNode fail [Apr 3, 2019 6:53:51 PM] (7813154+ajayydv) HDDS-1377. OM failed to start with incorrect hostname set as ip address [Apr 3, 2019 6:59:39 PM] (mackrorysd) HADOOP-16210. Update guava to 27.0-jre in hadoop-project trunk. [Apr 3, 2019 8:20:51 PM] (arp7) HDDS-1330 : Add a docker compose for Ozone deployment with Recon. (#669) [Apr 3, 2019 8:23:40 PM] (github) HADOOP-16233. S3AFileStatus to declare that isEncrypted() is always true [Apr 3, 2019 9:29:52 PM] (weichiu) HADOOP-16011. OsSecureRandom very slow compared to other SecureRandom [Apr 3, 2019 9:52:06 PM] (arp7) HDDS-1358 : Recon Server REST API not working as expected. (#668) [Apr 3, 2019 10:02:00 PM] (github) HDDS-1211. Test SCMChillMode failing randomly in Jenkins run (#543) [Apr 3, 2019 11:02:19 PM] (bharat) HDDS-1324. TestOzoneManagerHA tests are flaky (#676) [Apr 3, 2019 11:11:13 PM] (inigoiri) HDFS-14327. Using FQDN instead of IP to access servers with DNS -1 overall The following subsystems voted -1: asflicense findbugs hadolint pathlen unit The following subsystems voted -1 but were configured to be filtered/ignored: cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace The following subsystems are considered long running: (runtime bigger than 1h 0m 0s) unit Specific tests: FindBugs : module:hadoop-common-project/hadoop-kms Null passed for non-null parameter of com.google.common.base.Strings.isNullOrEmpty(String) in org.apache.hadoop.crypto.key.kms.server.KMSAudit.op(KMSAuditLogger$OpStatus, Object, UserGroupInformation, String, String, String) Method invoked at KMSAudit.java:of com.google.common.base.Strings.isNullOrEmpty(String) in org.apache.hadoop.crypto.key.kms.server.KMSAudit.op(KMSAuditLogger$OpStatus, Object, UserGroupInformation, String, String, String) Method invoked at KMSAudit.java:[line 195] FindBugs : module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager Null passed for non-null parameter of com.google.common.base.Strings.emptyToNull(String) in org.apache.hadoop.yarn.server.nodemanager.NodeHealthCheckerService.getHealthReport() Method invoked at NodeHealthCheckerService.java:of com.google.common.base.Strings.emptyToNull(String) in org.apache.hadoop.yarn.server.nodemanager.NodeHealthCheckerService.getHealthReport() Method invoked at NodeHealthCheckerService.java:[line 66] Null passed for non-null parameter of com.google.common.base.Strings.emptyToNull(String) in org.apache.hadoop.yarn.server.nodemanager.NodeHealthCheckerService.getHealthReport() Method invoked at NodeHealthCheckerService.java:of com.google.common.base.Strings.emptyToNull(String) in org.apache.hadoop.yarn.server.nodemanager.NodeHealthCheckerService.getHealthReport() Method invoked at NodeHealthCheckerService.java:[line 72] FindBugs : module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager Null passed for non-null parameter of com.google.common.util.concurrent.SettableFuture.set(Object) in org.apache.hadoop.yarn.server.resourcemanager.recovery.RMStateStore$UpdateAppTransition.transition(RMStateStore, RMStateStoreEvent) At RMStateStore.java:of com.google.common.util.concurrent.SettableFuture.set(Object) in org.apache.hadoop.yarn.server.resourcemanager.recovery.RMStateStore$UpdateAppTransition.transition(RMStateStore, RMStateStoreEvent) At RMStateStore.java:[line 291] Null passed for non-null parameter of com.google.common.util.concurrent.SettableFuture.set(Object) in org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.updateApplicationPriority(Priority, ApplicationId, SettableFuture, UserGroupInformation) At CapacityScheduler.java:of com.google.common.util.concurrent.SettableFuture.set(Object) in org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.updateApplicationPriority(Priority, ApplicationId, SettableFuture, UserGroupInformation) At CapacityScheduler.java:[line 2647] FindBugs :
Tool to view current status (applicability) of patches
Hi, I developed a tool recently, that can help to see the status of patches and checks if they can be applied to trunk (or any specified branch). The motivation for the project was quite trivial: It's very cumbersome to keep track of the status of all our pending patches with our YARN team. We already have a sheet to keep track of pending patches of our upstream work, so there came my idea: Let's write a script that checks if the patches still apply to trunk (or any specified branch / branches). The project currently has jira and Google Sheets integration (read/write). On a more longer-term, I'm planning to provide some javascript script that would place Red/Green status lights next to the patches that indicate their applicability to trunk. I would also pay attention to minimize requests sent to jira, so I'm planning to introduce some caching and provide a "force-refresh status" button to get the current status of the patch. Do you guys think this is a good idea and it's worth to spend some more time on this? This would require a moderate amount of work but my main concern is where to host this service. Is there an Apache server (or any other infra) that could host this application? The memory / cpu footprint is quite moderate, it requires some network bandwidth, though. Here's the link for the git repo of the project: https://github.com/szilard-nemeth/hadoop-reviewsync The project is still in a PoC phase so the code is not the cleanest, I'm planning to improve this in the near-future. Please feel free to share your thoughs, feedback, ideas, anything! Thanks, Szilard ╒═══╤╤═══╤╤══╤═══╤╤═╤══╤═══╕ │ Row │ Issue │ Patch apply │ Owner │ Patch file │ Branch│ Explicit │ Result │ Number of conflicted files │ Overall result │ ╞═══╪╪═══╪╪══╪═══╪╪═╪══╪═══╡ │ 1 │ YARN-8553 │ 1 │ Szilard Nemeth │ YARN-8553.003.patch │ origin/trunk │ Yes│ CONFLICT │ 1│ origin/trunk: CONFLICT, origin/branch-3.2: OK, origin/branch-3.1: CONFLICT│ ├───┼┼───┼┼──┼───┼┼─┼──┼───┤ │ 2 │ YARN-8553 │ 2 │ Szilard Nemeth │ YARN-8553.003.patch │ origin/branch-3.2 │ No │ APPLIES CLEANLY │ N/A │ origin/trunk: CONFLICT, origin/branch-3.2: OK, origin/branch-3.1: CONFLICT│ ├───┼┼───┼┼──┼───┼┼─┼──┼───┤ │ 3 │ YARN-8553 │ 3 │ Szilard Nemeth │ YARN-8553.003.patch │ origin/branch-3.1 │ No │ CONFLICT │ 1│ origin/trunk: CONFLICT, origin/branch-3.2: OK, origin/branch-3.1: CONFLICT│ ├───┼┼───┼┼──┼───┼┼─┼──┼───┤ │ 4 │ YARN-5464 │ 1 │ Antal Bálint Steinbach │ YARN-5464.005.patch │ origin/trunk │ Yes│ CONFLICT │ 3│ origin/trunk: CONFLICT, origin/branch-3.2: CONFLICT, origin/branch-3.1: CONFLICT │ ├───┼┼───┼┼──┼───┼┼─┼──┼───┤ │ 5 │ YARN-5464 │ 2 │ Antal Bálint Steinbach │ YARN-5464.005.patch │ origin/branch-3.2 │ No │ CONFLICT │ 12 │ origin/trunk: CONFLICT, origin/branch-3.2: CONFLICT, origin/branch-3.1: CONFLICT │
[jira] [Created] (YARN-9441) Component name should start with Apache Hadoop for consistency
Weiwei Yang created YARN-9441:
-
Summary: Component name should start with Apache Hadoop for
consistency
Key: YARN-9441
URL: https://issues.apache.org/jira/browse/YARN-9441
Project: Hadoop YARN
Issue Type: Sub-task
Reporter: Weiwei Yang
When I build from source, I found
{noformat}
[INFO] Apache Hadoop MapReduce HistoryServer .. SUCCESS [ 2.360 s]
[INFO] Apache Hadoop MapReduce JobClient .. SUCCESS [ 3.980 s]
[INFO] Apache Hadoop Mini-Cluster . SUCCESS [ 2.049 s]
[INFO] Apache Hadoop YARN Services SUCCESS [ 0.082 s]
[INFO] Apache Hadoop YARN Services Core ... SUCCESS [ 3.063 s]
[INFO] Apache Hadoop YARN Services API SUCCESS [ 1.885 s]
[INFO] YARN Application Catalog ... SUCCESS [ 0.086 s]
[INFO] YARN Application Catalog Webapp FAILURE [ 5.386 s]
[INFO] YARN Application Catalog Docker Image .. SKIPPED
{noformat}
The pom modules should be named with "Apache Hadoop" prefix for consistency.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-9440) Improve diagnostics for scheduler and app activities
Tao Yang created YARN-9440: -- Summary: Improve diagnostics for scheduler and app activities Key: YARN-9440 URL: https://issues.apache.org/jira/browse/YARN-9440 Project: Hadoop YARN Issue Type: Sub-task Components: capacityscheduler Reporter: Tao Yang Assignee: Tao Yang [Design doc|https://docs.google.com/document/d/1pwf-n3BCLW76bGrmNPM4T6pQ3vC4dVMcN2Ud1hq1t2M/edit#heading=h.d2ru7sigsi7j] -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-9439) Support asynchronized scheduling mode and multi-node lookup mechanism for app activities
Tao Yang created YARN-9439: -- Summary: Support asynchronized scheduling mode and multi-node lookup mechanism for app activities Key: YARN-9439 URL: https://issues.apache.org/jira/browse/YARN-9439 Project: Hadoop YARN Issue Type: Sub-task Components: capacityscheduler Reporter: Tao Yang Assignee: Tao Yang [Design doc|https://docs.google.com/document/d/1pwf-n3BCLW76bGrmNPM4T6pQ3vC4dVMcN2Ud1hq1t2M/edit#heading=h.d2ru7sigsi7j] -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
