[jira] [Created] (HADOOP-18616) Java 11 JavaDoc fails due to missing package comments
Steve Vaughan created HADOOP-18616: -- Summary: Java 11 JavaDoc fails due to missing package comments Key: HADOOP-18616 URL: https://issues.apache.org/jira/browse/HADOOP-18616 Project: Hadoop Common Issue Type: Improvement Components: common Affects Versions: 3.4.0, 3.3.5, 3.3.9 Environment: Yetus Java 11 OpenJDK JavaDoc Reporter: Steve Vaughan Assignee: Steve Vaughan Fix For: 3.4.0, 3.3.9 Submissions to `hadoop-common` fail in Yetus due to Java 11 JavaDoc errors: ``` [ERROR] /home/builder/src/hadoop/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/concurrent/package-info.java:21: error: unknown tag: InterfaceAudience.Private [ERROR] @InterfaceAudience.Private [ERROR] ^ [ERROR] /home/builder/src/hadoop/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/concurrent/package-info.java:22: error: unknown tag: InterfaceStability.Unstable [ERROR] @InterfaceStability.Unstable [ERROR] ^ ``` -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-18614) Ensure that the config writer are closed
Steve Vaughan created HADOOP-18614: -- Summary: Ensure that the config writer are closed Key: HADOOP-18614 URL: https://issues.apache.org/jira/browse/HADOOP-18614 Project: Hadoop Common Issue Type: Test Components: common Affects Versions: 3.4.0, 3.3.5, 3.3.9 Environment: Tested using the Hadoop development environment docker image. Reporter: Steve Vaughan Assignee: Steve Vaughan Fix For: 3.4.0, 3.3.9 Use AutoCloseable to ensure that the config writers are closed between tests. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-18612) Avoid mixing canonical and non-canonical when performing comparisons
Steve Vaughan created HADOOP-18612: -- Summary: Avoid mixing canonical and non-canonical when performing comparisons Key: HADOOP-18612 URL: https://issues.apache.org/jira/browse/HADOOP-18612 Project: Hadoop Common Issue Type: Bug Components: common Affects Versions: 3.4.0, 3.3.5, 3.3.9 Environment: Tests were run using the Hadoop development environment docker image. Reporter: Steve Vaughan Assignee: Steve Vaughan Fix For: 3.4.0, 3.3.9 The tests mix canonical and non-canonical paths and then perform comparisons. We can avoid unexpected failures by ensuring that comparisons are always made against canonical forms. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-18450) JavaKeyStoreProvider should throw FileNotFoundException in renameOrFail
Steve Vaughan created HADOOP-18450:
--
Summary: JavaKeyStoreProvider should throw FileNotFoundException
in renameOrFail
Key: HADOOP-18450
URL: https://issues.apache.org/jira/browse/HADOOP-18450
Project: Hadoop Common
Issue Type: New Feature
Components: common
Affects Versions: 3.4.0, 3.3.9
Environment: UBI 8 running Java 11
Reporter: Steve Vaughan
Assignee: Steve Vaughan
Attempting to create a key a KMS is configured with the JavaKeystoreProvider
and an HDFS store. The calls to:
{noformat}
renameOrFail(Path src, Path dest) throws IOException {noformat}
... fails with an IOException when it attempts to rename a file. The calling
code catches FileNotFoundException since the src file may not exist.
Example:
{noformat}
$ hadoop key create sample
java.io.IOException: Rename unsuccessful :
'hdfs://mycluster/security/kms.jks_NEW' to
'hdfs://mycluster/security/kms.jks_NEW_ORPHANED_1662946593691'{noformat}
Update the implementation to check for the file, throwing a
FileNotFoundException.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-18431) Unit test can fail due to unexpected host resolution
Steve Vaughan created HADOOP-18431: -- Summary: Unit test can fail due to unexpected host resolution Key: HADOOP-18431 URL: https://issues.apache.org/jira/browse/HADOOP-18431 Project: Hadoop Common Issue Type: Bug Components: test Affects Versions: 3.4.0, 3.3.9 Environment: Running using both Maven Surefire and an IDE results in a test failure. Switching the name to "bogus.invalid" results in the expected behavior, which depends on an UnknownHostException. Reporter: Steve Vaughan Assignee: Steve Vaughan Tests that want to use an unresolvable address may actually resolve in some environments. Replacing host names like "bogus" with a IETF RFC 2606 domain name avoids the issue. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-18419) Don't fail when using -DNoUnitTests
[ https://issues.apache.org/jira/browse/HADOOP-18419?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Vaughan resolved HADOOP-18419. Resolution: Duplicate > Don't fail when using -DNoUnitTests > --- > > Key: HADOOP-18419 > URL: https://issues.apache.org/jira/browse/HADOOP-18419 > Project: Hadoop Common > Issue Type: Bug > Components: build >Affects Versions: 3.4.0 >Reporter: Steve Vaughan >Assignee: Steve Vaughan >Priority: Critical > Labels: pull-request-available > > There are commands in hadoop.sh that use -Dtest=NoUnitTests. I believe this > is intended to replace -DskipTest, by not matching any tests. This causes an > issue since the default for surefire.failIfNoSpecifiedTests is true. > Updating those commands to override the default will address the issue. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-18419) Don't fail when using -DNoUnitTests
Steve Vaughan created HADOOP-18419: -- Summary: Don't fail when using -DNoUnitTests Key: HADOOP-18419 URL: https://issues.apache.org/jira/browse/HADOOP-18419 Project: Hadoop Common Issue Type: Bug Components: build Affects Versions: 3.4.0 Reporter: Steve Vaughan Assignee: Steve Vaughan There are commands in hadoop.sh that use -Dtest=NoUnitTests. I believe this is intended to replace -DskipTest, by not matching any tests. This causes an issue since the default for surefire.failIfNoSpecifiedTests is true. Updating those commands to override the default will address the issue. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-18417) Upgrade Maven Surefire plugin
Steve Vaughan created HADOOP-18417: -- Summary: Upgrade Maven Surefire plugin Key: HADOOP-18417 URL: https://issues.apache.org/jira/browse/HADOOP-18417 Project: Hadoop Common Issue Type: Improvement Components: build Affects Versions: 3.4.0, 3.3.9 Reporter: Steve Vaughan Assignee: Steve Vaughan The Maven Surefire plugin 3.0.0-M1 doesn't always including the launcher as part of it's setup, which can cause problems with Yarn tests. Some of the Yarn modules use Jupiter, which may be a complicating factor. Switching to 3.0.0-M7 fixes the issue. This is currently blocking MAPREDUCE-7386 -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-18396) Issues running in dynamic / managed environments
Steve Vaughan created HADOOP-18396: -- Summary: Issues running in dynamic / managed environments Key: HADOOP-18396 URL: https://issues.apache.org/jira/browse/HADOOP-18396 Project: Hadoop Common Issue Type: Improvement Affects Versions: 3.4.0, 3.3.9, 3.3.4 Environment: Running an HA configuration in Kubernetes, using Java 11. Reporter: Steve Vaughan Assignee: Steve Vaughan Running in dynamic or managed environments is a challenge because we can't assume that all services will have DNS entries, will be started in a specific order, will maintain constant IP addresses, etc. I'm using the following assumptions to guide the changes necessary to operate in this kind of environment: # The configuration files are an expression of desired state # If a referenced service instance is not resolvable or reachable at a moment in time, it will be eventually and should be able to participate in the future, as if it had been there originally, without requiring manual intervention # IP address changes should be handled in a way that no only allows distributed calls to continue to function, but avoids having to re-resolve the address over and over # Code that requires resolved names (Kerberos and DataNode registration) should fall back to DNS reverse lookups to work around temporary issues caused by caching. Example: The DataNode registration is only performed at startup, and yet the extra check that allows it to succeed in registering with the NameNode isn’t performed # If an HA system is supposed to only require a quorum, then we shouldn’t require the full set, allowing the called service to bring the remaining instances into compliance # Managing a service should be independent of other services. Example: You should be able to perform a rolling restart of JournalNodes without worrying about causing an issue with NameNodes as long as a quorum is present. A proof of these concepts would be the ability to: * Start with less that the full replica count of a service, while still providing the required quorum or minimal count, should still allow a cluster to start and function. Example: 2 out of 3 configured JournalNodes should still allow the NameNode to format, function, rollover to the standby, etc. * Introduce missing instances should join the existing cluster without manual intervention. Example: Starting the 3rd JournalNode should automatically be formatted and brought up to date * Perform rolling restarts of individual services without negatively impacting other services (causing failures, restarts, etc.). Example: Rolling restarts of JournalNodes shouldn't cause problems in NameNodes; Rolling restarts of NameNodes shouldn't cause problems with DataNodes * Logs should only report updated IP addresses once (per dependent), avoiding costly re-resolution -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-18394) Enable parallel building in pre-commit jobs
Steve Vaughan created HADOOP-18394: -- Summary: Enable parallel building in pre-commit jobs Key: HADOOP-18394 URL: https://issues.apache.org/jira/browse/HADOOP-18394 Project: Hadoop Common Issue Type: Sub-task Components: build Reporter: Steve Vaughan Assignee: Steve Vaughan Once MAPREDUCE-7386 is merged, we can update the pre-commit jobs to take advantage of the parallel builds. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-18365) Updated addresses are still accessed using the old IP address
Steve Vaughan created HADOOP-18365: -- Summary: Updated addresses are still accessed using the old IP address Key: HADOOP-18365 URL: https://issues.apache.org/jira/browse/HADOOP-18365 Project: Hadoop Common Issue Type: Improvement Components: common Environment: Demonstrated in a Kubernetes environment running Java 11. Reporter: Steve Vaughan When the IPC Client recognizes that an IP address has changed, it updates the server field and logs a message: Address change detected. Old: journalnode-1.journalnode.hdfs.svc.cluster.local/10.1.0.178:8485 New: journalnode-1.journalnode.hdfs.svc.cluster.local/10.1.0.182:8485 Although the change is detected, the client will continue to connect to the old IP address, resulting in repeated log messages. This is seen in managed environments when JournalNode syncing is enabled and a JournalNode is restarted, with the remaining nodes in the set repeatedly logging this message when syncing to the restarted JournalNode. The source of the problem is that the remoteId.address is not updated. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-18311) Upgrade dependencies to address several CVEs
Steve Vaughan created HADOOP-18311: -- Summary: Upgrade dependencies to address several CVEs Key: HADOOP-18311 URL: https://issues.apache.org/jira/browse/HADOOP-18311 Project: Hadoop Common Issue Type: Improvement Components: common Affects Versions: 3.3.3, 3.3.4 Reporter: Steve Vaughan Fix For: 3.3.4 The following CVEs can be addressed by upgrading dependencies within the build. This includes a replacement of HTrace with a noop implementation. * CVE-2018-7489 * CVE-2020-10663 * CVE-2020-28491 * CVE-2020-35490 * CVE-2020-35491 * CVE-2020-36518 * PRISMA-2021-0182 This addresses all of the CVEs from 3.3.3 except for ones that would require upgrading Netty to 4.x. I'll be submitting a pull request for 3.3.4. -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-18279) Cancel fileMonitoringTimer even if trustManager isn't defined
Steve Vaughan created HADOOP-18279: -- Summary: Cancel fileMonitoringTimer even if trustManager isn't defined Key: HADOOP-18279 URL: https://issues.apache.org/jira/browse/HADOOP-18279 Project: Hadoop Common Issue Type: Bug Components: common, test Affects Versions: 3.4.0, 3.3.4 Environment: This problem was identified using the Hadoop Development Environment during unit testing. Reporter: Steve Vaughan The key stores factory starts a timer for monitoring file changes that should be reloaded. The call to destroy() doesn't cancel the timer when a trust manager isn't defined. This leaves the timer running, during shutdown. This can be seen in unit tests that do not stop when the test completes. -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-17418) Generally support PEM keystores
Steve Vaughan created HADOOP-17418:
--
Summary: Generally support PEM keystores
Key: HADOOP-17418
URL: https://issues.apache.org/jira/browse/HADOOP-17418
Project: Hadoop Common
Issue Type: Improvement
Components: common
Reporter: Steve Vaughan
Several Apache projects have added some minimal PEM support. Usually this is
presented as part of supporting generated certificates for a trust store. By
adding a PEM Java KeyStore to Hadoop common, we could provide this support for
many clients.
Example:
{code:java}
ssl.server.truststore.type
PEM
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
