Apache Hadoop qbt Report: branch-3.3+JDK8 on Linux/x86_64
For more details, see https://ci-hadoop.apache.org/job/hadoop-qbt-branch-3.3-java8-linux-x86_64/64/ [Jul 7, 2022 9:04:15 PM] (Eric Payne) YARN-10997. Revisit allocation and reservation logging. Contributed by Andras Gyori [Jul 11, 2022 7:55:18 PM] (Owen O'Malley) HDFS-16591. Setup JaasConfiguration in ZKCuratorManager when SASL is enabled [Jul 13, 2022 11:37:32 AM] (Steve Loughran) HADOOP-18217. ExitUtil synchronized blocks reduced. #4255 [Jul 13, 2022 11:57:28 AM] (Steve Loughran) HADOOP-18336. Tag FSDataInputStream.getWrappedStream() @Public/@Stable (#4555) -1 overall The following subsystems voted -1: blanks pathlen unit xml The following subsystems voted -1 but were configured to be filtered/ignored: cc checkstyle javac javadoc pylint shellcheck The following subsystems are considered long running: (runtime bigger than 1h 0m 0s) unit Specific tests: XML : Parsing Error(s): hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-excerpt.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-missing-tags.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-missing-tags2.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-sample-output.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/fair-scheduler-invalid.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/yarn-site-with-invalid-allocation-file-ref.xml Failed junit tests : hadoop.hdfs.server.balancer.TestBalancerRPCDelay hadoop.hdfs.server.datanode.TestDataNodeRollingUpgrade hadoop.hdfs.server.federation.router.TestRouterRpc hadoop.hdfs.server.federation.router.TestRouterRpcMultiDestination hadoop.yarn.client.api.impl.TestAMRMClient cc: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-3.3-java8-linux-x86_64/64/artifact/out/results-compile-cc-root.txt [48K] javac: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-3.3-java8-linux-x86_64/64/artifact/out/results-compile-javac-root.txt [376K] blanks: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-3.3-java8-linux-x86_64/64/artifact/out/blanks-eol.txt [13M] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-3.3-java8-linux-x86_64/64/artifact/out/blanks-tabs.txt [2.0M] checkstyle: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-3.3-java8-linux-x86_64/64/artifact/out/results-checkstyle-root.txt [14M] pathlen: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-3.3-java8-linux-x86_64/64/artifact/out/results-pathlen.txt [16K] pylint: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-3.3-java8-linux-x86_64/64/artifact/out/results-pylint.txt [20K] shellcheck: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-3.3-java8-linux-x86_64/64/artifact/out/results-shellcheck.txt [20K] xml: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-3.3-java8-linux-x86_64/64/artifact/out/xml.txt [24K] javadoc: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-3.3-java8-linux-x86_64/64/artifact/out/results-javadoc-javadoc-root.txt [1.1M] unit: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-3.3-java8-linux-x86_64/64/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt [544K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-3.3-java8-linux-x86_64/64/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt [60K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-3.3-java8-linux-x86_64/64/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt [52K] Powered by Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86_64
For more details, see https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/921/ [Jul 13, 2022 11:35:44 AM] (noreply) HADOOP-18217. ExitUtil synchronized blocks reduced. #4255 [Jul 13, 2022 11:56:56 AM] (noreply) HADOOP-18336.Tag FSDataInputStream.getWrappedStream() @Public/@Stable (#4555) -1 overall The following subsystems voted -1: blanks pathlen xml The following subsystems voted -1 but were configured to be filtered/ignored: cc checkstyle javac javadoc pylint shellcheck The following subsystems are considered long running: (runtime bigger than 1h 0m 0s) unit Specific tests: XML : Parsing Error(s): hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-excerpt.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-missing-tags.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-missing-tags2.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-sample-output.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/fair-scheduler-invalid.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/yarn-site-with-invalid-allocation-file-ref.xml cc: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/921/artifact/out/results-compile-cc-root.txt [96K] javac: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/921/artifact/out/results-compile-javac-root.txt [540K] blanks: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/921/artifact/out/blanks-eol.txt [13M] https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/921/artifact/out/blanks-tabs.txt [2.0M] checkstyle: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/921/artifact/out/results-checkstyle-root.txt [14M] pathlen: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/921/artifact/out/results-pathlen.txt [16K] pylint: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/921/artifact/out/results-pylint.txt [20K] shellcheck: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/921/artifact/out/results-shellcheck.txt [28K] xml: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/921/artifact/out/xml.txt [24K] javadoc: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/921/artifact/out/results-javadoc-javadoc-root.txt [400K] Powered by Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-18296) Memory fragmentation in ChecksumFileSystem Vectored IO implementation.
[ https://issues.apache.org/jira/browse/HADOOP-18296?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mukund Thakur resolved HADOOP-18296. Resolution: Won't Fix > Memory fragmentation in ChecksumFileSystem Vectored IO implementation. > -- > > Key: HADOOP-18296 > URL: https://issues.apache.org/jira/browse/HADOOP-18296 > Project: Hadoop Common > Issue Type: Sub-task > Components: common >Affects Versions: 3.4.0 >Reporter: Mukund Thakur >Priority: Minor > Labels: fs > > As we have implemented merging of ranges in the ChecksumFSInputChecker > implementation of vectored IO api, it can lead to memory fragmentation. Let > me explain by example. > > Suppose client requests for 3 ranges. > 0-500, 700-1000 and 1200-1500. > Now because of merging, all the above ranges will get merged into one and we > will allocate a big byte buffer of 0-1500 size but return sliced byte buffers > for the desired ranges. > Now once the client is done reading all the ranges, it will only be able to > free the memory for requested ranges and memory of the gaps will never be > released for eg here (500-700 and 1000-1200). > > Note this only happens for direct byte buffers. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
Re: [VOTE] Release Apache Hadoop 3.2.4 - RC0
+1 from myself. * skimmed the contents of site documentation. * built the source tarball on Rocky Linux 8 (x86_64) by OpenJDK 8 with `-Pnative`. * launched pseudo distributed cluster including kms and httpfs with Kerberos and SSL enabled. * created encryption zone, put and read files via httpfs. * ran example MR wordcount over encryption zone. * launched 3-node docker cluster with NN-HA and RM-HA enabled and ran some example MR jobs. * built HBase 2.4.11, Hive 3.1.2 and Spark 3.1.2 against Hadoop 3.2.4 RC0 on CentOS 7 (x86_64) by using Bigtop branch-3.1 and ran smoke-tests. https://github.com/apache/bigtop/pull/942 * Hive needs updating exclusion rule to address HADOOP-18088 (migration to reload4j). * built Spark 3.3.0 against Hadoop 3.2.4 RC0 using the staging repository:: staged staged-releases https://repository.apache.org/content/repositories/orgapachehadoop-1354 true true Thanks, Masatake Iwasaki On 2022/07/13 1:14, Masatake Iwasaki wrote: Hi all, Here's Hadoop 3.2.4 release candidate #0: The RC is available at: https://home.apache.org/~iwasakims/hadoop-3.2.4-RC0/ The RC tag is at: https://github.com/apache/hadoop/releases/tag/release-3.2.4-RC0 The Maven artifacts are staged at: https://repository.apache.org/content/repositories/orgapachehadoop-1354 You can find my public key at: https://downloads.apache.org/hadoop/common/KEYS Please evaluate the RC and vote. The vote will be open for (at least) 5 days. Thanks, Masatake Iwasaki - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-18074) Partial/Incomplete groups list can be returned in LDAP groups lookup
[
https://issues.apache.org/jira/browse/HADOOP-18074?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran resolved HADOOP-18074.
-
Fix Version/s: 3.4.0
3.3.5
Resolution: Fixed
> Partial/Incomplete groups list can be returned in LDAP groups lookup
>
>
> Key: HADOOP-18074
> URL: https://issues.apache.org/jira/browse/HADOOP-18074
> Project: Hadoop Common
> Issue Type: Bug
> Components: security
>Reporter: Philippe Lanoe
>Assignee: Larry McCay
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0, 3.3.5
>
> Time Spent: 3h 20m
> Remaining Estimate: 0h
>
> Hello,
> The
> {code:java}
> Set doGetGroups(String user, int goUpHierarchy) {code}
> method in
> [https://github.com/apache/hadoop/blob/b27732c69b114f24358992a5a4d170bc94e2ceaf/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/LdapGroupsMapping.java#L476]
> Looks like having an issue if in the middle of the loop a *NamingException*
> is caught:
> The groups variable is not reset in the catch clause and therefore the
> fallback lookup cannot be executed (when goUpHierarchy==0 at least):
> ||
> {code:java}
> if (groups.isEmpty() || goUpHierarchy > 0) {
> groups = lookupGroup(result, c, goUpHierarchy);
> }
> {code}
>
> Consequence is that only a partial list of groups is returned, which is not
> correct.
> Following options could be used as solution:
> * Reset the group to an empty list in the catch clause, to trigger the
> fallback query.
> * Add an option flag to enable ignoring groups with Naming Exception (since
> they are not groups most probably)
> Independently, would any issue also occur (and therefore full list cannot be
> returned) in the first lookup as well as in the fallback query, the method
> should/could(with option flag) throw an Exception, because in some scenario
> accuracy is important.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
Apache Hadoop qbt Report: branch-2.10+JDK7 on Linux/x86_64
For more details, see https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/722/ No changes -1 overall The following subsystems voted -1: asflicense hadolint mvnsite pathlen unit The following subsystems voted -1 but were configured to be filtered/ignored: cc checkstyle javac javadoc pylint shellcheck whitespace The following subsystems are considered long running: (runtime bigger than 1h 0m 0s) unit Specific tests: Failed junit tests : hadoop.fs.TestFileUtil hadoop.hdfs.qjournal.server.TestJournalNodeRespectsBindHostKeys hadoop.hdfs.server.blockmanagement.TestReplicationPolicyWithUpgradeDomain hadoop.contrib.bkjournal.TestBookKeeperHACheckpoints hadoop.contrib.bkjournal.TestBookKeeperHACheckpoints hadoop.hdfs.server.federation.resolver.order.TestLocalResolver hadoop.hdfs.server.federation.router.TestRouterQuota hadoop.hdfs.server.federation.router.TestRouterNamenodeHeartbeat hadoop.hdfs.server.federation.resolver.TestMultipleDestinationResolver hadoop.yarn.server.resourcemanager.scheduler.fair.TestContinuousScheduling hadoop.yarn.server.resourcemanager.monitor.invariants.TestMetricsInvariantChecker hadoop.yarn.server.resourcemanager.TestClientRMService hadoop.mapreduce.lib.input.TestLineRecordReader hadoop.mapreduce.jobhistory.TestHistoryViewerPrinter hadoop.mapred.TestLineRecordReader hadoop.yarn.sls.TestSLSRunner hadoop.resourceestimator.service.TestResourceEstimatorService hadoop.resourceestimator.solver.impl.TestLpSolver cc: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/722/artifact/out/diff-compile-cc-root.txt [4.0K] javac: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/722/artifact/out/diff-compile-javac-root.txt [488K] checkstyle: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/722/artifact/out/diff-checkstyle-root.txt [14M] hadolint: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/722/artifact/out/diff-patch-hadolint.txt [4.0K] mvnsite: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/722/artifact/out/patch-mvnsite-root.txt [568K] pathlen: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/722/artifact/out/pathlen.txt [12K] pylint: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/722/artifact/out/diff-patch-pylint.txt [20K] shellcheck: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/722/artifact/out/diff-patch-shellcheck.txt [72K] whitespace: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/722/artifact/out/whitespace-eol.txt [12M] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/722/artifact/out/whitespace-tabs.txt [1.3M] javadoc: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/722/artifact/out/patch-javadoc-root.txt [40K] unit: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/722/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt [220K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/722/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt [428K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/722/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs_src_contrib_bkjournal.txt [16K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/722/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt [36K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/722/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common.txt [20K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/722/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt [132K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/722/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt [104K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/722/artifact/out/patch-unit-hadoop-tools_hadoop-azure.txt [20K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/722/artifact/out/patch-unit-hadoop-tools_hadoop-sls.txt [28K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/722/artifact/out/patch-unit-hadoop-tools_hadoop-resourceestimator.txt [16K] asflicense:
