Re: [VOTE] Release Apache Hadoop 3.3.2 - RC2
I'll find time to check out the RC bits.
I just feel bad that the tarball is now more than 600MB in size.
On Fri, Jan 21, 2022 at 2:23 AM Steve Loughran
wrote:
> *+1 binding.*
>
> reviewed binaries, source, artifacts in the staging maven repository in
> downstream builds. all good.
>
> *## test run*
>
> checked out the asf github repo at commit 6da346a358c into a location
> already set up with aws and azure test credentials
>
> ran the hadoop-aws tests with -Dparallel-tests -DtestsThreadCount=6
> -Dmarkers=delete -Dscale
> and hadoop-azure against azure cardiff with -Dparallel-tests=abfs
> -DtestsThreadCount=6
>
> all happy
>
>
>
> *## binary*
> downloaded KEYS and imported, so adding your key to my list (also signed
> this and updated the key servers)
>
> downloaded rc tar and verified
> ```
> > gpg2 --verify hadoop-3.3.2.tar.gz.asc hadoop-3.3.2.tar.gz
> gpg: Signature made Sat Jan 15 23:41:10 2022 GMT
> gpg:using RSA key DE7FA241EB298D027C97B2A1D8F1A97BE51ECA98
> gpg: Good signature from "Chao Sun (CODE SIGNING KEY) >"
> [full]
>
>
> > cat hadoop-3.3.2.tar.gz.sha512
> SHA512 (hadoop-3.3.2.tar.gz) =
>
> cdd3d9298ba7d6e63ed63f93c159729ea14d2b7d5e3a0640b1761c86c7714a721f88bdfa8cb1d8d3da316f616e4f0ceaace4f32845ee4441e6aaa7a12b8c647d
>
> > shasum -a 512 hadoop-3.3.2.tar.gz
>
> cdd3d9298ba7d6e63ed63f93c159729ea14d2b7d5e3a0640b1761c86c7714a721f88bdfa8cb1d8d3da316f616e4f0ceaace4f32845ee4441e6aaa7a12b8c647d
> hadoop-3.3.2.tar.gz
> ```
>
>
> *# cloudstore against staged artifacts*
> ```
> cd ~/.m2/repository/org/apache/hadoop
> find . -name \*3.3.2\* -print | xargs rm -r
> ```
> ensures no local builds have tainted the repo.
>
> in cloudstore mvn build without tests
> ```
> mci -Pextra -Phadoop-3.3.2 -Psnapshots-and-staging
> ```
> this fetches all from asf staging
>
> ```
> Downloading from ASF Staging:
>
> https://repository.apache.org/content/groups/staging/org/apache/hadoop/hadoop-client/3.3.2/hadoop-client-3.3.2.pom
> Downloaded from ASF Staging:
>
> https://repository.apache.org/content/groups/staging/org/apache/hadoop/hadoop-client/3.3.2/hadoop-client-3.3.2.pom
> (11 kB at 20 kB/s)
> ```
> there's no tests there, but it did audit the download process. FWIW, that
> project has switched to logback, so I now have all hadoop imports excluding
> slf4j and log4j. it takes too much effort right now.
>
> build works.
>
> tested abfs and s3a storediags, all happy
>
>
>
>
> *### google GCS against staged artifacts*
>
> gcs is now java 11 only, so I had to switch JVMs here.
>
> had to add a snapshots and staging profile, after which I could build and
> test.
>
> ```
> -Dhadoop.three.version=3.3.2 -Psnapshots-and-staging
> ```
> two test failures were related to auth failures where the tests were trying
> to raise exceptions but things failed differently
> ```
> [ERROR] Failures:
> [ERROR]
>
> GoogleHadoopFileSystemTest.eagerInitialization_fails_withInvalidCredentialsConfiguration:122
> unexpected exception type thrown; expected:
> but was:
> [ERROR]
>
> GoogleHadoopFileSystemTest.lazyInitialization_deleteCall_fails_withInvalidCredentialsConfiguration:100
> value of: throwable.getMessage()
> expected: Failed to create GCS FS
> but was : A JSON key file may not be specified at the same time as
> credentials via configuration.
>
> ```
>
> I'm not worried here.
>
> ran cloudstore's diagnostics against gcs.
>
> Nice to see they are now collecting IOStatistics on their input streams. we
> really need to get this collected through the parquet/orc libs and then
> through the query engines.
>
> ```
> > bin/hadoop jar $CLOUDSTORE storediag gs://stevel-london/
>
> ...
> 2022-01-20 17:52:47,447 [main] INFO diag.StoreDiag
> (StoreDurationInfo.java:(56)) - Starting: Reading a file
> gs://stevel-london/dir-9cbfc774-76ff-49c0-b216-d7800369c3e1/file
> input stream summary: org.apache.hadoop.fs.FSDataInputStream@6cfd9a54:
> com.google.cloud.hadoop.fs.gcs.GoogleHadoopFSInputStream@78c1372d
> {counters=((stream_read_close_operations=1)
> (stream_read_seek_backward_operations=0) (stream_read_total_bytes=7)
> (stream_read_bytes=7) (stream_read_exceptions=0)
> (stream_read_seek_operations=0) (stream_read_seek_bytes_skipped=0)
> (stream_read_operations=3) (stream_read_bytes_backwards_on_seek=0)
> (stream_read_seek_forward_operations=0)
> (stream_read_operations_incomplete=1));
> gauges=();
> minimums=();
> maximums=();
> means=();
> }
> ...
> ```
>
> *### source*
>
> once I'd done builds and tests which fetched from staging, I did a local
> build and test
>
> repeated download/validate of source tarball, unzip/untar
>
> build with java11.
>
> I've not done the test run there, because that directory tree doesn't have
> the credentials, and this mornings run was good.
>
> altogether then: very happy. tests good, downstream libraries building and
> linking.
>
> On Wed, 19 Jan 2022 at 17:50, Chao Sun wrote:
>
> > Hi all,
> >
> > I've put together Hadoop 3.3.2 RC2 below:
> >
> > The RC is available at:
> >
[jira] [Created] (HADOOP-18088) Replace log4j 1.x with reload4j
Wei-Chiu Chuang created HADOOP-18088: Summary: Replace log4j 1.x with reload4j Key: HADOOP-18088 URL: https://issues.apache.org/jira/browse/HADOOP-18088 Project: Hadoop Common Issue Type: Improvement Reporter: Wei-Chiu Chuang As proposed in the dev mailing list (https://lists.apache.org/thread/fdzkv80mzkf3w74z9120l0k0rc3v7kqk) let's replace log4j 1 with reload4j in the maintenance releases (i.e. 3.3.x, 3.2.x and 2.10.x) -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
Re: [DISCUSS] Migrate hadoop from log4j1 to log4j2
+1 I think it makes sense to use reload4j in maint releases. I have a draft PR doing this (https://github.com/apache/hadoop/pull/3906) log4j2 in Hadoop 3.4.0 makes sense to me. There could be incompatibilities introduced by log4j2, but I feel we should at least make it 3.4.0 a "preview" release, and try to address the incompat in later versions (e.g. 3.4.1) On Fri, Jan 21, 2022 at 8:42 AM Duo Zhang wrote: > For maintenance release line I also support we switch to reload4j to > address the security issues first. We could file an issue for it. > > Andrew Purtell 于2022年1月21日 周五01:15写道： > > > Just to clarify: I think you want to upgrade to Log4J2 (or switch to > > LogBack) as a strategy for new releases, but you have the option in > > maintenance releases to use Reload4J to maintain Appender API and > > operational compatibility, and users who want to minimize risks in > > production while mitigating the security issues will prefer that. > > > > > On Jan 20, 2022, at 8:59 AM, Andrew Purtell > > wrote: > > > > > > Reload4J has fixed all of those CVEs without requiring an upgrade. > > > > > >> On Jan 20, 2022, at 5:56 AM, Duo Zhang wrote: > > >> > > >> There are 3 new CVEs for log4j1 reported recently[1][2][3]. So I > think > > it > > >> is time to speed up the migration to log4j2 work[4] now. > > >> > > >> You can see the discussion on the jira issue[4], our goal is to fully > > >> migrate to log4j2 and the current most blocking issue is lack of the > > >> "log4j.rootLogger=INFO,Console" grammer support for log4j2. I've > already > > >> started a discussion thread on the log4j dev mailing list[5] and the > > result > > >> is optimistic and I've filed an issue for log4j2[6], but I do not > think > > it > > >> could be addressed and released soon. If we want to fully migrate to > > >> log4j2, then either we introduce new environment variables or split > the > > old > > >> HADOOP_ROOT_LOGGER variable in the startup scripts. And considering > the > > >> complexity of our current startup scripts, the work is not easy and it > > will > > >> also break lots of other hadoop deployment systems if they do not use > > our > > >> startup scripts... > > >> > > >> So after reconsidering the current situation, I prefer we use the > > log4j1.2 > > >> bridge to remove the log4j1 dependency first, and once LOG4J2-3341 is > > >> addressed and released, we start to fully migrate to log4j2. Of course > > we > > >> have other problems for log4j1.2 bridge too, as we have > TaskLogAppender, > > >> ContainerLogAppender and ContainerRollingLogAppender which inherit > > >> FileAppender and RollingFileAppender in log4j1, which are not part of > > the > > >> log4j1.2 bridge. But anyway, at least we could just copy the source > > code to > > >> hadoop as we have WriteAppender in log4j1.2 bridge, and these two > > classes > > >> do not have related CVEs. > > >> > > >> Thoughts? For me I would like us to make a new 3.4.x release line to > > remove > > >> the log4j1 dependencies ASAP. > > >> > > >> Thanks. > > >> > > >> 1. https://nvd.nist.gov/vuln/detail/CVE-2022-23302 > > >> 2. https://nvd.nist.gov/vuln/detail/CVE-2022-23305 > > >> 3. https://nvd.nist.gov/vuln/detail/CVE-2022-23307 > > >> 4. https://issues.apache.org/jira/browse/HADOOP-16206 > > >> 5. https://lists.apache.org/thread/gvfb3jkg6t11cyds4jmpo7lrswmx28w3 > > >> 6. https://issues.apache.org/jira/browse/LOG4J2-3341 > > >
[jira] [Created] (HADOOP-18087) fix bug in handling CNAME chain records
YUBI LEE created HADOOP-18087:
-
Summary: fix bug in handling CNAME chain records
Key: HADOOP-18087
URL: https://issues.apache.org/jira/browse/HADOOP-18087
Project: Hadoop Common
Issue Type: Bug
Components: registry
Affects Versions: 3.1.2
Reporter: YUBI LEE
When query A record which is chained by CNAME, YARN Registry DNS Server does
not properly respond. Some CNAME records are missing.
For example, "repo.maven.apache.org" is chaned as follows:
repo.maven.apache.org. 21317 IN CNAME repo.apache.maven.org.
repo.apache.maven.org. 20114 IN CNAME maven.map.fastly.net.
maven.map.fastly.net. 7 IN A 199.232.192.215
maven.map.fastly.net. 7 IN A 199.232.196.215
If ask A record for "repo.maven.apache.org" using "dig" or "nslookup", YARN
Registry DNS Server will give answers similar to this:
(10.1.2.3, 10.8.8.8 IP is virtual)
{code}
$ nslookup repo.maven.apache.org 10.1.2.3
Server: 10.1.2.3
Address:10.1.2.3#53
Non-authoritative answer:
repo.maven.apache.org canonical name = repo.apache.maven.org.
Name: maven.map.fastly.net
Address: 151.101.196.215
** server can't find repo.apache.maven.org: NXDOMAIN
{code}
The reason why you can see "NXDOMAIN", "nslookup" will query "A" & ""
records.
If there is no answer from other dns server, "answers == null" but YARN
Registry DNS Server has a bug. There is no null handling.
{code:java}
// Forward lookup to primary DNS servers
Record[] answers = getRecords(name, type);
try {
for (Record r : answers) {
if (!response.findRecord(r)) {
if (r.getType() == Type.SOA) {
response.addRecord(r, Section.AUTHORITY);
} else {
response.addRecord(r, Section.ANSWER);
}
}
if (r.getType() == Type.CNAME) {
Name cname = r.getName();
if (iterations < 6) {
remoteLookup(response, cname, type, iterations + 1);
}
}
}
} catch (NullPointerException e) {
return Rcode.NXDOMAIN;
} catch (Throwable e) {
return Rcode.SERVFAIL;
}
return Rcode.NOERROR;
{code}
It should be like this:
{code}
nslookup repo.maven.apache.org 10.8.8.8
Server: 10.8.8.8
Address:10.8.8.8#53
Non-authoritative answer:
repo.maven.apache.org canonical name = repo.apache.maven.org.
repo.apache.maven.org canonical name = maven.map.fastly.net.
Name: maven.map.fastly.net
Address: 151.101.196.215
{code}
I will make a pull request at https://github.com/apache/hadoop soon.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
Apache Hadoop qbt Report: branch-3.3+JDK8 on Linux/x86_64
For more details, see https://ci-hadoop.apache.org/job/hadoop-qbt-branch-3.3-java8-linux-x86_64/39/ No changes -1 overall The following subsystems voted -1: blanks pathlen unit xml The following subsystems voted -1 but were configured to be filtered/ignored: cc checkstyle javac javadoc pylint shellcheck The following subsystems are considered long running: (runtime bigger than 1h 0m 0s) unit Specific tests: XML : Parsing Error(s): hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-excerpt.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-missing-tags.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-missing-tags2.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-sample-output.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/fair-scheduler-invalid.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/yarn-site-with-invalid-allocation-file-ref.xml Failed junit tests : hadoop.security.TestGroupsCaching hadoop.hdfs.server.namenode.ha.TestHAAppend hadoop.hdfs.server.datanode.TestDataNodeRollingUpgrade hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacitySchedulerAutoQueueCreation hadoop.yarn.applications.distributedshell.TestDistributedShell hadoop.yarn.csi.client.TestCsiClient cc: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-3.3-java8-linux-x86_64/39/artifact/out/results-compile-cc-root.txt [48K] javac: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-3.3-java8-linux-x86_64/39/artifact/out/results-compile-javac-root.txt [388K] blanks: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-3.3-java8-linux-x86_64/39/artifact/out/blanks-eol.txt [13M] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-3.3-java8-linux-x86_64/39/artifact/out/blanks-tabs.txt [2.0M] checkstyle: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-3.3-java8-linux-x86_64/39/artifact/out/results-checkstyle-root.txt [14M] pathlen: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-3.3-java8-linux-x86_64/39/artifact/out/results-pathlen.txt [16K] pylint: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-3.3-java8-linux-x86_64/39/artifact/out/results-pylint.txt [20K] shellcheck: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-3.3-java8-linux-x86_64/39/artifact/out/results-shellcheck.txt [20K] xml: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-3.3-java8-linux-x86_64/39/artifact/out/xml.txt [24K] javadoc: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-3.3-java8-linux-x86_64/39/artifact/out/results-javadoc-javadoc-root.txt [1.1M] unit: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-3.3-java8-linux-x86_64/39/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt [256K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-3.3-java8-linux-x86_64/39/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt [536K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-3.3-java8-linux-x86_64/39/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt [172K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-3.3-java8-linux-x86_64/39/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-applications_hadoop-yarn-applications-distributedshell.txt [12K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-3.3-java8-linux-x86_64/39/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-csi.txt [16K] Powered by Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
Re: [DISCUSS] Migrate hadoop from log4j1 to log4j2
For maintenance release line I also support we switch to reload4j to address the security issues first. We could file an issue for it. Andrew Purtell 于2022年1月21日 周五01:15写道： > Just to clarify: I think you want to upgrade to Log4J2 (or switch to > LogBack) as a strategy for new releases, but you have the option in > maintenance releases to use Reload4J to maintain Appender API and > operational compatibility, and users who want to minimize risks in > production while mitigating the security issues will prefer that. > > > On Jan 20, 2022, at 8:59 AM, Andrew Purtell > wrote: > > > > Reload4J has fixed all of those CVEs without requiring an upgrade. > > > >> On Jan 20, 2022, at 5:56 AM, Duo Zhang wrote: > >> > >> There are 3 new CVEs for log4j1 reported recently[1][2][3]. So I think > it > >> is time to speed up the migration to log4j2 work[4] now. > >> > >> You can see the discussion on the jira issue[4], our goal is to fully > >> migrate to log4j2 and the current most blocking issue is lack of the > >> "log4j.rootLogger=INFO,Console" grammer support for log4j2. I've already > >> started a discussion thread on the log4j dev mailing list[5] and the > result > >> is optimistic and I've filed an issue for log4j2[6], but I do not think > it > >> could be addressed and released soon. If we want to fully migrate to > >> log4j2, then either we introduce new environment variables or split the > old > >> HADOOP_ROOT_LOGGER variable in the startup scripts. And considering the > >> complexity of our current startup scripts, the work is not easy and it > will > >> also break lots of other hadoop deployment systems if they do not use > our > >> startup scripts... > >> > >> So after reconsidering the current situation, I prefer we use the > log4j1.2 > >> bridge to remove the log4j1 dependency first, and once LOG4J2-3341 is > >> addressed and released, we start to fully migrate to log4j2. Of course > we > >> have other problems for log4j1.2 bridge too, as we have TaskLogAppender, > >> ContainerLogAppender and ContainerRollingLogAppender which inherit > >> FileAppender and RollingFileAppender in log4j1, which are not part of > the > >> log4j1.2 bridge. But anyway, at least we could just copy the source > code to > >> hadoop as we have WriteAppender in log4j1.2 bridge, and these two > classes > >> do not have related CVEs. > >> > >> Thoughts? For me I would like us to make a new 3.4.x release line to > remove > >> the log4j1 dependencies ASAP. > >> > >> Thanks. > >> > >> 1. https://nvd.nist.gov/vuln/detail/CVE-2022-23302 > >> 2. https://nvd.nist.gov/vuln/detail/CVE-2022-23305 > >> 3. https://nvd.nist.gov/vuln/detail/CVE-2022-23307 > >> 4. https://issues.apache.org/jira/browse/HADOOP-16206 > >> 5. https://lists.apache.org/thread/gvfb3jkg6t11cyds4jmpo7lrswmx28w3 > >> 6. https://issues.apache.org/jira/browse/LOG4J2-3341 >
Re: [DISCUSS] Migrate hadoop from log4j1 to log4j2
The EventCounter class has already been removed in HADOOP-17524. And on the filters, by default log4j1.2 bridge has log4j1 filter support, but as said above, maybe it is not fully functional if you have some advanced usage. So mind providing more information about how we use filters in Hadoop? Thanks. Arpit Agarwal 于2022年1月21日 周五00:44写道： > Hi Duo, > > Thank you for starting this discussion. Log4j1.2 bridge seems like a > practical short-term solution. However the bridge will silently affect > applications that add appenders or filters. NameNode audit logger and > metrics come to mind. There may be others. > > Thanks, > Arpit > > > > On Jan 20, 2022, at 5:55 AM, Duo Zhang wrote: > > > > There are 3 new CVEs for log4j1 reported recently[1][2][3]. So I think it > > is time to speed up the migration to log4j2 work[4] now. > > > > You can see the discussion on the jira issue[4], our goal is to fully > > migrate to log4j2 and the current most blocking issue is lack of the > > "log4j.rootLogger=INFO,Console" grammer support for log4j2. I've already > > started a discussion thread on the log4j dev mailing list[5] and the > result > > is optimistic and I've filed an issue for log4j2[6], but I do not think > it > > could be addressed and released soon. If we want to fully migrate to > > log4j2, then either we introduce new environment variables or split the > old > > HADOOP_ROOT_LOGGER variable in the startup scripts. And considering the > > complexity of our current startup scripts, the work is not easy and it > will > > also break lots of other hadoop deployment systems if they do not use our > > startup scripts... > > > > So after reconsidering the current situation, I prefer we use the > log4j1.2 > > bridge to remove the log4j1 dependency first, and once LOG4J2-3341 is > > addressed and released, we start to fully migrate to log4j2. Of course we > > have other problems for log4j1.2 bridge too, as we have TaskLogAppender, > > ContainerLogAppender and ContainerRollingLogAppender which inherit > > FileAppender and RollingFileAppender in log4j1, which are not part of the > > log4j1.2 bridge. But anyway, at least we could just copy the source code > to > > hadoop as we have WriteAppender in log4j1.2 bridge, and these two classes > > do not have related CVEs. > > > > Thoughts? For me I would like us to make a new 3.4.x release line to > remove > > the log4j1 dependencies ASAP. > > > > Thanks. > > > > 1. https://nvd.nist.gov/vuln/detail/CVE-2022-23302 > > 2. https://nvd.nist.gov/vuln/detail/CVE-2022-23305 > > 3. https://nvd.nist.gov/vuln/detail/CVE-2022-23307 > > 4. https://issues.apache.org/jira/browse/HADOOP-16206 > > 5. https://lists.apache.org/thread/gvfb3jkg6t11cyds4jmpo7lrswmx28w3 > > 6. https://issues.apache.org/jira/browse/LOG4J2-3341 > > > - > To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org > For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org > >
Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86_64
For more details, see https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/756/ [Jan 19, 2022 4:42:33 AM] (noreply) HDFS-16426. Fix nextBlockReportTime when trigger full block report force (#3887) [Jan 19, 2022 6:10:39 AM] (noreply) HDFS-16399. Reconfig cache report parameters for datanode (#3841) [Jan 19, 2022 8:59:42 AM] (noreply) HDFS-16423. Balancer should not get blocks on stale storages (#3883) [Jan 19, 2022 10:13:13 AM] (noreply) HADOOP-18084. ABFS: Add testfilePath while verifying test contents are read correctly (#3903) -1 overall The following subsystems voted -1: blanks pathlen unit xml The following subsystems voted -1 but were configured to be filtered/ignored: cc checkstyle javac javadoc pylint shellcheck The following subsystems are considered long running: (runtime bigger than 1h 0m 0s) unit Specific tests: XML : Parsing Error(s): hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-excerpt.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-missing-tags.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-missing-tags2.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-sample-output.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/fair-scheduler-invalid.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/yarn-site-with-invalid-allocation-file-ref.xml Failed junit tests : hadoop.ipc.TestIPC hadoop.hdfs.server.datanode.TestDataNodeHotSwapVolumes hadoop.hdfs.rbfbalance.TestRouterDistCpProcedure hadoop.yarn.csi.client.TestCsiClient cc: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/756/artifact/out/results-compile-cc-root.txt [96K] javac: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/756/artifact/out/results-compile-javac-root.txt [336K] blanks: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/756/artifact/out/blanks-eol.txt [13M] https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/756/artifact/out/blanks-tabs.txt [2.0M] checkstyle: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/756/artifact/out/results-checkstyle-root.txt [14M] pathlen: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/756/artifact/out/results-pathlen.txt [16K] pylint: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/756/artifact/out/results-pylint.txt [20K] shellcheck: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/756/artifact/out/results-shellcheck.txt [28K] xml: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/756/artifact/out/xml.txt [24K] javadoc: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/756/artifact/out/results-javadoc-javadoc-root.txt [404K] unit: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/756/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt [216K] https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/756/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt [520K] https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/756/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt [108K] https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/756/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-csi.txt [20K] Powered by Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
Re: [VOTE] Release Apache Hadoop 3.3.2 - RC2
*+1 binding.*
reviewed binaries, source, artifacts in the staging maven repository in
downstream builds. all good.
*## test run*
checked out the asf github repo at commit 6da346a358c into a location
already set up with aws and azure test credentials
ran the hadoop-aws tests with -Dparallel-tests -DtestsThreadCount=6
-Dmarkers=delete -Dscale
and hadoop-azure against azure cardiff with -Dparallel-tests=abfs
-DtestsThreadCount=6
all happy
*## binary*
downloaded KEYS and imported, so adding your key to my list (also signed
this and updated the key servers)
downloaded rc tar and verified
```
> gpg2 --verify hadoop-3.3.2.tar.gz.asc hadoop-3.3.2.tar.gz
gpg: Signature made Sat Jan 15 23:41:10 2022 GMT
gpg:using RSA key DE7FA241EB298D027C97B2A1D8F1A97BE51ECA98
gpg: Good signature from "Chao Sun (CODE SIGNING KEY) "
[full]
> cat hadoop-3.3.2.tar.gz.sha512
SHA512 (hadoop-3.3.2.tar.gz) =
cdd3d9298ba7d6e63ed63f93c159729ea14d2b7d5e3a0640b1761c86c7714a721f88bdfa8cb1d8d3da316f616e4f0ceaace4f32845ee4441e6aaa7a12b8c647d
> shasum -a 512 hadoop-3.3.2.tar.gz
cdd3d9298ba7d6e63ed63f93c159729ea14d2b7d5e3a0640b1761c86c7714a721f88bdfa8cb1d8d3da316f616e4f0ceaace4f32845ee4441e6aaa7a12b8c647d
hadoop-3.3.2.tar.gz
```
*# cloudstore against staged artifacts*
```
cd ~/.m2/repository/org/apache/hadoop
find . -name \*3.3.2\* -print | xargs rm -r
```
ensures no local builds have tainted the repo.
in cloudstore mvn build without tests
```
mci -Pextra -Phadoop-3.3.2 -Psnapshots-and-staging
```
this fetches all from asf staging
```
Downloading from ASF Staging:
https://repository.apache.org/content/groups/staging/org/apache/hadoop/hadoop-client/3.3.2/hadoop-client-3.3.2.pom
Downloaded from ASF Staging:
https://repository.apache.org/content/groups/staging/org/apache/hadoop/hadoop-client/3.3.2/hadoop-client-3.3.2.pom
(11 kB at 20 kB/s)
```
there's no tests there, but it did audit the download process. FWIW, that
project has switched to logback, so I now have all hadoop imports excluding
slf4j and log4j. it takes too much effort right now.
build works.
tested abfs and s3a storediags, all happy
*### google GCS against staged artifacts*
gcs is now java 11 only, so I had to switch JVMs here.
had to add a snapshots and staging profile, after which I could build and
test.
```
-Dhadoop.three.version=3.3.2 -Psnapshots-and-staging
```
two test failures were related to auth failures where the tests were trying
to raise exceptions but things failed differently
```
[ERROR] Failures:
[ERROR]
GoogleHadoopFileSystemTest.eagerInitialization_fails_withInvalidCredentialsConfiguration:122
unexpected exception type thrown; expected:
but was:
[ERROR]
GoogleHadoopFileSystemTest.lazyInitialization_deleteCall_fails_withInvalidCredentialsConfiguration:100
value of: throwable.getMessage()
expected: Failed to create GCS FS
but was : A JSON key file may not be specified at the same time as
credentials via configuration.
```
I'm not worried here.
ran cloudstore's diagnostics against gcs.
Nice to see they are now collecting IOStatistics on their input streams. we
really need to get this collected through the parquet/orc libs and then
through the query engines.
```
> bin/hadoop jar $CLOUDSTORE storediag gs://stevel-london/
...
2022-01-20 17:52:47,447 [main] INFO diag.StoreDiag
(StoreDurationInfo.java:(56)) - Starting: Reading a file
gs://stevel-london/dir-9cbfc774-76ff-49c0-b216-d7800369c3e1/file
input stream summary: org.apache.hadoop.fs.FSDataInputStream@6cfd9a54:
com.google.cloud.hadoop.fs.gcs.GoogleHadoopFSInputStream@78c1372d{counters=((stream_read_close_operations=1)
(stream_read_seek_backward_operations=0) (stream_read_total_bytes=7)
(stream_read_bytes=7) (stream_read_exceptions=0)
(stream_read_seek_operations=0) (stream_read_seek_bytes_skipped=0)
(stream_read_operations=3) (stream_read_bytes_backwards_on_seek=0)
(stream_read_seek_forward_operations=0)
(stream_read_operations_incomplete=1));
gauges=();
minimums=();
maximums=();
means=();
}
...
```
*### source*
once I'd done builds and tests which fetched from staging, I did a local
build and test
repeated download/validate of source tarball, unzip/untar
build with java11.
I've not done the test run there, because that directory tree doesn't have
the credentials, and this mornings run was good.
altogether then: very happy. tests good, downstream libraries building and
linking.
On Wed, 19 Jan 2022 at 17:50, Chao Sun wrote:
> Hi all,
>
> I've put together Hadoop 3.3.2 RC2 below:
>
> The RC is available at:
> http://people.apache.org/~sunchao/hadoop-3.3.2-RC2/
> The RC tag is at:
> https://github.com/apache/hadoop/releases/tag/release-3.3.2-RC2
> The Maven artifacts are staged at:
> https://repository.apache.org/content/repositories/orgapachehadoop-1332
>
> You can find my public key at:
> https://downloads.apache.org/hadoop/common/KEYS
>
> I've done the following tests and they look good:
> - Ran all the unit tests
> - Started a single node HDFS cluster and
Re: [DISCUSS] Migrate hadoop from log4j1 to log4j2
On Thu, 20 Jan 2022 at 17:15, Andrew Purtell wrote: > Just to clarify: I think you want to upgrade to Log4J2 (or switch to > LogBack) as a strategy for new releases, but you have the option in > maintenance releases to use Reload4J to maintain Appender API and > operational compatibility, and users who want to minimize risks in > production while mitigating the security issues will prefer that. i like this > > > > On Jan 20, 2022, at 8:59 AM, Andrew Purtell > wrote: > > > > Reload4J has fixed all of those CVEs without requiring an upgrade. > > > >> On Jan 20, 2022, at 5:56 AM, Duo Zhang wrote: > >> > >> There are 3 new CVEs for log4j1 reported recently[1][2][3]. So I think > it > >> is time to speed up the migration to log4j2 work[4] now. > >> > >> You can see the discussion on the jira issue[4], our goal is to fully > >> migrate to log4j2 and the current most blocking issue is lack of the > >> "log4j.rootLogger=INFO,Console" grammer support for log4j2. I've already > >> started a discussion thread on the log4j dev mailing list[5] and the > result > >> is optimistic and I've filed an issue for log4j2[6], but I do not think > it > >> could be addressed and released soon. If we want to fully migrate to > >> log4j2, then either we introduce new environment variables or split the > old > >> HADOOP_ROOT_LOGGER variable in the startup scripts. And considering the > >> complexity of our current startup scripts, the work is not easy and it > will > >> also break lots of other hadoop deployment systems if they do not use > our > >> startup scripts... > >> > >> So after reconsidering the current situation, I prefer we use the > log4j1.2 > >> bridge to remove the log4j1 dependency first, and once LOG4J2-3341 is > >> addressed and released, we start to fully migrate to log4j2. Of course > we > >> have other problems for log4j1.2 bridge too, as we have TaskLogAppender, > >> ContainerLogAppender and ContainerRollingLogAppender which inherit > >> FileAppender and RollingFileAppender in log4j1, which are not part of > the > >> log4j1.2 bridge. But anyway, at least we could just copy the source > code to > >> hadoop as we have WriteAppender in log4j1.2 bridge, and these two > classes > >> do not have related CVEs. > >> > >> Thoughts? For me I would like us to make a new 3.4.x release line to > remove > >> the log4j1 dependencies ASAP. > >> > >> Thanks. > >> > >> 1. https://nvd.nist.gov/vuln/detail/CVE-2022-23302 > >> 2. https://nvd.nist.gov/vuln/detail/CVE-2022-23305 > >> 3. https://nvd.nist.gov/vuln/detail/CVE-2022-23307 > >> 4. https://issues.apache.org/jira/browse/HADOOP-16206 > >> 5. https://lists.apache.org/thread/gvfb3jkg6t11cyds4jmpo7lrswmx28w3 > >> 6. https://issues.apache.org/jira/browse/LOG4J2-3341 > > - > To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org > For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org > >
Re: [DISCUSS] Migrate hadoop from log4j1 to log4j2
Just to clarify: I think you want to upgrade to Log4J2 (or switch to LogBack) as a strategy for new releases, but you have the option in maintenance releases to use Reload4J to maintain Appender API and operational compatibility, and users who want to minimize risks in production while mitigating the security issues will prefer that. > On Jan 20, 2022, at 8:59 AM, Andrew Purtell wrote: > > Reload4J has fixed all of those CVEs without requiring an upgrade. > >> On Jan 20, 2022, at 5:56 AM, Duo Zhang wrote: >> >> There are 3 new CVEs for log4j1 reported recently[1][2][3]. So I think it >> is time to speed up the migration to log4j2 work[4] now. >> >> You can see the discussion on the jira issue[4], our goal is to fully >> migrate to log4j2 and the current most blocking issue is lack of the >> "log4j.rootLogger=INFO,Console" grammer support for log4j2. I've already >> started a discussion thread on the log4j dev mailing list[5] and the result >> is optimistic and I've filed an issue for log4j2[6], but I do not think it >> could be addressed and released soon. If we want to fully migrate to >> log4j2, then either we introduce new environment variables or split the old >> HADOOP_ROOT_LOGGER variable in the startup scripts. And considering the >> complexity of our current startup scripts, the work is not easy and it will >> also break lots of other hadoop deployment systems if they do not use our >> startup scripts... >> >> So after reconsidering the current situation, I prefer we use the log4j1.2 >> bridge to remove the log4j1 dependency first, and once LOG4J2-3341 is >> addressed and released, we start to fully migrate to log4j2. Of course we >> have other problems for log4j1.2 bridge too, as we have TaskLogAppender, >> ContainerLogAppender and ContainerRollingLogAppender which inherit >> FileAppender and RollingFileAppender in log4j1, which are not part of the >> log4j1.2 bridge. But anyway, at least we could just copy the source code to >> hadoop as we have WriteAppender in log4j1.2 bridge, and these two classes >> do not have related CVEs. >> >> Thoughts? For me I would like us to make a new 3.4.x release line to remove >> the log4j1 dependencies ASAP. >> >> Thanks. >> >> 1. https://nvd.nist.gov/vuln/detail/CVE-2022-23302 >> 2. https://nvd.nist.gov/vuln/detail/CVE-2022-23305 >> 3. https://nvd.nist.gov/vuln/detail/CVE-2022-23307 >> 4. https://issues.apache.org/jira/browse/HADOOP-16206 >> 5. https://lists.apache.org/thread/gvfb3jkg6t11cyds4jmpo7lrswmx28w3 >> 6. https://issues.apache.org/jira/browse/LOG4J2-3341 - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
Re: [DISCUSS] Migrate hadoop from log4j1 to log4j2
Reload4J has fixed all of those CVEs without requiring an upgrade. > On Jan 20, 2022, at 5:56 AM, Duo Zhang wrote: > > There are 3 new CVEs for log4j1 reported recently[1][2][3]. So I think it > is time to speed up the migration to log4j2 work[4] now. > > You can see the discussion on the jira issue[4], our goal is to fully > migrate to log4j2 and the current most blocking issue is lack of the > "log4j.rootLogger=INFO,Console" grammer support for log4j2. I've already > started a discussion thread on the log4j dev mailing list[5] and the result > is optimistic and I've filed an issue for log4j2[6], but I do not think it > could be addressed and released soon. If we want to fully migrate to > log4j2, then either we introduce new environment variables or split the old > HADOOP_ROOT_LOGGER variable in the startup scripts. And considering the > complexity of our current startup scripts, the work is not easy and it will > also break lots of other hadoop deployment systems if they do not use our > startup scripts... > > So after reconsidering the current situation, I prefer we use the log4j1.2 > bridge to remove the log4j1 dependency first, and once LOG4J2-3341 is > addressed and released, we start to fully migrate to log4j2. Of course we > have other problems for log4j1.2 bridge too, as we have TaskLogAppender, > ContainerLogAppender and ContainerRollingLogAppender which inherit > FileAppender and RollingFileAppender in log4j1, which are not part of the > log4j1.2 bridge. But anyway, at least we could just copy the source code to > hadoop as we have WriteAppender in log4j1.2 bridge, and these two classes > do not have related CVEs. > > Thoughts? For me I would like us to make a new 3.4.x release line to remove > the log4j1 dependencies ASAP. > > Thanks. > > 1. https://nvd.nist.gov/vuln/detail/CVE-2022-23302 > 2. https://nvd.nist.gov/vuln/detail/CVE-2022-23305 > 3. https://nvd.nist.gov/vuln/detail/CVE-2022-23307 > 4. https://issues.apache.org/jira/browse/HADOOP-16206 > 5. https://lists.apache.org/thread/gvfb3jkg6t11cyds4jmpo7lrswmx28w3 > 6. https://issues.apache.org/jira/browse/LOG4J2-3341 - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
Re: [DISCUSS] Migrate hadoop from log4j1 to log4j2
Hi Duo, Thank you for starting this discussion. Log4j1.2 bridge seems like a practical short-term solution. However the bridge will silently affect applications that add appenders or filters. NameNode audit logger and metrics come to mind. There may be others. Thanks, Arpit > On Jan 20, 2022, at 5:55 AM, Duo Zhang wrote: > > There are 3 new CVEs for log4j1 reported recently[1][2][3]. So I think it > is time to speed up the migration to log4j2 work[4] now. > > You can see the discussion on the jira issue[4], our goal is to fully > migrate to log4j2 and the current most blocking issue is lack of the > "log4j.rootLogger=INFO,Console" grammer support for log4j2. I've already > started a discussion thread on the log4j dev mailing list[5] and the result > is optimistic and I've filed an issue for log4j2[6], but I do not think it > could be addressed and released soon. If we want to fully migrate to > log4j2, then either we introduce new environment variables or split the old > HADOOP_ROOT_LOGGER variable in the startup scripts. And considering the > complexity of our current startup scripts, the work is not easy and it will > also break lots of other hadoop deployment systems if they do not use our > startup scripts... > > So after reconsidering the current situation, I prefer we use the log4j1.2 > bridge to remove the log4j1 dependency first, and once LOG4J2-3341 is > addressed and released, we start to fully migrate to log4j2. Of course we > have other problems for log4j1.2 bridge too, as we have TaskLogAppender, > ContainerLogAppender and ContainerRollingLogAppender which inherit > FileAppender and RollingFileAppender in log4j1, which are not part of the > log4j1.2 bridge. But anyway, at least we could just copy the source code to > hadoop as we have WriteAppender in log4j1.2 bridge, and these two classes > do not have related CVEs. > > Thoughts? For me I would like us to make a new 3.4.x release line to remove > the log4j1 dependencies ASAP. > > Thanks. > > 1. https://nvd.nist.gov/vuln/detail/CVE-2022-23302 > 2. https://nvd.nist.gov/vuln/detail/CVE-2022-23305 > 3. https://nvd.nist.gov/vuln/detail/CVE-2022-23307 > 4. https://issues.apache.org/jira/browse/HADOOP-16206 > 5. https://lists.apache.org/thread/gvfb3jkg6t11cyds4jmpo7lrswmx28w3 > 6. https://issues.apache.org/jira/browse/LOG4J2-3341 - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[DISCUSS] Migrate hadoop from log4j1 to log4j2
There are 3 new CVEs for log4j1 reported recently[1][2][3]. So I think it is time to speed up the migration to log4j2 work[4] now. You can see the discussion on the jira issue[4], our goal is to fully migrate to log4j2 and the current most blocking issue is lack of the "log4j.rootLogger=INFO,Console" grammer support for log4j2. I've already started a discussion thread on the log4j dev mailing list[5] and the result is optimistic and I've filed an issue for log4j2[6], but I do not think it could be addressed and released soon. If we want to fully migrate to log4j2, then either we introduce new environment variables or split the old HADOOP_ROOT_LOGGER variable in the startup scripts. And considering the complexity of our current startup scripts, the work is not easy and it will also break lots of other hadoop deployment systems if they do not use our startup scripts... So after reconsidering the current situation, I prefer we use the log4j1.2 bridge to remove the log4j1 dependency first, and once LOG4J2-3341 is addressed and released, we start to fully migrate to log4j2. Of course we have other problems for log4j1.2 bridge too, as we have TaskLogAppender, ContainerLogAppender and ContainerRollingLogAppender which inherit FileAppender and RollingFileAppender in log4j1, which are not part of the log4j1.2 bridge. But anyway, at least we could just copy the source code to hadoop as we have WriteAppender in log4j1.2 bridge, and these two classes do not have related CVEs. Thoughts? For me I would like us to make a new 3.4.x release line to remove the log4j1 dependencies ASAP. Thanks. 1. https://nvd.nist.gov/vuln/detail/CVE-2022-23302 2. https://nvd.nist.gov/vuln/detail/CVE-2022-23305 3. https://nvd.nist.gov/vuln/detail/CVE-2022-23307 4. https://issues.apache.org/jira/browse/HADOOP-16206 5. https://lists.apache.org/thread/gvfb3jkg6t11cyds4jmpo7lrswmx28w3 6. https://issues.apache.org/jira/browse/LOG4J2-3341
[jira] [Resolved] (HADOOP-18086) Remove org.checkerframework.dataflow from hadoop-shaded-guava artifact (GNU GPLv2 license)
[
https://issues.apache.org/jira/browse/HADOOP-18086?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira Ajisaka resolved HADOOP-18086.
Resolution: Not A Problem
> Remove org.checkerframework.dataflow from hadoop-shaded-guava artifact (GNU
> GPLv2 license)
> --
>
> Key: HADOOP-18086
> URL: https://issues.apache.org/jira/browse/HADOOP-18086
> Project: Hadoop Common
> Issue Type: Bug
> Components: build
>Reporter: László Bodor
>Priority: Major
>
> Please refer to TEZ-4378 for further details:
> {code}
> jar tf
> ./hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-catalog/hadoop-yarn-applications-catalog-webapp/target/app/WEB-INF/lib/hadoop-shaded-guava-1.1.1.jar
> | grep "dataflow"
> org/apache/hadoop/thirdparty/org/checkerframework/dataflow/
> org/apache/hadoop/thirdparty/org/checkerframework/dataflow/qual/
> org/apache/hadoop/thirdparty/org/checkerframework/dataflow/qual/Deterministic.class
> org/apache/hadoop/thirdparty/org/checkerframework/dataflow/qual/Pure$Kind.class
> org/apache/hadoop/thirdparty/org/checkerframework/dataflow/qual/Pure.class
> org/apache/hadoop/thirdparty/org/checkerframework/dataflow/qual/SideEffectFree.class
> org/apache/hadoop/thirdparty/org/checkerframework/dataflow/qual/TerminatesExecution.class
> {code}
> I can see that checker-qual LICENSE.txt was removed in the scope of
> HADOOP-17648, but it has nothing to do with the license itself, only for
> [resolving a shading
> error|https://github.com/apache/hadoop-thirdparty/pull/9#issuecomment-822398949]
> my understanding is that in the current way an Apache licensed package (guava
> shaded jar) will contain a GPLv2 licensed software, which makes it a subject
> of GPLv2, also triggers license violations in security tools (like BlackDuck)
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
Re: [VOTE] Release Apache Hadoop 3.3.2 - RC2
thanks, i'm on it..will run the aws and azure tests and then play with the artifacts On Wed, 19 Jan 2022 at 17:50, Chao Sun wrote: > Hi all, > > I've put together Hadoop 3.3.2 RC2 below: > > The RC is available at: > http://people.apache.org/~sunchao/hadoop-3.3.2-RC2/ > The RC tag is at: > https://github.com/apache/hadoop/releases/tag/release-3.3.2-RC2 > The Maven artifacts are staged at: > https://repository.apache.org/content/repositories/orgapachehadoop-1332 > > You can find my public key at: > https://downloads.apache.org/hadoop/common/KEYS > > I've done the following tests and they look good: > - Ran all the unit tests > - Started a single node HDFS cluster and tested a few simple commands > - Ran all the tests in Spark using the RC2 artifacts > > Please evaluate the RC and vote, thanks! > > Best, > Chao >
Apache Hadoop qbt Report: branch-2.10+JDK7 on Linux/x86_64
For more details, see https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/548/ No changes -1 overall The following subsystems voted -1: asflicense hadolint mvnsite pathlen unit The following subsystems voted -1 but were configured to be filtered/ignored: cc checkstyle javac javadoc pylint shellcheck whitespace The following subsystems are considered long running: (runtime bigger than 1h 0m 0s) unit Specific tests: Failed junit tests : hadoop.io.compress.snappy.TestSnappyCompressorDecompressor hadoop.fs.TestFileUtil hadoop.fs.TestTrash hadoop.hdfs.server.namenode.ha.TestInitializeSharedEdits hadoop.hdfs.qjournal.server.TestJournalNodeRespectsBindHostKeys hadoop.hdfs.server.blockmanagement.TestReplicationPolicyWithUpgradeDomain hadoop.contrib.bkjournal.TestBookKeeperHACheckpoints hadoop.contrib.bkjournal.TestBookKeeperHACheckpoints hadoop.hdfs.server.federation.resolver.order.TestLocalResolver hadoop.hdfs.server.federation.router.TestRouterQuota hadoop.hdfs.server.federation.router.TestRouterNamenodeHeartbeat hadoop.hdfs.server.federation.resolver.TestMultipleDestinationResolver hadoop.yarn.server.resourcemanager.monitor.invariants.TestMetricsInvariantChecker hadoop.yarn.server.resourcemanager.TestClientRMService hadoop.mapreduce.lib.input.TestLineRecordReader hadoop.mapreduce.jobhistory.TestHistoryViewerPrinter hadoop.mapred.TestLineRecordReader hadoop.yarn.sls.appmaster.TestAMSimulator hadoop.yarn.sls.TestSLSRunner hadoop.resourceestimator.service.TestResourceEstimatorService hadoop.resourceestimator.solver.impl.TestLpSolver cc: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/548/artifact/out/diff-compile-cc-root.txt [4.0K] javac: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/548/artifact/out/diff-compile-javac-root.txt [476K] checkstyle: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/548/artifact/out/diff-checkstyle-root.txt [14M] hadolint: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/548/artifact/out/diff-patch-hadolint.txt [4.0K] mvnsite: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/548/artifact/out/patch-mvnsite-root.txt [560K] pathlen: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/548/artifact/out/pathlen.txt [12K] pylint: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/548/artifact/out/diff-patch-pylint.txt [20K] shellcheck: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/548/artifact/out/diff-patch-shellcheck.txt [72K] whitespace: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/548/artifact/out/whitespace-eol.txt [12M] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/548/artifact/out/whitespace-tabs.txt [1.3M] javadoc: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/548/artifact/out/patch-javadoc-root.txt [40K] unit: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/548/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt [224K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/548/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt [440K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/548/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs_src_contrib_bkjournal.txt [12K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/548/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt [36K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/548/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common.txt [20K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/548/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt [128K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/548/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt [104K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/548/artifact/out/patch-unit-hadoop-tools_hadoop-azure.txt [20K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/548/artifact/out/patch-unit-hadoop-tools_hadoop-sls.txt [32K]
[jira] [Created] (HADOOP-18086) Remove org.checkerframework.dataflow from hadoop-shaded-guava artifact (GNU GPLv2 license)
László Bodor created HADOOP-18086: - Summary: Remove org.checkerframework.dataflow from hadoop-shaded-guava artifact (GNU GPLv2 license) Key: HADOOP-18086 URL: https://issues.apache.org/jira/browse/HADOOP-18086 Project: Hadoop Common Issue Type: Wish Reporter: László Bodor -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
