Re: [VOTE] Release Apache Hadoop 3.3.6 RC1

2023-06-20 Thread Wei-Chiu Chuang 
Bumping this thread to the top.
If you are verifying the release, please vote on this thread. RC0 and RC1
are exactly the same. The only material difference is the Changelog.

Thanks!!

On Sun, Jun 18, 2023 at 5:52 PM Wei-Chiu Chuang  wrote:

> I am inviting anyone to try and vote on this release candidate.
>
> Note:
> This is exactly the same as RC0, except the CHANGELOG.
>
> The RC is available at:
> https://home.apache.org/~weichiu/hadoop-3.3.6-RC1-amd64/ (for amd64)
> https://home.apache.org/~weichiu/hadoop-3.3.6-RC1-arm64/ (for arm64)
>
> Git tag: release-3.3.6-RC1
> https://github.com/apache/hadoop/releases/tag/release-3.3.6-RC1
>
> Maven artifacts is built by x86 machine and are staged at
> https://repository.apache.org/content/repositories/orgapachehadoop-1380/
>
> My public key:
> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>
> Changelog:
> https://home.apache.org/~weichiu/hadoop-3.3.6-RC1-amd64/CHANGELOG.md
>
> Release notes:
> https://home.apache.org/~weichiu/hadoop-3.3.6-RC1-amd64/RELEASENOTES.md
>
> This is a relatively small release (by Hadoop standard) containing about
> 120 commits.
> Please give it a try, this RC vote will run for 7 days.
>
>
> Feature highlights:
>
> SBOM artifacts
> 
> Starting from this release, Hadoop publishes Software Bill of Materials
> (SBOM) using
> CycloneDX Maven plugin. For more information about SBOM, please go to
> [SBOM](https://cwiki.apache.org/confluence/display/COMDEV/SBOM).
>
> HDFS RBF: RDBMS based token storage support
> 
> HDFS Router-Router Based Federation now supports storing delegation tokens
> on MySQL,
> [HADOOP-18535](https://issues.apache.org/jira/browse/HADOOP-18535)
> which improves token operation through over the original Zookeeper-based
> implementation.
>
>
> New File System APIs
> 
> [HADOOP-18671](https://issues.apache.org/jira/browse/HADOOP-18671) moved
> a number of
> HDFS-specific APIs to Hadoop Common to make it possible for certain
> applications that
> depend on HDFS semantics to run on other Hadoop compatible file systems.
>
> In particular, recoverLease() and isFileClosed() are exposed through
> LeaseRecoverable
> interface. While setSafeMode() is exposed through SafeMode interface.
>
>
>

Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86_64

2023-06-20 Thread Apache Jenkins Server 
For more details, see 
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1263/

[Jun 19, 2023, 9:19:23 AM] (github) HDFS-17053. Optimize method 
BlockInfoStriped#findSlot to reduce time complexity. (#5757). Contributed by 
farmmamba.




-1 overall


The following subsystems voted -1:
blanks hadolint pathlen unit xml


The following subsystems voted -1 but
were configured to be filtered/ignored:
cc checkstyle javac javadoc pylint shellcheck


The following subsystems are considered long running:
(runtime bigger than 1h  0m  0s)
unit


Specific tests:

XML :

   Parsing Error(s): 
   
hadoop-common-project/hadoop-common/src/test/resources/xml/external-dtd.xml 
   
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-excerpt.xml
 
   
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-missing-tags.xml
 
   
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-missing-tags2.xml
 
   
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-sample-output.xml
 
   
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/fair-scheduler-invalid.xml
 
   
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/yarn-site-with-invalid-allocation-file-ref.xml
 

Failed junit tests :

   hadoop.hdfs.server.datanode.TestDirectoryScanner 
   hadoop.yarn.server.timelineservice.security.TestTimelineAuthFilterForV2 
   hadoop.mapreduce.v2.TestUberAM 
   hadoop.mapreduce.v2.TestMRJobsWithProfiler 
   hadoop.mapreduce.v2.TestMRJobs 
  

   cc:

  
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1263/artifact/out/results-compile-cc-root.txt
 [96K]

   javac:

  
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1263/artifact/out/results-compile-javac-root.txt
 [12K]

   blanks:

  
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1263/artifact/out/blanks-eol.txt
 [14M]
  
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1263/artifact/out/blanks-tabs.txt
 [2.0M]

   checkstyle:

  
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1263/artifact/out/results-checkstyle-root.txt
 [13M]

   hadolint:

  
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1263/artifact/out/results-hadolint.txt
 [20K]

   pathlen:

  
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1263/artifact/out/results-pathlen.txt
 [16K]

   pylint:

  
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1263/artifact/out/results-pylint.txt
 [20K]

   shellcheck:

  
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1263/artifact/out/results-shellcheck.txt
 [24K]

   xml:

  
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1263/artifact/out/xml.txt
 [24K]

   javadoc:

  
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1263/artifact/out/results-javadoc-javadoc-root.txt
 [244K]

   unit:

  
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1263/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
 [404K]
  
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1263/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests.txt
 [24K]
  
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1263/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt
 [72K]

Powered by Apache Yetus 0.14.0-SNAPSHOT   https://yetus.apache.org

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Apache Hadoop qbt Report: branch-2.10+JDK7 on Linux/x86_64

2023-06-20 Thread Apache Jenkins Server 
For more details, see 
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1064/

No changes


ERROR: File 'out/email-report.txt' does not exist

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 3.3.6 RC1

2023-06-20 Thread Masatake Iwasaki 

+1

+ verified the signature and checksum of the source tarball.
+ built from the source tarball on Rocky Linux 8 (x86_64) and OpenJDK 8 with 
native profile enabled.
  + launched pseudo distributed cluster including kms and httpfs with Kerberos 
and SSL enabled.
  + created encryption zone, put and read files via httpfs.
+ built RPM packages by Bigtop (modified to use ZooKeepr 3.6) on Rocky Linux 8 
(x86_64).
  + built HBase and Hive against Hadoop 3.3.6.
  + ran smoke-tests of hdfs, yarn, mapreduce, hbase and hive.
+ skimmed the contents of site documentation.

Thanks,
Masatake Iwasaki

On 2023/06/21 8:07, Wei-Chiu Chuang wrote:

Bumping this thread to the top.
If you are verifying the release, please vote on this thread. RC0 and RC1
are exactly the same. The only material difference is the Changelog.

Thanks!!

On Sun, Jun 18, 2023 at 5:52 PM Wei-Chiu Chuang  wrote:


I am inviting anyone to try and vote on this release candidate.

Note:
This is exactly the same as RC0, except the CHANGELOG.

The RC is available at:
https://home.apache.org/~weichiu/hadoop-3.3.6-RC1-amd64/ (for amd64)
https://home.apache.org/~weichiu/hadoop-3.3.6-RC1-arm64/ (for arm64)

Git tag: release-3.3.6-RC1
https://github.com/apache/hadoop/releases/tag/release-3.3.6-RC1

Maven artifacts is built by x86 machine and are staged at
https://repository.apache.org/content/repositories/orgapachehadoop-1380/

My public key:
https://dist.apache.org/repos/dist/release/hadoop/common/KEYS

Changelog:
https://home.apache.org/~weichiu/hadoop-3.3.6-RC1-amd64/CHANGELOG.md

Release notes:
https://home.apache.org/~weichiu/hadoop-3.3.6-RC1-amd64/RELEASENOTES.md

This is a relatively small release (by Hadoop standard) containing about
120 commits.
Please give it a try, this RC vote will run for 7 days.


Feature highlights:

SBOM artifacts

Starting from this release, Hadoop publishes Software Bill of Materials
(SBOM) using
CycloneDX Maven plugin. For more information about SBOM, please go to
[SBOM](https://cwiki.apache.org/confluence/display/COMDEV/SBOM).

HDFS RBF: RDBMS based token storage support

HDFS Router-Router Based Federation now supports storing delegation tokens
on MySQL,
[HADOOP-18535](https://issues.apache.org/jira/browse/HADOOP-18535)
which improves token operation through over the original Zookeeper-based
implementation.


New File System APIs

[HADOOP-18671](https://issues.apache.org/jira/browse/HADOOP-18671) moved
a number of
HDFS-specific APIs to Hadoop Common to make it possible for certain
applications that
depend on HDFS semantics to run on other Hadoop compatible file systems.

In particular, recoverLease() and isFileClosed() are exposed through
LeaseRecoverable
interface. While setSafeMode() is exposed through SafeMode interface.







-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 3.3.6 RC1

2023-06-20 Thread Tak Lon (Stephen) Wu 
+1 (non-binding), and thanks a lot for driving the vote.

* Signature of sources and binaries: ok
* Checksum of sources and binaries: ok
* Rat check (1.8.0_362): okie
 - mvn clean apache-rat:check
* Built from source (1.8.0_362): ok
 - mvn clean install -DskipTests
* Run Pseudo-Distributed mode with HDFS and YARN (1.8.0_362): ok
* Run Shell command (mkdir/put/ls/get) (1.8.0_362) : ok
* Run MR examples applications and check the result (1.8.0_362): ok
 - bin/hadoop jar
share/hadoop/mapreduce/hadoop-mapreduce-examples-3.3.6.jar grep input
output 'dfs[a-z.]+'

-Stephen

On Tue, Jun 20, 2023 at 6:56 PM Masatake Iwasaki 
wrote:

> +1
>
> + verified the signature and checksum of the source tarball.
> + built from the source tarball on Rocky Linux 8 (x86_64) and OpenJDK 8
> with native profile enabled.
>+ launched pseudo distributed cluster including kms and httpfs with
> Kerberos and SSL enabled.
>+ created encryption zone, put and read files via httpfs.
> + built RPM packages by Bigtop (modified to use ZooKeepr 3.6) on Rocky
> Linux 8 (x86_64).
>+ built HBase and Hive against Hadoop 3.3.6.
>+ ran smoke-tests of hdfs, yarn, mapreduce, hbase and hive.
> + skimmed the contents of site documentation.
>
> Thanks,
> Masatake Iwasaki
>
> On 2023/06/21 8:07, Wei-Chiu Chuang wrote:
> > Bumping this thread to the top.
> > If you are verifying the release, please vote on this thread. RC0 and RC1
> > are exactly the same. The only material difference is the Changelog.
> >
> > Thanks!!
> >
> > On Sun, Jun 18, 2023 at 5:52 PM Wei-Chiu Chuang 
> wrote:
> >
> >> I am inviting anyone to try and vote on this release candidate.
> >>
> >> Note:
> >> This is exactly the same as RC0, except the CHANGELOG.
> >>
> >> The RC is available at:
> >> https://home.apache.org/~weichiu/hadoop-3.3.6-RC1-amd64/ (for amd64)
> >> https://home.apache.org/~weichiu/hadoop-3.3.6-RC1-arm64/ (for arm64)
> >>
> >> Git tag: release-3.3.6-RC1
> >> https://github.com/apache/hadoop/releases/tag/release-3.3.6-RC1
> >>
> >> Maven artifacts is built by x86 machine and are staged at
> >>
> https://repository.apache.org/content/repositories/orgapachehadoop-1380/
> >>
> >> My public key:
> >> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> >>
> >> Changelog:
> >> https://home.apache.org/~weichiu/hadoop-3.3.6-RC1-amd64/CHANGELOG.md
> >>
> >> Release notes:
> >> https://home.apache.org/~weichiu/hadoop-3.3.6-RC1-amd64/RELEASENOTES.md
> >>
> >> This is a relatively small release (by Hadoop standard) containing about
> >> 120 commits.
> >> Please give it a try, this RC vote will run for 7 days.
> >>
> >>
> >> Feature highlights:
> >>
> >> SBOM artifacts
> >> 
> >> Starting from this release, Hadoop publishes Software Bill of Materials
> >> (SBOM) using
> >> CycloneDX Maven plugin. For more information about SBOM, please go to
> >> [SBOM](https://cwiki.apache.org/confluence/display/COMDEV/SBOM).
> >>
> >> HDFS RBF: RDBMS based token storage support
> >> 
> >> HDFS Router-Router Based Federation now supports storing delegation
> tokens
> >> on MySQL,
> >> [HADOOP-18535](https://issues.apache.org/jira/browse/HADOOP-18535)
> >> which improves token operation through over the original Zookeeper-based
> >> implementation.
> >>
> >>
> >> New File System APIs
> >> 
> >> [HADOOP-18671](https://issues.apache.org/jira/browse/HADOOP-18671)
> moved
> >> a number of
> >> HDFS-specific APIs to Hadoop Common to make it possible for certain
> >> applications that
> >> depend on HDFS semantics to run on other Hadoop compatible file systems.
> >>
> >> In particular, recoverLease() and isFileClosed() are exposed through
> >> LeaseRecoverable
> >> interface. While setSafeMode() is exposed through SafeMode interface.
> >>
> >>
> >>
> >
>
> -
> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: common-dev-h...@hadoop.apache.org
>
>

Apache Hadoop qbt Report: trunk+JDK11 on Linux/x86_64

2023-06-20 Thread Apache Jenkins Server 
For more details, see 
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java11-linux-x86_64/507/

[Jun 18, 2023, 6:08:38 AM] (github) HADOOP-18774. Add .vscode to gitignore. 
(#5756). Contributed by Xiaoqiao He.
[Jun 18, 2023, 9:09:25 AM] (github) YARN-11506. The formatted yarn queue list 
is displayed on CLI (#5716). Contributed by Lu Yuan.

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Resolved] (HDFS-17049) EC: Fix duplicate block group IDs generated by SequentialBlockGroupIdGenerator

2023-06-20 Thread Xiaoqiao He (Jira) 


 [ 
https://issues.apache.org/jira/browse/HDFS-17049?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xiaoqiao He resolved HDFS-17049.

Resolution: Not A Problem

> EC: Fix duplicate block group IDs generated by SequentialBlockGroupIdGenerator
> --
>
> Key: HDFS-17049
> URL: https://issues.apache.org/jira/browse/HDFS-17049
> Project: Hadoop HDFS
>  Issue Type: Bug
>Reporter: Shuyan Zhang
>Assignee: Shuyan Zhang
>Priority: Major
>  Labels: pull-request-available
>
> When I used multiple clients to write EC files concurrently, I found that 
> NameNode generated the same block group ID for different files:
> ```
> 2023-06-13 20:09:59,514 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* 
> allocate blk_-9223372036854697568_14389 for /ec-test/10/4068034329705654124
> 2023-06-13 20:09:59,514 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* 
> allocate blk_-9223372036854697568_14390 for /ec-test/19/7042966144171770731
> ```
> After diving into `SequentialBlockGroupIdGenerator`, I found that the current 
> implementation of `nextValue` is not thread-safe.
> This problem must be fixed.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Apache Hadoop qbt Report: branch-2.10+JDK7 on Linux/x86_64

2023-06-20 Thread Apache Jenkins Server 
For more details, see 
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1063/

No changes




-1 overall


The following subsystems voted -1:
asflicense compile golang hadolint mvnsite pathlen unit


The following subsystems voted -1 but
were configured to be filtered/ignored:
cc checkstyle javac javadoc pylint shellcheck whitespace


The following subsystems are considered long running:
(runtime bigger than 1h  0m  0s)
unit


Specific tests:

Failed junit tests :

   hadoop.ipc.TestIPC 
   hadoop.fs.TestFileUtil 
   hadoop.contrib.bkjournal.TestBookKeeperHACheckpoints 
   hadoop.hdfs.qjournal.server.TestJournalNodeRespectsBindHostKeys 
   hadoop.hdfs.server.namenode.snapshot.TestSnapshotBlocksMap 
   hadoop.hdfs.TestFileLengthOnClusterRestart 
   hadoop.hdfs.TestLeaseRecovery2 
   
hadoop.hdfs.server.blockmanagement.TestReplicationPolicyWithUpgradeDomain 
   hadoop.hdfs.server.namenode.snapshot.TestSnapshotDeletion 
   hadoop.hdfs.server.balancer.TestBalancerWithHANameNodes 
   hadoop.hdfs.server.federation.resolver.order.TestLocalResolver 
   hadoop.hdfs.server.federation.router.TestRouterNamenodeHeartbeat 
   hadoop.hdfs.server.federation.resolver.TestMultipleDestinationResolver 
   hadoop.hdfs.server.federation.router.TestRouterQuota 
   hadoop.contrib.bkjournal.TestBookKeeperHACheckpoints 
   hadoop.mapreduce.jobhistory.TestHistoryViewerPrinter 
   hadoop.mapreduce.lib.input.TestLineRecordReader 
   hadoop.mapred.TestLineRecordReader 
   hadoop.resourceestimator.service.TestResourceEstimatorService 
   hadoop.resourceestimator.solver.impl.TestLpSolver 
   hadoop.yarn.sls.TestSLSRunner 
   hadoop.yarn.client.api.impl.TestAMRMClient 
   
hadoop.yarn.server.nodemanager.containermanager.linux.resources.TestNumaResourceHandlerImpl
 
   
hadoop.yarn.server.nodemanager.containermanager.linux.resources.TestNumaResourceAllocator
 
   
hadoop.yarn.server.resourcemanager.monitor.invariants.TestMetricsInvariantChecker
 
   hadoop.yarn.server.resourcemanager.recovery.TestFSRMStateStore 
   hadoop.yarn.server.resourcemanager.TestClientRMService 
  

   compile:

   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1063/artifact/out/patch-compile-root.txt
  [680K]

   cc:

   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1063/artifact/out/patch-compile-root.txt
  [680K]

   golang:

   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1063/artifact/out/patch-compile-root.txt
  [680K]

   javac:

   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1063/artifact/out/patch-compile-root.txt
  [680K]

   checkstyle:

   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1063/artifact/out/diff-checkstyle-root.txt
  [14M]

   hadolint:

   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1063/artifact/out/diff-patch-hadolint.txt
  [4.0K]

   mvnsite:

   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1063/artifact/out/patch-mvnsite-root.txt
  [588K]

   pathlen:

   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1063/artifact/out/pathlen.txt
  [12K]

   pylint:

   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1063/artifact/out/diff-patch-pylint.txt
  [20K]

   shellcheck:

   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1063/artifact/out/diff-patch-shellcheck.txt
  [72K]

   whitespace:

   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1063/artifact/out/whitespace-eol.txt
  [12M]
   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1063/artifact/out/whitespace-tabs.txt
  [1.3M]

   javadoc:

   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1063/artifact/out/patch-javadoc-root.txt
  [40K]

   unit:

   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1063/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt
  [244K]
   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1063/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
  [456K]
   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1063/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt
  [36K]
   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1063/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs_src_contrib_bkjournal.txt
  [16K]
   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1063/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt
  [104K]

Re: [VOTE] Release Apache Hadoop 3.3.6 RC0

2023-06-20 Thread Ashutosh Gupta 
Hi

Thanks Wei-Chiu for driving the release.

+1 (non-binding)

* Builds from source looks good.
* Checksums and signatures look good.
* Running basic HDFS commands and running simple MapReduce jobs looks good.
* hadoop-tools/hadoop-aws UTs and ITs looks good

Thanks,
Ash

On Tue, Jun 20, 2023 at 5:18 PM Mukund Madhav Thakur
 wrote:

> Hi Wei-Chiu,
> Thanks for driving the release.
>
> +1 (binding)
> Verified checksum and signature.
> Built from source successfully.
> Ran aws Itests
> Ran azure Itests
> Compiled hadoop-api-shim
> Compiled google cloud storage.
>
>
> I did see the two test failures in GCS connector as well but those are
> harmless.
>
>
>
> On Thu, Jun 15, 2023 at 8:21 PM Wei-Chiu Chuang
>  wrote:
>
> > Overall so far so good.
> >
> > hadoop-api-shim:
> > built, tested successfully.
> >
> > cloudstore:
> > built successfully.
> >
> > Spark:
> > built successfully. Passed hadoop-cloud tests.
> >
> > Ozone:
> > One test failure due to unrelated Ozone issue. This test is being
> disabled
> > in the latest Ozone code.
> >
> > org.apache.hadoop.hdds.utils.NativeLibraryNotLoadedException: Unable
> > to load library ozone_rocksdb_tools from both java.library.path &
> > resource file libozone_rocksdb_t
> > ools.so from jar.
> > at
> >
> org.apache.hadoop.hdds.utils.db.managed.ManagedSSTDumpTool.(ManagedSSTDumpTool.java:49)
> >
> >
> > Google gcs:
> > There are two test failures. The tests were added recently by
> HADOOP-18724
> >  in Hadoop 3.3.6.
> This
> > is okay. Not production code problem. Can be addressed in GCS code.
> >
> > [ERROR] Errors:
> > [ERROR]
> >
> >
> TestInMemoryGoogleContractOpen>AbstractContractOpenTest.testFloatingPointLength:403
> > » IllegalArgument Unknown mandatory key for gs://fake-in-memory-test-buck
> > et/contract-test/testFloatingPointLength "fs.option.openfile.length"
> > [ERROR]
> >
> >
> TestInMemoryGoogleContractOpen>AbstractContractOpenTest.testOpenFileApplyAsyncRead:341
> > » IllegalArgument Unknown mandatory key for gs://fake-in-memory-test-b
> > ucket/contract-test/testOpenFileApplyAsyncRead
> "fs.option.openfile.length"
> >
> >
> >
> >
> >
> > On Wed, Jun 14, 2023 at 5:01 PM Wei-Chiu Chuang 
> > wrote:
> >
> > > The hbase-filesystem tests passed after reverting HADOOP-18596
> > >  and HADOOP-18633
> > >  from my local
> tree.
> > > So I think it's a matter of the default behavior being changed. It's
> not
> > > the end of the world. I think we can address it by adding an
> incompatible
> > > change flag and a release note.
> > >
> > > On Wed, Jun 14, 2023 at 3:55 PM Wei-Chiu Chuang 
> > > wrote:
> > >
> > >> Cross referenced git history and jira. Changelog needs some update
> > >>
> > >> Not in the release
> > >>
> > >>1. HDFS-16858 
> > >>
> > >>
> > >>1. HADOOP-18532 <
> https://issues.apache.org/jira/browse/HADOOP-18532>
> > >>2.
> > >>   1. HDFS-16861  >
> > >>  2.
> > >> 1. HDFS-16866
> > >> 
> > >> 2.
> > >>1. HADOOP-18320
> > >>
> > >>2.
> > >>
> > >> Updated fixed version. Will generate. new Changelog in the next RC.
> > >>
> > >> Was able to build HBase and hbase-filesystem without any code change.
> > >>
> > >> hbase has one unit test failure. This one is reproducible even with
> > >> Hadoop 3.3.5, so maybe a red herring. Local env or something.
> > >>
> > >> [ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time
> elapsed:
> > >> 9.007 s <<< FAILURE! - in
> > >> org.apache.hadoop.hbase.regionserver.TestSyncTimeRangeTracker
> > >> [ERROR]
> > >>
> >
> org.apache.hadoop.hbase.regionserver.TestSyncTimeRangeTracker.testConcurrentIncludeTimestampCorrectness
> > >>  Time elapsed: 3.13 s  <<< ERROR!
> > >> java.lang.OutOfMemoryError: Java heap space
> > >> at
> > >>
> >
> org.apache.hadoop.hbase.regionserver.TestSyncTimeRangeTracker$RandomTestData.(TestSyncTimeRangeTracker.java:91)
> > >> at
> > >>
> >
> org.apache.hadoop.hbase.regionserver.TestSyncTimeRangeTracker.testConcurrentIncludeTimestampCorrectness(TestSyncTimeRangeTracker.java:156)
> > >>
> > >> hbase-filesystem has three test failures in TestHBOSSContractDistCp,
> and
> > >> is not reproducible with Hadoop 3.3.5.
> > >> [ERROR] Failures: [ERROR]
> > >>
> >
> TestHBOSSContractDistCp>AbstractContractDistCpTest.testDistCpUpdateCheckFileSkip:976->Assert.fail:88
> > >> 10 errors in file of length 10
> > >> [ERROR]
> > >>
> >
> TestHBOSSContractDistCp>AbstractContractDistCpTest.testUpdateDeepDirectoryStructureNoChange:270->AbstractContractDistCpTest.assertCounterInRange:290->Assert.assertTrue:41->Assert.fail:88
> > >> Files

Re: [VOTE] Release Apache Hadoop 3.3.6 RC0

2023-06-20 Thread Ayush Saxena 
Folks you had put your vote on the wrong thread, this is the RC0 thead, you 
need to put on the RC1,
https://lists.apache.org/thread/022p4rml5tvsx9xpq6t7b3n1td8lzz1d

-Ayush

Sent from my iPhone

> On 20-Jun-2023, at 10:03 PM, Ashutosh Gupta  
> wrote:
> 
> Hi
> 
> Thanks Wei-Chiu for driving the release.
> 
> +1 (non-binding)
> 
> * Builds from source looks good.
> * Checksums and signatures look good.
> * Running basic HDFS commands and running simple MapReduce jobs looks good.
> * hadoop-tools/hadoop-aws UTs and ITs looks good
> 
> Thanks,
> Ash
> 
>> On Tue, Jun 20, 2023 at 5:18 PM Mukund Madhav Thakur
>>  wrote:
>> 
>> Hi Wei-Chiu,
>> Thanks for driving the release.
>> 
>> +1 (binding)
>> Verified checksum and signature.
>> Built from source successfully.
>> Ran aws Itests
>> Ran azure Itests
>> Compiled hadoop-api-shim
>> Compiled google cloud storage.
>> 
>> 
>> I did see the two test failures in GCS connector as well but those are
>> harmless.
>> 
>> 
>> 
>> On Thu, Jun 15, 2023 at 8:21 PM Wei-Chiu Chuang
>>  wrote:
>> 
>>> Overall so far so good.
>>> 
>>> hadoop-api-shim:
>>> built, tested successfully.
>>> 
>>> cloudstore:
>>> built successfully.
>>> 
>>> Spark:
>>> built successfully. Passed hadoop-cloud tests.
>>> 
>>> Ozone:
>>> One test failure due to unrelated Ozone issue. This test is being
>> disabled
>>> in the latest Ozone code.
>>> 
>>> org.apache.hadoop.hdds.utils.NativeLibraryNotLoadedException: Unable
>>> to load library ozone_rocksdb_tools from both java.library.path &
>>> resource file libozone_rocksdb_t
>>> ools.so from jar.
>>>at
>>> 
>> org.apache.hadoop.hdds.utils.db.managed.ManagedSSTDumpTool.(ManagedSSTDumpTool.java:49)
>>> 
>>> 
>>> Google gcs:
>>> There are two test failures. The tests were added recently by
>> HADOOP-18724
>>>  in Hadoop 3.3.6.
>> This
>>> is okay. Not production code problem. Can be addressed in GCS code.
>>> 
>>> [ERROR] Errors:
>>> [ERROR]
>>> 
>>> 
>> TestInMemoryGoogleContractOpen>AbstractContractOpenTest.testFloatingPointLength:403
>>> » IllegalArgument Unknown mandatory key for gs://fake-in-memory-test-buck
>>> et/contract-test/testFloatingPointLength "fs.option.openfile.length"
>>> [ERROR]
>>> 
>>> 
>> TestInMemoryGoogleContractOpen>AbstractContractOpenTest.testOpenFileApplyAsyncRead:341
>>> » IllegalArgument Unknown mandatory key for gs://fake-in-memory-test-b
>>> ucket/contract-test/testOpenFileApplyAsyncRead
>> "fs.option.openfile.length"
>>> 
>>> 
>>> 
>>> 
>>> 
>>> On Wed, Jun 14, 2023 at 5:01 PM Wei-Chiu Chuang 
>>> wrote:
>>> 
 The hbase-filesystem tests passed after reverting HADOOP-18596
  and HADOOP-18633
  from my local
>> tree.
 So I think it's a matter of the default behavior being changed. It's
>> not
 the end of the world. I think we can address it by adding an
>> incompatible
 change flag and a release note.
 
 On Wed, Jun 14, 2023 at 3:55 PM Wei-Chiu Chuang 
 wrote:
 
> Cross referenced git history and jira. Changelog needs some update
> 
> Not in the release
> 
>   1. HDFS-16858 
> 
> 
>   1. HADOOP-18532 <
>> https://issues.apache.org/jira/browse/HADOOP-18532>
>   2.
>  1. HDFS-16861 >> 
> 2.
>1. HDFS-16866
>
>2.
>   1. HADOOP-18320
>   
>   2.
> 
> Updated fixed version. Will generate. new Changelog in the next RC.
> 
> Was able to build HBase and hbase-filesystem without any code change.
> 
> hbase has one unit test failure. This one is reproducible even with
> Hadoop 3.3.5, so maybe a red herring. Local env or something.
> 
> [ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time
>> elapsed:
> 9.007 s <<< FAILURE! - in
> org.apache.hadoop.hbase.regionserver.TestSyncTimeRangeTracker
> [ERROR]
> 
>>> 
>> org.apache.hadoop.hbase.regionserver.TestSyncTimeRangeTracker.testConcurrentIncludeTimestampCorrectness
> Time elapsed: 3.13 s  <<< ERROR!
> java.lang.OutOfMemoryError: Java heap space
> at
> 
>>> 
>> org.apache.hadoop.hbase.regionserver.TestSyncTimeRangeTracker$RandomTestData.(TestSyncTimeRangeTracker.java:91)
> at
> 
>>> 
>> org.apache.hadoop.hbase.regionserver.TestSyncTimeRangeTracker.testConcurrentIncludeTimestampCorrectness(TestSyncTimeRangeTracker.java:156)
> 
> hbase-filesystem has three test failures in TestHBOSSContractDistCp,
>> and
> is not reproducible with Hadoop 3.3.5.
> [ERROR] Failures: [ERROR]
> 
>>> 
>>

Re: [VOTE] Release Apache Hadoop 3.3.6 RC0

2023-06-20 Thread Mukund Madhav Thakur 
Hi Wei-Chiu,
Thanks for driving the release.

+1 (binding)
Verified checksum and signature.
Built from source successfully.
Ran aws Itests
Ran azure Itests
Compiled hadoop-api-shim
Compiled google cloud storage.


I did see the two test failures in GCS connector as well but those are
harmless.



On Thu, Jun 15, 2023 at 8:21 PM Wei-Chiu Chuang
 wrote:

> Overall so far so good.
>
> hadoop-api-shim:
> built, tested successfully.
>
> cloudstore:
> built successfully.
>
> Spark:
> built successfully. Passed hadoop-cloud tests.
>
> Ozone:
> One test failure due to unrelated Ozone issue. This test is being disabled
> in the latest Ozone code.
>
> org.apache.hadoop.hdds.utils.NativeLibraryNotLoadedException: Unable
> to load library ozone_rocksdb_tools from both java.library.path &
> resource file libozone_rocksdb_t
> ools.so from jar.
> at
> org.apache.hadoop.hdds.utils.db.managed.ManagedSSTDumpTool.(ManagedSSTDumpTool.java:49)
>
>
> Google gcs:
> There are two test failures. The tests were added recently by HADOOP-18724
>  in Hadoop 3.3.6. This
> is okay. Not production code problem. Can be addressed in GCS code.
>
> [ERROR] Errors:
> [ERROR]
>
> TestInMemoryGoogleContractOpen>AbstractContractOpenTest.testFloatingPointLength:403
> » IllegalArgument Unknown mandatory key for gs://fake-in-memory-test-buck
> et/contract-test/testFloatingPointLength "fs.option.openfile.length"
> [ERROR]
>
> TestInMemoryGoogleContractOpen>AbstractContractOpenTest.testOpenFileApplyAsyncRead:341
> » IllegalArgument Unknown mandatory key for gs://fake-in-memory-test-b
> ucket/contract-test/testOpenFileApplyAsyncRead "fs.option.openfile.length"
>
>
>
>
>
> On Wed, Jun 14, 2023 at 5:01 PM Wei-Chiu Chuang 
> wrote:
>
> > The hbase-filesystem tests passed after reverting HADOOP-18596
> >  and HADOOP-18633
> >  from my local tree.
> > So I think it's a matter of the default behavior being changed. It's not
> > the end of the world. I think we can address it by adding an incompatible
> > change flag and a release note.
> >
> > On Wed, Jun 14, 2023 at 3:55 PM Wei-Chiu Chuang 
> > wrote:
> >
> >> Cross referenced git history and jira. Changelog needs some update
> >>
> >> Not in the release
> >>
> >>1. HDFS-16858 
> >>
> >>
> >>1. HADOOP-18532 
> >>2.
> >>   1. HDFS-16861 
> >>  2.
> >> 1. HDFS-16866
> >> 
> >> 2.
> >>1. HADOOP-18320
> >>
> >>2.
> >>
> >> Updated fixed version. Will generate. new Changelog in the next RC.
> >>
> >> Was able to build HBase and hbase-filesystem without any code change.
> >>
> >> hbase has one unit test failure. This one is reproducible even with
> >> Hadoop 3.3.5, so maybe a red herring. Local env or something.
> >>
> >> [ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed:
> >> 9.007 s <<< FAILURE! - in
> >> org.apache.hadoop.hbase.regionserver.TestSyncTimeRangeTracker
> >> [ERROR]
> >>
> org.apache.hadoop.hbase.regionserver.TestSyncTimeRangeTracker.testConcurrentIncludeTimestampCorrectness
> >>  Time elapsed: 3.13 s  <<< ERROR!
> >> java.lang.OutOfMemoryError: Java heap space
> >> at
> >>
> org.apache.hadoop.hbase.regionserver.TestSyncTimeRangeTracker$RandomTestData.(TestSyncTimeRangeTracker.java:91)
> >> at
> >>
> org.apache.hadoop.hbase.regionserver.TestSyncTimeRangeTracker.testConcurrentIncludeTimestampCorrectness(TestSyncTimeRangeTracker.java:156)
> >>
> >> hbase-filesystem has three test failures in TestHBOSSContractDistCp, and
> >> is not reproducible with Hadoop 3.3.5.
> >> [ERROR] Failures: [ERROR]
> >>
> TestHBOSSContractDistCp>AbstractContractDistCpTest.testDistCpUpdateCheckFileSkip:976->Assert.fail:88
> >> 10 errors in file of length 10
> >> [ERROR]
> >>
> TestHBOSSContractDistCp>AbstractContractDistCpTest.testUpdateDeepDirectoryStructureNoChange:270->AbstractContractDistCpTest.assertCounterInRange:290->Assert.assertTrue:41->Assert.fail:88
> >> Files Skipped value 0 too below minimum 1
> >> [ERROR]
> >>
> TestHBOSSContractDistCp>AbstractContractDistCpTest.testUpdateDeepDirectoryStructureToRemote:259->AbstractContractDistCpTest.distCpUpdateDeepDirectoryStructure:334->AbstractContractDistCpTest.assertCounterInRange:294->Assert.assertTrue:41->Assert.fail:88
> >> Files Copied value 2 above maximum 1
> >> [INFO]
> >> [ERROR] Tests run: 240, Failures: 3, Errors: 0, Skipped: 58
> >>
> >>
> >> Ozone
> >> test in progress. Will report back.
> >>
> >>
> >> On Tue, Jun 13, 2023 at 11:27 PM Wei-Chiu Chuang 
> >> wrote:
> >>
> >>> I am inviting anyone to try and vote on this