I'm sort of leaning towards a -1, though the regression is actually in
3.4.0, and surfacing as a failure to release buffers in localfs IO.
[ERROR] org.apache.parquet.hadoop.TestParquetReader.testRangeFiltering[2]
-- Time elapsed: 0.033 s <<< ERROR!
org.apache.parquet.bytes.TrackingByteBufferAllocator$LeakedByteBufferException:
4 ByteBuffer object(s) is/are remained unreleased after closing this
allocator.
at
org.apache.parquet.bytes.TrackingByteBufferAllocator.close(TrackingByteBufferAllocator.java:160)
at
org.apache.parquet.hadoop.TestParquetReader.closeAllocator(TestParquetReader.java:175)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
Method)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
Caused by:
org.apache.parquet.bytes.TrackingByteBufferAllocator$ByteBufferAllocationStacktraceException:
Set org.apache.parquet.bytes.TrackingByteBufferAllocator.DEBUG = true for
more info
1. This is very much the vector io stuff.
2. It's not a regression since 3.4.0
3. but it's indicative of a memory leak. Surprised and Annoyed I hadn't
spotted this before...the tests went in last year.
On Tue, 3 Jun 2025 at 10:55, Steve Loughran <ste...@cloudera.com> wrote:
>
> Ahmar, no need for prs there, it's a "commit then review" repo, as in
> other people can revoke if you break things.
>
> will do the merge and test
>
> On Mon, 2 Jun 2025 at 16:51, Suhail, Ahmar <ahma...@amazon.co.uk.invalid>
> wrote:
>
>> Steve - I created a PR on hadoop-release-support with my properties
>> file: https://github.com/apache/hadoop-release-support/pull/4
>>
>>
>> Masatake - Yes, sounds good, will update the documentation for the new RC
>> if it is created.
>>
>>
>> Mukund - I realised too late that I messed up the commit for that one.
>> Will figure out how to fix..
>>
>> ________________________________
>> From: Steve Loughran <ste...@cloudera.com>
>> Sent: Monday, June 2, 2025 4:25:00 PM
>> To: Ahmar Suhail
>> Cc: common-...@hadoop.apache.org; hdfs-dev@hadoop.apache.org;
>> mapreduce-...@hadoop.apache.org; yarn-...@hadoop.apache.org
>> Subject: RE: [EXTERNAL] [VOTE] Release Apache Hadoop 3.4.2
>>
>>
>> CAUTION: This email originated from outside of the organization. Do not
>> click links or open attachments unless you can confirm the sender and know
>> the content is safe.
>>
>>
>>
>> can you put up the relevant changes to the hadoop-release-support module;
>> I'd like to use it as part of my validation and I'm assuming you have that
>> src/releases/release-info-3.4.2.properties file
>>
>> On Wed, 28 May 2025 at 13:25, Ahmar Suhail <ah...@apache.org<mailto:
>> ah...@apache.org>> wrote:
>> Hey all,
>>
>> The first release candidate for Hadoop 3.4.2 is now available for voting.
>>
>> There are a couple of things to note:
>>
>> 1/ No Arm64 artifacts. This is due to previously reported issues:
>> https://issues.apache.org/jira/projects/YARN/issues/YARN-11712 and
>> ttps://issues.apache.org/jira/projects/YARN/issues/YARN-11713<
>> http://issues.apache.org/jira/projects/YARN/issues/YARN-11713>
>> <https://issues.apache.org/jira/projects/YARN/issues/YARN-11713>, which
>> mean that the build fails on arm64.
>>
>> 2/ Relevant for anyone testing S3A: We've removed the AWS SDK bundle
>> from hadoop-3.4.2.tar.gz. This is because the SDK bundle is now ~600MB,
>> which makes the size of tar > 1GB, and it can no longer be uploaded to
>> SVN.
>> For S3A, download SDK bundle v2.29.52 from:
>> https://mvnrepository.com/artifact/software.amazon.awssdk/bundle/2.29.52,
>> and drop it into /share/hadoop/common/lib. Release notes will be updated
>> with these instructions.
>>
>>
>> The RC is available at:
>>
>> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.2-RC1/
>>
>> The git tag is release-3.4.2-RC1, commit
>> 09870840ec35b48cd107972eb24d25e8aece04c9
>>
>> The maven artifacts are staged at:
>>
>> https://repository.apache.org/content/repositories/orgapachehadoop-1437
>>
>>
>> You can find my public key (02085AFB652F796A3B01D11FD737A6F52281FA98) at:
>>
>> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>>
>>
>> This release has been created off of branch-3.4. Key changes include:
>>
>> * S3A: Integration with S3 Analytics Accelerator input stream
>> * S3A: Support for S3 conditional writes
>> * ABFS: Deprecation of WASB driver
>> * ABFS: Support for Non-Heirarchical Namespace Accounts on ABFS Driver
>>
>>
>> This is my first attempt at managing a release, please do test the release
>> and let me know in case of any issues.
>>
>> Thanks,
>> Ahmar
>>
>
