I have a WiP PR to pull all the object store artifacts into share/common/lib; once we have the basic build working I"m going to extend the dist profile with an aws-sdk profile that includes the AWS SDK.
1. our public builds will skip the bundle 2. s3a, abfs gs and other connectors will be on the classpath! 3. for s3a, you'll need to add any version of the bundle.jar to common/lib we wouldn't have a full/lean tar up online, just a single distro without the bundle.jar file and a doc saying "download from X and drop into Y. We tested using version Z, later versions may nor may not work. " please review https://github.com/apache/hadoop/pull/7980 On Fri, 19 Sept 2025 at 13:10, Suhail, Ahmar <ahma...@amazon.co.uk> wrote: > Thanks Shilun, > > > > I agree, For 3.4.2, we can move these projects to use the lean-tar. > > > > Separately, working with Steve to update the packaging so the full tar > excludes the AWS bundle by default, so from the next release onwards, we > will just have the full tars, and will no longer need the lean-tar. > > > > So two options for now: > > > > - We add a new hadoop-3.4.2.tar.gz, which is just a copy of > hadoop-3.4.2-lean.tar.gz. > (I would prefer not to do this as this requires some manual changes to the > release directory) > - We update Hive and Ozone to use the hadoop-3.4.2-lean.tar.gz for > now. And from the next release, can go back to using hadoop-3.4.2.tar.gz. > > > > What do others think? > > > > *From: *slfan1989 <slfan1...@apache.org> > *Date: *Friday, 19 September 2025 at 04:20 > *To: *"Suhail, Ahmar" <ahma...@amazon.co.uk> > *Cc: *"Suhail, Ahmar" <ahma...@amazon.co.uk.invalid>, "ah...@apache.org" < > ah...@apache.org>, Ayush Saxena <ayush...@gmail.com>, Xiaoqiao He < > hexiaoq...@apache.org>, Steve Loughran <ste...@cloudera.com>, Chris > Nauroth <cnaur...@apache.org>, Hadoop Common <common-dev@hadoop.apache.org>, > yarn-dev <yarn-...@hadoop.apache.org>, mapreduce-dev < > mapreduce-...@hadoop.apache.org>, Hdfs-dev <hdfs-...@hadoop.apache.org>, > "<priv...@hadoop.apache.org>" <priv...@hadoop.apache.org> > *Subject: *RE: [EXTERNAL] Missing full tarball for Hadoop 3.4.2 > > > > *CAUTION*: This email originated from outside of the organization. Do not > click links or open attachments unless you can confirm the sender and know > the content is safe. > > > > Thank you for your feedback. I think we do need to make a collective > decision regarding the uploaded packages. At present, the absence of the > full tar package prevents Ozone and Hive from using the latest Hadoop 3.4.2 > release. From my perspective, using the minimal package as a substitute > for the full package could be a reasonable option. > > > > Best regards, > Shilun Fan. > > > > On Wed, Sep 10, 2025 at 5:56 PM Suhail, Ahmar <ahma...@amazon.co.uk> > wrote: > > Hey all, > > Heard back from Infra: > > "The 1GB increase in INFRA-25423 was temporary. Your project needs to > reduce the size of these binaries. In the previous temporary space > increase, your project stated they would be removing extraneous > dependencies/SDKs. We cannot keep increasing this allocation forever. What > is your plan for reducing the object size? Packages of this size are > difficult for users to download, and increase our bandwidth costs." > > So they're not too keen on increasing our limit. What do we want to do? We > could rename the lean tars to hadoop-3.4.2.tar.gz, the consequence is that > anyone downloading and trying to use S3A will hit errors, which might be ok > as our release notes say the release no longer includes the SDK. > > > On 01/09/2025, 09:40, "Suhail, Ahmar" <ahma...@amazon.co.uk.inva <mailto: > ahma...@amazon.co.uk.inva>LID> wrote: > > > CAUTION: This email originated from outside of the organization. Do not > click links or open attachments unless you can confirm the sender and know > the content is safe. > > > > > > > Thanks Shilun, > > > I’ve cut: https://issues.apache.org/jira/browse/INFRA-27182 < > https://issues.apache.org/jira/browse/INFRA-27182>. > > > Will upload the full tarball once this is resolved. > > > From: slfan1989 <slfan1...@apache.org <mailto:slfan1...@apache.org>> > Date: Saturday, 30 August 2025 at 09:29 > To: "ah...@apache.org <mailto:ah...@apache.org>" <ah...@apache.org > <mailto:ah...@apache.org>> > Cc: Ayush Saxena <ayush...@gmail.com <mailto:ayush...@gmail.com>>, > Xiaoqiao He <hexiaoq...@apache.org <mailto:hexiaoq...@apache.org>>, Steve > Loughran <ste...@cloudera.com <mailto:ste...@cloudera.com>>, Chris > Nauroth <cnaur...@apache.org <mailto:cnaur...@apache.org>>, Hadoop Common > <common-dev@hadoop.apache.org <mailto:common-dev@hadoop.apache.org>>, > yarn-dev <yarn-...@hadoop.apache.org <mailto:yarn-...@hadoop.apache.org>>, > mapreduce-dev <mapreduce-...@hadoop.apache.org <mailto: > mapreduce-...@hadoop.apache.org>>, Hdfs-dev <hdfs-...@hadoop.apache.org > <mailto:hdfs-...@hadoop.apache.org>>, "<priv...@hadoop.apache.org <mailto: > priv...@hadoop.apache.org>>" <priv...@hadoop.apache.org <mailto: > priv...@hadoop.apache.org>> > Subject: [EXTERNAL] Missing full tarball for Hadoop 3.4.2 > > > > > CAUTION: This email originated from outside of the organization. Do not > click links or open attachments unless you can confirm the sender and know > the content is safe. > > > Hi all, > > > First of all, thanks to everyone for the efforts that led to the > successful release of Hadoop 3.4.2. > > > While trying to use the new dependency in the Hive project, Ayush noticed > that the full Hadoop 3.4.2 tarball was not uploaded. > > > We can find the reference discussion here: > https://github.com/apache/hive/pull/6049 < > https://github.com/apache/hive/pull/6049> > > > After discussing with Ahmar, we plan to report this issue to INFRA, since > they previously helped us by temporarily extending the storage quota for > the release directory. > > > Here is the JIRA I have submitted: > https://issues.apache.org/jira/projects/INFRA/issues/INFRA-25423 < > https://issues.apache.org/jira/projects/INFRA/issues/INFRA-25423> > > > Best regards, > Shilun Fan. > > >
