Re: [Discuss] Analytics Accelerator Library for Amazon S3 as default S3 Input Stream

[Stubbs, Michael]

Thanks to everyone involved so far!
We added in the read support work that was agreed upon: 
https://github.com/apache/iceberg/pull/13997/

I have cut a pr that has the most recent changes for AAL with read vectored 
here: https://github.com/apache/iceberg/pull/14247

We have also started having conversations with Netflix on testing AAL with 
their workloads.
Could we create a follow up community sync to just go over our closed/in 
progress action items and to make sure we all are still aligned on the path 
forward.

Kind regards
Mike Stubbs – Amazon S3

From: Stubbs, Michael 
Date: Friday, 5 September 2025 at 15:44
To: dev@iceberg.apache.org 
Subject: RE: [EXTERNAL] [Discuss] Analytics Accelerator Library for Amazon S3 
as default S3 Input Stream

CAUTION: This email originated from outside of the organization. Do not click 
links or open attachments unless you can confirm the sender and know the 
content is safe.

Hi everyone we have a pr up for the first action item assigned to us:
https://github.com/apache/iceberg/pull/13997/

We will get started on the async client action item next.

From: Stubbs, Michael 
Date: Friday, 29 August 2025 at 09:34
To: dev@iceberg.apache.org 
Subject: RE: [EXTERNAL] [Discuss] Analytics Accelerator Library for Amazon S3 
as default S3 Input Stream

CAUTION: This email originated from outside of the organization. Do not click 
links or open attachments unless you can confirm the sender and know the 
content is safe.

Thanks to everyone who attended.
Notes/AI's from the meeting:

  *   Create epic on Iceberg Repo for this

  *   Look at implementation of read vectored

  *   Look at extending customer testing

  *   Async client when and when not to use it

  *   A Spark-Level Heap usage analysis with and without AAL

  *Public document on how to benchmark AAL would help folks to start using 
it

  *   Reach out to 3rd party store owners to test with AAL


Let me know if there is anything I missed!
Looking forward to the ongoing discussions.


From: Stubbs, Michael 
Date: Monday, 25 August 2025 at 20:23
To: dev@iceberg.apache.org 
Subject: RE: [EXTERNAL] [Discuss] Analytics Accelerator Library for Amazon S3 
as default S3 Input Stream

CAUTION: This email originated from outside of the organization. Do not click 
links or open attachments unless you can confirm the sender and know the 
content is safe.

Hi all I created meeting: 
https://calendar.google.com/calendar/u/0/share?slt=1AQOvs_VoS7SBdbchLznvLXrMfryyXGtIaAggTbcukopMCFyFVCWI7RDHoZpGSztnFA7AvkGaU57MAA
Please come with any questions you have for S3FileIO and the Analytics 
Accelerator Library.

From: "Basik, Fuat" 
Reply to: "dev@iceberg.apache.org" 
Date: Thursday, 21 August 2025 at 09:37
To: "dev@iceberg.apache.org" 
Subject: RE: [EXTERNAL] [Discuss] Analytics Accelerator Library for Amazon S3 
as default S3 Input Stream


CAUTION: This email originated from outside of the organization. Do not click 
links or open attachments unless you can confirm the sender and know the 
content is safe.

I would like to join and 27th works well for me.

Best,
Fuat.

From: "Stubbs, Michael" 
Reply-To: "dev@iceberg.apache.org" 
Date: Tuesday, 19 August 2025 at 14:34
To: "dev@iceberg.apache.org" 
Subject: RE: [EXTERNAL] [Discuss] Analytics Accelerator Library for Amazon S3 
as default S3 Input Stream


CAUTION: This email originated from outside of the organization. Do not click 
links or open attachments unless you can confirm the sender and know the 
content is safe.

Okay I am looking at scheduling a discussion on this for Wednesday the 27th at 
4:00 – 5:00pm UTC. Does this not work for anyone who would like to attend?

From: Tushar Choudhary 
Reply to: "dev@iceberg.apache.org" 
Date: Thursday, 14 August 2025 at 04:39
To: "dev@iceberg.apache.org" 
Subject: RE: [EXTERNAL] [Discuss] Analytics Accelerator Library for Amazon S3 
as default S3 Input Stream


CAUTION: This email originated from outside of the organization. Do not click 
links or open attachments unless you can confirm the sender and know the 
content is safe.

I agree
Cheers,
Tushar Choudhary


On Wed, 13 Aug 2025 at 10:39 PM, Kevin Liu 
mailto:kevinjq...@apache.org>> wrote:
Hey everyone,

As discussed on the community sync today, there's enough interest around AAL, 
S3FileIO, and FileIO in general that we would like to schedule an ad hoc sync 
for this topic.

I'll work with Michael to find a suitable time. We'll add it to the "Iceberg 
Dev 
Events"
 and also post on the devlist when we figure out more details.

Best,
Kevin Liu

On Mon, Aug 11, 2025 at 8:00 AM Kevin Liu 
mailto:kevinjq...@apache.org>> wrote:
Let's bring this up in the next community sync on Wed 8/13 and see if it 
warrants another adhoc sync about FileIO.
Best,
Kevin Liu

On Wed, Aug 6, 2025 at 1:21 PM Stubbs, Michael  
wrote:
I think Kevin has raised a few good points on the future of FileIO and the 
maintainability of the pro

Re: [Discuss] Analytics Accelerator Library for Amazon S3 as default S3 Input Stream

[Stubbs, Michael]

Hi everyone we have a pr up for the first action item assigned to us:
https://github.com/apache/iceberg/pull/13997/

We will get started on the async client action item next.

From: Stubbs, Michael 
Date: Friday, 29 August 2025 at 09:34
To: dev@iceberg.apache.org 
Subject: RE: [EXTERNAL] [Discuss] Analytics Accelerator Library for Amazon S3 
as default S3 Input Stream


CAUTION: This email originated from outside of the organization. Do not click 
links or open attachments unless you can confirm the sender and know the 
content is safe.

Thanks to everyone who attended.
Notes/AI's from the meeting:

  *
Create epic on Iceberg Repo for this
  *
Look at implementation of read vectored
  *
Look at extending customer testing
  *
Async client when and when not to use it
  *
A Spark-Level Heap usage analysis with and without AAL
  *
 Public document on how to benchmark AAL would help folks to start using it
  *
Reach out to 3rd party store owners to test with AAL



Let me know if there is anything I missed!
Looking forward to the ongoing discussions.


From: Stubbs, Michael 
Date: Monday, 25 August 2025 at 20:23
To: dev@iceberg.apache.org 
Subject: RE: [EXTERNAL] [Discuss] Analytics Accelerator Library for Amazon S3 
as default S3 Input Stream


CAUTION: This email originated from outside of the organization. Do not click 
links or open attachments unless you can confirm the sender and know the 
content is safe.

Hi all I created meeting: 
https://calendar.google.com/calendar/u/0/share?slt=1AQOvs_VoS7SBdbchLznvLXrMfryyXGtIaAggTbcukopMCFyFVCWI7RDHoZpGSztnFA7AvkGaU57MAA
Please come with any questions you have for S3FileIO and the Analytics 
Accelerator Library.

From: "Basik, Fuat" 
Reply to: "dev@iceberg.apache.org" 
Date: Thursday, 21 August 2025 at 09:37
To: "dev@iceberg.apache.org" 
Subject: RE: [EXTERNAL] [Discuss] Analytics Accelerator Library for Amazon S3 
as default S3 Input Stream


CAUTION: This email originated from outside of the organization. Do not click 
links or open attachments unless you can confirm the sender and know the 
content is safe.

I would like to join and 27th works well for me.

Best,
Fuat.

From: "Stubbs, Michael" 
Reply-To: "dev@iceberg.apache.org" 
Date: Tuesday, 19 August 2025 at 14:34
To: "dev@iceberg.apache.org" 
Subject: RE: [EXTERNAL] [Discuss] Analytics Accelerator Library for Amazon S3 
as default S3 Input Stream


CAUTION: This email originated from outside of the organization. Do not click 
links or open attachments unless you can confirm the sender and know the 
content is safe.

Okay I am looking at scheduling a discussion on this for Wednesday the 27th at 
4:00 – 5:00pm UTC. Does this not work for anyone who would like to attend?

From: Tushar Choudhary 
Reply to: "dev@iceberg.apache.org" 
Date: Thursday, 14 August 2025 at 04:39
To: "dev@iceberg.apache.org" 
Subject: RE: [EXTERNAL] [Discuss] Analytics Accelerator Library for Amazon S3 
as default S3 Input Stream


CAUTION: This email originated from outside of the organization. Do not click 
links or open attachments unless you can confirm the sender and know the 
content is safe.

I agree

Cheers,
Tushar Choudhary


On Wed, 13 Aug 2025 at 10:39 PM, Kevin Liu 
mailto:kevinjq...@apache.org>> wrote:
Hey everyone,

As discussed on the community sync today, there's enough interest around AAL, 
S3FileIO, and FileIO in general that we would like to schedule an ad hoc sync 
for this topic.

I'll work with Michael to find a suitable time. We'll add it to the "Iceberg 
Dev 
Events"
 and also post on the devlist when we figure out more details.

Best,
Kevin Liu

On Mon, Aug 11, 2025 at 8:00 AM Kevin Liu 
mailto:kevinjq...@apache.org>> wrote:
Let's bring this up in the next community sync on Wed 8/13 and see if it 
warrants another adhoc sync about FileIO.
Best,
Kevin Liu

On Wed, Aug 6, 2025 at 1:21 PM Stubbs, Michael  
wrote:
I think Kevin has raised a few good points on the future of FileIO and the 
maintainability of the project going forwarded with the AAL default on Proposal.
I think we should schedule community sync about this.

Thank you!

On 2025/07/31 13:12:48 Steve Loughran wrote:
> On Fri, 25 Jul 2025 at 17:28, Kevin Liu 
> mailto:ke...@apache.org>> wrote:
>
> *> I think it would be great to also make these improvements available to
> older Iceberg clients.*
>
> Use the S3A connector and turn on vector reads through parquert and you
> currently get the same performance, about at 30% speedup in TPC benchmarks
> (I know, but what else do we have?). S3A connector is going to to move to
> making the AAL input stream the default in a future release because it's a
> better architeture overall.
>
> the vector IO stuff can also do speedup on azire and abfs if their
> connectors support it. (oh and local fs too, FWIW). scatter/gather IO for
> the win.
>
> what AAL adds is format awareness, which could allow for extra
> opportunities.
>
> As an aside, it'

Re: [Discuss] Analytics Accelerator Library for Amazon S3 as default S3 Input Stream

[Stubbs, Michael]

Thanks to everyone who attended.
Notes/AI's from the meeting:

  *
Create epic on Iceberg Repo for this
  *
Look at implementation of read vectored
  *
Look at extending customer testing
  *
Async client when and when not to use it
  *
A Spark-Level Heap usage analysis with and without AAL
  *
 Public document on how to benchmark AAL would help folks to start using it
  *
Reach out to 3rd party store owners to test with AAL



Let me know if there is anything I missed!
Looking forward to the ongoing discussions.


From: Stubbs, Michael 
Date: Monday, 25 August 2025 at 20:23
To: dev@iceberg.apache.org 
Subject: RE: [EXTERNAL] [Discuss] Analytics Accelerator Library for Amazon S3 
as default S3 Input Stream


CAUTION: This email originated from outside of the organization. Do not click 
links or open attachments unless you can confirm the sender and know the 
content is safe.

Hi all I created meeting: 
https://calendar.google.com/calendar/u/0/share?slt=1AQOvs_VoS7SBdbchLznvLXrMfryyXGtIaAggTbcukopMCFyFVCWI7RDHoZpGSztnFA7AvkGaU57MAA
Please come with any questions you have for S3FileIO and the Analytics 
Accelerator Library.

From: "Basik, Fuat" 
Reply to: "dev@iceberg.apache.org" 
Date: Thursday, 21 August 2025 at 09:37
To: "dev@iceberg.apache.org" 
Subject: RE: [EXTERNAL] [Discuss] Analytics Accelerator Library for Amazon S3 
as default S3 Input Stream


CAUTION: This email originated from outside of the organization. Do not click 
links or open attachments unless you can confirm the sender and know the 
content is safe.

I would like to join and 27th works well for me.

Best,
Fuat.

From: "Stubbs, Michael" 
Reply-To: "dev@iceberg.apache.org" 
Date: Tuesday, 19 August 2025 at 14:34
To: "dev@iceberg.apache.org" 
Subject: RE: [EXTERNAL] [Discuss] Analytics Accelerator Library for Amazon S3 
as default S3 Input Stream


CAUTION: This email originated from outside of the organization. Do not click 
links or open attachments unless you can confirm the sender and know the 
content is safe.

Okay I am looking at scheduling a discussion on this for Wednesday the 27th at 
4:00 – 5:00pm UTC. Does this not work for anyone who would like to attend?

From: Tushar Choudhary 
Reply to: "dev@iceberg.apache.org" 
Date: Thursday, 14 August 2025 at 04:39
To: "dev@iceberg.apache.org" 
Subject: RE: [EXTERNAL] [Discuss] Analytics Accelerator Library for Amazon S3 
as default S3 Input Stream


CAUTION: This email originated from outside of the organization. Do not click 
links or open attachments unless you can confirm the sender and know the 
content is safe.

I agree

Cheers,
Tushar Choudhary


On Wed, 13 Aug 2025 at 10:39 PM, Kevin Liu 
mailto:kevinjq...@apache.org>> wrote:
Hey everyone,

As discussed on the community sync today, there's enough interest around AAL, 
S3FileIO, and FileIO in general that we would like to schedule an ad hoc sync 
for this topic.

I'll work with Michael to find a suitable time. We'll add it to the "Iceberg 
Dev 
Events"
 and also post on the devlist when we figure out more details.

Best,
Kevin Liu

On Mon, Aug 11, 2025 at 8:00 AM Kevin Liu 
mailto:kevinjq...@apache.org>> wrote:
Let's bring this up in the next community sync on Wed 8/13 and see if it 
warrants another adhoc sync about FileIO.
Best,
Kevin Liu

On Wed, Aug 6, 2025 at 1:21 PM Stubbs, Michael  
wrote:
I think Kevin has raised a few good points on the future of FileIO and the 
maintainability of the project going forwarded with the AAL default on Proposal.
I think we should schedule community sync about this.

Thank you!

On 2025/07/31 13:12:48 Steve Loughran wrote:
> On Fri, 25 Jul 2025 at 17:28, Kevin Liu 
> mailto:ke...@apache.org>> wrote:
>
> *> I think it would be great to also make these improvements available to
> older Iceberg clients.*
>
> Use the S3A connector and turn on vector reads through parquert and you
> currently get the same performance, about at 30% speedup in TPC benchmarks
> (I know, but what else do we have?). S3A connector is going to to move to
> making the AAL input stream the default in a future release because it's a
> better architeture overall.
>
> the vector IO stuff can also do speedup on azire and abfs if their
> connectors support it. (oh and local fs too, FWIW). scatter/gather IO for
> the win.
>
> what AAL adds is format awareness, which could allow for extra
> opportunities.
>
> As an aside, it'd be really good if FileIO added an overload newInputFile()
> api call which passed in the file type too, so that AAL &c would know what
> type to optimise for, rather than just guess of the extension. knowing what
> the v1 and maybe v2 schema offsets would save that GET request on the
> footter. AAL does a GET of a range at the bottom with the goal of including
> the schema, but knowing the exact range would be better.
>
> *> BTW, we have one-off community syncs about specific topics, I would be
> interested to talk more about this as we

Re: [Discuss] Analytics Accelerator Library for Amazon S3 as default S3 Input Stream

[Kevin Liu]

Thanks Michael. As a follow up to the last community sync, we're creating a
dedicated meeting to discuss the Analytics Accelerator Library that the S3
team has been working on.

I just added the meeting to the "Iceberg Dev Event" calendar:

S3 Analytics and S3FileIO Dev Sync
Wednesday, August 27 · 9:00 – 10:00am
Time zone: America/Los_Angeles
Google Meet joining info
Video call link: https://meet.google.com/eeu-xyqh-muy

Looking forward to it.

Best,
Kevin Liu

On Mon, Aug 25, 2025 at 12:24 PM Stubbs, Michael
 wrote:

> Hi all I created meeting:
> https://calendar.google.com/calendar/u/0/share?slt=1AQOvs_VoS7SBdbchLznvLXrMfryyXGtIaAggTbcukopMCFyFVCWI7RDHoZpGSztnFA7AvkGaU57MAA
> Please come with any questions you have for S3FileIO and the Analytics
> Accelerator Library.
>
>
>
> *From: *"Basik, Fuat" 
> *Reply to: *"dev@iceberg.apache.org" 
> *Date: *Thursday, 21 August 2025 at 09:37
> *To: *"dev@iceberg.apache.org" 
> *Subject: *RE: [EXTERNAL] [Discuss] Analytics Accelerator Library for
> Amazon S3 as default S3 Input Stream
>
>
>
> *CAUTION*: This email originated from outside of the organization. Do not
> click links or open attachments unless you can confirm the sender and know
> the content is safe.
>
>
>
> I would like to join and 27th works well for me.
>
>
>
> Best,
>
> Fuat.
>
>
>
> *From: *"Stubbs, Michael" 
> *Reply-To: *"dev@iceberg.apache.org" 
> *Date: *Tuesday, 19 August 2025 at 14:34
> *To: *"dev@iceberg.apache.org" 
> *Subject: *RE: [EXTERNAL] [Discuss] Analytics Accelerator Library for
> Amazon S3 as default S3 Input Stream
>
>
>
> *CAUTION*: This email originated from outside of the organization. Do not
> click links or open attachments unless you can confirm the sender and know
> the content is safe.
>
>
>
> Okay I am looking at scheduling a discussion on this for Wednesday the 27
> th at 4:00 – 5:00pm UTC. Does this not work for anyone who would like to
> attend?
>
>
>
> *From: *Tushar Choudhary 
> *Reply to: *"dev@iceberg.apache.org" 
> *Date: *Thursday, 14 August 2025 at 04:39
> *To: *"dev@iceberg.apache.org" 
> *Subject: *RE: [EXTERNAL] [Discuss] Analytics Accelerator Library for
> Amazon S3 as default S3 Input Stream
>
>
>
> *CAUTION*: This email originated from outside of the organization. Do not
> click links or open attachments unless you can confirm the sender and know
> the content is safe.
>
>
>
> I agree
>
> Cheers,
>
> Tushar Choudhary
>
>
>
>
>
> On Wed, 13 Aug 2025 at 10:39 PM, Kevin Liu  wrote:
>
> Hey everyone,
>
>
>
> As discussed on the community sync today, there's enough interest around
> AAL, S3FileIO, and FileIO in general that we would like to schedule an ad
> hoc sync for this topic.
>
>
>
> I'll work with Michael to find a suitable time. We'll add it to the "Iceberg
> Dev Events"
>  and
> also post on the devlist when we figure out more details.
>
>
>
> Best,
>
> Kevin Liu
>
>
>
> On Mon, Aug 11, 2025 at 8:00 AM Kevin Liu  wrote:
>
> Let's bring this up in the next community sync on Wed 8/13 and see if it
> warrants another adhoc sync about FileIO.
>
> Best,
>
> Kevin Liu
>
>
>
> On Wed, Aug 6, 2025 at 1:21 PM Stubbs, Michael
>  wrote:
>
> I think Kevin has raised a few good points on the future of FileIO and the
> maintainability of the project going forwarded with the AAL default on
> Proposal.
>
> I think we should schedule community sync about this.
>
> Thank you!
>
>
>
> On 2025/07/31 13:12:48 Steve Loughran wrote:
>
> > On Fri, 25 Jul 2025 at 17:28, Kevin Liu  wrote:
>
> >
>
> > *> I think it would be great to also make these improvements available to
>
> > older Iceberg clients.*
>
> >
>
> > Use the S3A connector and turn on vector reads through parquert and you
>
> > currently get the same performance, about at 30% speedup in TPC
> benchmarks
>
> > (I know, but what else do we have?). S3A connector is going to to move to
>
> > making the AAL input stream the default in a future release because it's
> a
>
> > better architeture overall.
>
> >
>
> > the vector IO stuff can also do speedup on azire and abfs if their
>
> > connectors support it. (oh and local fs too, FWIW). scatter/gather IO for
>
> > the win.
>
> >
>
> > what AAL adds is format awareness, which could allow for extra
>
> > opportunities.
>
> >
>
> > As an aside, it'd be really good if FileIO added an overload
> newInputFile()
>
> > api call which passed in the file type too, so that AAL &c would know
> what
>
> > type to optimise for, rather than just guess of the extension. knowing
> what
>
> > the v1 and maybe v2 schema offsets would save that GET request on the
>
> > footter. AAL does a GET of a range at the bottom with the goal of
> including
>
> > the schema, but knowing the exact range would be better.
>
> >
>
> > *> BTW, we have one-off community syncs about specific topics, I would be
>
> > interested to talk more about this as well as other FileIOs. We use the
>
> > "Iceberg Dev Events" calendar for scheduling 

Re: [Discuss] Analytics Accelerator Library for Amazon S3 as default S3 Input Stream

[Stubbs, Michael]

Hi all I created meeting: 
https://calendar.google.com/calendar/u/0/share?slt=1AQOvs_VoS7SBdbchLznvLXrMfryyXGtIaAggTbcukopMCFyFVCWI7RDHoZpGSztnFA7AvkGaU57MAA
Please come with any questions you have for S3FileIO and the Analytics 
Accelerator Library.

From: "Basik, Fuat" 
Reply to: "dev@iceberg.apache.org" 
Date: Thursday, 21 August 2025 at 09:37
To: "dev@iceberg.apache.org" 
Subject: RE: [EXTERNAL] [Discuss] Analytics Accelerator Library for Amazon S3 
as default S3 Input Stream


CAUTION: This email originated from outside of the organization. Do not click 
links or open attachments unless you can confirm the sender and know the 
content is safe.

I would like to join and 27th works well for me.

Best,
Fuat.

From: "Stubbs, Michael" 
Reply-To: "dev@iceberg.apache.org" 
Date: Tuesday, 19 August 2025 at 14:34
To: "dev@iceberg.apache.org" 
Subject: RE: [EXTERNAL] [Discuss] Analytics Accelerator Library for Amazon S3 
as default S3 Input Stream


CAUTION: This email originated from outside of the organization. Do not click 
links or open attachments unless you can confirm the sender and know the 
content is safe.

Okay I am looking at scheduling a discussion on this for Wednesday the 27th at 
4:00 – 5:00pm UTC. Does this not work for anyone who would like to attend?

From: Tushar Choudhary 
Reply to: "dev@iceberg.apache.org" 
Date: Thursday, 14 August 2025 at 04:39
To: "dev@iceberg.apache.org" 
Subject: RE: [EXTERNAL] [Discuss] Analytics Accelerator Library for Amazon S3 
as default S3 Input Stream


CAUTION: This email originated from outside of the organization. Do not click 
links or open attachments unless you can confirm the sender and know the 
content is safe.

I agree

Cheers,
Tushar Choudhary


On Wed, 13 Aug 2025 at 10:39 PM, Kevin Liu 
mailto:kevinjq...@apache.org>> wrote:
Hey everyone,

As discussed on the community sync today, there's enough interest around AAL, 
S3FileIO, and FileIO in general that we would like to schedule an ad hoc sync 
for this topic.

I'll work with Michael to find a suitable time. We'll add it to the "Iceberg 
Dev 
Events"
 and also post on the devlist when we figure out more details.

Best,
Kevin Liu

On Mon, Aug 11, 2025 at 8:00 AM Kevin Liu 
mailto:kevinjq...@apache.org>> wrote:
Let's bring this up in the next community sync on Wed 8/13 and see if it 
warrants another adhoc sync about FileIO.
Best,
Kevin Liu

On Wed, Aug 6, 2025 at 1:21 PM Stubbs, Michael  
wrote:
I think Kevin has raised a few good points on the future of FileIO and the 
maintainability of the project going forwarded with the AAL default on Proposal.
I think we should schedule community sync about this.

Thank you!

On 2025/07/31 13:12:48 Steve Loughran wrote:
> On Fri, 25 Jul 2025 at 17:28, Kevin Liu 
> mailto:ke...@apache.org>> wrote:
>
> *> I think it would be great to also make these improvements available to
> older Iceberg clients.*
>
> Use the S3A connector and turn on vector reads through parquert and you
> currently get the same performance, about at 30% speedup in TPC benchmarks
> (I know, but what else do we have?). S3A connector is going to to move to
> making the AAL input stream the default in a future release because it's a
> better architeture overall.
>
> the vector IO stuff can also do speedup on azire and abfs if their
> connectors support it. (oh and local fs too, FWIW). scatter/gather IO for
> the win.
>
> what AAL adds is format awareness, which could allow for extra
> opportunities.
>
> As an aside, it'd be really good if FileIO added an overload newInputFile()
> api call which passed in the file type too, so that AAL &c would know what
> type to optimise for, rather than just guess of the extension. knowing what
> the v1 and maybe v2 schema offsets would save that GET request on the
> footter. AAL does a GET of a range at the bottom with the goal of including
> the schema, but knowing the exact range would be better.
>
> *> BTW, we have one-off community syncs about specific topics, I would be
> interested to talk more about this as well as other FileIOs. We use the
> "Iceberg Dev Events" calendar for scheduling if there's interest.*
>
> I"d like that too.
>
>
> >
>

Re: [Discuss] Analytics Accelerator Library for Amazon S3 as default S3 Input Stream

[Basik, Fuat]

I would like to join and 27th works well for me.

Best,
Fuat.

From: "Stubbs, Michael" 
Reply-To: "dev@iceberg.apache.org" 
Date: Tuesday, 19 August 2025 at 14:34
To: "dev@iceberg.apache.org" 
Subject: RE: [EXTERNAL] [Discuss] Analytics Accelerator Library for Amazon S3 
as default S3 Input Stream


CAUTION: This email originated from outside of the organization. Do not click 
links or open attachments unless you can confirm the sender and know the 
content is safe.

Okay I am looking at scheduling a discussion on this for Wednesday the 27th at 
4:00 – 5:00pm UTC. Does this not work for anyone who would like to attend?

From: Tushar Choudhary 
Reply to: "dev@iceberg.apache.org" 
Date: Thursday, 14 August 2025 at 04:39
To: "dev@iceberg.apache.org" 
Subject: RE: [EXTERNAL] [Discuss] Analytics Accelerator Library for Amazon S3 
as default S3 Input Stream


CAUTION: This email originated from outside of the organization. Do not click 
links or open attachments unless you can confirm the sender and know the 
content is safe.

I agree

Cheers,
Tushar Choudhary


On Wed, 13 Aug 2025 at 10:39 PM, Kevin Liu 
mailto:kevinjq...@apache.org>> wrote:
Hey everyone,

As discussed on the community sync today, there's enough interest around AAL, 
S3FileIO, and FileIO in general that we would like to schedule an ad hoc sync 
for this topic.

I'll work with Michael to find a suitable time. We'll add it to the "Iceberg 
Dev 
Events"
 and also post on the devlist when we figure out more details.

Best,
Kevin Liu

On Mon, Aug 11, 2025 at 8:00 AM Kevin Liu 
mailto:kevinjq...@apache.org>> wrote:
Let's bring this up in the next community sync on Wed 8/13 and see if it 
warrants another adhoc sync about FileIO.
Best,
Kevin Liu

On Wed, Aug 6, 2025 at 1:21 PM Stubbs, Michael  
wrote:
I think Kevin has raised a few good points on the future of FileIO and the 
maintainability of the project going forwarded with the AAL default on Proposal.
I think we should schedule community sync about this.

Thank you!

On 2025/07/31 13:12:48 Steve Loughran wrote:
> On Fri, 25 Jul 2025 at 17:28, Kevin Liu 
> mailto:ke...@apache.org>> wrote:
>
> *> I think it would be great to also make these improvements available to
> older Iceberg clients.*
>
> Use the S3A connector and turn on vector reads through parquert and you
> currently get the same performance, about at 30% speedup in TPC benchmarks
> (I know, but what else do we have?). S3A connector is going to to move to
> making the AAL input stream the default in a future release because it's a
> better architeture overall.
>
> the vector IO stuff can also do speedup on azire and abfs if their
> connectors support it. (oh and local fs too, FWIW). scatter/gather IO for
> the win.
>
> what AAL adds is format awareness, which could allow for extra
> opportunities.
>
> As an aside, it'd be really good if FileIO added an overload newInputFile()
> api call which passed in the file type too, so that AAL &c would know what
> type to optimise for, rather than just guess of the extension. knowing what
> the v1 and maybe v2 schema offsets would save that GET request on the
> footter. AAL does a GET of a range at the bottom with the goal of including
> the schema, but knowing the exact range would be better.
>
> *> BTW, we have one-off community syncs about specific topics, I would be
> interested to talk more about this as well as other FileIOs. We use the
> "Iceberg Dev Events" calendar for scheduling if there's interest.*
>
> I"d like that too.
>
>
> >
>

Re: [Discuss] Analytics Accelerator Library for Amazon S3 as default S3 Input Stream

[Stubbs, Michael]

Okay I am looking at scheduling a discussion on this for Wednesday the 27th at 
4:00 – 5:00pm UTC. Does this not work for anyone who would like to attend?

From: Tushar Choudhary 
Reply to: "dev@iceberg.apache.org" 
Date: Thursday, 14 August 2025 at 04:39
To: "dev@iceberg.apache.org" 
Subject: RE: [EXTERNAL] [Discuss] Analytics Accelerator Library for Amazon S3 
as default S3 Input Stream


CAUTION: This email originated from outside of the organization. Do not click 
links or open attachments unless you can confirm the sender and know the 
content is safe.

I agree

Cheers,
Tushar Choudhary


On Wed, 13 Aug 2025 at 10:39 PM, Kevin Liu 
mailto:kevinjq...@apache.org>> wrote:
Hey everyone,

As discussed on the community sync today, there's enough interest around AAL, 
S3FileIO, and FileIO in general that we would like to schedule an ad hoc sync 
for this topic.

I'll work with Michael to find a suitable time. We'll add it to the "Iceberg 
Dev 
Events"
 and also post on the devlist when we figure out more details.

Best,
Kevin Liu

On Mon, Aug 11, 2025 at 8:00 AM Kevin Liu 
mailto:kevinjq...@apache.org>> wrote:
Let's bring this up in the next community sync on Wed 8/13 and see if it 
warrants another adhoc sync about FileIO.
Best,
Kevin Liu

On Wed, Aug 6, 2025 at 1:21 PM Stubbs, Michael  
wrote:
I think Kevin has raised a few good points on the future of FileIO and the 
maintainability of the project going forwarded with the AAL default on Proposal.
I think we should schedule community sync about this.

Thank you!

On 2025/07/31 13:12:48 Steve Loughran wrote:
> On Fri, 25 Jul 2025 at 17:28, Kevin Liu 
> mailto:ke...@apache.org>> wrote:
>
> *> I think it would be great to also make these improvements available to
> older Iceberg clients.*
>
> Use the S3A connector and turn on vector reads through parquert and you
> currently get the same performance, about at 30% speedup in TPC benchmarks
> (I know, but what else do we have?). S3A connector is going to to move to
> making the AAL input stream the default in a future release because it's a
> better architeture overall.
>
> the vector IO stuff can also do speedup on azire and abfs if their
> connectors support it. (oh and local fs too, FWIW). scatter/gather IO for
> the win.
>
> what AAL adds is format awareness, which could allow for extra
> opportunities.
>
> As an aside, it'd be really good if FileIO added an overload newInputFile()
> api call which passed in the file type too, so that AAL &c would know what
> type to optimise for, rather than just guess of the extension. knowing what
> the v1 and maybe v2 schema offsets would save that GET request on the
> footter. AAL does a GET of a range at the bottom with the goal of including
> the schema, but knowing the exact range would be better.
>
> *> BTW, we have one-off community syncs about specific topics, I would be
> interested to talk more about this as well as other FileIOs. We use the
> "Iceberg Dev Events" calendar for scheduling if there's interest.*
>
> I"d like that too.
>
>
> >
>

Re: Re: [Discuss] Analytics Accelerator Library for Amazon S3 as default S3 Input Stream

[Tushar Choudhary]

I agree

Cheers,
Tushar Choudhary


On Wed, 13 Aug 2025 at 10:39 PM, Kevin Liu  wrote:

> Hey everyone,
>
> As discussed on the community sync today, there's enough interest around
> AAL, S3FileIO, and FileIO in general that we would like to schedule an ad
> hoc sync for this topic.
>
> I'll work with Michael to find a suitable time. We'll add it to the "Iceberg
> Dev Events"
>  and
> also post on the devlist when we figure out more details.
>
> Best,
> Kevin Liu
>
> On Mon, Aug 11, 2025 at 8:00 AM Kevin Liu  wrote:
>
>> Let's bring this up in the next community sync on Wed 8/13 and see if it
>> warrants another adhoc sync about FileIO.
>>
>> Best,
>> Kevin Liu
>>
>> On Wed, Aug 6, 2025 at 1:21 PM Stubbs, Michael
>>  wrote:
>>
>>> I think Kevin has raised a few good points on the future of FileIO and
>>> the maintainability of the project going forwarded with the AAL default on
>>> Proposal.
>>>
>>> I think we should schedule community sync about this.
>>>
>>> Thank you!
>>>
>>>
>>>
>>> On 2025/07/31 13:12:48 Steve Loughran wrote:
>>>
>>> > On Fri, 25 Jul 2025 at 17:28, Kevin Liu  wrote:
>>>
>>> >
>>>
>>> > *> I think it would be great to also make these improvements available
>>> to
>>>
>>> > older Iceberg clients.*
>>>
>>> >
>>>
>>> > Use the S3A connector and turn on vector reads through parquert and you
>>>
>>> > currently get the same performance, about at 30% speedup in TPC
>>> benchmarks
>>>
>>> > (I know, but what else do we have?). S3A connector is going to to move
>>> to
>>>
>>> > making the AAL input stream the default in a future release because
>>> it's a
>>>
>>> > better architeture overall.
>>>
>>> >
>>>
>>> > the vector IO stuff can also do speedup on azire and abfs if their
>>>
>>> > connectors support it. (oh and local fs too, FWIW). scatter/gather IO
>>> for
>>>
>>> > the win.
>>>
>>> >
>>>
>>> > what AAL adds is format awareness, which could allow for extra
>>>
>>> > opportunities.
>>>
>>> >
>>>
>>> > As an aside, it'd be really good if FileIO added an overload
>>> newInputFile()
>>>
>>> > api call which passed in the file type too, so that AAL &c would know
>>> what
>>>
>>> > type to optimise for, rather than just guess of the extension. knowing
>>> what
>>>
>>> > the v1 and maybe v2 schema offsets would save that GET request on the
>>>
>>> > footter. AAL does a GET of a range at the bottom with the goal of
>>> including
>>>
>>> > the schema, but knowing the exact range would be better.
>>>
>>> >
>>>
>>> > *> BTW, we have one-off community syncs about specific topics, I would
>>> be
>>>
>>> > interested to talk more about this as well as other FileIOs. We use the
>>>
>>> > "Iceberg Dev Events" calendar for scheduling if there's interest.*
>>>
>>> >
>>>
>>> > I"d like that too.
>>>
>>> >
>>>
>>> >
>>>
>>> > >
>>>
>>> >
>>>
>>

Re: Re: [Discuss] Analytics Accelerator Library for Amazon S3 as default S3 Input Stream

[Kevin Liu]

Hey everyone,

As discussed on the community sync today, there's enough interest around
AAL, S3FileIO, and FileIO in general that we would like to schedule an ad
hoc sync for this topic.

I'll work with Michael to find a suitable time. We'll add it to the "Iceberg
Dev Events"
 and
also post on the devlist when we figure out more details.

Best,
Kevin Liu

On Mon, Aug 11, 2025 at 8:00 AM Kevin Liu  wrote:

> Let's bring this up in the next community sync on Wed 8/13 and see if it
> warrants another adhoc sync about FileIO.
>
> Best,
> Kevin Liu
>
> On Wed, Aug 6, 2025 at 1:21 PM Stubbs, Michael
>  wrote:
>
>> I think Kevin has raised a few good points on the future of FileIO and
>> the maintainability of the project going forwarded with the AAL default on
>> Proposal.
>>
>> I think we should schedule community sync about this.
>>
>> Thank you!
>>
>>
>>
>> On 2025/07/31 13:12:48 Steve Loughran wrote:
>>
>> > On Fri, 25 Jul 2025 at 17:28, Kevin Liu  wrote:
>>
>> >
>>
>> > *> I think it would be great to also make these improvements available
>> to
>>
>> > older Iceberg clients.*
>>
>> >
>>
>> > Use the S3A connector and turn on vector reads through parquert and you
>>
>> > currently get the same performance, about at 30% speedup in TPC
>> benchmarks
>>
>> > (I know, but what else do we have?). S3A connector is going to to move
>> to
>>
>> > making the AAL input stream the default in a future release because
>> it's a
>>
>> > better architeture overall.
>>
>> >
>>
>> > the vector IO stuff can also do speedup on azire and abfs if their
>>
>> > connectors support it. (oh and local fs too, FWIW). scatter/gather IO
>> for
>>
>> > the win.
>>
>> >
>>
>> > what AAL adds is format awareness, which could allow for extra
>>
>> > opportunities.
>>
>> >
>>
>> > As an aside, it'd be really good if FileIO added an overload
>> newInputFile()
>>
>> > api call which passed in the file type too, so that AAL &c would know
>> what
>>
>> > type to optimise for, rather than just guess of the extension. knowing
>> what
>>
>> > the v1 and maybe v2 schema offsets would save that GET request on the
>>
>> > footter. AAL does a GET of a range at the bottom with the goal of
>> including
>>
>> > the schema, but knowing the exact range would be better.
>>
>> >
>>
>> > *> BTW, we have one-off community syncs about specific topics, I would
>> be
>>
>> > interested to talk more about this as well as other FileIOs. We use the
>>
>> > "Iceberg Dev Events" calendar for scheduling if there's interest.*
>>
>> >
>>
>> > I"d like that too.
>>
>> >
>>
>> >
>>
>> > >
>>
>> >
>>
>

Re: Re: [Discuss] Analytics Accelerator Library for Amazon S3 as default S3 Input Stream

[Kevin Liu]

Let's bring this up in the next community sync on Wed 8/13 and see if it
warrants another adhoc sync about FileIO.

Best,
Kevin Liu

On Wed, Aug 6, 2025 at 1:21 PM Stubbs, Michael
 wrote:

> I think Kevin has raised a few good points on the future of FileIO and the
> maintainability of the project going forwarded with the AAL default on
> Proposal.
>
> I think we should schedule community sync about this.
>
> Thank you!
>
>
>
> On 2025/07/31 13:12:48 Steve Loughran wrote:
>
> > On Fri, 25 Jul 2025 at 17:28, Kevin Liu  wrote:
>
> >
>
> > *> I think it would be great to also make these improvements available to
>
> > older Iceberg clients.*
>
> >
>
> > Use the S3A connector and turn on vector reads through parquert and you
>
> > currently get the same performance, about at 30% speedup in TPC
> benchmarks
>
> > (I know, but what else do we have?). S3A connector is going to to move to
>
> > making the AAL input stream the default in a future release because it's
> a
>
> > better architeture overall.
>
> >
>
> > the vector IO stuff can also do speedup on azire and abfs if their
>
> > connectors support it. (oh and local fs too, FWIW). scatter/gather IO for
>
> > the win.
>
> >
>
> > what AAL adds is format awareness, which could allow for extra
>
> > opportunities.
>
> >
>
> > As an aside, it'd be really good if FileIO added an overload
> newInputFile()
>
> > api call which passed in the file type too, so that AAL &c would know
> what
>
> > type to optimise for, rather than just guess of the extension. knowing
> what
>
> > the v1 and maybe v2 schema offsets would save that GET request on the
>
> > footter. AAL does a GET of a range at the bottom with the goal of
> including
>
> > the schema, but knowing the exact range would be better.
>
> >
>
> > *> BTW, we have one-off community syncs about specific topics, I would be
>
> > interested to talk more about this as well as other FileIOs. We use the
>
> > "Iceberg Dev Events" calendar for scheduling if there's interest.*
>
> >
>
> > I"d like that too.
>
> >
>
> >
>
> > >
>
> >
>

RE: Re: [Discuss] Analytics Accelerator Library for Amazon S3 as default S3 Input Stream

[Stubbs, Michael]

I think Kevin has raised a few good points on the future of FileIO and the 
maintainability of the project going forwarded with the AAL default on Proposal.
I think we should schedule community sync about this.

Thank you!


On 2025/07/31 13:12:48 Steve Loughran wrote:
> On Fri, 25 Jul 2025 at 17:28, Kevin Liu 
> mailto:ke...@apache.org>> wrote:
>
> *> I think it would be great to also make these improvements available to
> older Iceberg clients.*
>
> Use the S3A connector and turn on vector reads through parquert and you
> currently get the same performance, about at 30% speedup in TPC benchmarks
> (I know, but what else do we have?). S3A connector is going to to move to
> making the AAL input stream the default in a future release because it's a
> better architeture overall.
>
> the vector IO stuff can also do speedup on azire and abfs if their
> connectors support it. (oh and local fs too, FWIW). scatter/gather IO for
> the win.
>
> what AAL adds is format awareness, which could allow for extra
> opportunities.
>
> As an aside, it'd be really good if FileIO added an overload newInputFile()
> api call which passed in the file type too, so that AAL &c would know what
> type to optimise for, rather than just guess of the extension. knowing what
> the v1 and maybe v2 schema offsets would save that GET request on the
> footter. AAL does a GET of a range at the bottom with the goal of including
> the schema, but knowing the exact range would be better.
>
> *> BTW, we have one-off community syncs about specific topics, I would be
> interested to talk more about this as well as other FileIOs. We use the
> "Iceberg Dev Events" calendar for scheduling if there's interest.*
>
> I"d like that too.
>
>
> >
>

Re: [Discuss] Analytics Accelerator Library for Amazon S3 as default S3 Input Stream

[Steve Loughran]

On Fri, 25 Jul 2025 at 17:28, Kevin Liu  wrote:

*> I think it would be great to also make these improvements available to
older Iceberg clients.*

Use the S3A connector and turn on vector reads through parquert and you
currently get the same performance, about at 30% speedup in TPC benchmarks
(I know, but what else do we have?). S3A connector is going to to move to
making the AAL input stream the default in a future release because it's a
better architeture overall.

the vector IO stuff can also do speedup on azire and abfs if their
connectors support it. (oh and local fs too, FWIW). scatter/gather IO for
the win.

what AAL adds is format awareness, which could allow for extra
opportunities.

As an aside, it'd be really good if FileIO added an overload newInputFile()
api call which passed in the file type too, so that AAL &c would know what
type to optimise for, rather than just guess of the extension. knowing what
the v1 and maybe v2 schema offsets would save that GET request on the
footter. AAL does a GET of a range at the bottom with the goal of including
the schema, but knowing the exact range would be better.

*> BTW, we have one-off community syncs about specific topics, I would be
interested to talk more about this as well as other FileIOs. We use the
"Iceberg Dev Events" calendar for scheduling if there's interest.*

I"d like that too.


>

Re: [Discuss] Analytics Accelerator Library for Amazon S3 as default S3 Input Stream

[Kevin Liu]

Thank you for the write up. This is super interesting. I've added a few
comments on the doc.
I think it would be great to also make these improvements available to
older Iceberg clients.

BTW, we have one-off community syncs about specific topics, I would be
interested to talk more about this as well as other FileIOs. We use
the "Iceberg
Dev Events" calendar
 for
scheduling if there's interest.

Best,
Kevin Liu

On Fri, Jul 25, 2025 at 7:47 AM Stubbs, Michael
 wrote:

> Hi Iceberg community,
>
>
>
> Following up on this thread
> ,
>
> we've done additional testing and would like to propose making the
> Analytics stream the new default stream for S3FileIO.
>
>
>
> To provide more context and outline the reasoning, we've put together a
> proposal document:
>
> 📄 Proposal: Default to Analytics Stream in S3FileIO
> 
>
>
>
> We welcome your feedback — feel free to comment directly in the document
> or reply on the dev list with any thoughts or suggestions.
>
>
>
> Looking forward to the discussion
>
> Michael Stubbs
>