Hi all, This is great news! I agree with Dmitri - the current conversation regarding AWS CN partition support would also be easily resolved if we were able to run against real AWS servers.
+1 also to Dmitri's point: "I believe it would be appropriate to make a new test class / suite dedicated to this kind of integration testing." CloudWatch has a LocalStack container which allows us to test e2e through a local Docker container, so we wouldn't need to use AWS credits on those tests. But, in the future, if we support other similar services in GCP or Azure, credits with those CSPs would come in handy to do this type of testing (not sure if ASF has a partnership with them though). Best, Adnan Hemani On Mon, Nov 17, 2025 at 6:38 AM Dmitri Bourlatchkov <[email protected]> wrote: > Hi All, > > It would be great to test on real AWS servers. > > JB: Is there a doc on how those "credits" are consumed? What are the > factors leading to their use? > > I echo the concern about overusing the AWS resources. I believe most of the > functionality can be tested with containerized S3 providers like Apache > Ozone. > > Real AWS is probably needed only for validating AssumeRole requests and > making a small number of R/W calls. I believe it would be appropriate to > make a new test class / suite dedicated to this kind of integration > testing.RestCatalogS3IT, specifically, might be doing more than what is > minimally required to ensure AWS integration works. > > I'm not up-to-date on Cloud Watch tests, but those may need to be included > too. > > Cheers, > Dmitri. > > On Mon, Nov 17, 2025 at 8:48 AM Alexandre Dutra <[email protected]> wrote: > > > Hi JB, > > > > That's a great idea, thanks for driving this! On our side, I agree > > with Robert that we should make sure we use these resources sparingly. > > > > Thanks, > > Alex > > > > On Mon, Nov 17, 2025 at 5:43 AM Robert Stupp <[email protected]> wrote: > > > > > > That's great! > > > > > > As credentials (secrets) are required, we can use those from (trusted) > > > GH workflows, but not for PRs (`on: pull_request`). > > > > > > I think we should define what should be integration tested, when those > > > tests shall run and how/where the test results are published. > > > > > > Robert > > > > > > > > > > > > On Mon, Nov 17, 2025 at 3:36 AM Jean-Baptiste Onofré <[email protected]> > > wrote: > > > > > > > > Hi folks, > > > > > > > > During the last Polaris Community Meeting, we briefly discussed using > > > > "concrete" AWS instances of integration tests (especially for S3). > > > > > > > > As AWS is a sponsor of the ASF, AWS provides "credits" to ASF > projects. > > > > The process to request/use AWS credits is to ask the ASF infra, who > > > > are the keeper of those credits. It all needs to go through Infra so > > > > that we (The ASF) can keep track of donations, for tax/non-profit > > > > purposes. Projects are discouraged from attempting to get that > > > > directly. > > > > > > > > So, if there are no objections, I will request credit for the Polaris > > > > podling at The ASF Infra. > > > > I will give you more "logistic details" about those credits (how to > > > > actually use them, etc). > > > > > > > > Regards > > > > JB > > >
