steveloughran commented on PR #10292: URL: https://github.com/apache/iceberg/pull/10292#issuecomment-2286808868
> This makes me wonder if we have tested s3 express with HadoopFileIO (where we require version hint and hence listing) as well as ran iceberg maintenance actions (like clean orphan files) with the same I am assuming these ops will require listing as well and if we have suff data points proving that it works out of the box with these subtle differences and no changes in iceberg lib is required to work like we are doing here to fix the integ tests. s3a on hadoop-3.4.0 supported s3 express since the morning of the re:invent keynote speech. Older releases use the v1 sdk, so don't. w.r.t clean orphan files, #18048 hands down bulk delete calls though to the s3a fs instance, albeith with some reflection-pain on the way > It depends, from what i am aware this is not prod recommended, since S3FileIO is the one we would be using. Your team might not be using it -but it is production quality and does appear to have better test coverage against s3express than S3FileIO. If you are working with S3 Express, the one thing which did cause problems is that the prefix of in-progress uploads are visible in list results, even though there are no objects are underneath. I'll leave you identify and address any problems which surface there. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For additional commands, e-mail: issues-h...@iceberg.apache.org
