d33bs opened a new pull request, #50960: URL: https://github.com/apache/arrow/pull/50960
### Rationale for this change S3 keys form a flat list. A key can look like a folder path, but S3 does not enforce real folders. Some tools create a zero-byte object to mark a folder. This marker can use the exact same key as a folder that also holds real files. `GetFileInfo` did not check for this case. It read the zero-byte marker as a file. Parquet reads then failed with the error `Invalid Parquet file size is 0 bytes`. See #24093 for the original report. ### What changes are included in this PR? - `GetFileInfo` now checks zero-byte objects for files nested under them. - If nested files exist, `GetFileInfo` reports the path as a directory, not a file. This change is in one function, in `cpp/src/arrow/filesystem/s3fs.cc`. The recursive directory listing already handled markers with a trailing slash. This fix covers the direct `GetFileInfo` lookup on a single path. ### Are these changes tested? Yes, with one limit. I ran the full `arrow-s3fs-test` suite against a local Minio server. All 65 tests pass. I did not add a new automated test for the exact bug case. Minio stores each S3 key as a real file on disk. It cannot store a key and a nested key under that key at the same time. Real S3 can store both. I checked the fix a different way. I wrote a small program that links against the patched library. I ran it against `moto`, a mock S3 server with a flat key space like real S3. Results: - A marker key with real files under it: now reported as a directory. Before the fix, it was reported as a file with size 0. - A real file with no files under it: still reported as a file with size 0. No regression here. ### Are there any user-facing changes? No public API change. `S3FileSystem::GetFileInfo` now returns a correct result for this one edge case. Closes #24093 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
