mohammadnaqvi04 opened a new pull request, #51321: URL: https://github.com/apache/arrow/pull/51321
### Rationale for this change `FileSystem::OpenAppendStream` is [documented](https://github.com/apache/arrow/blob/main/cpp/src/arrow/filesystem/filesystem.h#L339-L341) to create the target file if it doesn't already exist. On HDFS this instead raises a not-found error. `HadoopFileSystemImpl::OpenWritable` always opens in append mode when asked to append, but Hadoop's `FileSystem.append()` requires the file to already exist, unlike a local append-mode open. See #32438. ### What changes are included in this PR? - `OpenWritable` now only sets append mode when the target already exists; otherwise it falls back to a plain create, matching every other backend. Adds an existence check against the NameNode before an append-mode open, since that's the only way to know which mode to request. - Removed `TestHadoopFileSystemGeneric::allow_append_to_new_file`'s override to `false`, seemingly added when HDFS was first wired into the shared filesystem test suite. HDFS now runs the same append-to-new-file check as every other backend. ### Are these changes tested? Yes. With the override removed, `TestHadoopFileSystemGeneric.OpenAppendStream` fails on the unpatched code with the same not-found error from the issue, and passes with this change, run against the Docker-based HDFS test setup (`ci/scripts/integration_hdfs.sh`). The rest of the HDFS C++ suite and `pyarrow.tests.test_fs` pass with no regressions. Unrelated to this change: `ci/scripts/install_minio.sh` currently fails for anyone building the CI images locally, since MinIO discontinued the binary distribution it fetches from. I patched around it locally to get a working build. If this is fixed properly upstream and CI behaves differently as a result, I wouldn't have had visibility into that from my local setup. ### Are there any user-facing changes? Yes: appending to a nonexistent file on HDFS now succeeds and creates the file, instead of raising an error. **This PR contains a "Critical Fix".** It fixes a case where an operation documented to succeed instead raised an error. * GitHub Issue: #32438 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
