cxzl25 opened a new pull request, #2724: URL: https://github.com/apache/orc/pull/2724
### What changes were proposed in this pull request? When `addRowBatch` fails, `WriterImpl` currently calls `close()` from the catch block. This PR replaces that call with a private `abortOnFailure()` that only marks the writer closed, removes it from the `MemoryManager` and closes the `PhysicalWriter`. It also makes `addRowBatch` fail fast with an `IOException` when the writer is already closed. ### Why are the changes needed? Two problems come from calling `close()` on the failure path. 1. `close()` is virtual. Hive's `org.apache.hadoop.hive.ql.io.orc.WriterImpl` overrides it as `flushInternalBatch(); super.close();`, and `flushInternalBatch()` hands the buffered batch back to `addRowBatch`. When a batch fails (for example a dictionary `chunkIndex overflow` on a large string column), the sequence is`addRowBatch -> close() -> flushInternalBatch -> addRowBatch -> ...` with the same failing batch at every level, until `StackOverflowError`. Every level re-inserts the batch into the dictionary, so a single failure turns into hours of CPU before the task reports anything. We captured a live Spark task thread with 26,734 frames, 8,811 of them this loop. After the stack unwinds, Spark's `abort()` calls `close()` again and the whole recursion starts over. 2. Even without a subclass, `close()` flushes the current stripe and writes a footer for a writer that just failed mid-batch. The result is a file that reads fine but silently misses rows. A failed write should leave a file that is obviously incomplete, not one that masquerades as complete. This mirrors what Parquet did in PARQUET-2437 : record the failure, stop flushing, and let `close()` return without a footer. ### How was this patch tested? 1. Add UT ### Was this patch authored or co-authored using generative AI tooling? Yes, co-authored with Claude Code (Claude Fable 5.1) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
