Github user mallman commented on the issue:
https://github.com/apache/spark/pull/15538
Getting this redirection to work on remote executors was quite involved.
The additional complexities derive from the following:
1. Java doesn't call the default (or any) constructor in deserialization.
2. Spark doesn't use `ParquetFileFormat` at all in the write path.
To address the first issue, I added a deserialization hook to
`ParquetFileFormat` which calls
`ParquetFileFormat.ensureParquetLogRedirection`. To address the second, I added
deserialization hooks to a couple of classes the driver serializes when it
sends the output job to the executors.
This is messy, but it's the simplest, most direct way I found to tackle
these complexities.
Hopefully all of this awfulness can be removed when Spark updates its
dependency on parquet-mr to the 1.9 release, which uses slf4j for logging.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
