Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/8995#issuecomment-147872165
Hey @a-roberts,
How about this:
- Add a `private[spark]` method to the `private[spark]` `CompressionCodec`
companion object and have that method maintain the hardcoded list of
compression codecs which support concatenation of serialized streams. This
method should accept a `CompressionCodec` instance and perform the `instanceof`
check. I'd consider naming this something like
"supportsConcatenationOfSerializedStreams" to be very explicit and clear.
- Update `fastMergeIsSupported` to use this new static method.
I like this approach since it makes it very clear why we're only supporting
those two codecs.
I wouldn't worry about third-party / external compression codecs being able
to take advantage of this feature.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
