Github user a-roberts commented on the pull request:
https://github.com/apache/spark/pull/8995#issuecomment-147657380
Josh, apologies for the late response here and making sure I'm
understanding your proposal
For your first comment the naive way would to be add SnappyCompressionCodec
such that:
```
final boolean fastMergeIsSupported =
!compressionEnabled || compressionCodec instanceof
LZFCompressionCodec || compressionCodec instanceof SnappyCompressionCodec;
```
but this isn't scalable in the long term (will be prone to error if we
support more codecs and feels hacky - requiring this to be modified over time
based on new functionality or new codecs being available). Having said that, we
already have hard coded compression codec names in CompressionCodec.scala...
With your proposal we'd add
```
@Private
private[spark] def supportsSerializedStreams: Boolean = false
```
Alternatively a required method named "supportsSerializedStreams" - so when
a user defines their own codec to be used with Spark, this would be required.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
