Github user dongjoon-hyun commented on a diff in the pull request:
https://github.com/apache/spark/pull/22179#discussion_r214479500
--- Diff: pom.xml ---
@@ -1770,6 +1770,10 @@
<groupId>org.apache.hive</groupId>
<artifactId>hive-storage-api</artifactId>
</exclusion>
+ <exclusion>
+ <groupId> com.esotericsoftware</groupId>
+ <artifactId>kryo-shaded</artifactId>
--- End diff --
@srowen In short, the current Spark always uses the same Kryo version for
read/write `SearchArgument` and it's used only on runtime.
1. Old OrcFileFormat always uses
`org.spark-project.hive:hive-exec:1.2.1.spark2` which uses the shaded one in
`hive-exec`.
- `com.esotericsoftware.kryo:kryo:2.21`.
2. New OrcFileFormat uses `org.apache.orc` which uses the one provided by
Spark.
- `com.esotericsoftware:kryo-shaded:3.0.3` (All Spark/Orc/Hive uses
this version for now)
3. New OrcFileFormat (in this PR) uses `org.apache.orc` which uses the one
provided by Spark.
- `com.esotericsoftware:kryo-shaded:4.0.2`
So, (1) is unchanged by this PR. (2) and (3) also doesn't use a mixed
version of Kryo. So, it should be fine because Apache Spark doesn't allow a
mixed Spark version(master and executor). BTW, during investigation, there was
some performance issue in `createFilter`. I'll file a new JIRA for that.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
