jiayuasu commented on pull request #510:
URL: https://github.com/apache/incubator-sedona/pull/510#issuecomment-789994079
@netanel246 Thank you for your hard work!
I am not sure whether we should use WKB serializer now. It looks like the
existing Serde is around 4 - 5 times faster than WKB SerDe in terms of
serialization and 2 times faster in terms of deserialization. I believe this
will make a huge difference for big data processing.
Two directions to move forward. Please let me know what you think.
1. Fix the issues in the existing Serde. See my comment above.
2. Leave the WKB serializer as an optional serializer and use the old Serde
as the default SerDe for both Core and SQL. The user should only use it if they
use geometries that are currently not supported by the old Serde.
The user should be able to choose the serializer as follows
```
.config("spark.serializer", classOf[KryoSerializer].getName) //
org.apache.spark.serializer.KryoSerializer
.config("spark.kryo.registrator", classOf[SedonaKryoRegistrator].getName)
```
```
.config("spark.serializer", classOf[KryoSerializer].getName) //
org.apache.spark.serializer.KryoSerializer
.config("spark.kryo.registrator", classOf[SedonaWKBKryoRegistrator].getName)
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
