geoHeil commented on a change in pull request #536:
URL: https://github.com/apache/incubator-sedona/pull/536#discussion_r685799341
##########
File path: core/src/main/java/org/apache/sedona/core/spatialRDD/SpatialRDD.java
##########
@@ -564,7 +588,45 @@ public void saveAsWKT(String outputLocation)
}
}).saveAsTextFile(outputLocation);
}
-
+
+ public void saveAsParquet(JavaSparkContext sc,
+ String geometryColumn,
+ List<Field> userColumns,
+ String outputLocation,
+ String namespace,
+ String name) throws SedonaException, IOException
{
+ sc.hadoopConfiguration().setBoolean("parquet.enable.summary-metadata",
false);
+ final Job job = Job.getInstance(sc.hadoopConfiguration());
+ GeometryType geometryType = this.getGeometryType();
+ Schema schema = AvroUtils.getSchema(geometryType, geometryColumn,
userColumns,namespace,
+ name);
+// ParquetOutputFormat.setWriteSupportClass(job,
AvroWriteSupport.class);
+ AvroParquetOutputFormat.setSchema(job, schema);
+ this.rawSpatialRDD.mapPartitionsToPair(new
PairFlatMapFunction<Iterator<T>, Void, GenericRecord>() {
+ @Override
+ public Iterator<Tuple2<Void, GenericRecord>> call(Iterator<T>
iterator) throws Exception {
+ Schema schema = AvroUtils.getSchema(geometryType,
geometryColumn, userColumns, namespace,
+ name);
+ Iterable<T> recordIterable = () -> iterator;
+ try {
+ return StreamSupport.stream(recordIterable.spliterator(),
false).map(geometry -> {
+ try {
+ GenericRecord genericRecord =
+ AvroUtils.getRecord(geometry,
geometryType, geometryColumn, schema);
Review comment:
Why are we not simply choosing WKB for the serialization of the geometry
column and keep the rest as a normal parquet file?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
