duhizjame opened a new issue, #3097:
URL: https://github.com/apache/sedona/issues/3097
hi everyone!
I am getting an error when running a spark pipeline with sedona 1.7.2(also
with 1.8.0 and 1.6.1) on Spark 3.5.4.3.5.7191000.100-2(cloudera environment):
```
java.lang.NoSuchMethodError:
org.apache.spark.sql.execution.datasources.parquet.ParquetFooterReader.readFooter(Lorg/apache/hadoop/conf/Configuration;Lorg/apache/spark/sql/execution/datasources/PartitionedFile;Z)Lorg/apache/parquet/hadoop/metadata/ParquetMetadata;
at
org.apache.spark.sql.execution.datasources.parquet.GeoParquetFileFormat.$anonfun$buildReaderWithPartitionValues$2(GeoParquetFileFormat.scala:240)
at
org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.org$apache$spark$sql$execution$datasources$FileScanRDD$$anon$$readCurrentFile(FileScanRDD.scala:219)
at
org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.nextIterator(FileScanRDD.scala:282)
at
org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.hasNext(FileScanRDD.scala:131)
at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:460)
at
org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown
Source)
at
org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
at
org.apache.spark.sql.execution.WholeStageCodegenEvaluatorFactory$WholeStageCodegenPartitionEvaluator$$anon$1.hasNext(WholeStageCodegenEvaluatorFactory.scala:43)
at
org.apache.spark.sql.execution.UnsafeExternalRowSorter.sort(UnsafeExternalRowSorter.java:225)
at
org.apache.spark.sql.execution.exchange.ShuffleExchangeExec$.$anonfun$prepareShuffleDependency$10(ShuffleExchangeExec.scala:377)
at
org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2(RDD.scala:922)
at
org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2$adapted(RDD.scala:922)
```
I have inspected the stack trace from:
```
val footerFileMetaData =
ParquetFooterReader
.readFooter(sharedConf, file, ParquetFooterReader.SKIP_ROW_GROUPS)
.getFileMetaData
```
the only difference that I see between 3.5.4 spark and cloudera flavor of
the ParquetFooterReader is that this part is missing from the cloudera flavour,
which matches the signature of the exception:
```
public class ParquetFooterReader {
public static final boolean SKIP_ROW_GROUPS = true;
public static final boolean WITH_ROW_GROUPS = false;
/**
* Reads footer for the input Parquet file 'split'. If 'skipRowGroup' is
true,
* this will skip reading the Parquet row group metadata.
*
* @param file a part (i.e. "block") of a single file that should be read
* @param configuration hadoop configuration of file
* @param skipRowGroup If true, skip reading row groups;
* if false, read row groups according to the file
split range
*/
public static ParquetMetadata readFooter(
Configuration configuration,
PartitionedFile file,
boolean skipRowGroup) throws IOException {
long fileStart = file.start();
ParquetMetadataConverter.MetadataFilter filter;
if (skipRowGroup) {
filter = ParquetMetadataConverter.SKIP_ROW_GROUPS;
} else {
filter = HadoopReadOptions.builder(configuration, file.toPath())
.withRange(fileStart, fileStart + file.length())
.build()
.getMetadataFilter();
}
return readFooter(configuration, file.toPath(), filter);
}
```
Has anyone encountered this issue on Cloudera environment? without shading
this specific class, does any version work with CDS spark 3.5?
is there a way to use the unshaded sedona jars without conflict on CDP?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
