amygbAI commented on PR #488: URL: https://github.com/apache/incubator-graphar/pull/488#issuecomment-2157113357
Hi @acezen, after rebasing, the compilation with changes for datasources-32 ( spark 3.2 ) worked perfectly ! no more trouble with JSONOptions. But then i wanted to compile datasources-33 ( spark 3.3.4 ) and it ran into several issues , none related to the changes i made .. i went ahead and checked a few of them and found that the variables its cribbing about dont exist on the spark 3.3 branch BUT are present in 3.4 onwards .. the issue is that if i continue without making changes to datasources-33 the PR will be incomplete ; please advise ``` [ERROR] [Error] /datadrive/APACHE_GRAPHAR/incubator-graphar/maven-projects/spark/datasources-33/src/main/scala/org/apache/spark/sql/graphar/GarScan.scala:141: one more argument than can be applied to method apply: (sqlConf: org.apache.spark.sql.internal.SQLConf, broadcastedConf: org.apache.spark.broadcast.Broadcast[org.apache.spark.util.SerializableConfiguration], dataSchema: org.apache.spark.sql.types.StructType, readDataSchema: org.apache.spark.sql.types.StructType, partitionSchema: org.apache.spark.sql.types.StructType, filters: Array[org.apache.spark.sql.sources.Filter])org.apache.spark.sql.execution.datasources.v2.orc.OrcPartitionReaderFactory in object OrcPartitionReaderFactory Note that 'aggregation' is not a parameter name of the invoked method. [ERROR] [Error] /datadrive/APACHE_GRAPHAR/incubator-graphar/maven-projects/spark/datasources-33/src/main/scala/org/apache/spark/sql/graphar/GarScan.scala:182: value LEGACY_PARQUET_NANOS_AS_LONG is not a member of object org.apache.spark.sql.internal.SQLConf [ERROR] [Error] /datadrive/APACHE_GRAPHAR/incubator-graphar/maven-projects/spark/datasources-33/src/main/scala/org/apache/spark/sql/graphar/GarScan.scala:183: value legacyParquetNanosAsLong is not a member of org.apache.spark.sql.internal.SQLConf [ERROR] [Error] /datadrive/APACHE_GRAPHAR/incubator-graphar/maven-projects/spark/datasources-33/src/main/scala/org/apache/spark/sql/graphar/GarScan.scala:186: value PARQUET_FIELD_ID_READ_ENABLED is not a member of object org.apache.spark.sql.internal.SQLConf [ERROR] [Error] /datadrive/APACHE_GRAPHAR/incubator-graphar/maven-projects/spark/datasources-33/src/main/scala/org/apache/spark/sql/graphar/GarScan.scala:187: value parquetFieldIdReadEnabled is not a member of org.apache.spark.sql.internal.SQLConf [ERROR] [Error] /datadrive/APACHE_GRAPHAR/incubator-graphar/maven-projects/spark/datasources-33/src/main/scala/org/apache/spark/sql/graphar/GarScan.scala:202: one more argument than can be applied to method apply: (sqlConf: org.apache.spark.sql.internal.SQLConf, broadcastedConf: org.apache.spark.broadcast.Broadcast[org.apache.spark.util.SerializableConfiguration], dataSchema: org.apache.spark.sql.types.StructType, readDataSchema: org.apache.spark.sql.types.StructType, partitionSchema: org.apache.spark.sql.types.StructType, filters: Array[org.apache.spark.sql.sources.Filter], parquetOptions: org.apache.spark.sql.execution.datasources.parquet.ParquetOptions)org.apache.spark.sql.execution.datasources.v2.parquet.ParquetPartitionReaderFactory in object ParquetPartitionReaderFactory Note that 'aggregation' is not a parameter name of the invoked method. [WARNING] [Warn] /datadrive/APACHE_GRAPHAR/incubator-graphar/maven-projects/spark/datasources-33/src/main/scala/org/apache/spark/sql/graphar/GarScan.scala:244: parameter value sparkSession in method getFilePartitions is never used [ERROR] [Error] /datadrive/APACHE_GRAPHAR/incubator-graphar/maven-projects/spark/datasources-33/src/main/scala/org/apache/spark/sql/graphar/GarScanBuilder.scala:68: value pushDataFilters is not a member of org.apache.spark.sql.execution.datasources.v2.parquet.ParquetScanBuilder [ERROR] [Error] /datadrive/APACHE_GRAPHAR/incubator-graphar/maven-projects/spark/datasources-33/src/main/scala/org/apache/spark/sql/graphar/GarScanBuilder.scala:79: value pushDataFilters is not a member of org.apache.spark.sql.execution.datasources.v2.orc.OrcScanBuilder [ERROR] [Error] /datadrive/APACHE_GRAPHAR/incubator-graphar/maven-projects/spark/datasources-33/src/main/scala/org/apache/spark/sql/graphar/GarScanBuilder.scala:103: not found: value pushedDataFilters [ERROR] [Error] /datadrive/APACHE_GRAPHAR/incubator-graphar/maven-projects/spark/datasources-33/src/main/scala/org/apache/spark/sql/graphar/GarWriteBuilder.scala:162: not found: value bucketSpec [WARNING] [Warn] /datadrive/APACHE_GRAPHAR/incubator-graphar/maven-projects/spark/datasources-33/src/main/scala/org/apache/spark/sql/graphar/GarWriteBuilder.scala:107: local val pathName in method validateInputs is never used [WARNING] [Warn] /datadrive/APACHE_GRAPHAR/incubator-graphar/maven-projects/spark/datasources-33/src/main/scala/org/apache/spark/sql/graphar/GarWriteBuilder.scala:97: parameter value caseSensitiveAnalysis in method validateInputs is never used [ERROR] [Error] /datadrive/APACHE_GRAPHAR/incubator-graphar/maven-projects/spark/datasources-33/src/main/scala/org/apache/spark/sql/graphar/parquet/ParquetWriteBuilder.scala:114: value PARQUET_FIELD_ID_WRITE_ENABLED is not a member of object org.apache.spark.sql.internal.SQLConf [ERROR] [Error] /datadrive/APACHE_GRAPHAR/incubator-graphar/maven-projects/spark/datasources-33/src/main/scala/org/apache/spark/sql/graphar/parquet/ParquetWriteBuilder.scala:115: value parquetFieldIdWriteEnabled is not a member of org.apache.spark.sql.internal.SQLConf ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
