damccorm opened a new issue, #19704: URL: https://github.com/apache/beam/issues/19704
Much of the code that depends on Avro (notably the wrappers built with [BeamSQL](https://github.com/apache/beam/blob/ae83448597f64474c3f5754d7b8e3f6b02347a6b/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/utils/AvroUtils.java#L34) but also [some](https://github.com/apache/beam/blob/ae83448597f64474c3f5754d7b8e3f6b02347a6b/sdks/java/io/parquet/src/main/java/org/apache/beam/sdk/io/parquet/ParquetIO.java) [connectors](https://github.com/apache/beam/blob/ae83448597f64474c3f5754d7b8e3f6b02347a6b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryAvroUtils.java#L42)) require a version \> 1.8.x This library is not present in Spark 2.2 and Spark 2.3 clusters, which are meant to be supported. These pipelines will fail with ClassNotFoundException / MethodNotFoundExceptions. Spark 2.4**** should be unaffected. Relocating or vendoring is probably not appropriate, since Avro is frequently exposed in the API through parameters and potentially in generated specific records. Imported from Jira [BEAM-7979](https://issues.apache.org/jira/browse/BEAM-7979). Original Jira may contain additional context. Reported by: rskraba. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
