Matthieu Maitre created SPARK-22824:
---------------------------------------
Summary: Spark Structured Streaming Source trait breaking change
Key: SPARK-22824
URL: https://issues.apache.org/jira/browse/SPARK-22824
Project: Spark
Issue Type: Bug
Components: Structured Streaming
Affects Versions: 2.3.0
Environment: Azure Databricks 3.4
Reporter: Matthieu Maitre
org.apache.spark.sql.execution.streaming.Offset was moved to
org.apache.spark.sql.sources.v2.reader.Offset on the Source trait:
https://github.com/apache/spark/commit/f8c7c1f21aa9d1fd38b584ca8c4adf397966e9f7#diff-56453fdb4dc2d7666c7ab237cb102189
This broke our custom sources that are called in Spark jobs running on Azure
Databricks 3.4 since the Maven package does not match the hosted Spark bits. We
use the following Maven version to be able to deploy on Azure Databricks:
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-sql_2.11</artifactId>
<version>2.3.0-SNAPSHOT</version>
</dependency>
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
