SemyonSinchenko commented on issue #593: URL: https://github.com/apache/incubator-graphar/issues/593#issuecomment-2288273648
@yecol Our target is to support multiple versions of Apache Spark. Unfortunately, the `DataSource` API of Apache Spark is a Developer API and changing dramatically from one version of spark to another. And sometimes changes are so big, that reflection is not enough. We [made](https://github.com/apache/incubator-graphar/issues/320) a decision to separate datasource implementation into a maven subpackage. And we [have](https://github.com/apache/incubator-graphar/blob/main/maven-projects/spark/pom.xml#L51C1-L99C16) the following maven profiles: ```xml <profiles> <profile> <id>datasources-32</id> <properties> <sbt.project.name>graphar</sbt.project.name> <spark.version>3.2.4</spark.version> </properties> <modules> <module>graphar</module> <module>datasources-32</module> </modules> </profile> <profile> <id>datasources-33</id> <properties> <sbt.project.name>graphar</sbt.project.name> <spark.version>3.3.4</spark.version> </properties> <modules> <module>graphar</module> <module>datasources-33</module> </modules> </profile> <profile> <id>datasources-34</id> <properties> <sbt.project.name>graphar</sbt.project.name> <spark.version>3.4.3</spark.version> </properties> <modules> <module>graphar</module> <module>datasources-34</module> </modules> </profile> <profile> <id>datasources-35</id> <properties> <sbt.project.name>graphar</sbt.project.name> <spark.version>3.5.1</spark.version> </properties> <modules> <module>graphar</module> <module>datasources-35</module> </modules> <activation> <activeByDefault>true</activeByDefault> </activation> </profile> </profiles> ``` so, using that approach we are able to build GraphAr Spark for a different version of spark itself. At the moment, that approach is used in our CI when we are running tests for all the supported Maven profiles. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
