Hi all, I am concerned that the API-breaking changes in SPARK-25908 (as well as SPARK-16775, and potentially others) will make the migration process from Spark 2 to Spark 3 unnecessarily painful. For example, the removal of SQLContext.getOrCreate will break a large number of libraries currently built on Spark 2.
Even if library developers do not use deprecated APIs, API changes between 2.x and 3.x will result in inconsistencies that require hacking around. For a fairly small and new (2.4.3+) genomics library, I had to create a number of shims (https://github.com/projectglow/glow/pull/155) for the source and test code due to API changes in SPARK-25393, SPARK-27328, SPARK-28744. It would be best practice to avoid breaking existing APIs to ease library development. To avoid dealing with similar deprecated API issues down the road, we should practice more prudence when considering new API proposals. I'd love to see more discussion on this. -- Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/ --------------------------------------------------------------------- To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
