PerilousApricot commented on issue #28159: [WIP][SPARK-31363][SQL] Add DataSourceRegisterV2 URL: https://github.com/apache/spark/pull/28159#issuecomment-611871616 I did a POC with Spark2.4 as well, which can be found at https://github.com/PerilousApricot/spark/tree/feature/registerv2-24 I compiled my datasource and both Spark patches and verified that the patch correctly loads the right version of the plugin. You can verify it yourself with the following jar http://mirror.accre.vanderbilt.edu/spark/laurelin-both.jar and input file https://github.com/spark-root/laurelin/raw/master/testdata/stdvector.root ``` scala> val df = spark.read.format("root").option("tree","tvec").load("stdvector.root") val df = spark.read.format("root").option("tree","tvec").load("stdvector.root") df: org.apache.spark.sql.DataFrame = [vpx: array<float>, vpy: array<float> ... 7 more fields] scala> df.select("vpx").show() +--------------------+ | vpx| +--------------------+ | [-2.409915]| |[-0.39048654, 0.3...| |[-0.0517636, 0.06...| |[-0.4088529, 0.23...| |[0.22863834, -0.3...| |[1.5184512, -1.10...| |[-1.2615219, 1.17...| |[0.26021498, -0.3...| |[-1.2066879, 0.91...| |[0.54583037, -0.4...| +--------------------+ ```
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
