dmsuehir commented on issue #9328: URL: https://github.com/apache/incubator-gluten/issues/9328#issuecomment-2971739717
I was able to reproduce the error that's being reported here. The NullPointerException is happening when trying to [get files from a directory](https://github.com/apache/incubator-gluten/blob/main/gluten-ut/spark35/src/test/scala/org/apache/spark/sql/GlutenSQLQueryTestSuite.scala#L787) that does not exist. I found that this happens if the `--spark_home` arg is pre-built spark package, which is what I was originally using. The [instructions](https://github.com/apache/incubator-gluten/blob/main/docs/velox-backend-support-progress.md#function-support-status) for using `gen-function-support-docs.py` say to use Spark source code, which has also been built from source. I was also able to use my pre-built spark package, but I had to copy in the `spark/sql/core/src/test/resource` from Spark source code since the unit tests are using those resource files. After making this change in my environment, I was able to successfully run the `gen-function-support-docs.py` script and unit tests. I posted a PR to update the documentation to clarify what needs to be in the `--spark_home` directory, and I also updated the test script to ensure that the path exists before trying to get all the files. This prevents the NullPointerException and will instead give a `FileNotFoundException` with a clearer error, including the full path that the test was expecting. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
