HyukjinKwon opened a new pull request, #44617:
URL: https://github.com/apache/spark/pull/44617
### What changes were proposed in this pull request?
This PR proposes to log full exception when failed to lookup Python Data
Sources
### Why are the changes needed?
In my internal testing it logs something like:
```
...
24/01/05 03:49:49 WARN DataSourceManager: Skipping the lookup of Python Data
Sources due to the failure: java.lang.StackOverflowError
24/01/05 03:49:49 WARN DataSourceManager: Skipping the lookup of Python Data
Sources due to the failure: java.lang.StackOverflowError
24/01/05 03:49:49 WARN WSFSWorkerManager: Files feature is disabled because
com.databricks.backend.daemon.driver.WSFSManager is not available
24/01/05 03:49:49 WARN DataSourceManager: Skipping the lookup of Python Data
Sources due to the failure: java.lang.StackOverflowError
24/01/05 03:49:49 WARN DataSourceManager: Skipping the lookup of Python Data
Sources due to the failure: java.lang.StackOverflowError
24/01/05 03:49:49 WARN PythonWorkerFactory: Failed to open socket to Python
daemon:
java.net.ConnectException: Connection refused
at sun.nio.ch.Net.connect0(Native Method)
...
```
which is hard to debug. It should show the full error messages so developers
can debug.
### Does this PR introduce _any_ user-facing change?
No, the main change has not been released out yet.
### How was this patch tested?
Manually.
### Was this patch authored or co-authored using generative AI tooling?
No.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
