GitHub user JoshRosen opened a pull request:
https://github.com/apache/spark/pull/7710
[SPARK-9393] [SQL] Fix several error-handling bugs in ScriptTransform
operator
SparkSQL's ScriptTransform operator has several serious bugs which make
debugging fairly difficult:
- If exceptions are thrown in the writing thread then the child process
will not be killed, leading to a deadlock because the reader thread will block
while waiting for input that will never arrive.
- TaskContext is not propagated to the writer thread, which may cause
errors in upstream pipelined operators.
- Exceptions which occur in the writer thread are not propagated to the
main reader thread, which may cause upstream errors to be silently ignored
instead of killing the job. This can lead to silently incorrect query results.
- The writer thread is not a daemon thread, but it should be.
In addition, the code in this file is extremely messy:
- Lots of fields are nullable but the nullability isn't clearly explained.
- Many confusing variable names: for instance, there are variables named
`ite` and `iterator` that are defined in the same scope.
- Some code was misindented.
- The `*serdeClass` variables are actually expected to be single-quoted
strings, which is really confusing: I feel that this parsing / extraction
should be performed in the analyzer, not in the operator itself.
- There were no unit tests for the operator itself, only end-to-end tests.
This pull request addresses these issues, borrowing some error-handling
techniques from PySpark's PythonRDD.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/JoshRosen/spark script-transform
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/7710.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #7710
----
commit fa18d262513d763ac0616971d9288a94a54bc88f
Author: Josh Rosen <[email protected]>
Date: 2015-07-27T19:22:47Z
Add basic unit test for script transform with 'cat' command.
commit b43e4ec31b3bd4030c9e9985a64b2f186df17825
Author: Josh Rosen <[email protected]>
Date: 2015-07-27T21:36:55Z
Clean up nullability in ScriptTransformation
commit bd4c94887a3dbe63a4794c1ac8ea4457a0925887
Author: Josh Rosen <[email protected]>
Date: 2015-07-27T23:21:44Z
Skip launching of external command for empty partitions.
commit 4ee36a209a22ed35f51704fc2002c52646d0b70e
Author: Josh Rosen <[email protected]>
Date: 2015-07-27T23:48:33Z
Kill script transform subprocess when error occurs in input writer.
commit 8b162b6776f5cda19a0e80147e27a4fa53d1780f
Author: Josh Rosen <[email protected]>
Date: 2015-07-28T01:32:38Z
Add failing test which demonstrates exception masking issue
commit 88278de8d7539c63c67db79da798a850fc15c4c3
Author: Josh Rosen <[email protected]>
Date: 2015-07-28T01:47:06Z
Split ScriptTransformation writer thread into own class.
commit b31258dec638dcce37ff1542c9b31b20aedeaad7
Author: Josh Rosen <[email protected]>
Date: 2015-07-28T01:48:04Z
Rename iterator variables to disambiguate.
commit 323bb2b0e860ba64032a51632f6ba0ca0b9e45d9
Author: Josh Rosen <[email protected]>
Date: 2015-07-28T02:02:59Z
Fix error-swallowing bug
commit 494cde023ffb87a9308709dd536d9957105969d0
Author: Josh Rosen <[email protected]>
Date: 2015-07-28T02:10:14Z
Propagate TaskContext to writer thread
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
