Github user rdblue commented on the issue:
https://github.com/apache/spark/pull/17540
@cloud-fan, could you have another look at this?
There are a few new changes:
* withNewExecutionId now warns instead of throwing an exception, but still
throws exceptions if spark.testing is defined
* SQLExecution.nested allows nested execution IDs without test failures or
warnings. This is needed because several places will nest when
withNewExceptionId is called at the high-level operations. CacheTableCommand is
an example.
Over the last week, I've fixed nearly all of the tests. The remaining
failure, SQLExecutionSuite.concurrent query execution (SPARK-10548), is fixed
in maven, but fails in SBT. The problem is that exceptions are now only thrown
if `spark.testing` is defined, and for some reason adding it to the test's
SparkSession or SparkContext doesn't work on Jenkins. Because this test is
reproducing a case that now will never happen for two reasons (the original
multi-threading fix and throw only if spark.testing), I'd like to simply remove
it. Let me know what you think about that.
Other changes to look at:
* `SQLMetricsSuite.save metrics` started failing because there is a nested
execution ID. This is because there are two SQL physical plans. The first,
`ExecutedCommandExec` links in a logical plan that is turned into a second
physical plan *at runtime*. This means that the inner plan can't report the
metrics that will be collected when analyzing the outer plan because it doesn't
exist yet. The long-term solution is to fix `ExecutedCommandExec`, but for now
this accepts any metrics created by the inner plan.
* `StreamExecution` wasn't calling `withNewExecutionId` and was caught by
the new assertion. I added the call around the entire execution so that there
isn't a new SQL execution for every batch. This required creating a special
`queryExecution` to pass in.
* `DataFrameCallbackSuite` had to be updated to include commands that were
previously not registered in the SQL tab. The new SQL executions are for
dropping tables, so the result looks more correct than before.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
