I've got a case where the same structured query (it's union) gives 1 stage
for a run and 5 stages for another. I could not find any pattern yet (and
it's hard to reproduce it due to the volume and the application), but I'm
pretty certain that it's *never* possible that Spark 2.3 could come up with
1 vs 5 stages for the very same query plan (even if I changed number of
executors or number of cores or anything execution-related).

So my question is, is this possible that Spark SQL could give 1-stage
execution plan and 5-stage execution plan for the very same query?

(I am not saying that I'm 100% sure that the query is indeed the same since
I'm working on a reproducible test case and only when I got it I'll really

Sorry for the vague description, but I've got nothing more to share yet.

Jacek Laskowski
Mastering Spark SQL https://bit.ly/mastering-spark-sql
Spark Structured Streaming https://bit.ly/spark-structured-streaming
Mastering Kafka Streams https://bit.ly/mastering-kafka-streams
Follow me at https://twitter.com/jaceklaskowski

Reply via email to