milenkovicm commented on issue #17494: URL: https://github.com/apache/datafusion/issues/17494#issuecomment-3580827225
datafusion plan: https://gist.github.com/milenkovicm/8850460d772ce145e53d91a58a71ee25 , there is also spark plan attached. the biggest difference is that spark is taking broadcast join all way through, which is finishes query in around 15 seconds, compared to datafusions 3 mins 30 seconds (part of plan) ```text +- * BroadcastHashJoin Inner BuildRight (22) :- * Project (16) : +- * SortMergeJoin Inner (15) : :- * Sort (7) : : +- AQEShuffleRead (6) : : +- ShuffleQueryStage (5), Statistics(sizeInBytes=979.3 MiB, rowCount=1.43E+7) : : +- Exchange (4) : : +- * Filter (3) : : +- * ColumnarToRow (2) : : +- Scan parquet (1) : +- * Sort (14) : +- AQEShuffleRead (13) : +- ShuffleQueryStage (12), Statistics(sizeInBytes=4.7 GiB, rowCount=1.26E+8) : +- Exchange (11) : +- * Filter (10) : +- * ColumnarToRow (9) : +- Scan parquet (8) +- BroadcastQueryStage (21), Statistics(sizeInBytes=1024.1 KiB, rowCount=10) +- BroadcastExchange (20) +- * Filter (19) +- * ColumnarToRow (18) +- Scan parquet (17) ``` if you disable broadcast shuffle `spark.sql.autoBroadcastJoinThreshold=-1` spark gets to similar time (around 3 mins) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
