avantgardnerio commented on code in PR #2038:
URL:
https://github.com/apache/datafusion-ballista/pull/2038#discussion_r3611001150
##########
ballista/core/src/config.rs:
##########
@@ -342,16 +342,16 @@ static CONFIG_ENTRIES: LazyLock<HashMap<String,
ConfigEntry>> = LazyLock::new(||
ConfigEntry::new(
Review Comment:
I think in general it would be good to move towards 0, as it fully allows DF
internal parallelism to optimize things. But unfortunately with the state the
PR is in today, Andy saw a 20% perf regression on queries with very skewed
partitioning. Setting this to 1 is the other extreme, and probably the safe
step for existing Spark users.
What I'd like to do over time is to fix how this deals with skewed
partitioning (or possibly allow tasks to release vcores before completion), but
I think it's outside the scope of this PR.
So in the mean time the correct value is quite likely somewhere between 1
and infinity (0), which is why I think it's a knob not a boolean. Sorry, I
should have explained my reasoning the first time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
