viirya commented on pull request #30812: URL: https://github.com/apache/spark/pull/30812#issuecomment-750781778
> I'm wondering whether this would solve the issue. There are so many factors involved. I feel the initial even task distribution (assuming the executors are free so the task scheduler will respect the preferred locations) would become uneven quickly after some micro batches, caused by, such as, uneven partitions, executor lost, concurrent queries, etc... Did you verify that this would make a real workload that didn't work before become working? During running stateful streaming queries recently, it caused some troubles by bad initial locations of stores. I agree that this is not ideal, but it solves the problem I saw during I tested SS recently. It is simple and shouldn't have bad impact/regression to SS queries. Combined with task locality configuration, it makes SS queries more stable in my local test. > If the issue is about memory, a low memory state store implementation such as https://github.com/qubole/spark-state-store is a better solution. The problem is, we have a built-in in memory store. And, I don't think Rocksdb-based state store is the answer to all cases. Even with Rocksdb-based store, is it good to have skew stores on few executors? Then local disk space might be the next issue. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
