thinkharderdev opened a new pull request, #658:
URL: https://github.com/apache/arrow-ballista/pull/658
# Which issue does this PR close?
<!--
We generally require a GitHub issue to be filed for all bug fixes and
enhancements and this helps us generate change logs for our releases. You can
link an issue to this PR using the GitHub syntax. For example `Closes #123`
indicates that this PR will close issue #123.
-->
Closes #554
# Rationale for this change
<!--
Why are you proposing this change? If this is already explained clearly in
the issue then this section is not needed.
Explaining clearly why changes are proposed helps reviewers understand your
changes and offer better suggestions for fixes.
-->
See original ticket for full description but the tl;dr; is:
1. The existing interface for managing shared state among schedulers is too
low-level and both pushes too much complexity into other layers and prevents us
from taking advanatage of data store specific freatures (atomics, transactions,
etc) to avoid locking in the application layer.
2. By forcing everything through the KV interface we force all the
serialization overhead even if we are only using in-memory state.
# What changes are included in this PR?
<!--
There is no need to duplicate the description in the issue here but it is
sometimes worth providing a summary of the individual changes in this PR.
-->
There are a lot of changes because the `StateBackendClient` was baked in
everywhere but at a high level:
1. Create `JobState` trait to complement `ClusterState` created in last PR.
This in the interface for storing/managing global state w/r/t jobs and sessions.
2. Refactor in-memory implementation to not serialize things to protobuf
3. Provide `KeyValueState` implementation based on existing KV interface so
we can continue to use etcd and sled for state without any changes.
4. Take advantage of curated job architecture to minimize distributed
locking. Since jobs are owned by a single scheduler we don't need to lock
beyond the node level.
5. Cleanup configs.
6. Move everything related to state the `ballista_scheduler::cluster` module
I removed the separate configs for `cluster-backend` and `config-backend`
because it seems confusing. So currently if you want to use different
implementations for the `ClusterState` and `JobState` you would need to write
your own entrypoint. We might introduce standard "profiles" (standalone/ha/etc)
for setting up the two state backends on different implementations but for now
it seems too noisy config wise.
NOTE: I was not able to run the integration tests using the standard
configuration due to weird timeouts. When using push scheduling everything
worked fine so not sure what's going on. I also checked on `main` and see the
same issue so maybe something related to the M1 macbook?
# Are there any user-facing changes?
<!--
If there are user-facing changes then we may require documentation to be
updated before approving the PR.
-->
The storage layout for etcd and sled are simplified so existing state would
not be valid after upgrade.
<!--
If there are any breaking changes to public APIs, please add the `api
change` label.
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
