capistrant opened a new pull request, #19728:
URL: https://github.com/apache/druid/pull/19728
### Description
#### Unconditional application of partial `EMPTY_LOAD` requests for
clustered segments
Previously we would only send empty loads for segments who were conforming
to the clustered rule spec, were in a core partition set, matched no groups for
the rule, and had a sibling partition in the core partition set that matched
the rule. This was done to ensure we never failed to announce a full core
partition set, resulting in the broker not routing queries to these segments.
This approach broke down in two cases:
1. An entire core partition set was conforming but did not have any matching
groups for the rule. These empty loads would be discarded and as a result the
whole core partition set would never be announced by historicals and thus be
un-queryable even as on demand virtual segments.
2. Non core partitions with no matching groups for the rule were sent to
onCannotMatch and either fell through the cascade or FULL_LOAD
* FULL_LOAD is the real bad behavior, this is meant for opaque segments
(segments who do not conform to the shape required to partial load)
* FALL_THROUGH does give these segments the opportunity to match another
rule and potentially match groups, and eagerly load. So removing that ability
in this PR technically reduces some flexibility with a partial load rule
cascade setup, but I think it is acceptable trade off for now.
The two above cases are no longer problems because we now treat any
conforming segment that matches a rule but ends up with no underlying matching
groups as matched and empty loaded. These segments are announced according to
the tieredReplicant map of the rule and will be queryable for historicals using
virtual storage + partial loads via on demand download of the needed columns
for any inbound query.
#### Future change potential
As I alluded to above, we lose some flexibility when deciding to
unconditionally empty load anything that is conforming, matches a rule but has
0 matching cluster groups after inspecting the segments. I see there being
arguments for allowing partitions who matched a rule generally but then had no
matching groups be able to re-enter the cascade and continue searching for a
rule they match and have matching cluster groups. That could open the door to
multi-tiered partial loaded datasources, etc. There is added complexity here
because we still must be sure we complete all core partition sets so you cannot
have segments who end up falling through to nothingness and leaving things
un-queryable.
#### Release note
N/A Druid 39 should have a larger release note for clustered segments and
partial loads improvements
<hr>
##### Key changed/added classes in this PR
* `RunRules`
* `ClusterGroupPartialLoadMatcher`
<hr>
<!-- Check the items by putting "x" in the brackets for the done things. Not
all of these items apply to every PR. Remove the items which are not done or
not relevant to the PR. None of the items from the checklist below are strictly
necessary, but it would be very helpful if you at least self-review the PR. -->
This PR has:
- [ ] been self-reviewed.
- [x] added Javadocs for most classes and all non-trivial methods. Linked
related entities via Javadoc links.
- [x] added comments explaining the "why" and the intent of the code
wherever would not be obvious for an unfamiliar reader.
- [x] added unit tests or modified existing tests to cover new code paths,
ensuring the threshold for [code
coverage](https://github.com/apache/druid/blob/master/dev/code-review/code-coverage.md)
is met.
- [ ] added integration tests.
- [ ] been tested in a test Druid cluster.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
