kfaraz opened a new pull request #11973:
URL: https://github.com/apache/druid/pull/11973
### Description
This PR adds support for handling null dimension values while creating
partition boundaries
in range partitioning.
This means that we can now have partition boundaries like `[null, "abc"]` or
`["abc", null, "def"]`.
This fixes the following problems seen in production environments.
- In cases of very sparse data, where there is no row that contains non-null
values
for all partition dimensions, partitioning just doesn't work. This is
because all the rows
are ignored in the `PartialDimensionDistributionTask` and
`intervalToPartitions` turns
out to be empty.
- If there are too many rows where the first partition dimension is null,
all of them would
end up in the first partition thus bloating up that partition. This can
now be mitigated in
cases where the later dimensions have non-null values which vary over
these rows.
#### The tuple of nulls
Please note that even though `[null, null]` is also a valid boundary now, it
will never actually
be encountered in practice. This is because lexicographically, the null
tuple is
immediately followed the tuple of nulls. i.e. `null < [null, null]`
Thus, if the start and end of a partition were `null` and `[null, null]`
respectively,
that partition would contain zero rows (end is not inclusive). There would
be no
point in having such a partition and so it never occurs in practice.
### Changes
* Do not skip nulls in `PartialDimensionDistributionTask`
* Add a null-safe serde to use instead of `ArrayOfStringsSerde`
<hr>
This PR has:
- [ ] been self-reviewed.
- [ ] using the [concurrency
checklist](https://github.com/apache/druid/blob/master/dev/code-review/concurrency.md)
(Remove this item if the PR doesn't have any relation to concurrency.)
- [ ] added documentation for new or modified features or behaviors.
- [x] added Javadocs for most classes and all non-trivial methods. Linked
related entities via Javadoc links.
- [ ] added or updated version, license, or notice information in
[licenses.yaml](https://github.com/apache/druid/blob/master/dev/license.md)
- [x] added comments explaining the "why" and the intent of the code
wherever would not be obvious for an unfamiliar reader.
- [x] added unit tests or modified existing tests to cover new code paths,
ensuring the threshold for [code
coverage](https://github.com/apache/druid/blob/master/dev/code-review/code-coverage.md)
is met.
- [ ] added integration tests.
- [x] been tested in a test Druid cluster.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
