GitHub user ChengXiangLi opened a pull request:
https://github.com/apache/flink/pull/1255
[FLINK-7] [Runtime] Enable Range Partitioner.
This PR enable range partitioner for Flink follow the path of existing
other partitioners. It depends on the sample operator to random sample data
from `DataSet` and build range boundaries based on sampled data. 2 other hints
about PR:
1. Why execute the sample data job in `JobGraphGenerator` instead of
`PartitionOperator`?
i. launch another job in compile time would lead to infinite job
submission, because the `DataSink`s has not been cleared during compile time.
ii. we need the target stage parallelism to decide sample data size,
and `TypeSerializer`/`TypeComparator` to serialize/sort sampled data.
2. Expand the `DataDistribution` API, previous `DataDistribution` take
`Key[]` as range boundaries, there is not simple generic way to extract Key
from nested object, and `TypeComparator::compareAgainstReference()` is not
supported by current comparators. Use `DataSet` elements as the range
boundaries make everything much easier, we could use
'TypeComparator::compare()' directly for sort during build `DataDistribution`
and selecting channel.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/ChengXiangLi/flink rangepartitioner
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/1255.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1255
----
commit 8a41b18c6c40115d545271039e51ebad44300191
Author: chengxiang li <[email protected]>
Date: 2015-10-12T07:13:38Z
[FLINK-7] [Runtime] Enable Range Partitioner.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
