Github user fhueske commented on the pull request:
https://github.com/apache/flink/pull/1255#issuecomment-162935419
Thanks for the fast update!
Your last commit seems to include changes that have already been merged to
the master branch. Can you revert it and rebase the second last commit on the
current master?
I also got feedback from @StephanEwen about the use of the IntDistribution.
He shares your opinion, that it is not really needed and just incurs additional
overhead due to the second binary search. He proposes to use a
`PARTITION_CUSTOM` strategy instead of a `PARTITION_RANGE` strategy with a
`Partitioner` that simply returns the partition id. You basically can replace
`partChannel.setShipStrategy(ShipStrategyType.PARTITION_RANGE, keys,
sortDirection, null, DataExchangeMode.PIPELINED);`
by
`partChannel.setShipStrategy(ShipStrategyType.PARTITION_CUSTOM, keys, new
IdPartitioner(), DataExchangeMode.PIPELINED);`
and provide an implementation for IdPartitioner. This way, we have still
support for generic DataDistribution range partitioning (although not being
used ATM) and a more efficient automatic range partitioning.
On last very small remark. Could you please set the parallelism also for
the `OptimizerNodes` (`sipNode`, `sicNode`, etc.)? Otherwise, the JSON output
will have some `"parallelism": "default"` values.
I think we're almost there. :-)
Thanks a lot, Fabian
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
