Hi,
I agree with Robert that adding open/close support for partitioners
would mean additional complexity in the code base. We're currently not
thinking of supporting that.
Best,
Aljoscha
On 05.06.20 20:19, Arvid Heise wrote:
Hi Arnaud,
just to add up. The overhead of this additional map
Hi Arnaud,
just to add up. The overhead of this additional map is negligible if you
enable object reuse [1].
[1]
https://ci.apache.org/projects/flink/flink-docs-stable/dev/execution_configuration.html
On Tue, Jun 2, 2020 at 10:34 AM Robert Metzger wrote:
> I'm not 100% sure about this answer,
I'm not 100% sure about this answer, that's why I'm CCing Aljoscha to
correct me if needed:
Partitioners are not regular operators (like a map or window), thus they
are not included in the regular Task lifecycle methods (of open() / map()
etc. / close(), with the proper error handling, task
Hello,
Yes, that would definitely do the trick, with an extra mapper after keyBy to
remove the tuple so that it stays seamless. It’s less hacky that what I was
thinking of, thanks!
However, is there any plan in a future release to have rich partitioners ? That
would avoid adding overhead and
Hi Arnaud,
Maybe I don't fully understand the constraints, but what about
stream.map(new GetKuduPartitionMapper).keyBy(0).addSink(KuduSink());
The map(new GetKuduPartitionMapper) will be a regular RichMapFunction with
open() and close() where you can handle the connection with Kudu's
