[jira] [Commented] (FLINK-13864) StreamingFileSink: Allow inherited classes to extend StreamingFileSink correctly
[ https://issues.apache.org/jira/browse/FLINK-13864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16919902#comment-16919902 ] Ying Xu commented on FLINK-13864: - HI [~kkl0u] , this is Ying and I have been working with [~kailashhd] on this feature. Posted a PR [https://github.com/apache/flink/pull/9581] where you can find more details. We've tested a customized StreamingfileSink built on top of the new interface, and it was working fine. Would love your comments there. > StreamingFileSink: Allow inherited classes to extend StreamingFileSink > correctly > > > Key: FLINK-13864 > URL: https://issues.apache.org/jira/browse/FLINK-13864 > Project: Flink > Issue Type: Improvement > Components: Connectors / FileSystem >Reporter: Kailash Hassan Dayanand >Priority: Minor > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > Currently the StreamingFileSink can't be extended correctly as there are a > few issues [PR |[https://github.com/apache/flink/pull/8469]] merged for this > [Jira|https://issues.apache.org/jira/browse/FLINK-12539] > Mailing list discussion: > [http://mail-archives.apache.org/mod_mbox/flink-dev/201908.mbox/%3CCACGLQUAxXjr2mBOf-6hbXcwmWoH5ib_0YEy-Vyjj%3DEPyQ25Qiw%40mail.gmail.com%3E] > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (FLINK-13864) StreamingFileSink: Allow inherited classes to extend StreamingFileSink correctly
[
https://issues.apache.org/jira/browse/FLINK-13864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16917708#comment-16917708
]
Kailash Hassan Dayanand commented on FLINK-13864:
-
Hello [~kkl0u],
Thanks for your response. Wanted to include some code snippets which will make
the discussion easier. This is my current extended class:
public class ExtendedtStreamingFileSink extends StreamingFileSink {
public static StreamingFileSink.BulkFormatBuilder
forBulkFormat(
Path basePath, Factory writerFactory) {
return new ExtendedtStreamingFileSink.ExtendedBulkFormatBuilder(
basePath, writerFactory, new DateTimeBucketAssigner());
}
public static class ExtendedBulkFormatBuilder
extends StreamingFileSink.BulkFormatBuilder {
public ExtendedtStreamingFileSink.ExtendedBulkFormatBuilder
withBucketAssigner(
BucketAssigner assigner) {
return new ExtendedtStreamingFileSink.ExtendedBulkFormatBuilder(
super.basePath,
super.writerFactory,
(BucketAssigner) Preconditions.checkNotNull(assigner),
this.bucketCheckInterval,
new DefaultBucketFactoryImpl());
}
... similar function for other functions: withBucketCheckInterval,
// Constructors
public ExtendedBulkFormatBuilder(
Path basePath, Factory writerFactory, BucketAssigner
assigner) {
super(basePath, writerFactory, assigner);
}
private ExtendedBulkFormatBuilder(
Path basePath,
Factory writerFactory,
BucketAssigner assigner,
long bucketCheckInterval,
BucketFactory bucketFactory) {
super(basePath, writerFactory, assigner, bucketCheckInterval, bucketFactory);
}
}
Since I am using the private members of the StreamingFileSink.Builder class in
the extendedBuilder class, I changed them to protected. But this may not be
completely necessary and I could use some mechanism described here:
[https://stackoverflow.com/questions/17164375/subclassing-a-java-builder-class]
which may this recommended change unnecessary.
> StreamingFileSink: Allow inherited classes to extend StreamingFileSink
> correctly
>
>
> Key: FLINK-13864
> URL: https://issues.apache.org/jira/browse/FLINK-13864
> Project: Flink
> Issue Type: Improvement
> Components: Connectors / FileSystem
>Reporter: Kailash Hassan Dayanand
>Priority: Minor
>
> Currently the StreamingFileSink can't be extended correctly as there are a
> few issues [PR |[https://github.com/apache/flink/pull/8469]] merged for this
> [Jira|https://issues.apache.org/jira/browse/FLINK-12539]
> Mailing list discussion:
> [http://mail-archives.apache.org/mod_mbox/flink-dev/201908.mbox/%3CCACGLQUAxXjr2mBOf-6hbXcwmWoH5ib_0YEy-Vyjj%3DEPyQ25Qiw%40mail.gmail.com%3E]
>
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Commented] (FLINK-13864) StreamingFileSink: Allow inherited classes to extend StreamingFileSink correctly
[
https://issues.apache.org/jira/browse/FLINK-13864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16916469#comment-16916469
]
Kostas Kloudas commented on FLINK-13864:
Hi [~kailashhd]!
Could you please be more specific about:
1) what is the functionality you want to add?
2) which parts need to be adjusted so that the {{StreamingFileSink}} is more
"friendly" for extension (after the merging of the PRs you pointed to)?
In the PR you post in the mailing-list thread, you make all the members of the
FormatBuilders protected and all the factory interfaces public. Is this
necessary? Giving a bit more details about what you need can help us also
figure out the context under which to review your PR ;)
> StreamingFileSink: Allow inherited classes to extend StreamingFileSink
> correctly
>
>
> Key: FLINK-13864
> URL: https://issues.apache.org/jira/browse/FLINK-13864
> Project: Flink
> Issue Type: Improvement
> Components: Connectors / FileSystem
>Reporter: Kailash Hassan Dayanand
>Priority: Minor
>
> Currently the StreamingFileSink can't be extended correctly as there are a
> few issues [PR |[https://github.com/apache/flink/pull/8469]] merged for this
> [Jira|https://issues.apache.org/jira/browse/FLINK-12539]
> Mailing list discussion:
> [http://mail-archives.apache.org/mod_mbox/flink-dev/201908.mbox/%3CCACGLQUAxXjr2mBOf-6hbXcwmWoH5ib_0YEy-Vyjj%3DEPyQ25Qiw%40mail.gmail.com%3E]
>
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
