There is a Pull Request to enable the new streaming sink for Hadoop < 2.7,
so it may become an option in the next release.
Thanks for bearing with us!
Best,
Stephan
On Sat, Sep 22, 2018 at 2:27 PM Paul Lam wrote:
>
> Hi Stephan!
>
> It's bad that I'm using Hadoop 2.6, so I have to stick to
Hi Stephan!
It's bad that I'm using Hadoop 2.6, so I have to stick to the old bucketing
sink. I made it by explicitly setting Hadoop conf for the bucketing sink in
the user code.
Thank you very much!
Best,
Paul Lam
Stephan Ewen 于2018年9月21日周五 下午6:30写道：
> Hi!
>
> The old bucketing sink does
Hi Stefan, Stephan,
Yes, the `hadoop.security.group.mapping` option is explicitly set to
`org.apache.hadoop.security.LdapGroupsMapping`. Guess that was why the
classloader found an unshaded class.
I don’t have the permission to change the Hadoop cluster configurations so I
modified the
Hi!
A few questions to diagnose/fix this:
Do you explicitly configure the "hadoop.security.group.mapping"?
- If not, this setting may have leaked in from a Hadoop config in the
classpath. We are fixing this in Flink 1.7, to make this insensitive to
such settings leaking in.
- If yes, then
Hi,
I could not find any open Jira for the problem you describe. Could you please
open one?
Best,
Stefan
> Am 19.09.2018 um 09:54 schrieb Paul Lam :
>
> Hi,
>
> I’m using StreamingFileSink of 1.6.0 to write logs to S3 and encounter a
> classloader problem. It seems that there are conflicts
Hi,
I’m using StreamingFileSink of 1.6.0 to write logs to S3 and encounter a
classloader problem. It seems that there are conflicts in
flink-shaded-hadoop2-uber-1.6.0.jar and flink-s3-fs-hadoop-1.6.0.jar, and maybe
related to class loading orders.
Did anyone meet this problem? Thanks a lot!
