Hi Fred,
I see! Thanks for posting your solution here.
Best,
Max
On 06.11.18 03:49, K Fred wrote:
Hi Max,
I have resolved this issue. It's caused by the flink cluster kerberos
configuration. Just need to set some config on flink-conf.yaml can make
it work fine!
The settings is below:
security.kerberos.login.use-ticket-cache: false
security.kerberos.login.keytab: /etc/kafka/kafka.keytab
security.kerberos.login.principal: [email protected]
<mailto:[email protected]>
security.kerberos.login.contexts: Client,KafkaClient
Thanks,
Fred.
On Tue, Nov 6, 2018 at 2:56 AM Maximilian Michels <[email protected]
<mailto:[email protected]>> wrote:
Hi Fred,
Just to double check: Are you running this from a cluster or your local
machine? Asking because the stack trace indicates that the exception
occurs during job submission through the Flink command-line client. So
the machine you're running this on should also have the file located in
/etc.
Thanks,
Max
On 05.11.18 12:26, K Fred wrote:
> Hi Max,
>
> Yeah, The config is always located on the remote cluster. The
exception
> looks like that my application can find the config file, but
cannot find
> out the config's KafkaClient entry. So i guess the reason may be
related
> to flink cluster some settings!
>
> /These code depict some stack trace below:/
>
-----------------------------------------------------------------------------
> The program finished with the following exception:
>
> org.apache.flink.client.program.ProgramInvocationException: The main
> method caused an error.
> at
>
org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:545)
> at
>
org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:420)
> at
org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:404)
> at
>
org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:785)
> at
org.apache.flink.client.cli.CliFrontend.runProgram(CliFrontend.java:279)
> at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:214)
> at
>
org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:1025)
> at
>
org.apache.flink.client.cli.CliFrontend.lambda$main$9(CliFrontend.java:1101)
> at
>
org.apache.flink.runtime.security.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:30)
> at
org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1101)
> Caused by: java.lang.RuntimeException: Error while translating
> UnboundedSource:
org.apache.beam.sdk.io.kafka.KafkaUnboundedSource@7bc6d27a
> at
>
org.apache.beam.runners.flink.FlinkStreamingTransformTranslators$UnboundedReadSourceTranslator.translateNode(FlinkStreamingTransformTranslators.java:225)
> at
>
org.apache.beam.runners.flink.FlinkStreamingTransformTranslators$ReadSourceTranslator.translateNode(FlinkStreamingTransformTranslators.java:273)
> at
>
org.apache.beam.runners.flink.FlinkStreamingPipelineTranslator.applyStreamingTransform(FlinkStreamingPipelineTranslator.java:122)
> at
>
org.apache.beam.runners.flink.FlinkStreamingPipelineTranslator.visitPrimitiveTransform(FlinkStreamingPipelineTranslator.java:101)
> at
>
org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:657)
> at
>
org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:649)
> at
>
org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:649)
> at
>
org.apache.beam.sdk.runners.TransformHierarchy$Node.access$600(TransformHierarchy.java:311)
> at
>
org.apache.beam.sdk.runners.TransformHierarchy.visit(TransformHierarchy.java:245)
> at
org.apache.beam.sdk.Pipeline.traverseTopologically(Pipeline.java:458)
> at
>
org.apache.beam.runners.flink.FlinkPipelineTranslator.translate(FlinkPipelineTranslator.java:38)
> at
>
org.apache.beam.runners.flink.FlinkStreamingPipelineTranslator.translate(FlinkStreamingPipelineTranslator.java:53)
> at
>
org.apache.beam.runners.flink.FlinkPipelineExecutionEnvironment.translate(FlinkPipelineExecutionEnvironment.java:101)
> at
org.apache.beam.runners.flink.FlinkRunner.run(FlinkRunner.java:105)
> at org.apache.beam.sdk.Pipeline.run(Pipeline.java:313)
> at org.apache.beam.sdk.Pipeline.run(Pipeline.java:299)
> at
ac.cn.iie.process.RelationProcess.process(RelationProcess.java:119)
> at ac.cn.iie.Bootstrap.main(Bootstrap.java:16)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
>
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
>
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
>
org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:528)
> ... 9 more
> Caused by: org.apache.kafka.common.KafkaException: Failed to
construct
> kafka consumer
> at
>
org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:781)
> at
>
org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:604)
> at
>
org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:587)
> at
>
org.apache.beam.sdk.io.kafka.KafkaUnboundedSource.split(KafkaUnboundedSource.java:64)
> at
> org.apache.beam.runners.flink.translation.wrappers.streaming.io
<http://streaming.io>.UnboundedSourceWrapper.<init>(UnboundedSourceWrapper.java:150)
> at
>
org.apache.beam.runners.flink.FlinkStreamingTransformTranslators$UnboundedReadSourceTranslator.translateNode(FlinkStreamingTransformTranslators.java:206)
> ... 31 more
> Caused by: java.lang.IllegalArgumentException: Could not find a
> 'KafkaClient' entry in the JAAS configuration. System property
> 'java.security.auth.login.config' is /etc/kafka/kafka_sink_jaas.conf
> at
>
org.apache.kafka.common.security.JaasContext.defaultContext(JaasContext.java:131)
> at
org.apache.kafka.common.security.JaasContext.load(JaasContext.java:96)
> at
org.apache.kafka.common.security.JaasContext.load(JaasContext.java:78)
> at
>
org.apache.kafka.common.network.ChannelBuilders.create(ChannelBuilders.java:103)
> at
>
org.apache.kafka.common.network.ChannelBuilders.clientChannelBuilder(ChannelBuilders.java:61)
> at
>
org.apache.kafka.clients.ClientUtils.createChannelBuilder(ClientUtils.java:86)
> at
>
org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:702)
> ... 36 more
>
> Thanks!
>
> On Mon, Nov 5, 2018 at 6:29 PM Maximilian Michels <[email protected]
<mailto:[email protected]>
> <mailto:[email protected] <mailto:[email protected]>>> wrote:
>
> Hi Fred,
>
> This is hard to debug without further information. Maybe a
stack trace
> would help. Are you sure the config is also located on the remote
> cluster?
>
> Thanks,
> Max
>
> On 03.11.18 15:45, K Fred wrote:
> > Hi,
> >
> > I am running into a very strange issue that
> > 'Could not find a "KafkaClient" entry in the JAAS
configuration.
> System
> > property "java.security.auth.login.config" is
> > /etc/kafka/kafka_sink_jaas.conf'
> > on a single node flink cluster when i consume record from
kafka
> using
> > beam KafkaIO.
> >
> > The JAAS file contains 'KafkaClient', but the flink cluster
> cannot find
> > the entry. Does anyone know the reason caused?
> >
> > Thanks!
> > --
> >
> > Fred
> >
>
> --
>
> Fred
>
--
Fred
