Hi Fred,
Just to double check: Are you running this from a cluster or your local
machine? Asking because the stack trace indicates that the exception
occurs during job submission through the Flink command-line client. So
the machine you're running this on should also have the file located in
/etc.
Thanks,
Max
On 05.11.18 12:26, K Fred wrote:
Hi Max,
Yeah, The config is always located on the remote cluster. The exception
looks like that my application can find the config file, but cannot find
out the config's KafkaClient entry. So i guess the reason may be related
to flink cluster some settings!
/These code depict some stack trace below:/
-----------------------------------------------------------------------------
The program finished with the following exception:
org.apache.flink.client.program.ProgramInvocationException: The main
method caused an error.
at
org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:545)
at
org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:420)
at org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:404)
at
org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:785)
at org.apache.flink.client.cli.CliFrontend.runProgram(CliFrontend.java:279)
at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:214)
at
org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:1025)
at
org.apache.flink.client.cli.CliFrontend.lambda$main$9(CliFrontend.java:1101)
at
org.apache.flink.runtime.security.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:30)
at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1101)
Caused by: java.lang.RuntimeException: Error while translating
UnboundedSource: org.apache.beam.sdk.io.kafka.KafkaUnboundedSource@7bc6d27a
at
org.apache.beam.runners.flink.FlinkStreamingTransformTranslators$UnboundedReadSourceTranslator.translateNode(FlinkStreamingTransformTranslators.java:225)
at
org.apache.beam.runners.flink.FlinkStreamingTransformTranslators$ReadSourceTranslator.translateNode(FlinkStreamingTransformTranslators.java:273)
at
org.apache.beam.runners.flink.FlinkStreamingPipelineTranslator.applyStreamingTransform(FlinkStreamingPipelineTranslator.java:122)
at
org.apache.beam.runners.flink.FlinkStreamingPipelineTranslator.visitPrimitiveTransform(FlinkStreamingPipelineTranslator.java:101)
at
org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:657)
at
org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:649)
at
org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:649)
at
org.apache.beam.sdk.runners.TransformHierarchy$Node.access$600(TransformHierarchy.java:311)
at
org.apache.beam.sdk.runners.TransformHierarchy.visit(TransformHierarchy.java:245)
at org.apache.beam.sdk.Pipeline.traverseTopologically(Pipeline.java:458)
at
org.apache.beam.runners.flink.FlinkPipelineTranslator.translate(FlinkPipelineTranslator.java:38)
at
org.apache.beam.runners.flink.FlinkStreamingPipelineTranslator.translate(FlinkStreamingPipelineTranslator.java:53)
at
org.apache.beam.runners.flink.FlinkPipelineExecutionEnvironment.translate(FlinkPipelineExecutionEnvironment.java:101)
at org.apache.beam.runners.flink.FlinkRunner.run(FlinkRunner.java:105)
at org.apache.beam.sdk.Pipeline.run(Pipeline.java:313)
at org.apache.beam.sdk.Pipeline.run(Pipeline.java:299)
at ac.cn.iie.process.RelationProcess.process(RelationProcess.java:119)
at ac.cn.iie.Bootstrap.main(Bootstrap.java:16)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:528)
... 9 more
Caused by: org.apache.kafka.common.KafkaException: Failed to construct
kafka consumer
at
org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:781)
at
org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:604)
at
org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:587)
at
org.apache.beam.sdk.io.kafka.KafkaUnboundedSource.split(KafkaUnboundedSource.java:64)
at
org.apache.beam.runners.flink.translation.wrappers.streaming.io.UnboundedSourceWrapper.<init>(UnboundedSourceWrapper.java:150)
at
org.apache.beam.runners.flink.FlinkStreamingTransformTranslators$UnboundedReadSourceTranslator.translateNode(FlinkStreamingTransformTranslators.java:206)
... 31 more
Caused by: java.lang.IllegalArgumentException: Could not find a
'KafkaClient' entry in the JAAS configuration. System property
'java.security.auth.login.config' is /etc/kafka/kafka_sink_jaas.conf
at
org.apache.kafka.common.security.JaasContext.defaultContext(JaasContext.java:131)
at org.apache.kafka.common.security.JaasContext.load(JaasContext.java:96)
at org.apache.kafka.common.security.JaasContext.load(JaasContext.java:78)
at
org.apache.kafka.common.network.ChannelBuilders.create(ChannelBuilders.java:103)
at
org.apache.kafka.common.network.ChannelBuilders.clientChannelBuilder(ChannelBuilders.java:61)
at
org.apache.kafka.clients.ClientUtils.createChannelBuilder(ClientUtils.java:86)
at
org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:702)
... 36 more
Thanks!
On Mon, Nov 5, 2018 at 6:29 PM Maximilian Michels <[email protected]
<mailto:[email protected]>> wrote:
Hi Fred,
This is hard to debug without further information. Maybe a stack trace
would help. Are you sure the config is also located on the remote
cluster?
Thanks,
Max
On 03.11.18 15:45, K Fred wrote:
> Hi,
>
> I am running into a very strange issue that
> 'Could not find a "KafkaClient" entry in the JAAS configuration.
System
> property "java.security.auth.login.config" is
> /etc/kafka/kafka_sink_jaas.conf'
> on a single node flink cluster when i consume record from kafka
using
> beam KafkaIO.
>
> The JAAS file contains 'KafkaClient', but the flink cluster
cannot find
> the entry. Does anyone know the reason caused?
>
> Thanks!
> --
>
> Fred
>
--
Fred
