Hi Max, Yeah, The config is always located on the remote cluster. The exception looks like that my application can find the config file, but cannot find out the config's KafkaClient entry. So i guess the reason may be related to flink cluster some settings!
*These code depict some stack trace below:* ----------------------------------------------------------------------------- The program finished with the following exception: org.apache.flink.client.program.ProgramInvocationException: The main method caused an error. at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:545) at org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:420) at org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:404) at org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:785) at org.apache.flink.client.cli.CliFrontend.runProgram(CliFrontend.java:279) at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:214) at org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:1025) at org.apache.flink.client.cli.CliFrontend.lambda$main$9(CliFrontend.java:1101) at org.apache.flink.runtime.security.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:30) at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1101) Caused by: java.lang.RuntimeException: Error while translating UnboundedSource: org.apache.beam.sdk.io.kafka.KafkaUnboundedSource@7bc6d27a at org.apache.beam.runners.flink.FlinkStreamingTransformTranslators$UnboundedReadSourceTranslator.translateNode(FlinkStreamingTransformTranslators.java:225) at org.apache.beam.runners.flink.FlinkStreamingTransformTranslators$ReadSourceTranslator.translateNode(FlinkStreamingTransformTranslators.java:273) at org.apache.beam.runners.flink.FlinkStreamingPipelineTranslator.applyStreamingTransform(FlinkStreamingPipelineTranslator.java:122) at org.apache.beam.runners.flink.FlinkStreamingPipelineTranslator.visitPrimitiveTransform(FlinkStreamingPipelineTranslator.java:101) at org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:657) at org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:649) at org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:649) at org.apache.beam.sdk.runners.TransformHierarchy$Node.access$600(TransformHierarchy.java:311) at org.apache.beam.sdk.runners.TransformHierarchy.visit(TransformHierarchy.java:245) at org.apache.beam.sdk.Pipeline.traverseTopologically(Pipeline.java:458) at org.apache.beam.runners.flink.FlinkPipelineTranslator.translate(FlinkPipelineTranslator.java:38) at org.apache.beam.runners.flink.FlinkStreamingPipelineTranslator.translate(FlinkStreamingPipelineTranslator.java:53) at org.apache.beam.runners.flink.FlinkPipelineExecutionEnvironment.translate(FlinkPipelineExecutionEnvironment.java:101) at org.apache.beam.runners.flink.FlinkRunner.run(FlinkRunner.java:105) at org.apache.beam.sdk.Pipeline.run(Pipeline.java:313) at org.apache.beam.sdk.Pipeline.run(Pipeline.java:299) at ac.cn.iie.process.RelationProcess.process(RelationProcess.java:119) at ac.cn.iie.Bootstrap.main(Bootstrap.java:16) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:528) ... 9 more Caused by: org.apache.kafka.common.KafkaException: Failed to construct kafka consumer at org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:781) at org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:604) at org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:587) at org.apache.beam.sdk.io.kafka.KafkaUnboundedSource.split(KafkaUnboundedSource.java:64) at org.apache.beam.runners.flink.translation.wrappers.streaming.io.UnboundedSourceWrapper.<init>(UnboundedSourceWrapper.java:150) at org.apache.beam.runners.flink.FlinkStreamingTransformTranslators$UnboundedReadSourceTranslator.translateNode(FlinkStreamingTransformTranslators.java:206) ... 31 more Caused by: java.lang.IllegalArgumentException: Could not find a 'KafkaClient' entry in the JAAS configuration. System property 'java.security.auth.login.config' is /etc/kafka/kafka_sink_jaas.conf at org.apache.kafka.common.security.JaasContext.defaultContext(JaasContext.java:131) at org.apache.kafka.common.security.JaasContext.load(JaasContext.java:96) at org.apache.kafka.common.security.JaasContext.load(JaasContext.java:78) at org.apache.kafka.common.network.ChannelBuilders.create(ChannelBuilders.java:103) at org.apache.kafka.common.network.ChannelBuilders.clientChannelBuilder(ChannelBuilders.java:61) at org.apache.kafka.clients.ClientUtils.createChannelBuilder(ClientUtils.java:86) at org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:702) ... 36 more Thanks! On Mon, Nov 5, 2018 at 6:29 PM Maximilian Michels <[email protected]> wrote: > Hi Fred, > > This is hard to debug without further information. Maybe a stack trace > would help. Are you sure the config is also located on the remote cluster? > > Thanks, > Max > > On 03.11.18 15:45, K Fred wrote: > > Hi, > > > > I am running into a very strange issue that > > 'Could not find a "KafkaClient" entry in the JAAS configuration. System > > property "java.security.auth.login.config" is > > /etc/kafka/kafka_sink_jaas.conf' > > on a single node flink cluster when i consume record from kafka using > > beam KafkaIO. > > > > The JAAS file contains 'KafkaClient', but the flink cluster cannot find > > the entry. Does anyone know the reason caused? > > > > Thanks! > > -- > > > > Fred > > > -- Fred
