Hi Max, I have resolved this issue. It's caused by the flink cluster kerberos configuration. Just need to set some config on flink-conf.yaml can make it work fine!
The settings is below: security.kerberos.login.use-ticket-cache: false security.kerberos.login.keytab: /etc/kafka/kafka.keytab security.kerberos.login.principal: [email protected] security.kerberos.login.contexts: Client,KafkaClient Thanks, Fred. On Tue, Nov 6, 2018 at 2:56 AM Maximilian Michels <[email protected]> wrote: > Hi Fred, > > Just to double check: Are you running this from a cluster or your local > machine? Asking because the stack trace indicates that the exception > occurs during job submission through the Flink command-line client. So > the machine you're running this on should also have the file located in > /etc. > > Thanks, > Max > > On 05.11.18 12:26, K Fred wrote: > > Hi Max, > > > > Yeah, The config is always located on the remote cluster. The exception > > looks like that my application can find the config file, but cannot find > > out the config's KafkaClient entry. So i guess the reason may be related > > to flink cluster some settings! > > > > /These code depict some stack trace below:/ > > > ----------------------------------------------------------------------------- > > The program finished with the following exception: > > > > org.apache.flink.client.program.ProgramInvocationException: The main > > method caused an error. > > at > > > org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:545) > > at > > > org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:420) > > at > org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:404) > > at > > > org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:785) > > at > org.apache.flink.client.cli.CliFrontend.runProgram(CliFrontend.java:279) > > at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:214) > > at > > > org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:1025) > > at > > > org.apache.flink.client.cli.CliFrontend.lambda$main$9(CliFrontend.java:1101) > > at > > > org.apache.flink.runtime.security.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:30) > > at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1101) > > Caused by: java.lang.RuntimeException: Error while translating > > UnboundedSource: > org.apache.beam.sdk.io.kafka.KafkaUnboundedSource@7bc6d27a > > at > > > org.apache.beam.runners.flink.FlinkStreamingTransformTranslators$UnboundedReadSourceTranslator.translateNode(FlinkStreamingTransformTranslators.java:225) > > at > > > org.apache.beam.runners.flink.FlinkStreamingTransformTranslators$ReadSourceTranslator.translateNode(FlinkStreamingTransformTranslators.java:273) > > at > > > org.apache.beam.runners.flink.FlinkStreamingPipelineTranslator.applyStreamingTransform(FlinkStreamingPipelineTranslator.java:122) > > at > > > org.apache.beam.runners.flink.FlinkStreamingPipelineTranslator.visitPrimitiveTransform(FlinkStreamingPipelineTranslator.java:101) > > at > > > org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:657) > > at > > > org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:649) > > at > > > org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:649) > > at > > > org.apache.beam.sdk.runners.TransformHierarchy$Node.access$600(TransformHierarchy.java:311) > > at > > > org.apache.beam.sdk.runners.TransformHierarchy.visit(TransformHierarchy.java:245) > > at org.apache.beam.sdk.Pipeline.traverseTopologically(Pipeline.java:458) > > at > > > org.apache.beam.runners.flink.FlinkPipelineTranslator.translate(FlinkPipelineTranslator.java:38) > > at > > > org.apache.beam.runners.flink.FlinkStreamingPipelineTranslator.translate(FlinkStreamingPipelineTranslator.java:53) > > at > > > org.apache.beam.runners.flink.FlinkPipelineExecutionEnvironment.translate(FlinkPipelineExecutionEnvironment.java:101) > > at org.apache.beam.runners.flink.FlinkRunner.run(FlinkRunner.java:105) > > at org.apache.beam.sdk.Pipeline.run(Pipeline.java:313) > > at org.apache.beam.sdk.Pipeline.run(Pipeline.java:299) > > at ac.cn.iie.process.RelationProcess.process(RelationProcess.java:119) > > at ac.cn.iie.Bootstrap.main(Bootstrap.java:16) > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > > at > > > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > > at > > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > > at java.lang.reflect.Method.invoke(Method.java:498) > > at > > > org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:528) > > ... 9 more > > Caused by: org.apache.kafka.common.KafkaException: Failed to construct > > kafka consumer > > at > > > org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:781) > > at > > > org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:604) > > at > > > org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:587) > > at > > > org.apache.beam.sdk.io.kafka.KafkaUnboundedSource.split(KafkaUnboundedSource.java:64) > > at > > org.apache.beam.runners.flink.translation.wrappers.streaming.io > .UnboundedSourceWrapper.<init>(UnboundedSourceWrapper.java:150) > > at > > > org.apache.beam.runners.flink.FlinkStreamingTransformTranslators$UnboundedReadSourceTranslator.translateNode(FlinkStreamingTransformTranslators.java:206) > > ... 31 more > > Caused by: java.lang.IllegalArgumentException: Could not find a > > 'KafkaClient' entry in the JAAS configuration. System property > > 'java.security.auth.login.config' is /etc/kafka/kafka_sink_jaas.conf > > at > > > org.apache.kafka.common.security.JaasContext.defaultContext(JaasContext.java:131) > > at org.apache.kafka.common.security.JaasContext.load(JaasContext.java:96) > > at org.apache.kafka.common.security.JaasContext.load(JaasContext.java:78) > > at > > > org.apache.kafka.common.network.ChannelBuilders.create(ChannelBuilders.java:103) > > at > > > org.apache.kafka.common.network.ChannelBuilders.clientChannelBuilder(ChannelBuilders.java:61) > > at > > > org.apache.kafka.clients.ClientUtils.createChannelBuilder(ClientUtils.java:86) > > at > > > org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:702) > > ... 36 more > > > > Thanks! > > > > On Mon, Nov 5, 2018 at 6:29 PM Maximilian Michels <[email protected] > > <mailto:[email protected]>> wrote: > > > > Hi Fred, > > > > This is hard to debug without further information. Maybe a stack > trace > > would help. Are you sure the config is also located on the remote > > cluster? > > > > Thanks, > > Max > > > > On 03.11.18 15:45, K Fred wrote: > > > Hi, > > > > > > I am running into a very strange issue that > > > 'Could not find a "KafkaClient" entry in the JAAS configuration. > > System > > > property "java.security.auth.login.config" is > > > /etc/kafka/kafka_sink_jaas.conf' > > > on a single node flink cluster when i consume record from kafka > > using > > > beam KafkaIO. > > > > > > The JAAS file contains 'KafkaClient', but the flink cluster > > cannot find > > > the entry. Does anyone know the reason caused? > > > > > > Thanks! > > > -- > > > > > > Fred > > > > > > > -- > > > > Fred > > > -- Fred
