Related to another thread: Is there a value in posting issues that get put here (with follow up solutions, like this thread, which indeed was excellent to have shared the solution with the list) in Stack Overflow? Again, for ease of discoverability, for those that face similar issues. Not sure how would formalize, but bringing up nonetheless.
On Tue, Nov 6, 2018 at 1:49 AM Maximilian Michels <[email protected]> wrote: > Hi Fred, > > I see! Thanks for posting your solution here. > > Best, > Max > > On 06.11.18 03:49, K Fred wrote: > > Hi Max, > > > > I have resolved this issue. It's caused by the flink cluster kerberos > > configuration. Just need to set some config on flink-conf.yaml can make > > it work fine! > > > > The settings is below: > > > > security.kerberos.login.use-ticket-cache: false > > security.kerberos.login.keytab: /etc/kafka/kafka.keytab > > security.kerberos.login.principal: [email protected] > > <mailto:[email protected]> > > security.kerberos.login.contexts: Client,KafkaClient > > > > > > Thanks, > > Fred. > > > > On Tue, Nov 6, 2018 at 2:56 AM Maximilian Michels <[email protected] > > <mailto:[email protected]>> wrote: > > > > Hi Fred, > > > > Just to double check: Are you running this from a cluster or your > local > > machine? Asking because the stack trace indicates that the exception > > occurs during job submission through the Flink command-line client. > So > > the machine you're running this on should also have the file located > in > > /etc. > > > > Thanks, > > Max > > > > On 05.11.18 12:26, K Fred wrote: > > > Hi Max, > > > > > > Yeah, The config is always located on the remote cluster. The > > exception > > > looks like that my application can find the config file, but > > cannot find > > > out the config's KafkaClient entry. So i guess the reason may be > > related > > > to flink cluster some settings! > > > > > > /These code depict some stack trace below:/ > > > > > > ----------------------------------------------------------------------------- > > > The program finished with the following exception: > > > > > > org.apache.flink.client.program.ProgramInvocationException: The > main > > > method caused an error. > > > at > > > > > > > org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:545) > > > at > > > > > > > org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:420) > > > at > > > org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:404) > > > at > > > > > > org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:785) > > > at > > > org.apache.flink.client.cli.CliFrontend.runProgram(CliFrontend.java:279) > > > at > org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:214) > > > at > > > > > > > org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:1025) > > > at > > > > > > org.apache.flink.client.cli.CliFrontend.lambda$main$9(CliFrontend.java:1101) > > > at > > > > > > > org.apache.flink.runtime.security.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:30) > > > at > > org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1101) > > > Caused by: java.lang.RuntimeException: Error while translating > > > UnboundedSource: > > org.apache.beam.sdk.io.kafka.KafkaUnboundedSource@7bc6d27a > > > at > > > > > > > org.apache.beam.runners.flink.FlinkStreamingTransformTranslators$UnboundedReadSourceTranslator.translateNode(FlinkStreamingTransformTranslators.java:225) > > > at > > > > > > > org.apache.beam.runners.flink.FlinkStreamingTransformTranslators$ReadSourceTranslator.translateNode(FlinkStreamingTransformTranslators.java:273) > > > at > > > > > > > org.apache.beam.runners.flink.FlinkStreamingPipelineTranslator.applyStreamingTransform(FlinkStreamingPipelineTranslator.java:122) > > > at > > > > > > > org.apache.beam.runners.flink.FlinkStreamingPipelineTranslator.visitPrimitiveTransform(FlinkStreamingPipelineTranslator.java:101) > > > at > > > > > > > org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:657) > > > at > > > > > > > org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:649) > > > at > > > > > > > org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:649) > > > at > > > > > > > org.apache.beam.sdk.runners.TransformHierarchy$Node.access$600(TransformHierarchy.java:311) > > > at > > > > > > > org.apache.beam.sdk.runners.TransformHierarchy.visit(TransformHierarchy.java:245) > > > at > > org.apache.beam.sdk.Pipeline.traverseTopologically(Pipeline.java:458) > > > at > > > > > > > org.apache.beam.runners.flink.FlinkPipelineTranslator.translate(FlinkPipelineTranslator.java:38) > > > at > > > > > > > org.apache.beam.runners.flink.FlinkStreamingPipelineTranslator.translate(FlinkStreamingPipelineTranslator.java:53) > > > at > > > > > > > org.apache.beam.runners.flink.FlinkPipelineExecutionEnvironment.translate(FlinkPipelineExecutionEnvironment.java:101) > > > at > > org.apache.beam.runners.flink.FlinkRunner.run(FlinkRunner.java:105) > > > at org.apache.beam.sdk.Pipeline.run(Pipeline.java:313) > > > at org.apache.beam.sdk.Pipeline.run(Pipeline.java:299) > > > at > > ac.cn.iie.process.RelationProcess.process(RelationProcess.java:119) > > > at ac.cn.iie.Bootstrap.main(Bootstrap.java:16) > > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > > > at > > > > > > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > > > at > > > > > > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > > > at java.lang.reflect.Method.invoke(Method.java:498) > > > at > > > > > > > org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:528) > > > ... 9 more > > > Caused by: org.apache.kafka.common.KafkaException: Failed to > > construct > > > kafka consumer > > > at > > > > > > > org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:781) > > > at > > > > > > > org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:604) > > > at > > > > > > > org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:587) > > > at > > > > > > > org.apache.beam.sdk.io.kafka.KafkaUnboundedSource.split(KafkaUnboundedSource.java:64) > > > at > > > org.apache.beam.runners.flink.translation.wrappers.streaming.io > > <http://streaming.io > >.UnboundedSourceWrapper.<init>(UnboundedSourceWrapper.java:150) > > > at > > > > > > > org.apache.beam.runners.flink.FlinkStreamingTransformTranslators$UnboundedReadSourceTranslator.translateNode(FlinkStreamingTransformTranslators.java:206) > > > ... 31 more > > > Caused by: java.lang.IllegalArgumentException: Could not find a > > > 'KafkaClient' entry in the JAAS configuration. System property > > > 'java.security.auth.login.config' is > /etc/kafka/kafka_sink_jaas.conf > > > at > > > > > > > org.apache.kafka.common.security.JaasContext.defaultContext(JaasContext.java:131) > > > at > > > org.apache.kafka.common.security.JaasContext.load(JaasContext.java:96) > > > at > > > org.apache.kafka.common.security.JaasContext.load(JaasContext.java:78) > > > at > > > > > > > org.apache.kafka.common.network.ChannelBuilders.create(ChannelBuilders.java:103) > > > at > > > > > > > org.apache.kafka.common.network.ChannelBuilders.clientChannelBuilder(ChannelBuilders.java:61) > > > at > > > > > > > org.apache.kafka.clients.ClientUtils.createChannelBuilder(ClientUtils.java:86) > > > at > > > > > > > org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:702) > > > ... 36 more > > > > > > Thanks! > > > > > > On Mon, Nov 5, 2018 at 6:29 PM Maximilian Michels <[email protected] > > <mailto:[email protected]> > > > <mailto:[email protected] <mailto:[email protected]>>> wrote: > > > > > > Hi Fred, > > > > > > This is hard to debug without further information. Maybe a > > stack trace > > > would help. Are you sure the config is also located on the > remote > > > cluster? > > > > > > Thanks, > > > Max > > > > > > On 03.11.18 15:45, K Fred wrote: > > > > Hi, > > > > > > > > I am running into a very strange issue that > > > > 'Could not find a "KafkaClient" entry in the JAAS > > configuration. > > > System > > > > property "java.security.auth.login.config" is > > > > /etc/kafka/kafka_sink_jaas.conf' > > > > on a single node flink cluster when i consume record from > > kafka > > > using > > > > beam KafkaIO. > > > > > > > > The JAAS file contains 'KafkaClient', but the flink cluster > > > cannot find > > > > the entry. Does anyone know the reason caused? > > > > > > > > Thanks! > > > > -- > > > > > > > > Fred > > > > > > > > > > -- > > > > > > Fred > > > > > > > -- > > > > Fred > > >
