Jan Tichý created GOBBLIN-1061:
----------------------------------
Summary: Kafka consumer Kerberos config
Key: GOBBLIN-1061
URL: https://issues.apache.org/jira/browse/GOBBLIN-1061
Project: Apache Gobblin
Issue Type: Bug
Affects Versions: 0.14.0
Environment: OS RHEL 7.5, Cloudera kafka 3.1.0 (kafka 1.0.1), CDH
5.16.2
Reporter: Jan Tichý
Attachments: application.conf, sip_voice_raw.pull
Does the gobblin support kerberos auth for kafka client consumer? I have right
kafka consumer settings with sasl_plaintext configuration but if it tries to
kinit it fails with exception below:
{code:java}
2020-02-20 05:44:00 PST INFO [DefaultQuartzScheduler_Worker-1]
org.apache.kafka.common.config.AbstractConfig - ConsumerConfig values:
metric.reporters = []
metadata.max.age.ms = 300000
value.deserializer = class io.confluent.kafka.serializers.KafkaAvroDeserializer
group.id = kafka09
partition.assignment.strategy =
[org.apache.kafka.clients.consumer.RangeAssignor]
reconnect.backoff.ms = 50
sasl.kerberos.ticket.renew.window.factor = 0.8
max.partition.fetch.bytes = 1048576
bootstrap.servers = [czrtim1hr.oskarmobil.cz:9092, czrtim2hr.oskarmobil.cz:9092]
retry.backoff.ms = 100
sasl.kerberos.kinit.cmd = /usr/bin/kinit
sasl.kerberos.service.name = kafka
sasl.kerberos.ticket.renew.jitter = 0.05
ssl.keystore.type = JKS
ssl.trustmanager.algorithm = PKIX
enable.auto.commit = false
ssl.key.password = null
fetch.max.wait.ms = 500
sasl.kerberos.min.time.before.relogin = 60000
connections.max.idle.ms = 540000
ssl.truststore.password = null
session.timeout.ms = 30000
metrics.num.samples = 2
client.id =
ssl.endpoint.identification.algorithm = null
key.deserializer = class io.confluent.kafka.serializers.KafkaAvroDeserializer
ssl.protocol = TLS
check.crcs = true
request.timeout.ms = 40000
ssl.provider = null
ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1]
ssl.keystore.location = null
heartbeat.interval.ms = 3000
auto.commit.interval.ms = 5000
receive.buffer.bytes = 32768
ssl.cipher.suites = null
ssl.truststore.type = JKS
security.protocol = SASL_PLAINTEXT
ssl.truststore.location = null
ssl.keystore.password = null
ssl.keymanager.algorithm = SunX509
metrics.sample.window.ms = 30000
fetch.min.bytes = 1
send.buffer.bytes = 131072
auto.offset.reset = latest2020-02-20 05:44:00 PST ERROR
[DefaultQuartzScheduler_Worker-1] org.apache.gobblin.runtime.SourceDecorator -
Failed to get work units for job job_SipVoiceRaw_1582206240042
org.apache.kafka.common.KafkaException: Failed to construct kafka consumer
at
org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:648)
at
org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:542)
at
org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:524)
at
org.apache.gobblin.kafka.client.Kafka09ConsumerClient.<init>(Kafka09ConsumerClient.java:116)
at
org.apache.gobblin.kafka.client.Kafka09ConsumerClient.<init>(Kafka09ConsumerClient.java:69)
at
org.apache.gobblin.kafka.client.Kafka09ConsumerClient$Factory.create(Kafka09ConsumerClient.java:224)
at
org.apache.gobblin.source.extractor.extract.kafka.KafkaSource.getWorkunits(KafkaSource.java:210)
at
org.apache.gobblin.runtime.SourceDecorator.getWorkunitStream(SourceDecorator.java:81)
at
org.apache.gobblin.runtime.AbstractJobLauncher.launchJob(AbstractJobLauncher.java:411)
at
org.apache.gobblin.cluster.GobblinHelixJobLauncher.launchJob(GobblinHelixJobLauncher.java:378)
at org.apache.gobblin.scheduler.JobScheduler.runJob(JobScheduler.java:487)
at
org.apache.gobblin.cluster.HelixRetriggeringJobCallable.runJobLauncherLoop(HelixRetriggeringJobCallable.java:203)
at
org.apache.gobblin.cluster.HelixRetriggeringJobCallable.call(HelixRetriggeringJobCallable.java:159)
at
org.apache.gobblin.cluster.GobblinHelixJobScheduler.runJob(GobblinHelixJobScheduler.java:228)
at
org.apache.gobblin.cluster.GobblinHelixJob.executeImpl(GobblinHelixJob.java:61)
at org.apache.gobblin.scheduler.BaseGobblinJob.execute(BaseGobblinJob.java:58)
at org.quartz.core.JobRunShell.run(JobRunShell.java:202)
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573)
Caused by: org.apache.kafka.common.KafkaException:
java.lang.IllegalArgumentException: You must pass
java.security.auth.login.config in secure mode.
at
org.apache.kafka.common.network.SaslChannelBuilder.configure(SaslChannelBuilder.java:74)
at
org.apache.kafka.common.network.ChannelBuilders.create(ChannelBuilders.java:60){code}
I have these conf options in my config for kerberos auth:
gobblin.yarn.keytab.file.path="/home/morpheus2/.keytab"
gobblin.yarn.keytab.principal.name=morpheus2
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
