cdmikechen created HUDI-3113:
--------------------------------
Summary: Kafka Connect create Multiple Embedded Timeline Services
Key: HUDI-3113
URL: https://issues.apache.org/jira/browse/HUDI-3113
Project: Apache Hudi
Issue Type: Bug
Reporter: cdmikechen
After Kafka Connect started, I've found that hudi will create not one Embedded
Timeline Service.
{code}
[2021-12-28 07:52:57,154] INFO Starting Timeline service !!
(org.apache.hudi.client.embedded.EmbeddedTimelineService)
[2021-12-28 07:52:57,155] WARN Unable to find driver bind address from spark
config (org.apache.hudi.client.embedded.EmbeddedTimelineService)
[2021-12-28 07:52:57,169] INFO Creating View Manager with storage type :MEMORY
(org.apache.hudi.common.table.view.FileSystemViewManager)
[2021-12-28 07:52:57,170] INFO Creating in-memory based Table View
(org.apache.hudi.common.table.view.FileSystemViewManager)
[2021-12-28 07:52:57,184] INFO Logging initialized @27658ms to
org.apache.hudi.org.eclipse.jetty.util.log.Slf4jLog
(org.apache.hudi.org.eclipse.jetty.util.log)
[2021-12-28 07:52:57,502] INFO
__ __ _
/ /____ _ _ __ ____ _ / /(_)____
__ / // __ `/| | / // __ `// // // __ \
/ /_/ // /_/ / | |/ // /_/ // // // / / /
\____/ \__,_/ |___/ \__,_//_//_//_/ /_/
https://javalin.io/documentation
(io.javalin.Javalin)
[2021-12-28 07:52:57,504] INFO Starting Javalin ... (io.javalin.Javalin)
[2021-12-28 07:52:57,650] INFO Listening on http://localhost:43691/
(io.javalin.Javalin)
[2021-12-28 07:52:57,650] INFO Javalin started in 151ms \o/ (io.javalin.Javalin)
[2021-12-28 07:52:57,650] INFO Starting Timeline server on port :43691
(org.apache.hudi.timeline.service.TimelineService)
[2021-12-28 07:52:57,650] INFO Started embedded timeline server at
172.17.0.7:43691 (org.apache.hudi.client.embedded.EmbeddedTimelineService)
[2021-12-28 07:52:57,661] INFO Start Transaction Coordinator for topic
hudi-test-topic partition 0
(org.apache.hudi.connect.transaction.ConnectTransactionCoordinator)
[2021-12-28 07:52:57,690] INFO Loaded instants upto :
Option\{val=[==>20211228075022280__commit__INFLIGHT]}
(org.apache.hudi.common.table.timeline.HoodieActiveTimeline)
[2021-12-28 07:52:57,822] INFO Retrieved Raw Kafka offsets from Hudi Commit
File 0=100 (org.apache.hudi.connect.transaction.ConnectTransactionCoordinator)
[2021-12-28 07:52:57,823] INFO Initialized the kafka offset commits \{0=100}
(org.apache.hudi.connect.transaction.ConnectTransactionCoordinator)
[2021-12-28 07:52:57,858] INFO The value of
hoodie.datasource.write.keygenerator.type is empty, using SIMPLE
(org.apache.hudi.keygen.factory.HoodieAvroKeyGeneratorFactory)
[2021-12-28 07:52:57,928] INFO AdminClientConfig values:
bootstrap.servers = [10.3.101.60:9092]
client.dns.lookup = use_all_dns_ips
client.id =
connections.max.idle.ms = 300000
default.api.timeout.ms = 60000
metadata.max.age.ms = 300000
metric.reporters = []
metrics.num.samples = 2
metrics.recording.level = INFO
metrics.sample.window.ms = 30000
receive.buffer.bytes = 65536
reconnect.backoff.max.ms = 1000
reconnect.backoff.ms = 50
request.timeout.ms = 30000
retries = 2147483647
retry.backoff.ms = 100
sasl.client.callback.handler.class = null
sasl.jaas.config = null
sasl.kerberos.kinit.cmd = /usr/bin/kinit
sasl.kerberos.min.time.before.relogin = 60000
sasl.kerberos.service.name = null
sasl.kerberos.ticket.renew.jitter = 0.05
sasl.kerberos.ticket.renew.window.factor = 0.8
sasl.login.callback.handler.class = null
sasl.login.class = null
sasl.login.refresh.buffer.seconds = 300
sasl.login.refresh.min.period.seconds = 60
sasl.login.refresh.window.factor = 0.8
sasl.login.refresh.window.jitter = 0.05
sasl.mechanism = GSSAPI
security.protocol = PLAINTEXT
security.providers = null
send.buffer.bytes = 131072
socket.connection.setup.timeout.max.ms = 127000
socket.connection.setup.timeout.ms = 10000
ssl.cipher.suites = null
ssl.enabled.protocols = [TLSv1.2, TLSv1.3]
ssl.endpoint.identification.algorithm = https
ssl.engine.factory.class = null
ssl.key.password = null
ssl.keymanager.algorithm = SunX509
ssl.keystore.certificate.chain = null
ssl.keystore.key = null
ssl.keystore.location = null
ssl.keystore.password = null
ssl.keystore.type = JKS
ssl.protocol = TLSv1.3
ssl.provider = null
ssl.secure.random.implementation = null
ssl.trustmanager.algorithm = PKIX
ssl.truststore.certificates = null
ssl.truststore.location = null
ssl.truststore.password = null
ssl.truststore.type = JKS
(org.apache.kafka.clients.admin.AdminClientConfig)
[2021-12-28 07:52:57,930] INFO Kafka version: 6.1.1-ccs
(org.apache.kafka.common.utils.AppInfoParser)
[2021-12-28 07:52:57,930] INFO Kafka commitId: c209f70c6c2e52ae
(org.apache.kafka.common.utils.AppInfoParser)
[2021-12-28 07:52:57,930] INFO Kafka startTimeMs: 1640677977929
(org.apache.kafka.common.utils.AppInfoParser)
[2021-12-28 07:52:57,937] INFO Starting Timeline service !!
(org.apache.hudi.client.embedded.EmbeddedTimelineService)
[2021-12-28 07:52:57,938] WARN Unable to find driver bind address from spark
config (org.apache.hudi.client.embedded.EmbeddedTimelineService)
[2021-12-28 07:52:57,938] INFO Creating View Manager with storage type :MEMORY
(org.apache.hudi.common.table.view.FileSystemViewManager)
[2021-12-28 07:52:57,938] INFO Creating in-memory based Table View
(org.apache.hudi.common.table.view.FileSystemViewManager)
[2021-12-28 07:52:57,943] INFO Latest number of partitions for topic
hudi-test-topic is 1 (org.apache.hudi.connect.utils.KafkaConnectUtils)
[2021-12-28 07:52:57,947] INFO
__ __ _
/ /____ _ _ __ ____ _ / /(_)____
__ / // __ `/| | / // __ `// // // __ \
/ /_/ // /_/ / | |/ // /_/ // // // / / /
\____/ \__,_/ |___/ \__,_//_//_//_/ /_/
https://javalin.io/documentation
(io.javalin.Javalin)
[2021-12-28 07:52:57,947] INFO Starting Javalin ... (io.javalin.Javalin)
[2021-12-28 07:52:57,959] INFO Loading HoodieTableMetaClient from
hdfs://hdp-syzh-cluster/hive/warehouse/default.db/hudi_test_topic
(org.apache.hudi.common.table.HoodieTableMetaClient)
[2021-12-28 07:52:57,960] INFO Listening on http://localhost:41798/
(io.javalin.Javalin)
[2021-12-28 07:52:57,960] INFO Javalin started in 14ms \o/ (io.javalin.Javalin)
[2021-12-28 07:52:57,960] INFO Starting Timeline server on port :41798
(org.apache.hudi.timeline.service.TimelineService)
[2021-12-28 07:52:57,962] INFO Started embedded timeline server at
172.17.0.7:41798 (org.apache.hudi.client.embedded.EmbeddedTimelineService)
{code}
I think a Kafka Connect should only start one Embedded Timeline Service in the
current environment.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
