[
https://issues.apache.org/jira/browse/HUDI-3113?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
sivabalan narayanan updated HUDI-3113:
--------------------------------------
Component/s: kafka-connect
> Kafka Connect create Multiple Embedded Timeline Services
> --------------------------------------------------------
>
> Key: HUDI-3113
> URL: https://issues.apache.org/jira/browse/HUDI-3113
> Project: Apache Hudi
> Issue Type: Task
> Components: kafka-connect
> Reporter: cdmikechen
> Priority: Major
>
> After Kafka Connect started, I've found that hudi will create not one
> Embedded Timeline Service.
> {code}
> [2021-12-28 07:52:57,154] INFO Starting Timeline service !!
> (org.apache.hudi.client.embedded.EmbeddedTimelineService)
> [2021-12-28 07:52:57,155] WARN Unable to find driver bind address from spark
> config (org.apache.hudi.client.embedded.EmbeddedTimelineService)
> [2021-12-28 07:52:57,169] INFO Creating View Manager with storage type
> :MEMORY (org.apache.hudi.common.table.view.FileSystemViewManager)
> [2021-12-28 07:52:57,170] INFO Creating in-memory based Table View
> (org.apache.hudi.common.table.view.FileSystemViewManager)
> [2021-12-28 07:52:57,184] INFO Logging initialized @27658ms to
> org.apache.hudi.org.eclipse.jetty.util.log.Slf4jLog
> (org.apache.hudi.org.eclipse.jetty.util.log)
> [2021-12-28 07:52:57,502] INFO
> __ __ _
> / /____ _ _ __ ____ _ / /(_)____
> __ / // __ `/| | / // __ `// // // __ \
> / /_/ // /_/ / | |/ // /_/ // // // / / /
> \____/ \__,_/ |___/ \__,_//_//_//_/ /_/
> https://javalin.io/documentation
> (io.javalin.Javalin)
> [2021-12-28 07:52:57,504] INFO Starting Javalin ... (io.javalin.Javalin)
> [2021-12-28 07:52:57,650] INFO Listening on http://localhost:43691/
> (io.javalin.Javalin)
> [2021-12-28 07:52:57,650] INFO Javalin started in 151ms \o/
> (io.javalin.Javalin)
> [2021-12-28 07:52:57,650] INFO Starting Timeline server on port :43691
> (org.apache.hudi.timeline.service.TimelineService)
> [2021-12-28 07:52:57,650] INFO Started embedded timeline server at
> 172.17.0.7:43691 (org.apache.hudi.client.embedded.EmbeddedTimelineService)
> [2021-12-28 07:52:57,661] INFO Start Transaction Coordinator for topic
> hudi-test-topic partition 0
> (org.apache.hudi.connect.transaction.ConnectTransactionCoordinator)
> [2021-12-28 07:52:57,690] INFO Loaded instants upto :
> Option\{val=[==>20211228075022280__commit__INFLIGHT]}
> (org.apache.hudi.common.table.timeline.HoodieActiveTimeline)
> [2021-12-28 07:52:57,822] INFO Retrieved Raw Kafka offsets from Hudi Commit
> File 0=100 (org.apache.hudi.connect.transaction.ConnectTransactionCoordinator)
> [2021-12-28 07:52:57,823] INFO Initialized the kafka offset commits \{0=100}
> (org.apache.hudi.connect.transaction.ConnectTransactionCoordinator)
> [2021-12-28 07:52:57,858] INFO The value of
> hoodie.datasource.write.keygenerator.type is empty, using SIMPLE
> (org.apache.hudi.keygen.factory.HoodieAvroKeyGeneratorFactory)
> [2021-12-28 07:52:57,928] INFO AdminClientConfig values:
> bootstrap.servers = [10.3.101.60:9092]
> client.dns.lookup = use_all_dns_ips
> client.id =
> connections.max.idle.ms = 300000
> default.api.timeout.ms = 60000
> metadata.max.age.ms = 300000
> metric.reporters = []
> metrics.num.samples = 2
> metrics.recording.level = INFO
> metrics.sample.window.ms = 30000
> receive.buffer.bytes = 65536
> reconnect.backoff.max.ms = 1000
> reconnect.backoff.ms = 50
> request.timeout.ms = 30000
> retries = 2147483647
> retry.backoff.ms = 100
> sasl.client.callback.handler.class = null
> sasl.jaas.config = null
> sasl.kerberos.kinit.cmd = /usr/bin/kinit
> sasl.kerberos.min.time.before.relogin = 60000
> sasl.kerberos.service.name = null
> sasl.kerberos.ticket.renew.jitter = 0.05
> sasl.kerberos.ticket.renew.window.factor = 0.8
> sasl.login.callback.handler.class = null
> sasl.login.class = null
> sasl.login.refresh.buffer.seconds = 300
> sasl.login.refresh.min.period.seconds = 60
> sasl.login.refresh.window.factor = 0.8
> sasl.login.refresh.window.jitter = 0.05
> sasl.mechanism = GSSAPI
> security.protocol = PLAINTEXT
> security.providers = null
> send.buffer.bytes = 131072
> socket.connection.setup.timeout.max.ms = 127000
> socket.connection.setup.timeout.ms = 10000
> ssl.cipher.suites = null
> ssl.enabled.protocols = [TLSv1.2, TLSv1.3]
> ssl.endpoint.identification.algorithm = https
> ssl.engine.factory.class = null
> ssl.key.password = null
> ssl.keymanager.algorithm = SunX509
> ssl.keystore.certificate.chain = null
> ssl.keystore.key = null
> ssl.keystore.location = null
> ssl.keystore.password = null
> ssl.keystore.type = JKS
> ssl.protocol = TLSv1.3
> ssl.provider = null
> ssl.secure.random.implementation = null
> ssl.trustmanager.algorithm = PKIX
> ssl.truststore.certificates = null
> ssl.truststore.location = null
> ssl.truststore.password = null
> ssl.truststore.type = JKS
> (org.apache.kafka.clients.admin.AdminClientConfig)
> [2021-12-28 07:52:57,930] INFO Kafka version: 6.1.1-ccs
> (org.apache.kafka.common.utils.AppInfoParser)
> [2021-12-28 07:52:57,930] INFO Kafka commitId: c209f70c6c2e52ae
> (org.apache.kafka.common.utils.AppInfoParser)
> [2021-12-28 07:52:57,930] INFO Kafka startTimeMs: 1640677977929
> (org.apache.kafka.common.utils.AppInfoParser)
> [2021-12-28 07:52:57,937] INFO Starting Timeline service !!
> (org.apache.hudi.client.embedded.EmbeddedTimelineService)
> [2021-12-28 07:52:57,938] WARN Unable to find driver bind address from spark
> config (org.apache.hudi.client.embedded.EmbeddedTimelineService)
> [2021-12-28 07:52:57,938] INFO Creating View Manager with storage type
> :MEMORY (org.apache.hudi.common.table.view.FileSystemViewManager)
> [2021-12-28 07:52:57,938] INFO Creating in-memory based Table View
> (org.apache.hudi.common.table.view.FileSystemViewManager)
> [2021-12-28 07:52:57,943] INFO Latest number of partitions for topic
> hudi-test-topic is 1 (org.apache.hudi.connect.utils.KafkaConnectUtils)
> [2021-12-28 07:52:57,947] INFO
> __ __ _
> / /____ _ _ __ ____ _ / /(_)____
> __ / // __ `/| | / // __ `// // // __ \
> / /_/ // /_/ / | |/ // /_/ // // // / / /
> \____/ \__,_/ |___/ \__,_//_//_//_/ /_/
> https://javalin.io/documentation
> (io.javalin.Javalin)
> [2021-12-28 07:52:57,947] INFO Starting Javalin ... (io.javalin.Javalin)
> [2021-12-28 07:52:57,959] INFO Loading HoodieTableMetaClient from
> hdfs://hdp-syzh-cluster/hive/warehouse/default.db/hudi_test_topic
> (org.apache.hudi.common.table.HoodieTableMetaClient)
> [2021-12-28 07:52:57,960] INFO Listening on http://localhost:41798/
> (io.javalin.Javalin)
> [2021-12-28 07:52:57,960] INFO Javalin started in 14ms \o/
> (io.javalin.Javalin)
> [2021-12-28 07:52:57,960] INFO Starting Timeline server on port :41798
> (org.apache.hudi.timeline.service.TimelineService)
> [2021-12-28 07:52:57,962] INFO Started embedded timeline server at
> 172.17.0.7:41798 (org.apache.hudi.client.embedded.EmbeddedTimelineService)
> {code}
>
> I think a Kafka Connect should only start one Embedded Timeline Service in
> the current environment.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
