hudi-bot opened a new issue, #14974:
URL: https://github.com/apache/hudi/issues/14974
After Kafka Connect started, I've found that hudi will create not one
Embedded Timeline Service.
{code}
[2021-12-28 07:52:57,154] INFO Starting Timeline service !!
(org.apache.hudi.client.embedded.EmbeddedTimelineService)
[2021-12-28 07:52:57,155] WARN Unable to find driver bind address from spark
config (org.apache.hudi.client.embedded.EmbeddedTimelineService)
[2021-12-28 07:52:57,169] INFO Creating View Manager with storage type
:MEMORY (org.apache.hudi.common.table.view.FileSystemViewManager)
[2021-12-28 07:52:57,170] INFO Creating in-memory based Table View
(org.apache.hudi.common.table.view.FileSystemViewManager)
[2021-12-28 07:52:57,184] INFO Logging initialized @27658ms to
org.apache.hudi.org.eclipse.jetty.util.log.Slf4jLog
(org.apache.hudi.org.eclipse.jetty.util.log)
[2021-12-28 07:52:57,502] INFO
__ __ _
/ /____ _ _ __ ____ _ / /(_)____
__ / // __ `/| | / // __ `// // // __ \
/ /_/ // /_/ / | |/ // /_/ // // // / / /
\____/ \__,_/ |___/ \__,_//_//_//_/ /_/
https://javalin.io/documentation
(io.javalin.Javalin)
[2021-12-28 07:52:57,504] INFO Starting Javalin ... (io.javalin.Javalin)
[2021-12-28 07:52:57,650] INFO Listening on http://localhost:43691/
(io.javalin.Javalin)
[2021-12-28 07:52:57,650] INFO Javalin started in 151ms \o/
(io.javalin.Javalin)
[2021-12-28 07:52:57,650] INFO Starting Timeline server on port :43691
(org.apache.hudi.timeline.service.TimelineService)
[2021-12-28 07:52:57,650] INFO Started embedded timeline server at
172.17.0.7:43691 (org.apache.hudi.client.embedded.EmbeddedTimelineService)
[2021-12-28 07:52:57,661] INFO Start Transaction Coordinator for topic
hudi-test-topic partition 0
(org.apache.hudi.connect.transaction.ConnectTransactionCoordinator)
[2021-12-28 07:52:57,690] INFO Loaded instants upto :
Option\{val=[==>20211228075022280__commit__INFLIGHT]}
(org.apache.hudi.common.table.timeline.HoodieActiveTimeline)
[2021-12-28 07:52:57,822] INFO Retrieved Raw Kafka offsets from Hudi Commit
File 0=100 (org.apache.hudi.connect.transaction.ConnectTransactionCoordinator)
[2021-12-28 07:52:57,823] INFO Initialized the kafka offset commits \{0=100}
(org.apache.hudi.connect.transaction.ConnectTransactionCoordinator)
[2021-12-28 07:52:57,858] INFO The value of
hoodie.datasource.write.keygenerator.type is empty, using SIMPLE
(org.apache.hudi.keygen.factory.HoodieAvroKeyGeneratorFactory)
[2021-12-28 07:52:57,928] INFO AdminClientConfig values:
bootstrap.servers = [10.3.101.60:9092]
client.dns.lookup = use_all_dns_ips
client.id =
connections.max.idle.ms = 300000
default.api.timeout.ms = 60000
metadata.max.age.ms = 300000
metric.reporters = []
metrics.num.samples = 2
metrics.recording.level = INFO
metrics.sample.window.ms = 30000
receive.buffer.bytes = 65536
reconnect.backoff.max.ms = 1000
reconnect.backoff.ms = 50
request.timeout.ms = 30000
retries = 2147483647
retry.backoff.ms = 100
sasl.client.callback.handler.class = null
sasl.jaas.config = null
sasl.kerberos.kinit.cmd = /usr/bin/kinit
sasl.kerberos.min.time.before.relogin = 60000
sasl.kerberos.service.name = null
sasl.kerberos.ticket.renew.jitter = 0.05
sasl.kerberos.ticket.renew.window.factor = 0.8
sasl.login.callback.handler.class = null
sasl.login.class = null
sasl.login.refresh.buffer.seconds = 300
sasl.login.refresh.min.period.seconds = 60
sasl.login.refresh.window.factor = 0.8
sasl.login.refresh.window.jitter = 0.05
sasl.mechanism = GSSAPI
security.protocol = PLAINTEXT
security.providers = null
send.buffer.bytes = 131072
socket.connection.setup.timeout.max.ms = 127000
socket.connection.setup.timeout.ms = 10000
ssl.cipher.suites = null
ssl.enabled.protocols = [TLSv1.2, TLSv1.3]
ssl.endpoint.identification.algorithm = https
ssl.engine.factory.class = null
ssl.key.password = null
ssl.keymanager.algorithm = SunX509
ssl.keystore.certificate.chain = null
ssl.keystore.key = null
ssl.keystore.location = null
ssl.keystore.password = null
ssl.keystore.type = JKS
ssl.protocol = TLSv1.3
ssl.provider = null
ssl.secure.random.implementation = null
ssl.trustmanager.algorithm = PKIX
ssl.truststore.certificates = null
ssl.truststore.location = null
ssl.truststore.password = null
ssl.truststore.type = JKS
(org.apache.kafka.clients.admin.AdminClientConfig)
[2021-12-28 07:52:57,930] INFO Kafka version: 6.1.1-ccs
(org.apache.kafka.common.utils.AppInfoParser)
[2021-12-28 07:52:57,930] INFO Kafka commitId: c209f70c6c2e52ae
(org.apache.kafka.common.utils.AppInfoParser)
[2021-12-28 07:52:57,930] INFO Kafka startTimeMs: 1640677977929
(org.apache.kafka.common.utils.AppInfoParser)
[2021-12-28 07:52:57,937] INFO Starting Timeline service !!
(org.apache.hudi.client.embedded.EmbeddedTimelineService)
[2021-12-28 07:52:57,938] WARN Unable to find driver bind address from spark
config (org.apache.hudi.client.embedded.EmbeddedTimelineService)
[2021-12-28 07:52:57,938] INFO Creating View Manager with storage type
:MEMORY (org.apache.hudi.common.table.view.FileSystemViewManager)
[2021-12-28 07:52:57,938] INFO Creating in-memory based Table View
(org.apache.hudi.common.table.view.FileSystemViewManager)
[2021-12-28 07:52:57,943] INFO Latest number of partitions for topic
hudi-test-topic is 1 (org.apache.hudi.connect.utils.KafkaConnectUtils)
[2021-12-28 07:52:57,947] INFO
__ __ _
/ /____ _ _ __ ____ _ / /(_)____
__ / // __ `/| | / // __ `// // // __ \
/ /_/ // /_/ / | |/ // /_/ // // // / / /
\____/ \__,_/ |___/ \__,_//_//_//_/ /_/
https://javalin.io/documentation
(io.javalin.Javalin)
[2021-12-28 07:52:57,947] INFO Starting Javalin ... (io.javalin.Javalin)
[2021-12-28 07:52:57,959] INFO Loading HoodieTableMetaClient from
hdfs://hdp-syzh-cluster/hive/warehouse/default.db/hudi_test_topic
(org.apache.hudi.common.table.HoodieTableMetaClient)
[2021-12-28 07:52:57,960] INFO Listening on http://localhost:41798/
(io.javalin.Javalin)
[2021-12-28 07:52:57,960] INFO Javalin started in 14ms \o/
(io.javalin.Javalin)
[2021-12-28 07:52:57,960] INFO Starting Timeline server on port :41798
(org.apache.hudi.timeline.service.TimelineService)
[2021-12-28 07:52:57,962] INFO Started embedded timeline server at
172.17.0.7:41798 (org.apache.hudi.client.embedded.EmbeddedTimelineService)
{code}
I think a Kafka Connect should only start one Embedded Timeline Service in
the current environment.
## JIRA info
- Link: https://issues.apache.org/jira/browse/HUDI-3113
- Type: Improvement
- Epic: https://issues.apache.org/jira/browse/HUDI-2324
- Fix version(s):
- 1.1.0
---
## Comments
30/Jan/22 19:23;shivnarayan;CC [~guoyihua] ;;;
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
