STREAMING_SALES_TABLE table reads messages from Kafka topic kylin_demoļ¼but got 0 message.
Could you check if the topic has incoming message: bin/kafka-console-consumer.sh --zookeeper localhost:2181 --bootstrap-server localhost:9092 --topic kylin_demo 2017-10-12 17:19 GMT+08:00 Kumar, Manoj H <[email protected]>: > Pls. find below information about consumer config from Kylin log file. > > > > 2017-10-11 02:11:43,787 INFO [pool-8-thread-1] > threadpool.DefaultScheduler:123 : Job Fetcher: 0 should running, 0 actual > running, 0 stopped, 0 ready, 1 already succeed, 0 error, 0 discarded, 0 > others > > 2017-10-11 02:12:13,783 INFO [pool-8-thread-1] > threadpool.DefaultScheduler:123 : Job Fetcher: 0 should running, 0 actual > running, 0 stopped, 0 ready, 1 already succeed, 0 error, 0 discarded, 0 > others > > 2017-10-11 02:12:40,734 INFO [http-bio-7070-exec-3] > streaming.StreamingManager:222 : Reloading Streaming Metadata from folder > kylin_metadata(key='/streaming')@kylin_metadata@hbase > > 2017-10-11 02:12:40,760 DEBUG [http-bio-7070-exec-3] > streaming.StreamingManager:247 : Loaded 1 StreamingConfig(s) > > 2017-10-11 02:12:43,789 INFO [pool-8-thread-1] > threadpool.DefaultScheduler:123 : Job Fetcher: 0 should running, 0 actual > running, 0 stopped, 0 ready, 1 already succeed, 0 error, 0 discarded, 0 > others > > 2017-10-11 02:13:13,788 INFO [pool-8-thread-1] > threadpool.DefaultScheduler:123 : Job Fetcher: 0 should running, 0 actual > running, 0 stopped, 0 ready, 1 already succeed, 0 error, 0 discarded, 0 > others > > 2017-10-11 02:13:43,785 INFO [pool-8-thread-1] > threadpool.DefaultScheduler:123 : Job Fetcher: 0 should running, 0 actual > running, 0 stopped, 0 ready, 1 already succeed, 0 error, 0 discarded, 0 > others > > 2017-10-11 02:14:13,789 INFO [pool-8-thread-1] > threadpool.DefaultScheduler:123 : Job Fetcher: 0 should running, 0 actual > running, 0 stopped, 0 ready, 1 already succeed, 0 error, 0 discarded, 0 > others > > 2017-10-11 02:14:43,796 INFO [pool-8-thread-1] > threadpool.DefaultScheduler:123 : Job Fetcher: 0 should running, 0 actual > running, 0 stopped, 0 ready, 1 already succeed, 0 error, 0 discarded, 0 > others > > 2017-10-11 02:15:03,911 DEBUG [http-bio-7070-exec-1] > controller.StreamingController:255 > : Saving StreamingConfig {"uuid":"8613b0e1-40ac-438c- > bdf5-72be4d91c230","last_modified":1507705685859," > version":"2.1.0","name":"DEFAULT.STREAMING_SALES_TABLE","type":"kafka"} > > 2017-10-11 02:15:03,913 DEBUG [http-bio-7070-exec-1] > controller.StreamingController:273 > : Saving KafkaConfig {"uuid":"87dc6ab5-5141-4bd8-8e00-c16ec86dce41","last_ > modified":1507705685916,"version":"2.1.0","name":" > DEFAULT.STREAMING_SALES_TABLE","clusters":[{"brokers":[{"id" > :"1","host":"sandbox","port":"9092"}]}],"topic":"kylin_demo" > ,"timeout":60000,"parserName":"org.apache.kylin.source. > kafka.TimedJsonStreamParser","parserTimeStampField":null," > margin":0,"parserProperties":"tsColName=order_time"} > > 2017-10-11 02:15:03,963 DEBUG [pool-7-thread-1] cachesync.Broadcaster:132 > : Servers in the cluster: [localhost:7070] > > 2017-10-11 02:15:04,000 DEBUG [pool-7-thread-1] cachesync.Broadcaster:139 > : Announcing new broadcast event: BroadcastEvent{entity=streaming, > event=update, cacheKey=DEFAULT.STREAMING_SALES_TABLE} > > 2017-10-11 02:15:04,009 DEBUG [pool-7-thread-1] cachesync.Broadcaster:132 > : Servers in the cluster: [localhost:7070] > > 2017-10-11 02:15:04,009 DEBUG [pool-7-thread-1] cachesync.Broadcaster:139 > : Announcing new broadcast event: BroadcastEvent{entity=kafka, > event=update, cacheKey=DEFAULT.STREAMING_SALES_TABLE} > > 2017-10-11 02:15:04,164 DEBUG [http-bio-7070-exec-9] > cachesync.Broadcaster:236 : Done broadcasting metadata change: > entity=streaming, event=UPDATE, cacheKey=DEFAULT.STREAMING_SALES_TABLE > > 2017-10-11 02:15:04,192 DEBUG [http-bio-7070-exec-10] > cachesync.Broadcaster:236 : Done broadcasting metadata change: > entity=kafka, event=UPDATE, cacheKey=DEFAULT.STREAMING_SALES_TABLE > > 2017-10-11 02:15:13,789 INFO [pool-8-thread-1] > threadpool.DefaultScheduler:123 : Job Fetcher: 0 should running, 0 actual > running, 0 stopped, 0 ready, 1 already succeed, 0 error, 0 discarded, 0 > others > > 2017-10-11 02:15:23,780 DEBUG [http-bio-7070-exec-7] kafka.KafkaSource:83 > : Last segment doesn't exist, and didn't initiate the start offset, will > seek from topic's earliest offset. > > > > 2017-10-11 20:50:42,558 INFO [http-bio-7070-exec-8] > utils.AppInfoParser:83 : Kafka version : 0.10.2-kafka-2.2.0 > > 2017-10-11 20:50:42,563 INFO [http-bio-7070-exec-8] > utils.AppInfoParser:84 : Kafka commitId : unknown > > 2017-10-11 20:50:42,570 DEBUG [http-bio-7070-exec-8] kafka.KafkaSource:105 > : Seek end offsets from topic > > 2017-10-11 20:50:42,570 INFO [http-bio-7070-exec-8] > consumer.ConsumerConfig:196 : ConsumerConfig values: > > auto.commit.interval.ms = 5000 > > auto.offset.reset = latest > > bootstrap.servers = [localhost:9092] > > check.crcs = true > > client.id = > > connections.max.idle.ms = 540000 > > enable.auto.commit = false > > exclude.internal.topics = true > > fetch.max.bytes = 52428800 > > fetch.max.wait.ms = 500 > > fetch.min.bytes = 1 > > group.id = streaming_cube > > heartbeat.interval.ms = 3000 > > interceptor.classes = null > > internal.leave.group.on.close = true > > key.deserializer = class org.apache.kafka.common.serialization. > StringDeserializer > > max.partition.fetch.bytes = 1048576 > > max.poll.interval.ms = 300000 > > max.poll.records = 500 > > metadata.max.age.ms = 300000 > > metric.reporters = [] > > metrics.num.samples = 2 > > metrics.recording.level = INFO > > metrics.sample.window.ms = 30000 > > partition.assignment.strategy = [class org.apache.kafka.clients. > consumer.RangeAssignor] > > receive.buffer.bytes = 65536 > > reconnect.backoff.ms = 50 > > request.timeout.ms = 305000 > > retry.backoff.ms = 100 > > sasl.jaas.config = null > > sasl.kerberos.kinit.cmd = /usr/bin/kinit > > sasl.kerberos.min.time.before.relogin = 60000 > > sasl.kerberos.service.name = null > > sasl.kerberos.ticket.renew.jitter = 0.05 > > request.timeout.ms = 305000 > > retry.backoff.ms = 100 > > sasl.jaas.config = null > > sasl.kerberos.kinit.cmd = /usr/bin/kinit > > sasl.kerberos.min.time.before.relogin = 60000 > > sasl.kerberos.service.name = null > > sasl.kerberos.ticket.renew.jitter = 0.05 > > sasl.kerberos.ticket.renew.window.factor = 0.8 > > sasl.mechanism = GSSAPI > > security.protocol = PLAINTEXT > > send.buffer.bytes = 131072 > > session.timeout.ms = 10000 > > ssl.cipher.suites = null > > ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1] > > ssl.endpoint.identification.algorithm = null > > ssl.key.password = null > > ssl.keymanager.algorithm = SunX509 > > ssl.keystore.location = null > > ssl.keystore.password = null > > ssl.keystore.type = JKS > > ssl.protocol = TLS > > ssl.provider = null > > ssl.secure.random.implementation = null > > ssl.trustmanager.algorithm = PKIX > > ssl.truststore.location = null > > ssl.truststore.password = null > > ssl.truststore.type = JKS > > value.deserializer = class org.apache.kafka.common.serialization. > StringDeserializer > > > > 2017-10-11 20:50:42,573 INFO [http-bio-7070-exec-8] > utils.AppInfoParser:83 : Kafka version : 0.10.2-kafka-2.2.0 > > 2017-10-11 20:50:42,573 INFO [http-bio-7070-exec-8] > utils.AppInfoParser:84 : Kafka commitId : unknown > > 2017-10-11 20:50:42,586 DEBUG [http-bio-7070-exec-8] kafka.KafkaSource:107 > : The end offsets are {0=0} > > 2017-10-11 20:50:42,588 ERROR [http-bio-7070-exec-8] > controller.CubeController:305 : No new message comes, startOffset = > endOffset:0 > > java.lang.IllegalArgumentException: No new message comes, startOffset = > endOffset:0 > > at org.apache.kylin.source.kafka.KafkaSource. > enrichSourcePartitionBeforeBuild(KafkaSource.java:134) > > at org.apache.kylin.rest.service.JobService.submitJobInternal( > JobService.java:236) > > Regards, > > Manoj > > > > *From:* Billy Liu [mailto:[email protected]] > *Sent:* Thursday, October 12, 2017 1:06 PM > *To:* user > *Subject:* Re: Kafka Streaming data - Error while building the Cube > > > > Hi Kumar, > > > > Could you paste more Kafka Consumer related log in kylin.log? And also > check from the Kafka broker side, if the Kylin client has connected to > Broker. > > > > 2017-10-12 14:29 GMT+08:00 Kumar, Manoj H <[email protected]>: > > Building the Cube from Kylin UI - Although Messages are there in Kafka > topic but Kylin is not able read the offset. Can someone help on this? > > 2017-10-11 20:50:42,573 INFO [http-bio-7070-exec-8] > utils.AppInfoParser:83 : Kafka version : 0.10.2-kafka-2.2.0 > 2017-10-11 20:50:42,573 INFO [http-bio-7070-exec-8] > utils.AppInfoParser:84 : Kafka commitId : unknown > 2017-10-11 20:50:42,586 DEBUG [http-bio-7070-exec-8] kafka.KafkaSource:107 > : The end offsets are {0=0} > 2017-10-11 20:50:42,588 ERROR [http-bio-7070-exec-8] > controller.CubeController:305 : No new message comes, startOffset = > endOffset:0 > java.lang.IllegalArgumentException: No new message comes, startOffset = > endOffset:0 > at org.apache.kylin.source.kafka.KafkaSource. > enrichSourcePartitionBeforeBuild(KafkaSource.java:134) > at org.apache.kylin.rest.service.JobService.submitJobInternal( > JobService.java:236) > at org.apache.kylin.rest.service.JobService.submitJob( > JobService.java:208) > at org.apache.kylin.rest.service.JobService$$ > FastClassBySpringCGLIB$$83a44b2a.invoke(<generated>) > > Regards, > Manoj > > > This message is confidential and subject to terms at: > http://www.jpmorgan.com/emaildisclaimer including on confidentiality, > legal privilege, viruses and monitoring of electronic messages. If you are > not the intended recipient, please delete this message and notify the > sender immediately. Any unauthorized use is strictly prohibited. > > > > This message is confidential and subject to terms at: http:// > www.jpmorgan.com/emaildisclaimer including on confidentiality, legal > privilege, viruses and monitoring of electronic messages. If you are not > the intended recipient, please delete this message and notify the sender > immediately. Any unauthorized use is strictly prohibited. >
