And in Kylin tutorial, the topic is kylindemo, in your sample, the topic is kylin_demo. Please double check the topic name.
2017-10-13 14:27 GMT+08:00 Billy Liu <[email protected]>: > If you could package the source code, please try to add more debug log > when retrieving partition info by Kafka consumer. Check which topic and how > many partitions you got. > > 2017-10-12 23:19 GMT+08:00 Kumar, Manoj H <[email protected]>: > >> I guess KafkaSource Class file where it has >> enrichSourcePartitionBeforeBuild() to get partition values. There its >> getting Error out. Do we know how can we test out to find why its coming as >> 0 for start & end offset? >> >> >> >> Regards, >> >> Manoj >> >> >> >> *From:* Kumar, Manoj H >> *Sent:* Thursday, October 12, 2017 3:35 PM >> *To:* '[email protected]' >> *Subject:* RE: Kafka Streaming data - Error while building the Cube >> >> >> >> Yes its there.. I could see the messages.. >> >> >> >> Regards, >> >> Manoj >> >> >> >> *From:* Billy Liu [mailto:[email protected] <[email protected]>] >> *Sent:* Thursday, October 12, 2017 3:11 PM >> >> *To:* user >> *Subject:* Re: Kafka Streaming data - Error while building the Cube >> >> >> >> STREAMING_SALES_TABLE table reads messages from Kafka topic kylin_demoļ¼but >> got 0 message. >> >> >> >> Could you check if the topic has incoming message: >> bin/kafka-console-consumer.sh >> --zookeeper localhost:2181 --bootstrap-server localhost:9092 --topic >> kylin_demo >> >> >> >> 2017-10-12 17:19 GMT+08:00 Kumar, Manoj H <[email protected]>: >> >> Pls. find below information about consumer config from Kylin log file. >> >> >> >> 2017-10-11 02:11:43,787 INFO [pool-8-thread-1] >> threadpool.DefaultScheduler:123 : Job Fetcher: 0 should running, 0 >> actual running, 0 stopped, 0 ready, 1 already succeed, 0 error, 0 >> discarded, 0 others >> >> 2017-10-11 02:12:13,783 INFO [pool-8-thread-1] >> threadpool.DefaultScheduler:123 : Job Fetcher: 0 should running, 0 >> actual running, 0 stopped, 0 ready, 1 already succeed, 0 error, 0 >> discarded, 0 others >> >> 2017-10-11 02:12:40,734 INFO [http-bio-7070-exec-3] >> streaming.StreamingManager:222 : Reloading Streaming Metadata from folder >> kylin_metadata(key='/streaming')@kylin_metadata@hbase >> >> 2017-10-11 02:12:40,760 DEBUG [http-bio-7070-exec-3] >> streaming.StreamingManager:247 : Loaded 1 StreamingConfig(s) >> >> 2017-10-11 02:12:43,789 INFO [pool-8-thread-1] >> threadpool.DefaultScheduler:123 : Job Fetcher: 0 should running, 0 >> actual running, 0 stopped, 0 ready, 1 already succeed, 0 error, 0 >> discarded, 0 others >> >> 2017-10-11 02:13:13,788 INFO [pool-8-thread-1] >> threadpool.DefaultScheduler:123 : Job Fetcher: 0 should running, 0 >> actual running, 0 stopped, 0 ready, 1 already succeed, 0 error, 0 >> discarded, 0 others >> >> 2017-10-11 02:13:43,785 INFO [pool-8-thread-1] >> threadpool.DefaultScheduler:123 : Job Fetcher: 0 should running, 0 >> actual running, 0 stopped, 0 ready, 1 already succeed, 0 error, 0 >> discarded, 0 others >> >> 2017-10-11 02:14:13,789 INFO [pool-8-thread-1] >> threadpool.DefaultScheduler:123 : Job Fetcher: 0 should running, 0 >> actual running, 0 stopped, 0 ready, 1 already succeed, 0 error, 0 >> discarded, 0 others >> >> 2017-10-11 02:14:43,796 INFO [pool-8-thread-1] >> threadpool.DefaultScheduler:123 : Job Fetcher: 0 should running, 0 >> actual running, 0 stopped, 0 ready, 1 already succeed, 0 error, 0 >> discarded, 0 others >> >> 2017-10-11 02:15:03,911 DEBUG [http-bio-7070-exec-1] >> controller.StreamingController:255 : Saving StreamingConfig >> {"uuid":"8613b0e1-40ac-438c-bdf5-72be4d91c230","last_modifie >> d":1507705685859,"version":"2.1.0","name":"DEFAULT. >> STREAMING_SALES_TABLE","type":"kafka"} >> >> 2017-10-11 02:15:03,913 DEBUG [http-bio-7070-exec-1] >> controller.StreamingController:273 : Saving KafkaConfig >> {"uuid":"87dc6ab5-5141-4bd8-8e00-c16ec86dce41","last_modifie >> d":1507705685916,"version":"2.1.0","name":"DEFAULT. >> STREAMING_SALES_TABLE","clusters":[{"brokers":[{"id":" >> 1","host":"sandbox","port":"9092"}]}],"topic":"kylin_demo"," >> timeout":60000,"parserName":"org.apache.kylin.source.kafka. >> TimedJsonStreamParser","parserTimeStampField":null,"margin": >> 0,"parserProperties":"tsColName=order_time"} >> >> 2017-10-11 02:15:03,963 DEBUG [pool-7-thread-1] cachesync.Broadcaster:132 >> : Servers in the cluster: [localhost:7070] >> >> 2017-10-11 02:15:04,000 DEBUG [pool-7-thread-1] cachesync.Broadcaster:139 >> : Announcing new broadcast event: BroadcastEvent{entity=streaming, >> event=update, cacheKey=DEFAULT.STREAMING_SALES_TABLE} >> >> 2017-10-11 02:15:04,009 DEBUG [pool-7-thread-1] cachesync.Broadcaster:132 >> : Servers in the cluster: [localhost:7070] >> >> 2017-10-11 02:15:04,009 DEBUG [pool-7-thread-1] cachesync.Broadcaster:139 >> : Announcing new broadcast event: BroadcastEvent{entity=kafka, >> event=update, cacheKey=DEFAULT.STREAMING_SALES_TABLE} >> >> 2017-10-11 02:15:04,164 DEBUG [http-bio-7070-exec-9] >> cachesync.Broadcaster:236 : Done broadcasting metadata change: >> entity=streaming, event=UPDATE, cacheKey=DEFAULT.STREAMING_SALES_TABLE >> >> 2017-10-11 02:15:04,192 DEBUG [http-bio-7070-exec-10] >> cachesync.Broadcaster:236 : Done broadcasting metadata change: >> entity=kafka, event=UPDATE, cacheKey=DEFAULT.STREAMING_SALES_TABLE >> >> 2017-10-11 02:15:13,789 INFO [pool-8-thread-1] >> threadpool.DefaultScheduler:123 : Job Fetcher: 0 should running, 0 >> actual running, 0 stopped, 0 ready, 1 already succeed, 0 error, 0 >> discarded, 0 others >> >> 2017-10-11 02:15:23,780 DEBUG [http-bio-7070-exec-7] kafka.KafkaSource:83 >> : Last segment doesn't exist, and didn't initiate the start offset, will >> seek from topic's earliest offset. >> >> >> >> 2017-10-11 20:50:42,558 INFO [http-bio-7070-exec-8] >> utils.AppInfoParser:83 : Kafka version : 0.10.2-kafka-2.2.0 >> >> 2017-10-11 20:50:42,563 INFO [http-bio-7070-exec-8] >> utils.AppInfoParser:84 : Kafka commitId : unknown >> >> 2017-10-11 20:50:42,570 DEBUG [http-bio-7070-exec-8] >> kafka.KafkaSource:105 : Seek end offsets from topic >> >> 2017-10-11 20:50:42,570 INFO [http-bio-7070-exec-8] >> consumer.ConsumerConfig:196 : ConsumerConfig values: >> >> auto.commit.interval.ms = 5000 >> >> auto.offset.reset = latest >> >> bootstrap.servers = [localhost:9092] >> >> check.crcs = true >> >> client.id = >> >> connections.max.idle.ms = 540000 >> >> enable.auto.commit = false >> >> exclude.internal.topics = true >> >> fetch.max.bytes = 52428800 >> >> fetch.max.wait.ms = 500 >> >> fetch.min.bytes = 1 >> >> group.id = streaming_cube >> >> heartbeat.interval.ms = 3000 >> >> interceptor.classes = null >> >> internal.leave.group.on.close = true >> >> key.deserializer = class org.apache.kafka.common.serial >> ization.StringDeserializer >> >> max.partition.fetch.bytes = 1048576 >> >> max.poll.interval.ms = 300000 >> >> max.poll.records = 500 >> >> metadata.max.age.ms = 300000 >> >> metric.reporters = [] >> >> metrics.num.samples = 2 >> >> metrics.recording.level = INFO >> >> metrics.sample.window.ms = 30000 >> >> partition.assignment.strategy = [class >> org.apache.kafka.clients.consumer.RangeAssignor] >> >> receive.buffer.bytes = 65536 >> >> reconnect.backoff.ms = 50 >> >> request.timeout.ms = 305000 >> >> retry.backoff.ms = 100 >> >> sasl.jaas.config = null >> >> sasl.kerberos.kinit.cmd = /usr/bin/kinit >> >> sasl.kerberos.min.time.before.relogin = 60000 >> >> sasl.kerberos.service.name = null >> >> sasl.kerberos.ticket.renew.jitter = 0.05 >> >> request.timeout.ms = 305000 >> >> retry.backoff.ms = 100 >> >> sasl.jaas.config = null >> >> sasl.kerberos.kinit.cmd = /usr/bin/kinit >> >> sasl.kerberos.min.time.before.relogin = 60000 >> >> sasl.kerberos.service.name = null >> >> sasl.kerberos.ticket.renew.jitter = 0.05 >> >> sasl.kerberos.ticket.renew.window.factor = 0.8 >> >> sasl.mechanism = GSSAPI >> >> security.protocol = PLAINTEXT >> >> send.buffer.bytes = 131072 >> >> session.timeout.ms = 10000 >> >> ssl.cipher.suites = null >> >> ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1] >> >> ssl.endpoint.identification.algorithm = null >> >> ssl.key.password = null >> >> ssl.keymanager.algorithm = SunX509 >> >> ssl.keystore.location = null >> >> ssl.keystore.password = null >> >> ssl.keystore.type = JKS >> >> ssl.protocol = TLS >> >> ssl.provider = null >> >> ssl.secure.random.implementation = null >> >> ssl.trustmanager.algorithm = PKIX >> >> ssl.truststore.location = null >> >> ssl.truststore.password = null >> >> ssl.truststore.type = JKS >> >> value.deserializer = class org.apache.kafka.common.serial >> ization.StringDeserializer >> >> >> >> 2017-10-11 20:50:42,573 INFO [http-bio-7070-exec-8] >> utils.AppInfoParser:83 : Kafka version : 0.10.2-kafka-2.2.0 >> >> 2017-10-11 20:50:42,573 INFO [http-bio-7070-exec-8] >> utils.AppInfoParser:84 : Kafka commitId : unknown >> >> 2017-10-11 20:50:42,586 DEBUG [http-bio-7070-exec-8] >> kafka.KafkaSource:107 : The end offsets are {0=0} >> >> 2017-10-11 20:50:42,588 ERROR [http-bio-7070-exec-8] >> controller.CubeController:305 : No new message comes, startOffset = >> endOffset:0 >> >> java.lang.IllegalArgumentException: No new message comes, startOffset = >> endOffset:0 >> >> at org.apache.kylin.source.kafka.KafkaSource.enrichSourcePartit >> ionBeforeBuild(KafkaSource.java:134) >> >> at org.apache.kylin.rest.service.JobService.submitJobInternal(J >> obService.java:236) >> >> Regards, >> >> Manoj >> >> >> >> *From:* Billy Liu [mailto:[email protected]] >> *Sent:* Thursday, October 12, 2017 1:06 PM >> *To:* user >> *Subject:* Re: Kafka Streaming data - Error while building the Cube >> >> >> >> Hi Kumar, >> >> >> >> Could you paste more Kafka Consumer related log in kylin.log? And also >> check from the Kafka broker side, if the Kylin client has connected to >> Broker. >> >> >> >> 2017-10-12 14:29 GMT+08:00 Kumar, Manoj H <[email protected]>: >> >> Building the Cube from Kylin UI - Although Messages are there in Kafka >> topic but Kylin is not able read the offset. Can someone help on this? >> >> 2017-10-11 20:50:42,573 INFO [http-bio-7070-exec-8] >> utils.AppInfoParser:83 : Kafka version : 0.10.2-kafka-2.2.0 >> 2017-10-11 20:50:42,573 INFO [http-bio-7070-exec-8] >> utils.AppInfoParser:84 : Kafka commitId : unknown >> 2017-10-11 20:50:42,586 DEBUG [http-bio-7070-exec-8] >> kafka.KafkaSource:107 : The end offsets are {0=0} >> 2017-10-11 20:50:42,588 ERROR [http-bio-7070-exec-8] >> controller.CubeController:305 : No new message comes, startOffset = >> endOffset:0 >> java.lang.IllegalArgumentException: No new message comes, startOffset = >> endOffset:0 >> at org.apache.kylin.source.kafka.KafkaSource.enrichSourcePartit >> ionBeforeBuild(KafkaSource.java:134) >> at org.apache.kylin.rest.service.JobService.submitJobInternal(J >> obService.java:236) >> at org.apache.kylin.rest.service.JobService.submitJob(JobServic >> e.java:208) >> at org.apache.kylin.rest.service.JobService$$FastClassBySpringC >> GLIB$$83a44b2a.invoke(<generated>) >> >> Regards, >> Manoj >> >> >> This message is confidential and subject to terms at: >> http://www.jpmorgan.com/emaildisclaimer including on confidentiality, >> legal privilege, viruses and monitoring of electronic messages. If you are >> not the intended recipient, please delete this message and notify the >> sender immediately. Any unauthorized use is strictly prohibited. >> >> >> >> This message is confidential and subject to terms at: http:// >> www.jpmorgan.com/emaildisclaimer including on confidentiality, legal >> privilege, viruses and monitoring of electronic messages. If you are not >> the intended recipient, please delete this message and notify the sender >> immediately. Any unauthorized use is strictly prohibited. >> >> >> >> This message is confidential and subject to terms at: http:// >> www.jpmorgan.com/emaildisclaimer including on confidentiality, legal >> privilege, viruses and monitoring of electronic messages. If you are not >> the intended recipient, please delete this message and notify the sender >> immediately. Any unauthorized use is strictly prohibited. >> > >
