>From the stacktrace, it's all HDFS code. Have you verified the Hadoop cluster is basically working? By run a sample MR job for example.
On Tue, Dec 13, 2016 at 4:01 PM, JunZhang <[email protected]> wrote: > Hi , 根据官方文档用Kylin1.6 + Kafka1.0 测试Streaming cube build时， 在 第一步 就反复报如下 > 错误，部分日志信息如下，请指导 ！2016-12-12 23:37:10,080 INFO [pool-9-thread-1] > common.AbstractHadoopJob:499 : HDFS meta dir is: > file:///opt/kylin/bin/../tomcat/temp/kylin_job_meta2568880879058796161/ > meta2016-12-12 > 23:37:10,080 INFO [pool-9-thread-1] common.AbstractHadoopJob:372 : Job > 'tmpfiles' updated -- > file:///opt/kylin/bin/../tomcat/temp/kylin_job_meta2568880879058796161/ > meta2016-12-12 > 23:37:10,399 INFO [pool-9-thread-1] client.RMProxy:98 : Connecting to > ResourceManager at quickstart.cloudera/127.0.0.1:80322016-12-12 > 23:37:14,158 > WARN [DataStreamer for file > /user/root/.staging/job_1481606707079_0001/files/meta/ > cube_desc/StreamingCube21.json] > hdfs.DFSClient:864 : *Caught exception java.lang.InterruptedException* > at java.lang.Object.wait(Native Method) at > java.lang.Thread.join(Thread.java:1281) at > java.lang.Thread.join(Thread.java:1355) at > org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.closeResponder( > DFSOutputStream.java:862) > at > org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer. > endBlock(DFSOutputStream.java:600) > at > org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer. > run(DFSOutputStream.java:789)2016-12-12 > 23:37:14,264 WARN [DataStreamer for file > /user/root/.staging/job_1481606707079_0001/files/meta/ > model_desc/StreamingModel21.json > block BP-1288191314-127.0.0.1-1470859564889:blk_1073743599_2783] > hdfs.DFSClient:864 : Caught exception java.lang.InterruptedException > at java.lang.Object.wait(Native Method) at > java.lang.Thread.join(Thread.java:1281) at > java.lang.Thread.join(Thread.java:1355) at > org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.closeResponder( > DFSOutputStream.java:862) > at > org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer. > closeInternal(DFSOutputStream.java:830) > at > org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer. > run(DFSOutputStream.java:826)2016-12-12 > 23:37:14,308 WARN [DataStreamer for file > /user/root/.staging/job_1481606707079_0001/files/meta/kylin.properties > block > BP-1288191314-127.0.0.1-1470859564889:blk_1073743600_2784] > hdfs.DFSClient:864 : Caught exception java.lang.InterruptedException > at java.lang.Object.wait(Native Method) at > java.lang.Thread.join(Thread.java:1281) at > java.lang.Thread.join(Thread.java:1355) at > org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.closeResponder( > DFSOutputStream.java:862) > at > org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer. > closeInternal(DFSOutputStream.java:830) > at > org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer. > run(DFSOutputStream.java:826)2016-12-12 > 23:37:14,371 WARN [DataStreamer for file > /user/root/.staging/job_1481606707079_0001/files/meta/streaming/DEFAULT. > STREAMINGTABLE21.json > block BP-1288191314-127.0.0.1-1470859564889:blk_1073743601_2785] > hdfs.DFSClient:864 : Caught exception java.lang.InterruptedException > at java.lang.Object.wait(Native Method) at > java.lang.Thread.join(Thread.java:1281) at > java.lang.Thread.join(Thread.java:1355) at > org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.closeResponder( > DFSOutputStream.java:862) > at > org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer. > closeInternal(DFSOutputStream.java:830) > at > org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer. > run(DFSOutputStream.java:826)2016-12-12 > 23:37:17,615 INFO [pool-9-thread-1] consumer.ConsumerConfig:180 : > ConsumerConfig values: auto.commit.interval.ms = 5000 > auto.offset.reset = latest bootstrap.servers = [172.31.10.171:9092] > check.crcs = true client.id = connections.max.idle.ms = > 540000 enable.auto.commit = false exclude.internal.topics = > true fetch.max.bytes = 52428800 fetch.max.wait.ms = 500 > fetch.min.bytes = 1 group.id = StreamingCube21 > heartbeat.interval.ms = 3000 interceptor.classes = null > key.deserializer = class > org.apache.kafka.common.serialization.StringDeserializer > max.partition.fetch.bytes = 1048576 max.poll.interval.ms = 300000 > max.poll.records = 500 metadata.max.age.ms = 300000 > metric.reporters = [] metrics.num.samples = 2 > metrics.sample.window.ms = 30000 partition.assignment.strategy = > [class org.apache.kafka.clients.consumer.RangeAssignor] > receive.buffer.bytes = 65536 reconnect.backoff.ms = 50 > request.timeout.ms = 305000 retry.backoff.ms = 100 > sasl.kerberos.kinit.cmd = /usr/bin/kinit > sasl.kerberos.min.time.before.relogin = 60000 > sasl.kerberos.service.name = null sasl.kerberos.ticket.renew.jitter > = > 0.05 sasl.kerberos.ticket.renew.window.factor = 0.8 > sasl.mechanism = GSSAPI security.protocol = PLAINTEXT > send.buffer.bytes = 131072 session.timeout.ms = 30000 > ssl.cipher.suites = null ssl.enabled.protocols = [TLSv1.2, TLSv1.1, > TLSv1] ssl.endpoint.identification.algorithm = null > ssl.key.password = null ssl.keymanager.algorithm = SunX509 > ssl.keystore.location = null ssl.keystore.password = null > ssl.keystore.type = JKS ssl.protocol = TLS ssl.provider = > null > ssl.secure.random.implementation = null ssl.trustmanager.algorithm > = > PKIX ssl.truststore.location = null ssl.truststore.password = > null ssl.truststore.type = JKS value.deserializer = class > org.apache.kafka.common.serialization.StringDeserializer2016-12-12 > 23:37:17,617 INFO [pool-9-thread-1] consumer.ConsumerConfig:180 : > ConsumerConfig values: auto.commit.interval.ms = 5000 > auto.offset.reset = latest bootstrap.servers = [172.31.10.171:9092] > check.crcs = true client.id = consumer-4 > connections.max.idle.ms = 540000 enable.auto.commit = false > exclude.internal.topics = true fetch.max.bytes = 52428800 > fetch.max.wait.ms = 500 fetch.min.bytes = 1 group.id = > StreamingCube21 heartbeat.interval.ms = 3000 > interceptor.classes = null key.deserializer = class > org.apache.kafka.common.serialization.StringDeserializer > max.partition.fetch.bytes = 1048576 max.poll.interval.ms = 300000 > max.poll.records = 500 metadata.max.age.ms = 300000 > metric.reporters = [] metrics.num.samples = 2 > metrics.sample.window.ms = 30000 partition.assignment.strategy = > [class org.apache.kafka.clients.consumer.RangeAssignor] > receive.buffer.bytes = 65536 reconnect.backoff.ms = 50 > request.timeout.ms = 305000 retry.backoff.ms = 100 > sasl.kerberos.kinit.cmd = /usr/bin/kinit > sasl.kerberos.min.time.before.relogin = 60000 > sasl.kerberos.service.name = null sasl.kerberos.ticket.renew.jitter > = > 0.05 sasl.kerberos.ticket.renew.window.factor = 0.8 > sasl.mechanism = GSSAPI security.protocol = PLAINTEXT > send.buffer.bytes = 131072 session.timeout.ms = 30000 > ssl.cipher.suites = null ssl.enabled.protocols = [TLSv1.2, TLSv1.1, > TLSv1] ssl.endpoint.identification.algorithm = null > ssl.key.password = null ssl.keymanager.algorithm = SunX509 > ssl.keystore.location = null ssl.keystore.password = null > ssl.keystore.type = JKS ssl.protocol = TLS ssl.provider = > null > ssl.secure.random.implementation = null ssl.trustmanager.algorithm > = > PKIX ssl.truststore.location = null ssl.truststore.password = > null ssl.truststore.type = JKS value.deserializer = class > org.apache.kafka.common.serialization.StringDeserializer2016-12-12 > 23:37:17,640 INFO [pool-9-thread-1] utils.AppInfoParser:83 : Kafka version > : 0.10.1.02016-12-12 23:37:17,640 INFO [pool-9-thread-1] > utils.AppInfoParser:84 : Kafka commitId : 3402a74efb23d1d42016-12-12 > 23:37:17,973 WARN [DataStreamer for file > /user/root/.staging/job_1481606707079_0001/job.split] hdfs.DFSClient:864 : > Caught exception java.lang.InterruptedException at > java.lang.Object.wait(Native Method) at > java.lang.Thread.join(Thread.java:1281) at > java.lang.Thread.join(Thread.java:1355) at > org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.closeResponder( > DFSOutputStream.java:862) > at > org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer. > endBlock(DFSOutputStream.java:600) > at > org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer. > run(DFSOutputStream.java:789)2016-12-12 > 23:37:18,059 INFO [pool-9-thread-1] mapreduce.JobSubmitter:202 : number of > splits:12016-12-12 23:37:18,390 INFO [pool-9-thread-1] > mapreduce.JobSubmitter:291 : Submitting tokens for job: > job_1481606707079_00012016-12-12 23:37:20,904 INFO [pool-9-thread-1] > impl.YarnClientImpl:260 : Submitted application > application_1481606707079_00012016-12-12 23:37:21,376 INFO > [pool-9-thread-1] mapreduce.Job:1311 : The url to track the job: > http://quickstart.cloudera:8088/proxy/application_ > 1481606707079_0001/2016-12-12 > 23:37:21,377 INFO [pool-9-thread-1] common.AbstractHadoopJob:506 : > tempMetaFileString is : > file:///opt/kylin/bin/../tomcat/temp/kylin_job_meta2568880879058796161/ > meta2016-12-12 > 23:37:21,533 DEBUG [pool-9-thread-1] dao.ExecutableDao:210 : updating job > output, id: 33aa8bff-2ebf-412d-a48b-03ce56e09833-002016-12-12 23:37:21,542 > DEBUG [pool-9-thread-1] hbase.HBaseResourceStore:262 : Update row > /execute_output/33aa8bff-2ebf-412d-a48b-03ce56e09833-00 from oldTs: > 1481614628719, to newTs: 1481614641533, operation result: true2016-12-12 > 23:37:31,575 DEBUG [pool-9-thread-1] dao.ExecutableDao:210 : updating job > output, id: 33aa8bff-2ebf-412d-a48b-03ce56e09833-002016-12-12 23:37:31,585 > DEBUG [pool-9-thread-1] hbase.HBaseResourceStore:262 : Update row > /execute_output/33aa8bff-2ebf-412d-a48b-03ce56e09833-00 from oldTs: > 1481614641533, to newTs: 1481614651575, operation result: true2016-12-12 > 23:37:35,485 DEBUG [http-bio-7070-exec-8] controller.UserController:64 : > authentication.getPrincipal() is > org.springframework.security.core.userdetails.User@3b40b2f: Username: > ADMIN; > Password: [PROTECTED]; Enabled: true; AccountNonExpired: true; > credentialsNonExpired: true; AccountNonLocked: true; Granted Authorities: > ROLE_ADMIN,ROLE_ANALYST,ROLE_MODELER2016-12-12 23:37:36,003 DEBUG > [http-bio-7070-exec-9] badquery.BadQueryHistoryManager:84 : Loaded 0 Bad > Query(s)2016-12-12 23:37:41,620 DEBUG [pool-9-thread-1] > dao.ExecutableDao:210 : updating job output, id: > 33aa8bff-2ebf-412d-a48b-03ce56e09833-002016-12-12 23:37:41,630 DEBUG > [pool-9-thread-1] hbase.HBaseResourceStore:262 : Update row > /execute_output/33aa8bff-2ebf-412d-a48b-03ce56e09833-00 from oldTs: > 1481614651575, to newTs: 1481614661621, operation result: true2016-12-12 > 23:37:51,701 DEBUG [pool-9-thread-1] dao.ExecutableDao:210 : updating job > output, id: 33aa8bff-2ebf-412d-a48b-03ce56e09833-002016-12-12 23:37:51,720 > DEBUG [pool-9-thread-1] hbase.HBaseResourceStore:262 : Update row > /execute_output/33aa8bff-2ebf-412d-a48b-03ce56e09833-00 from oldTs: > 1481614661621, to newTs: 1481614671701, operation result: true2016-12-12 > 23:38:01,758 DEBUG [pool-9-thread-1] dao.ExecutableDao:210 : updating job > output, id: 33aa8bff-2ebf-412d-a48b-03ce56e09833-002016-12-12 23:38:01,772 > DEBUG [pool-9-thread-1] hbase.HBaseResourceStore:262 : Update row > /execute_output/33aa8bff-2ebf-412d-a48b-03ce56e09833-00 from oldTs: > 1481614671701, to newTs: 1481614681758, operation result: true2016-12-12 > 23:38:08,562 INFO [pool-8-thread-1] threadpool.DefaultScheduler:118 : Job > Fetcher: 1 should running, 1 actual running, 0 ready, 0 already succeed, 0 > error, 0 discarded, 0 others2016-12-12 23:38:11,799 INFO [pool-9-thread-1] > mapred.ClientServiceDelegate:277 : Application state is completed. > FinalApplicationStatus=FAILED. Redirecting to job history server2016-12-12 > 23:38:15,287 DEBUG [pool-9-thread-1] dao.ExecutableDao:210 : updating job > output, id: 33aa8bff-2ebf-412d-a48b-03ce56e09833-002016-12-12 23:38:15,295 > DEBUG [pool-9-thread-1] hbase.HBaseResourceStore:262 : Update row > /execute_output/33aa8bff-2ebf-412d-a48b-03ce56e09833-00 from oldTs: > 1481614681758, to newTs: 1481614695287, operation result: true2016-12-12 > 23:38:15,306 DEBUG [pool-9-thread-1] dao.ExecutableDao:210 : updating job > output, id: 33aa8bff-2ebf-412d-a48b-03ce56e09833-002016-12-12 23:38:15,313 > DEBUG [pool-9-thread-1] hbase.HBaseResourceStore:262 : Update row > /execute_output/33aa8bff-2ebf-412d-a48b-03ce56e09833-00 from oldTs: > 1481614695287, to newTs: 1481614695306, operation result: true2016-12-12 > 23:38:15,433 WARN [pool-9-thread-1] common.HadoopCmdOutput:89 : no > counters > for job job_1481606707079_00012016-12-12 23:38:15,441 DEBUG > [pool-9-thread-1] dao.ExecutableDao:210 : updating job output, id: > 33aa8bff-2ebf-412d-a48b-03ce56e09833-002016-12-12 23:38:15,449 DEBUG > [pool-9-thread-1] hbase.HBaseResourceStore:262 : Update row > /execute_output/33aa8bff-2ebf-412d-a48b-03ce56e09833-00 from oldTs: > 1481614695306, to newTs: 1481614695441, operation result: true2016-12-12 > 23:38:15,463 DEBUG [pool-9-thread-1] dao.ExecutableDao:210 : updating job > output, id: 33aa8bff-2ebf-412d-a48b-03ce56e09833-002016-12-12 23:38:15,473 > DEBUG [pool-9-thread-1] hbase.HBaseResourceStore:262 : Update row > /execute_output/33aa8bff-2ebf-412d-a48b-03ce56e09833-00 from oldTs: > 1481614695441, to newTs: 1481614695463, operation result: true2016-12-12 > 23:38:15,481 DEBUG [pool-9-thread-1] dao.ExecutableDao:210 : updating job > output, id: 33aa8bff-2ebf-412d-a48b-03ce56e09833-002016-12-12 23:38:15,490 > DEBUG [pool-9-thread-1] hbase.HBaseResourceStore:262 : Update row > /execute_output/33aa8bff-2ebf-412d-a48b-03ce56e09833-00 from oldTs: > 1481614695463, to newTs: 1481614695482, operation result: true2016-12-12 > 23:38:15,493 INFO [pool-9-thread-1] manager.ExecutableManager:292 : job > id:33aa8bff-2ebf-412d-a48b-03ce56e09833-00 from RUNNING to ERROR2016-12-12 > 23:38:15,504 DEBUG [pool-9-thread-1] dao.ExecutableDao:210 : updating job > output, id: 33aa8bff-2ebf-412d-a48b-03ce56e098332016-12-12 23:38:15,513 > DEBUG [pool-9-thread-1] hbase.HBaseResourceStore:262 : Update row > /execute_output/33aa8bff-2ebf-412d-a48b-03ce56e09833 from oldTs: > 1481614628669, to newTs: 1481614695504, operation result: true2016-12-12 > 23:38:15,524 DEBUG [pool-9-thread-1] dao.ExecutableDao:210 : updating job > output, id: 33aa8bff-2ebf-412d-a48b-03ce56e098332016-12-12 23:38:15,534 > DEBUG [pool-9-thread-1] hbase.HBaseResourceStore:262 : Update row > /execute_output/33aa8bff-2ebf-412d-a48b-03ce56e09833 from oldTs: > 1481614695504, to newTs: 1481614695524, operation result: true2016-12-12 > 23:38:15,544 DEBUG [pool-9-thread-1] dao.ExecutableDao:210 : updating job > output, id: 33aa8bff-2ebf-412d-a48b-03ce56e098332016-12-12 23:38:15,559 > DEBUG [pool-9-thread-1] hbase.HBaseResourceStore:262 : Update row > /execute_output/33aa8bff-2ebf-412d-a48b-03ce56e09833 from oldTs: > 1481614695524, to newTs: 1481614695544, operation result: true2016-12-12 > 23:38:15,559 INFO [pool-9-thread-1] manager.ExecutableManager:292 : job > id:33aa8bff-2ebf-412d-a48b-03ce56e09833 from RUNNING to ERROR2016-12-12 > 23:38:15,559 WARN [pool-9-thread-1] execution.AbstractExecutable:247 : no > need to send email, user list is empty2016-12-12 23:38:15,592 INFO > [pool-8-thread-1] threadpool.DefaultScheduler:118 : Job Fetcher: 0 should > running, 0 actual running, 0 ready, 0 already succeed, 1 error, 0 > discarded, > 0 others2016-12-12 23:39:08,552 INFO [pool-8-thread-1] > threadpool.DefaultScheduler:118 : Job Fetcher: 0 should running, 0 actual > running, 0 ready, 0 already succeed, 1 error, 0 discarded, 0 > others2016-12-12 23:40:08,543 INFO [pool-8-thread-1] > threadpool.DefaultScheduler:118 : Job Fetcher: 0 should running, 0 actual > running, 0 ready, 0 already succeed, 1 error, 0 discarded, 0 > others2016-12-12 23:41:08,583 INFO [pool-8-thread-1] > threadpool.DefaultScheduler:118 : Job Fetcher: 0 should running, 0 actual > running, 0 ready, 0 already succeed, 1 error, 0 discarded, 0 > others2016-12-12 23:42:08,557 INFO [pool-8-thread-1] > threadpool.DefaultScheduler:118 : Job Fetcher: 0 should running, 0 actual > running, 0 ready, 0 already succeed, 1 error, 0 discarded, 0 > others2016-12-12 23:43:08,564 INFO [pool-8-thread-1] > threadpool.DefaultScheduler:118 : Job Fetcher: 0 should running, 0 actual > running, 0 ready, 0 already succeed, 1 error, 0 discarded, 0 > others2016-12-12 23:44:08,549 INFO [pool-8-thread-1] > threadpool.DefaultScheduler:118 : Job Fetcher: 0 should running, 0 actual > running, 0 ready, 0 already succeed, 1 error, 0 discarded, 0 others > > -- > View this message in context: http://apache-kylin.74782.x6. > nabble.com/Streaimg-cube-build-error-in-the-first-step-tp6614.html > Sent from the Apache Kylin mailing list archive at Nabble.com. >
