您好,我最近在创建cube的时候遇到了一个问题,解决了好几天一直没有进展。非常感谢谁来帮助我一下,项目一直卡在这里。
问题描述:我的三张hive里面的表格中的数据量非常大,有一个能达到45万行,在构建cube的时候会一直卡到第一步,本来以为会因为数据量大而时间长,结果1520分钟过去进度还是0.我又重新换成小数据试验了一下,结果原来可以构建成功的小数据量的cube也失败了,是同样的问题。
日志:
一开始是这样:
2020-09-30 08:40:16,787 WARN  [FetcherRunner 1998130220-53] 
threadpool.FetcherRunner:56 : There are too many jobs running, Job Fetch will 
wait until next schedule time


后来是这样,一直这样
========================================================================= 772
2020-10-04 18:07:08,781 TRACE [streaming_job_submitter-thread-1] 
coordinate.BuildJobSubmitter:530 : Dump JobCheckList:
2020-10-04 18:07:08,781 DEBUG [streaming_job_submitter-thread-1] 
coordinator.ZookeeperStreamMetadataStore:673 : [StreamMetadataStoreStats]  read 
: 9 ; write: 1 ; read failed: 0 ; write failed: 0 .
2020-10-04 18:07:09,232 INFO  [BadQueryDetector] service.BadQueryDetector:148 : 
Detect bad query.
2020-10-04 18:07:11,133 INFO  [FetcherRunner 1662320485-53] 
threadpool.DefaultFetcherRunner:115 : Job Fetcher: 7 should running, 8 actual 
running, 0 stopped, 0 ready, 17 already succeed, 17 error, 0 discarded, 0 others
2020-10-04 18:07:41,133 INFO  [FetcherRunner 1662320485-53] 
threadpool.DefaultFetcherRunner:115 : Job Fetcher: 7 should running, 8 actual 
running, 0 stopped, 0 ready, 17 already succeed, 17 error, 0 discarded, 0 others
2020-10-04 18:08:09,233 INFO  [BadQueryDetector] service.BadQueryDetector:148 : 
Detect bad query.
2020-10-04 18:08:11,133 INFO  [FetcherRunner 1662320485-53] 
threadpool.DefaultFetcherRunner:115 : Job Fetcher: 7 should running, 8 actual 
running, 0 stopped, 0 ready, 17 already succeed, 17 error, 0 discarded, 0 others
2020-10-04 18:08:41,133 INFO  [FetcherRunner 1662320485-53] 
threadpool.DefaultFetcherRunner:115 : Job Fetcher: 7 should running, 8 actual 
running, 0 stopped, 0 ready, 17 already succeed, 17 error, 0 discarded, 0 others
2020-10-04 18:09:08,781 DEBUG [streaming_job_submitter-thread-1] 
coordinate.BuildJobSubmitter:142 :
========================================================================= 773
2020-10-04 18:09:08,781 TRACE [streaming_job_submitter-thread-1] 
coordinate.BuildJobSubmitter:530 : Dump JobCheckList:
2020-10-04 18:09:09,233 INFO  [BadQueryDetector] service.BadQueryDetector:148 : 
Detect bad query.
2020-10-04 18:09:11,133 INFO  [FetcherRunner 1662320485-53] 
threadpool.DefaultFetcherRunner:115 : Job Fetcher: 7 should running, 8 actual 
running, 0 stopped, 0 ready, 17 already succeed, 17 error, 0 discarded, 0 others
2020-10-04 18:09:41,133 INFO  [FetcherRunner 1662320485-53] 
threadpool.DefaultFetcherRunner:115 : Job Fetcher: 7 should running, 8 actual 
running, 0 stopped, 0 ready, 17 already succeed, 17 error, 0 discarded, 0 others
2020-10-04 18:10:09,233 INFO  [BadQueryDetector] service.BadQueryDetector:148 : 
Detect bad query.
2020-10-04 18:10:11,133 INFO  [FetcherRunner 1662320485-53] 
threadpool.DefaultFetcherRunner:115 : Job Fetcher: 7 should running, 8 actual 
running, 0 stopped, 0 ready, 17 already succeed, 17 error, 0 discarded, 0 others
2020-10-04 18:10:41,133 INFO  [FetcherRunner 1662320485-53] 
threadpool.DefaultFetcherRunner:115 : Job Fetcher: 7 should running, 8 actual 
running, 0 stopped, 0 ready, 17 already succeed, 17 error, 0 discarded, 0 others
                                                                               


Reply via email to