[
https://issues.apache.org/jira/browse/KYLIN-4442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17083775#comment-17083775
]
Xiaoxiang Yu edited comment on KYLIN-4442 at 4/15/20, 4:00 AM:
---------------------------------------------------------------
It is strange, isn't it?
We can see "Try to write 196 records" and "Try to use new partition content
path" in your log output, but we cannot see "Success to write ....", they
should exists together because they are in the same method, except some
exception were thrown in some step.
I didn't face such issue in Hadoop2, I will do another test in HDP3.0 (that is
Hadoop3.1) when I have a env.
{code:java}
2020-04-15 11:28:57,813 INFO [metrics-blocking-reservoir-scheduler-0]
hive.HiveReservoirReporter:131 : Try to write 11 records
2020-04-15 11:28:57,898 INFO [metrics-blocking-reservoir-scheduler-0]
hive.HiveProducer:121 : For hdfs://cdh-master:8020, supportAppend was set to
true
2020-04-15 11:28:58,165 DEBUG [metrics-blocking-reservoir-scheduler-0]
hive.HiveProducer:104 : Find table location for HIVE_METRICS_QUERY_CUBE_QA at
hdfs://cdh-master:8020/user/hive/warehouse/kylin.db/hive_metrics_query_cube_qa
2020-04-15 11:28:58,190 DEBUG [metrics-blocking-reservoir-scheduler-0]
hive.HiveProducer:185 : create partition by ALTER TABLE
KYLIN.HIVE_METRICS_QUERY_CUBE_QA ADD IF NOT EXISTS PARTITION
(kday_date='2020-04-15').
2020-04-15 11:29:00,114 INFO [metrics-blocking-reservoir-scheduler-0]
hive.HiveProducer:235 : Try to use new partition content path:
hdfs://cdh-master:8020/user/hive/warehouse/kylin.db/hive_metrics_query_cube_qa/kday_date=2020-04-15/cdh-master-1586921337897-part-00000
for metric: METRICS_QUERY_CUBE_QA
2020-04-15 11:29:00,182 DEBUG [metrics-blocking-reservoir-scheduler-0]
hive.HiveProducer:266 : Success to write 1 metrics (METRICS_QUERY_CUBE_QA) to
file
hdfs://cdh-master:8020/user/hive/warehouse/kylin.db/hive_metrics_query_cube_qa/kday_date=2020-04-15/cdh-master-1586921337897-part-00000
2020-04-15 11:29:00,250 INFO [metrics-blocking-reservoir-scheduler-0]
hive.HiveProducer:121 : For hdfs://cdh-master:8020, supportAppend was set to
true
2020-04-15 11:29:00,391 DEBUG [metrics-blocking-reservoir-scheduler-0]
hive.HiveProducer:104 : Find table location for HIVE_METRICS_JOB_QA at
hdfs://cdh-master:8020/user/hive/warehouse/kylin.db/hive_metrics_job_qa
2020-04-15 11:29:00,401 INFO [metrics-blocking-reservoir-scheduler-0]
hive.HiveProducer:235 : Try to use new partition content path:
hdfs://cdh-master:8020/user/hive/warehouse/kylin.db/hive_metrics_job_qa/kday_date=2020-04-15/cdh-master-1586921340250-part-00000
for metric: METRICS_JOB_QA
2020-04-15 11:29:00,536 DEBUG [metrics-blocking-reservoir-scheduler-0]
hive.HiveProducer:266 : Success to write 8 metrics (METRICS_JOB_QA) to file
hdfs://cdh-master:8020/user/hive/warehouse/kylin.db/hive_metrics_job_qa/kday_date=2020-04-15/cdh-master-1586921340250-part-00000
2020-04-15 11:29:00,614 INFO [metrics-blocking-reservoir-scheduler-0]
hive.HiveProducer:121 : For hdfs://cdh-master:8020, supportAppend was set to
true
2020-04-15 11:29:00,756 DEBUG [metrics-blocking-reservoir-scheduler-0]
hive.HiveProducer:104 : Find table location for HIVE_METRICS_QUERY_QA at
hdfs://cdh-master:8020/user/hive/warehouse/kylin.db/hive_metrics_query_qa
2020-04-15 11:29:00,765 DEBUG [metrics-blocking-reservoir-scheduler-0]
hive.HiveProducer:185 : create partition by ALTER TABLE
KYLIN.HIVE_METRICS_QUERY_QA ADD IF NOT EXISTS PARTITION
(kday_date='2020-04-15').
2020-04-15 11:29:01,430 INFO [metrics-blocking-reservoir-scheduler-0]
hive.HiveProducer:235 : Try to use new partition content path:
hdfs://cdh-master:8020/user/hive/warehouse/kylin.db/hive_metrics_query_qa/kday_date=2020-04-15/cdh-master-1586921340614-part-00000
for metric: METRICS_QUERY_QA
2020-04-15 11:29:01,470 DEBUG [metrics-blocking-reservoir-scheduler-0]
hive.HiveProducer:266 : Success to write 1 metrics (METRICS_QUERY_QA) to file
hdfs://cdh-master:8020/user/hive/warehouse/kylin.db/hive_metrics_query_qa/kday_date=2020-04-15/cdh-master-1586921340614-part-00000
2020-04-15 11:29:01,540 INFO [metrics-blocking-reservoir-scheduler-0]
hive.HiveProducer:121 : For hdfs://cdh-master:8020, supportAppend was set to
true
2020-04-15 11:29:01,679 DEBUG [metrics-blocking-reservoir-scheduler-0]
hive.HiveProducer:104 : Find table location for HIVE_METRICS_QUERY_RPC_QA at
hdfs://cdh-master:8020/user/hive/warehouse/kylin.db/hive_metrics_query_rpc_qa
2020-04-15 11:29:01,687 DEBUG [metrics-blocking-reservoir-scheduler-0]
hive.HiveProducer:185 : create partition by ALTER TABLE
KYLIN.HIVE_METRICS_QUERY_RPC_QA ADD IF NOT EXISTS PARTITION
(kday_date='2020-04-15').
2020-04-15 11:29:02,329 INFO [metrics-blocking-reservoir-scheduler-0]
hive.HiveProducer:235 : Try to use new partition content path:
hdfs://cdh-master:8020/user/hive/warehouse/kylin.db/hive_metrics_query_rpc_qa/kday_date=2020-04-15/cdh-master-1586921341540-part-00000
for metric: METRICS_QUERY_RPC_QA
2020-04-15 11:29:02,387 DEBUG [metrics-blocking-reservoir-scheduler-0]
hive.HiveProducer:266 : Success to write 1 metrics (METRICS_QUERY_RPC_QA) to
file
hdfs://cdh-master:8020/user/hive/warehouse/kylin.db/hive_metrics_query_rpc_qa/kday_date=2020-04-15/cdh-master-1586921341540-part-00000
2020-04-15 11:29:02,387 INFO [metrics-blocking-reservoir-scheduler-0]
impl.BlockingReservoir:159 : There's no record in the blocking queue.
2020-04-15 11:29:06,385 INFO [BadQueryDetector] service.BadQueryDetector:148 :
Detect bad query.
{code}
{code:java}
[root@cdh-master all]# hadoop fs -cat
/user/hive/warehouse/kylin.db/hive_metrics_query_rpc_qa/kday_date=2020-04-15/cdh-master-1586921341540-part-00000
cdh-master:10.1.3.90,LEARN_KYLIN,kylin_sales_SPARK,cdh-worker-2,NULL,62,3,1177,0,1174,1174,1586921283325,2020-01-01,2020-04-01,2020-04-12,11:28:03,11,28,3
cdh-master:10.1.3.90,LEARN_KYLIN,kylin_sales_SPARK,cdh-worker-2,NULL,5,3,1177,0,1174,1174,1586921474478,2020-01-01,2020-04-01,2020-04-12,11:31:14,11,31,14
cdh-master:10.1.3.90,LEARN_KYLIN,kylin_sales_SPARK,cdh-worker-2,NULL,4,3,1177,0,1174,1174,1586921477876,2020-01-01,2020-04-01,2020-04-12,11:31:17,11,31,17
cdh-master:10.1.3.90,LEARN_KYLIN,kylin_sales_SPARK,cdh-worker-2,NULL,6,3,1177,0,1174,1174,1586921479495,2020-01-01,2020-04-01,2020-04-12,11:31:19,11,31,19
cdh-master:10.1.3.90,LEARN_KYLIN,kylin_sales_SPARK,cdh-worker-2,NULL,4,3,1177,0,1174,1174,1586921481052,2020-01-01,2020-04-01,2020-04-12,11:31:21,11,31,21
cdh-master:10.1.3.90,LEARN_KYLIN,kylin_sales_SPARK,cdh-worker-2,NULL,3,3,1177,0,1174,1174,1586921482454,2020-01-01,2020-04-01,2020-04-12,11:31:22,11,31,22
cdh-master:10.1.3.90,LEARN_KYLIN,kylin_sales_SPARK,cdh-worker-2,NULL,4,3,1177,0,1174,1174,1586921483939,2020-01-01,2020-04-01,2020-04-12,11:31:23,11,31,23
cdh-master:10.1.3.90,LEARN_KYLIN,kylin_sales_SPARK,cdh-worker-2,NULL,5,3,1177,0,1174,1174,1586921485044,2020-01-01,2020-04-01,2020-04-12,11:31:25,11,31,25
cdh-master:10.1.3.90,LEARN_KYLIN,kylin_sales_SPARK,cdh-worker-2,NULL,6,3,1177,0,1174,1174,1586921488973,2020-01-01,2020-04-01,2020-04-12,11:31:28,11,31,28
cdh-master:10.1.3.90,LEARN_KYLIN,kylin_sales_SPARK,cdh-worker-2,NULL,3,3,1177,0,1174,1174,1586921490280,2020-01-01,2020-04-01,2020-04-12,11:31:30,11,31,30
cdh-master:10.1.3.90,LEARN_KYLIN,kylin_sales_SPARK,cdh-worker-2,NULL,4,3,1177,0,1174,1174,1586921491317,2020-01-01,2020-04-01,2020-04-12,11:31:31,11,31,31
{code}
{code:java}
hive> select * from kylin.hive_metrics_query_rpc_qa where kday_date =
'2020-04-15';
OK
cdh-master:10.1.3.90 LEARN_KYLIN kylin_sales_SPARK cdh-worker-2
NULL 62 3 1177 0 1174 1174 1586921283325
2020-01-01 2020-04-01 2020-04-12 11:28:03 11 28
3 2020-04-15
cdh-master:10.1.3.90 LEARN_KYLIN kylin_sales_SPARK cdh-worker-2
NULL 5 3 1177 0 1174 1174 1586921474478
2020-01-01 2020-04-01 2020-04-12 11:31:14 11 31
14 2020-04-15
cdh-master:10.1.3.90 LEARN_KYLIN kylin_sales_SPARK cdh-worker-2
NULL 4 3 1177 0 1174 1174 1586921477876
2020-01-01 2020-04-01 2020-04-12 11:31:17 11 31
17 2020-04-15
cdh-master:10.1.3.90 LEARN_KYLIN kylin_sales_SPARK cdh-worker-2
NULL 6 3 1177 0 1174 1174 1586921479495
2020-01-01 2020-04-01 2020-04-12 11:31:19 11 31
19 2020-04-15
cdh-master:10.1.3.90 LEARN_KYLIN kylin_sales_SPARK cdh-worker-2
NULL 4 3 1177 0 1174 1174 1586921481052
2020-01-01 2020-04-01 2020-04-12 11:31:21 11 31
21 2020-04-15
cdh-master:10.1.3.90 LEARN_KYLIN kylin_sales_SPARK cdh-worker-2
NULL 3 3 1177 0 1174 1174 1586921482454
2020-01-01 2020-04-01 2020-04-12 11:31:22 11 31
22 2020-04-15
cdh-master:10.1.3.90 LEARN_KYLIN kylin_sales_SPARK cdh-worker-2
NULL 4 3 1177 0 1174 1174 1586921483939
2020-01-01 2020-04-01 2020-04-12 11:31:23 11 31
23 2020-04-15
cdh-master:10.1.3.90 LEARN_KYLIN kylin_sales_SPARK cdh-worker-2
NULL 5 3 1177 0 1174 1174 1586921485044
2020-01-01 2020-04-01 2020-04-12 11:31:25 11 31
25 2020-04-15
cdh-master:10.1.3.90 LEARN_KYLIN kylin_sales_SPARK cdh-worker-2
NULL 6 3 1177 0 1174 1174 1586921488973
2020-01-01 2020-04-01 2020-04-12 11:31:28 11 31
28 2020-04-15
cdh-master:10.1.3.90 LEARN_KYLIN kylin_sales_SPARK cdh-worker-2
NULL 3 3 1177 0 1174 1174 1586921490280
2020-01-01 2020-04-01 2020-04-12 11:31:30 11 31
30 2020-04-15
cdh-master:10.1.3.90 LEARN_KYLIN kylin_sales_SPARK cdh-worker-2
NULL 4 3 1177 0 1174 1174 1586921491317
2020-01-01 2020-04-01 2020-04-12 11:31:31 11 31
31 2020-04-15
Time taken: 0.544 seconds, Fetched: 11 row(s)
{code}
was (Author: hit_lacus):
It is strange, isn't it?
We can see "Try to write 196 records" and "Try to use new partition content
path" in your log output, but we cannot see "Success to write ....", they
should exists together because they are in the same method, except some
exception were thrown in some step.
> Kylin System Cube Hive tables not able to update
> -------------------------------------------------
>
> Key: KYLIN-4442
> URL: https://issues.apache.org/jira/browse/KYLIN-4442
> Project: Kylin
> Issue Type: Bug
> Affects Versions: v3.0.0
> Reporter: Sonu Singh
> Priority: Blocker
> Attachments: kylin-metrics-reporter-hive-3.1.0-SNAPSHOT.jar
>
>
> Kylin System Cube Hive table is not updating. I am using Kylin hadoop3 code
> and running on vanilla hadoop3.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
