[
https://issues.apache.org/jira/browse/KYLIN-4442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17084997#comment-17084997
]
Sonu Singh commented on KYLIN-4442:
-----------------------------------
Hi [~hit_lacus],
I have added few logs in HiveProducer.Java and below is the logs;
2020-04-16 15:19:50,222 INFO [metrics-blocking-reservoir-scheduler-0]
hive.HiveReservoirReporter:124 : Try to write 17 records
2020-04-16 15:19:50,335 DEBUG [metrics-blocking-reservoir-scheduler-0]
hive.HiveProducer:120 : send final List: 1
2020-04-16 15:19:50,335 DEBUG [metrics-blocking-reservoir-scheduler-0]
hive.HiveProducer:280 : convertTo: METRICS_JOB_QA
2020-04-16 15:19:50,335 DEBUG [metrics-blocking-reservoir-scheduler-0]
hive.HiveProducer:282 : convertTo rawValue:
\{JOB_ID=155ebdd9-6533-2f4a-0a85-c0482e77bea7, STEP_DURATION_INMEM_CUBING=0,
HOST=hostname:xxx.xxx.xxx.xxx, KDAY_DATE=2020-04-16, KTIME_MINUTE=9,
PER_BYTES_TIME_COST=0.025792330503463745, STEP_DURATION_HFILE_CONVERT=63182,
KWEEK_BEGIN_DATE=2020-04-12, KMONTH_BEGIN_DATE=2020-04-01, KUSER=admin,
STEP_DURATION_DISTINCT_COLUMNS=83150, KTIME_HOUR=23, PROJECT=REDSHIFT_PROJECT,
CUBE_SIZE=32854226, STEP_DURATION_DICTIONARY=20350, KDAY_TIME=23:09:22,
WAIT_RESOURCE_TIME=13048, KTIMESTAMP=1587049762656, CUBING_TYPE=LAYER,
CUBE_NAME=redshift_cube_1, TABLE_SIZE=9640254, KTIME_SECOND=22, JOB_TYPE=BUILD,
KYEAR_BEGIN_DATE=2020-01-01, DURATION=865447}
2020-04-16 15:19:50,336 DEBUG [metrics-blocking-reservoir-scheduler-0]
hive.HiveProducer:285 : convertTo partitionKVs: {}
2020-04-16 15:19:50,336 DEBUG [metrics-blocking-reservoir-scheduler-0]
hive.HiveProducer:288 : convertTo partitionKVs value: 2020-04-16
2020-04-16 15:19:50,336 DEBUG [metrics-blocking-reservoir-scheduler-0]
hive.HiveProducer:295 : parseToHiveProducerRecord: KYLIN.HIVE_METRICS_JOB_QA
2020-04-16 15:19:50,336 DEBUG [metrics-blocking-reservoir-scheduler-0]
hive.HiveProducer:297 : parseToHiveProducerRecord tableNameSplits: KYLIN
HIVE_METRICS_JOB_QA
2020-04-16 15:19:50,490 DEBUG [metrics-blocking-reservoir-scheduler-0]
hive.HiveProducer:299 : parseToHiveProducerRecord fields:
[FieldSchema(name:job_id, type:string, comment:null), FieldSchema(name:host,
type:string, comment:null), FieldSchema(name:kuser, type:string, comment:null),
FieldSchema(name:project, type:string, comment:null),
FieldSchema(name:cube_name, type:string, comment:null),
FieldSchema(name:job_type, type:string, comment:null),
FieldSchema(name:cubing_type, type:string, comment:null),
FieldSchema(name:duration, type:bigint, comment:null),
FieldSchema(name:table_size, type:bigint, comment:null),
FieldSchema(name:cube_size, type:bigint, comment:null),
FieldSchema(name:per_bytes_time_cost, type:double, comment:null),
FieldSchema(name:wait_resource_time, type:bigint, comment:null),
FieldSchema(name:step_duration_distinct_columns, type:bigint, comment:null),
FieldSchema(name:step_duration_dictionary, type:bigint, comment:null),
FieldSchema(name:step_duration_inmem_cubing, type:bigint, comment:null),
FieldSchema(name:step_duration_hfile_convert, type:bigint, comment:null),
FieldSchema(name:ktimestamp, type:bigint, comment:null),
FieldSchema(name:kyear_begin_date, type:string, comment:null),
FieldSchema(name:kmonth_begin_date, type:string, comment:null),
FieldSchema(name:kweek_begin_date, type:string, comment:null),
FieldSchema(name:kday_time, type:string, comment:null),
FieldSchema(name:ktime_hour, type:int, comment:null),
FieldSchema(name:ktime_minute, type:int, comment:null),
FieldSchema(name:ktime_second, type:int, comment:null)]
2020-04-16 15:19:50,490 DEBUG [metrics-blocking-reservoir-scheduler-0]
hive.HiveProducer:301 : parseToHiveProducerRecord columnValues: []
2020-04-16 15:19:50,490 DEBUG [metrics-blocking-reservoir-scheduler-0]
hive.HiveProducer:303 : parseToHiveProducerRecord fieldSchema: job_id
2020-04-16 15:19:50,490 DEBUG [metrics-blocking-reservoir-scheduler-0]
hive.HiveProducer:303 : parseToHiveProducerRecord fieldSchema: host
2020-04-16 15:19:50,490 DEBUG [metrics-blocking-reservoir-scheduler-0]
hive.HiveProducer:303 : parseToHiveProducerRecord fieldSchema: kuser
2020-04-16 15:19:50,490 DEBUG [metrics-blocking-reservoir-scheduler-0]
hive.HiveProducer:303 : parseToHiveProducerRecord fieldSchema: project
2020-04-16 15:19:50,491 DEBUG [metrics-blocking-reservoir-scheduler-0]
hive.HiveProducer:303 : parseToHiveProducerRecord fieldSchema: cube_name
2020-04-16 15:19:50,491 DEBUG [metrics-blocking-reservoir-scheduler-0]
hive.HiveProducer:303 : parseToHiveProducerRecord fieldSchema: job_type
2020-04-16 15:19:50,491 DEBUG [metrics-blocking-reservoir-scheduler-0]
hive.HiveProducer:303 : parseToHiveProducerRecord fieldSchema: cubing_type
2020-04-16 15:19:50,491 DEBUG [metrics-blocking-reservoir-scheduler-0]
hive.HiveProducer:303 : parseToHiveProducerRecord fieldSchema: duration
2020-04-16 15:19:50,491 DEBUG [metrics-blocking-reservoir-scheduler-0]
hive.HiveProducer:303 : parseToHiveProducerRecord fieldSchema: table_size
2020-04-16 15:19:50,491 DEBUG [metrics-blocking-reservoir-scheduler-0]
hive.HiveProducer:303 : parseToHiveProducerRecord fieldSchema: cube_size
2020-04-16 15:19:50,491 DEBUG [metrics-blocking-reservoir-scheduler-0]
hive.HiveProducer:303 : parseToHiveProducerRecord fieldSchema:
per_bytes_time_cost
2020-04-16 15:19:50,491 DEBUG [metrics-blocking-reservoir-scheduler-0]
hive.HiveProducer:303 : parseToHiveProducerRecord fieldSchema:
wait_resource_time
2020-04-16 15:19:50,491 DEBUG [metrics-blocking-reservoir-scheduler-0]
hive.HiveProducer:303 : parseToHiveProducerRecord fieldSchema:
step_duration_distinct_columns
2020-04-16 15:19:50,491 DEBUG [metrics-blocking-reservoir-scheduler-0]
hive.HiveProducer:303 : parseToHiveProducerRecord fieldSchema:
step_duration_dictionary
2020-04-16 15:19:50,491 DEBUG [metrics-blocking-reservoir-scheduler-0]
hive.HiveProducer:303 : parseToHiveProducerRecord fieldSchema:
step_duration_inmem_cubing
2020-04-16 15:19:50,491 DEBUG [metrics-blocking-reservoir-scheduler-0]
hive.HiveProducer:303 : parseToHiveProducerRecord fieldSchema:
step_duration_hfile_convert
2020-04-16 15:19:50,491 DEBUG [metrics-blocking-reservoir-scheduler-0]
hive.HiveProducer:303 : parseToHiveProducerRecord fieldSchema: ktimestamp
2020-04-16 15:19:50,491 DEBUG [metrics-blocking-reservoir-scheduler-0]
hive.HiveProducer:303 : parseToHiveProducerRecord fieldSchema: kyear_begin_date
2020-04-16 15:19:50,491 DEBUG [metrics-blocking-reservoir-scheduler-0]
hive.HiveProducer:303 : parseToHiveProducerRecord fieldSchema: kmonth_begin_date
2020-04-16 15:19:50,491 DEBUG [metrics-blocking-reservoir-scheduler-0]
hive.HiveProducer:303 : parseToHiveProducerRecord fieldSchema: kweek_begin_date
2020-04-16 15:19:50,491 DEBUG [metrics-blocking-reservoir-scheduler-0]
hive.HiveProducer:303 : parseToHiveProducerRecord fieldSchema: kday_time
2020-04-16 15:19:50,491 DEBUG [metrics-blocking-reservoir-scheduler-0]
hive.HiveProducer:303 : parseToHiveProducerRecord fieldSchema: ktime_hour
2020-04-16 15:19:50,491 DEBUG [metrics-blocking-reservoir-scheduler-0]
hive.HiveProducer:303 : parseToHiveProducerRecord fieldSchema: ktime_minute
2020-04-16 15:19:50,491 DEBUG [metrics-blocking-reservoir-scheduler-0]
hive.HiveProducer:303 : parseToHiveProducerRecord fieldSchema: ktime_second
2020-04-16 15:19:50,491 DEBUG [metrics-blocking-reservoir-scheduler-0]
hive.HiveProducer:306 : parseToHiveProducerRecord return
2020-04-16 15:19:50,493 DEBUG [metrics-blocking-reservoir-scheduler-0]
hive.HiveProducer:124 : fhiveRecord:
HiveProducerRecord(key=RecordKey(database=KYLIN, table=HIVE_METRICS_JOB_QA,
partition=\{KDAY_DATE=2020-04-16}),
value=[155ebdd9-6533-2f4a-0a85-c0482e77bea7, hostname:XXXXXXX, admin,
REDSHIFT_PROJECT, redshift_cube_1, BUILD, LAYER, 865447, 9640254, 32854226,
0.025792330503463745, 13048, 83150, 20350, 0, 63182, 1587049762656, 2020-01-01,
2020-04-01, 2020-04-12, 23:09:22, 23, 9, 22])
2020-04-16 15:19:50,493 DEBUG [metrics-blocking-reservoir-scheduler-0]
hive.HiveProducer:126 : send(final List: hiveRecord.key():
RecordKey(database=KYLIN, table=HIVE_METRICS_JOB_QA,
partition=\{KDAY_DATE=2020-04-16})
2020-04-16 15:19:50,493 DEBUG [metrics-blocking-reservoir-scheduler-0]
hive.HiveProducer:130 : send(final List: recordMap: 1
2020-04-16 15:19:50,493 DEBUG [metrics-blocking-reservoir-scheduler-0]
hive.HiveProducer:134 : for (Map.Entry<RecordKey: RecordKey(database=KYLIN,
table=HIVE_METRICS_JOB_QA, partition=\{KDAY_DATE=2020-04-16})
2020-04-16 15:19:50,493 DEBUG [metrics-blocking-reservoir-scheduler-0]
hive.HiveProducer:140 : write entry: HIVE_METRICS_JOB_QA
2020-04-16 15:19:50,493 DEBUG [metrics-blocking-reservoir-scheduler-0]
hive.HiveProducer:144 : Step 1: write tableLocation:
hdfs://hostname:8020/apps/hive/warehouse/KYLIN.db/hive_metrics_job_qa
2020-04-16 15:19:50,493 DEBUG [metrics-blocking-reservoir-scheduler-0]
hive.HiveProducer:154 : Step 1: write partitionPath:
hdfs://hostname:8020/apps/hive/warehouse/KYLIN.db/hive_metrics_job_qa/kday_date=2020-04-16
2020-04-16 15:19:50,495 DEBUG [metrics-blocking-reservoir-scheduler-0]
hive.HiveProducer:165 : Step 2: create partition for hive:
hdfs://hostname-master:8020/apps/hive/warehouse/KYLIN.db/hive_metrics_job_qa/kday_date=2020-04-16
2020-04-16 15:19:50,495 DEBUG [metrics-blocking-reservoir-scheduler-0]
hive.HiveProducer:181 : create partition by ALTER TABLE
KYLIN.HIVE_METRICS_JOB_QA ADD IF NOT EXISTS PARTITION (kday_date='2020-04-16').
2020-04-16 15:19:50,522 DEBUG [metrics-blocking-reservoir-scheduler-0]
hive.HiveProducer:183 : Step 2: create partition for hive: driver:
org.apache.hadoop.hive.ql.Driver@7a56971e
2020-04-16 15:19:50,566 DEBUG [metrics-blocking-reservoir-scheduler-0]
hive.HiveProducer:185 : Step 2: create partition for hive: session:
org.apache.hadoop.hive.cli.CliSessionState@396793e5
2020-04-16 15:19:50,687 DEBUG [metrics-blocking-reservoir-scheduler-0]
hive.HiveProducer:187 : Step 2: create partition for hive: start session
> Kylin System Cube Hive tables not able to update
> -------------------------------------------------
>
> Key: KYLIN-4442
> URL: https://issues.apache.org/jira/browse/KYLIN-4442
> Project: Kylin
> Issue Type: Bug
> Affects Versions: v3.0.0
> Reporter: Sonu Singh
> Priority: Blocker
> Attachments: kylin-metrics-reporter-hive-3.1.0-SNAPSHOT.jar
>
>
> Kylin System Cube Hive table is not updating. I am using Kylin hadoop3 code
> and running on vanilla hadoop3.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
