Hi Team,
We recently upgraded from Kylin 2.1 to Kylin 2.3.1
Since then we are facing some issues with our cube building pipeline.
While building new segments, sometimes(not always), some of the existing
segment tables get deleted from Hbase.
For example here is the segment Json for empty segment, it shows me the table
name , but this table gets dropped from Hbase.
{ <>
"uuid":"90972f20-d64d-4224-a1f1-cdb6a0ddb69c",
"name":"20180402110000_20180402120000",
"storage_location_identifier":"KYLIN_Z3C9IU2QNY",
"date_range_start":1522666800000,
"date_range_end":1522670400000,
"source_offset_start":0,
"source_offset_end":0,
"status":"NEW",
"size_kb":0,
"input_records":0,
"input_records_size":0,
"last_build_time":0,
"last_build_job_id":null,
"create_time_utc":1522700641299,
"cuboid_shard_nums":{ <>
},
"total_shards":0,
"blackout_cuboids":[ <>
],
"binary_signature":null,
"dictionaries":{ <>
},
"snapshots":null,
"rowkey_stats":[ <>
]
}
This data loss is actually proving out very heavy business impact as we are
always going back and restoring previous day snapshots and building the new
segments again, hoping it doesn’t fails.
Here are my kylin.props
kylin.web.timezone=US/Pacific
kylin.metadata.url=kylin2.1MetadataProduction@hbase
kylin.storage.url=hbase
kylin.env.hdfs-working-dir=/tmp/kylin-2.1-prod
kylin.engine.mr.reduce-input-mb=300
kylin.server.mode=all
kylin.job.use-remote-cli=false
kylin.job.remote-cli-working-dir=/tmp/kylin-2.1
kylin.job.max-concurrent-jobs=10
kylin.engine.mr.yarn-check-interval-seconds=10
kylin.source.hive.database-for-flat-table=tmp_kylin
kylin.storage.hbase.table-name-prefix=KYLIN_
kylin.storage.hbase.compression-codec=lz4
kylin.storage.hbase.region-cut-gb=3
kylin.storage.hbase.min-region-count=1
kylin.storage.hbase.max-region-count=500
kylin.storage.partition.max-scan-bytes=16106127360
kylin.storage.hbase.coprocessor-mem-gb=6
kylin.security.profile=testing
kylin.query.cache-enabled=true
kylin.query.cache-threshold-duration=500
kylin.query.cache-threshold-scan-count=10240
kylin.storage.hbase.scan-cache-rows=4096
Any idea around how and why this corruption might happen, How can even data get
dropped while building some other segments.
Thanks,
Ketan@Exponential
