Hi Team,
We recently upgraded from Kylin 2.1 to Kylin 2.3.1

Since then we are facing some issues with our cube building pipeline.

While building new segments, sometimes(not always), some of the existing 
segment tables get deleted from Hbase.
For example here is the segment Json for empty segment, it shows me the table 
name , but this table gets dropped from Hbase.
{  <> 
         "uuid":"90972f20-d64d-4224-a1f1-cdb6a0ddb69c",
         "name":"20180402110000_20180402120000",
         "storage_location_identifier":"KYLIN_Z3C9IU2QNY",
         "date_range_start":1522666800000,
         "date_range_end":1522670400000,
         "source_offset_start":0,
         "source_offset_end":0,
         "status":"NEW",
         "size_kb":0,
         "input_records":0,
         "input_records_size":0,
         "last_build_time":0,
         "last_build_job_id":null,
         "create_time_utc":1522700641299,
         "cuboid_shard_nums":{  <> 

         },
         "total_shards":0,
         "blackout_cuboids":[  <> 

         ],
         "binary_signature":null,
         "dictionaries":{  <> 

         },
         "snapshots":null,
         "rowkey_stats":[  <> 

         ]
      }

This data loss is actually proving out very heavy business impact as we are 
always going back and restoring previous day snapshots and building the new 
segments again, hoping it doesn’t fails.
Here are my kylin.props
kylin.web.timezone=US/Pacific
kylin.metadata.url=kylin2.1MetadataProduction@hbase
kylin.storage.url=hbase
kylin.env.hdfs-working-dir=/tmp/kylin-2.1-prod
kylin.engine.mr.reduce-input-mb=300
kylin.server.mode=all
kylin.job.use-remote-cli=false
kylin.job.remote-cli-working-dir=/tmp/kylin-2.1
kylin.job.max-concurrent-jobs=10
kylin.engine.mr.yarn-check-interval-seconds=10
kylin.source.hive.database-for-flat-table=tmp_kylin
kylin.storage.hbase.table-name-prefix=KYLIN_
kylin.storage.hbase.compression-codec=lz4
kylin.storage.hbase.region-cut-gb=3
kylin.storage.hbase.min-region-count=1
kylin.storage.hbase.max-region-count=500
kylin.storage.partition.max-scan-bytes=16106127360
kylin.storage.hbase.coprocessor-mem-gb=6
kylin.security.profile=testing

kylin.query.cache-enabled=true
kylin.query.cache-threshold-duration=500
kylin.query.cache-threshold-scan-count=10240
kylin.storage.hbase.scan-cache-rows=4096


Any idea around how and why this corruption might happen, How can even data get 
dropped while building some other segments.

Thanks,
Ketan@Exponential

Reply via email to