hit-lacus edited a comment on pull request #1351:
URL: https://github.com/apache/kylin/pull/1351#issuecomment-671065519
1. Go to web page and re-enable Cube, you will see some failure, please
don't be in a panic.
```sh
2020-08-09 23:23:10,437 INFO [qtp193625667-35]
controller.AdminController:58 : receive assign
request:AssignRequest{cubeName='UserAnalysisCube', partitions=[Partition
[partitionId=0, partitionInfo=null], Partition [partitionId=5,
partitionInfo=null], Partition [partitionId=10, partitionInfo=null], Partition
[partitionId=15, partitionInfo=null], Partition [partitionId=13,
partitionInfo=null], Partition [partitionId=2, partitionInfo=null], Partition
[partitionId=8, partitionInfo=null], Partition [partitionId=12,
partitionInfo=null], Partition [partitionId=14, partitionInfo=null], Partition
[partitionId=9, partitionInfo=null], Partition [partitionId=11,
partitionInfo=null], Partition [partitionId=1, partitionInfo=null], Partition
[partitionId=4, partitionInfo=null], Partition [partitionId=6,
partitionInfo=null], Partition [partitionId=7, partitionInfo=null], Partition
[partitionId=3, partitionInfo=null]], startConsumers=true}
2020-08-09 23:23:10,437 INFO [qtp193625667-35] server.StreamingServer:351 :
create and start new consumer for cube:UserAnalysisCube
2020-08-09 23:23:10,468 INFO [qtp193625667-35] kafka.KafkaSource:141 :
Create StreamingConnector for Cube UserAnalysisCube, assignedPartitions
[Partition [partitionId=0, partitionInfo=null], Partition [partitionId=5,
partitionInfo=null], Partition [partitionId=10, partitionInfo=null], Partition
[partitionId=15, partitionInfo=null], Partition [partitionId=13,
partitionInfo=null], Partition [partitionId=2, partitionInfo=null], Partition
[partitionId=8, partitionInfo=null], Partition [partitionId=12,
partitionInfo=null], Partition [partitionId=14, partitionInfo=null], Partition
[partitionId=9, partitionInfo=null], Partition [partitionId=11,
partitionInfo=null], Partition [partitionId=1, partitionInfo=null], Partition
[partitionId=4, partitionInfo=null], Partition [partitionId=6,
partitionInfo=null], Partition [partitionId=7, partitionInfo=null], Partition
[partitionId=3, partitionInfo=null]], startProtocol null
2020-08-09 23:23:10,470 INFO [qtp193625667-35]
kafka.TimedJsonStreamParser:98 : Using parser field mapping by
{content_list=content_list, act_type=act_type, event_ts_2=event_ts_2,
event_ts=event_ts, devide_type=user_detail.devide_type,
location_city=user_detail.location.city, network_type=user_detail.network_type,
video_id=video_id, event_date_2=event_date_2, str_minute=str_minute,
video_type=video_type, play_times=play_times, pageview_id=pageview_id,
active_minutes=active_minutes, device_brand=device_brand,
str_minute_second=str_minute_second, play_duration=play_duration,
event_date=event_date, page_id=page_id, str_second=str_second, uid=uid}
2020-08-09 23:23:10,470 INFO [qtp193625667-35]
kafka.TimedJsonStreamParser:120 : TimedJsonStreamParser with formatTs false
tsColName event_date
2020-08-09 23:23:10,470 INFO [qtp193625667-35] consumer.ConsumerConfig:279
: ConsumerConfig values:
auto.commit.interval.ms = 5000
auto.offset.reset = latest
bootstrap.servers = [cdh-master:9092, cdh-worker-1:9092,
cdh-worker-2:9092]
check.crcs = true
client.dns.lookup = default
client.id =
connections.max.idle.ms = 540000
default.api.timeout.ms = 60000
enable.auto.commit = false
exclude.internal.topics = true
fetch.max.bytes = 52428800
fetch.max.wait.ms = 500
fetch.min.bytes = 1
group.id =
heartbeat.interval.ms = 3000
interceptor.classes = []
internal.leave.group.on.close = true
isolation.level = read_uncommitted
key.deserializer = class
org.apache.kafka.common.serialization.ByteArrayDeserializer
max.partition.fetch.bytes = 1048576
max.poll.interval.ms = 300000
max.poll.records = 500
metadata.max.age.ms = 300000
metric.reporters = []
metrics.num.samples = 2
metrics.recording.level = INFO
metrics.sample.window.ms = 30000
partition.assignment.strategy = [class
org.apache.kafka.clients.consumer.RangeAssignor]
receive.buffer.bytes = 65536
reconnect.backoff.max.ms = 1000
reconnect.backoff.ms = 50
request.timeout.ms = 30000
retry.backoff.ms = 100
sasl.client.callback.handler.class = null
sasl.jaas.config = null
sasl.kerberos.kinit.cmd = /usr/bin/kinit
sasl.kerberos.min.time.before.relogin = 60000
sasl.kerberos.service.name = null
sasl.kerberos.ticket.renew.jitter = 0.05
sasl.kerberos.ticket.renew.window.factor = 0.8
sasl.login.callback.handler.class = null
sasl.login.class = null
sasl.login.refresh.buffer.seconds = 300
sasl.login.refresh.min.period.seconds = 60
sasl.login.refresh.window.factor = 0.8
sasl.login.refresh.window.jitter = 0.05
sasl.mechanism = GSSAPI
security.protocol = PLAINTEXT
send.buffer.bytes = 131072
session.timeout.ms = 20000
ssl.cipher.suites = null
ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1]
ssl.endpoint.identification.algorithm = https
ssl.key.password = null
ssl.keymanager.algorithm = SunX509
ssl.keystore.location = null
ssl.keystore.password = null
ssl.keystore.type = JKS
ssl.protocol = TLS
ssl.provider = null
ssl.secure.random.implementation = null
ssl.trustmanager.algorithm = PKIX
ssl.truststore.location = null
ssl.truststore.password = null
ssl.truststore.type = JKS
value.deserializer = class
org.apache.kafka.common.serialization.ByteArrayDeserializer
2020-08-09 23:23:10,473 INFO [qtp193625667-35] utils.AppInfoParser:109 :
Kafka version : 2.1.1
2020-08-09 23:23:10,473 INFO [qtp193625667-35] utils.AppInfoParser:110 :
Kafka commitId : 21234bee31165527
2020-08-09 23:23:10,475 INFO [qtp193625667-35] kafka.KafkaSource:192 :
localConsumeStats from local checkpoint null, remoteConsumeStats from remote
checkpoint
{"0":363171,"1":363198,"2":363249,"3":363171,"4":363199,"5":363250,"6":363170,"7":363196,"8":363250,"9":363170}
2020-08-09 23:23:10,475 INFO [qtp193625667-35] kafka.KafkaSource:239 :
start kafka connector from specified
position:KafkaPosition{partitionOffsetMap={0=363171, 1=363198, 2=363249,
3=363171, 4=363199, 5=363250, 6=363170, 7=363196, 8=363250, 9=363170}}
2020-08-09 23:23:10,476 ERROR [qtp193625667-35] server.StreamingServer:357 :
consumer start fail for cube:UserAnalysisCube
java.lang.NullPointerException
at
org.apache.kylin.stream.source.kafka.consumer.KafkaConnector.open(KafkaConnector.java:86)
at
org.apache.kylin.stream.core.consumer.StreamingConsumerChannel.start(StreamingConsumerChannel.java:85)
at
org.apache.kylin.stream.server.StreamingServer.startConsumer(StreamingServer.java:355)
at
org.apache.kylin.stream.server.StreamingServer.startConsumers(StreamingServer.java:327)
at
org.apache.kylin.stream.server.rest.controller.AdminController.assign(AdminController.java:61)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205)
at
org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:133)
at
org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:97)
at
org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:849)
at
org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:760)
at
org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85)
at
org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:967)
at
org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:901)
at
org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970)
at
org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:872)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
at
org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
at
org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:755)
at
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:547)
at
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
at
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1297)
at
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
at
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:485)
at
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
at
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1212)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at
org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:221)
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
at org.eclipse.jetty.server.Server.handle(Server.java:500)
at
org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:383)
at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:547)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:375)
at
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:270)
at
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938)
at java.lang.Thread.run(Thread.java:748)
```
2. Let try doing a reassignment.
<img width="1517" alt="image"
src="https://user-images.githubusercontent.com/14030549/89735825-bd5b1000-da97-11ea-9944-ae54331002aa.png";>
3. I have to modify the source code and repackage. Here is my what I change:
```java
@Override
public void open() {
if (partitions == null || partitions.size() <= 0) {
throw new IllegalStateException("not assign partitions");
}
List<TopicPartition> topicPartitions = Lists.newArrayList();
for (Partition partition : partitions) {
topicPartitions.add(new TopicPartition(topic,
partition.getPartitionId()));
}
kafkaConsumer.assign(topicPartitions);
if (startMode == ConsumerStartMode.EARLIEST) {
kafkaConsumer.seekToBeginning(topicPartitions);
} else if (startMode == ConsumerStartMode.LATEST) {
kafkaConsumer.seekToEnd(topicPartitions);
} else {
for (TopicPartition topicPartition : topicPartitions) {
Long offset =
partitionOffsets.getOrDefault(topicPartition.partition(), 0L);
if (offset == 0L) {
logger.warn("Assume {} is a new added partition, let's
seek it to 0", topicPartition);
}
kafkaConsumer.seek(topicPartition, offset);
}
}
}
```
```sh
mvn clean package -skipTests
...
scp stream-receiver/target/kylin-stream-receiver-3.1.1-SNAPSHOT-all.jar
root@cdh-master:/root/xiaoxiang.yu/older_version/apache-kylin-3.1.0-bin-cdh57-receiver/lib/streaming
```
In receiver side, remove old kylin-stream-receiver-3.1.0-all.jar .
```
[root@cdh-master apache-kylin-3.1.0-bin-cdh57-receiver]# mv
lib/streaming/kylin-stream-receiver-3.1.0-all.jar
kylin-stream-receiver-3.1.0-all.jar.bak
```
4. Restart receiver and here is what we see:
```
2020-08-09 23:59:10,436 INFO [main] server.StreamingServer:725 : Using
/root/xiaoxiang.yu/older_version/apache-kylin-3.1.0-bin-cdh57-receiver/stream_index
to store local segment cache.
2020-08-09 23:59:10,437 INFO [main] server.StreamingServer:301 : register
receiver: cdh-master:7514
2020-08-09 23:59:10,438 WARN [main]
coordinator.ZookeeperStreamMetadataStore:488 :
/stream/receivers/cdh-master_7514 exists.
2020-08-09 23:59:10,458 INFO [main] server.StreamingServer:421 : add the
node to the replicaSet:0, join the group leader election.
2020-08-09 23:59:10,482 INFO [main] common.KylinConfig:476 : Creating new
manager instance of class org.apache.kylin.cube.CubeManager
2020-08-09 23:59:10,495 INFO [main] cube.CubeManager:136 : Initializing
CubeManager with config
REALTIME@jdbc,url=jdbc:mysql://localhost:3306/NightlyBuild,username=root,password=R00t@kylin,maxActive=10,maxIdle=10
2020-08-09 23:59:10,539 INFO [main] persistence.ResourceStore:90 : Using
metadata url
REALTIME@jdbc,url=jdbc:mysql://localhost:3306/NightlyBuild,username=root,password=R00t@kylin,maxActive=10,maxIdle=10
for resource store
2020-08-09 23:59:10,545 INFO [main] persistence.JDBCConnectionManager:92 :
Connecting to Jdbc with url:jdbc:mysql://localhost:3306/NightlyBuild by user
root
2020-08-09 23:59:10,559 INFO [Curator-LeaderSelector-0]
server.ReplicaSetLeaderSelector:66 : become the leader of the replicaSet:0
2020-08-09 23:59:10,559 INFO [Curator-LeaderSelector-0]
server.StreamingServer:575 : become leader of the replicaSet:0
2020-08-09 23:59:10,607 INFO [Curator-LeaderSelector-0]
client.HttpCoordinatorClient:81 : send replicaSet lead change notification to
coordinator
2020-08-09 23:59:11,194 INFO [main] persistence.JDBCConnectionManager:65 :
Connected to MySQL 5.7.26
2020-08-09 23:59:11,212 INFO [main] persistence.JDBCResourceStore:115 :
Table [REALTIME] already exists
2020-08-09 23:59:11,214 INFO [main] persistence.JDBCResourceStore:115 :
Table [REALTIME_log] already exists
2020-08-09 23:59:11,217 DEBUG [main] cachesync.CachedCrudAssist:122 :
Reloading CubeInstance from
REALTIME(key='/cube')@REALTIME@jdbc,url=jdbc:mysql://localhost:3306/NightlyBuild,username=root,password=R00t@kylin,maxActive=10,maxIdle=10
2020-08-09 23:59:11,306 INFO [main] common.KylinConfig:476 : Creating new
manager instance of class org.apache.kylin.cube.CubeDescManager
2020-08-09 23:59:11,307 INFO [main] cube.CubeDescManager:91 : Initializing
CubeDescManager with config
REALTIME@jdbc,url=jdbc:mysql://localhost:3306/NightlyBuild,username=root,password=R00t@kylin,maxActive=10,maxIdle=10
2020-08-09 23:59:11,307 DEBUG [main] cachesync.CachedCrudAssist:122 :
Reloading CubeDesc from
REALTIME(key='/cube_desc')@REALTIME@jdbc,url=jdbc:mysql://localhost:3306/NightlyBuild,username=root,password=R00t@kylin,maxActive=10,maxIdle=10
2020-08-09 23:59:11,358 INFO [main] common.KylinConfig:476 : Creating new
manager instance of class org.apache.kylin.metadata.project.ProjectManager
2020-08-09 23:59:11,359 INFO [main] project.ProjectManager:85 :
Initializing ProjectManager with metadata url
REALTIME@jdbc,url=jdbc:mysql://localhost:3306/NightlyBuild,username=root,password=R00t@kylin,maxActive=10,maxIdle=10
2020-08-09 23:59:11,360 DEBUG [main] cachesync.CachedCrudAssist:122 :
Reloading ProjectInstance from
REALTIME(key='/project')@REALTIME@jdbc,url=jdbc:mysql://localhost:3306/NightlyBuild,username=root,password=R00t@kylin,maxActive=10,maxIdle=10
2020-08-09 23:59:11,369 DEBUG [main] cachesync.CachedCrudAssist:155 : Loaded
1 ProjectInstance(s) out of 1 resource with 0 errors
2020-08-09 23:59:11,371 INFO [main] common.KylinConfig:476 : Creating new
manager instance of class org.apache.kylin.metadata.cachesync.Broadcaster
2020-08-09 23:59:11,373 DEBUG [main] cachesync.Broadcaster:102 : 1 nodes in
the cluster: [localhost:7514]
2020-08-09 23:59:11,377 INFO [main] common.KylinConfig:476 : Creating new
manager instance of class org.apache.kylin.metadata.model.DataModelManager
2020-08-09 23:59:11,381 INFO [main] common.KylinConfig:476 : Creating new
manager instance of class org.apache.kylin.metadata.TableMetadataManager
2020-08-09 23:59:11,382 DEBUG [main] cachesync.CachedCrudAssist:122 :
Reloading TableDesc from
REALTIME(key='/table')@REALTIME@jdbc,url=jdbc:mysql://localhost:3306/NightlyBuild,username=root,password=R00t@kylin,maxActive=10,maxIdle=10
2020-08-09 23:59:11,400 INFO [main] measure.MeasureTypeFactory:121 :
Checking custom measure types from kylin config
2020-08-09 23:59:11,401 INFO [main] measure.MeasureTypeFactory:150 :
registering COUNT_DISTINCT(hllc), class
org.apache.kylin.measure.hllc.HLLCMeasureType$Factory
2020-08-09 23:59:11,406 INFO [main] measure.MeasureTypeFactory:150 :
registering COUNT_DISTINCT(bitmap), class
org.apache.kylin.measure.bitmap.BitmapMeasureType$Factory
2020-08-09 23:59:11,410 INFO [main] measure.MeasureTypeFactory:150 :
registering TOP_N(topn), class
org.apache.kylin.measure.topn.TopNMeasureType$Factory
2020-08-09 23:59:11,412 INFO [main] measure.MeasureTypeFactory:150 :
registering RAW(raw), class org.apache.kylin.measure.raw.RawMeasureType$Factory
2020-08-09 23:59:11,414 INFO [main] measure.MeasureTypeFactory:150 :
registering EXTENDED_COLUMN(extendedcolumn), class
org.apache.kylin.measure.extendedcolumn.ExtendedColumnMeasureType$Factory
2020-08-09 23:59:11,415 INFO [main] measure.MeasureTypeFactory:150 :
registering PERCENTILE_APPROX(percentile), class
org.apache.kylin.measure.percentile.PercentileMeasureType$Factory
2020-08-09 23:59:11,416 INFO [main] measure.MeasureTypeFactory:150 :
registering COUNT_DISTINCT(dim_dc), class
org.apache.kylin.measure.dim.DimCountDistinctMeasureType$Factory
2020-08-09 23:59:11,418 INFO [main] measure.MeasureTypeFactory:150 :
registering STDDEV_SUM(stddev_sum), class
org.apache.kylin.measure.stddev.StdDevSumMeasureType$Factory
2020-08-09 23:59:11,431 INFO [main] measure.MeasureTypeFactory:150 :
registering COUNT_DISTINCT(bitmap_map), class
org.apache.kylin.measure.map.bitmap.BitmapMapMeasureType$Factory
2020-08-09 23:59:11,438 DEBUG [main] cachesync.CachedCrudAssist:155 : Loaded
1 TableDesc(s) out of 1 resource with 0 errors
2020-08-09 23:59:11,439 DEBUG [main] cachesync.CachedCrudAssist:122 :
Reloading TableExtDesc from
REALTIME(key='/table_exd')@REALTIME@jdbc,url=jdbc:mysql://localhost:3306/NightlyBuild,username=root,password=R00t@kylin,maxActive=10,maxIdle=10
2020-08-09 23:59:11,442 DEBUG [main] cachesync.CachedCrudAssist:155 : Loaded
0 TableExtDesc(s) out of 0 resource with 0 errors
2020-08-09 23:59:11,443 DEBUG [main] cachesync.CachedCrudAssist:122 :
Reloading ExternalFilterDesc from
REALTIME(key='/ext_filter')@REALTIME@jdbc,url=jdbc:mysql://localhost:3306/NightlyBuild,username=root,password=R00t@kylin,maxActive=10,maxIdle=10
2020-08-09 23:59:11,445 DEBUG [main] cachesync.CachedCrudAssist:155 : Loaded
0 ExternalFilterDesc(s) out of 0 resource with 0 errors
2020-08-09 23:59:11,445 DEBUG [main] cachesync.CachedCrudAssist:122 :
Reloading DataModelDesc from
REALTIME(key='/model_desc')@REALTIME@jdbc,url=jdbc:mysql://localhost:3306/NightlyBuild,username=root,password=R00t@kylin,maxActive=10,maxIdle=10
2020-08-09 23:59:11,472 DEBUG [main] cachesync.CachedCrudAssist:155 : Loaded
1 DataModelDesc(s) out of 1 resource with 0 errors
2020-08-09 23:59:11,494 DEBUG [main] cachesync.CachedCrudAssist:155 : Loaded
1 CubeDesc(s) out of 1 resource with 0 errors
2020-08-09 23:59:11,495 DEBUG [main] cachesync.CachedCrudAssist:155 : Loaded
1 CubeInstance(s) out of 1 resource with 0 errors
2020-08-09 23:59:11,518 INFO [main] storage.StreamingSegmentManager:239 :
restore segments from local files for cube:UserAnalysisCube
2020-08-09 23:59:11,522 WARN [main] storage.StreamingSegmentManager:253 :
no checkpoint for cube:UserAnalysisCube store state, remove all local segments
folders
2020-08-09 23:59:11,522 INFO [main] server.StreamingServer:351 : create and
start new consumer for cube:UserAnalysisCube
2020-08-09 23:59:11,550 INFO [main] common.KylinConfig:476 : Creating new
manager instance of class org.apache.kylin.cube.cuboid.CuboidManager
2020-08-09 23:59:11,566 INFO [main] kafka.KafkaSource:141 : Create
StreamingConnector for Cube UserAnalysisCube, assignedPartitions [Partition
[partitionId=0, partitionInfo=null], Partition [partitionId=5,
partitionInfo=null], Partition [partitionId=10, partitionInfo=null], Partition
[partitionId=15, partitionInfo=null], Partition [partitionId=13,
partitionInfo=null], Partition [partitionId=2, partitionInfo=null], Partition
[partitionId=8, partitionInfo=null], Partition [partitionId=12,
partitionInfo=null], Partition [partitionId=14, partitionInfo=null], Partition
[partitionId=9, partitionInfo=null], Partition [partitionId=11,
partitionInfo=null], Partition [partitionId=1, partitionInfo=null], Partition
[partitionId=4, partitionInfo=null], Partition [partitionId=6,
partitionInfo=null], Partition [partitionId=7, partitionInfo=null], Partition
[partitionId=3, partitionInfo=null]], startProtocol null
2020-08-09 23:59:11,571 INFO [main] kafka.TimedJsonStreamParser:98 : Using
parser field mapping by {content_list=content_list, act_type=act_type,
event_ts_2=event_ts_2, event_ts=event_ts, devide_type=user_detail.devide_type,
location_city=user_detail.location.city, network_type=user_detail.network_type,
video_id=video_id, event_date_2=event_date_2, str_minute=str_minute,
video_type=video_type, play_times=play_times, pageview_id=pageview_id,
active_minutes=active_minutes, device_brand=device_brand,
str_minute_second=str_minute_second, play_duration=play_duration,
event_date=event_date, page_id=page_id, str_second=str_second, uid=uid}
2020-08-09 23:59:11,584 INFO [main] kafka.TimedJsonStreamParser:120 :
TimedJsonStreamParser with formatTs false tsColName event_date
2020-08-09 23:59:11,663 INFO [main] consumer.ConsumerConfig:279 :
ConsumerConfig values:
auto.commit.interval.ms = 5000
auto.offset.reset = latest
bootstrap.servers = [cdh-master:9092, cdh-worker-1:9092,
cdh-worker-2:9092]
check.crcs = true
client.dns.lookup = default
client.id =
connections.max.idle.ms = 540000
default.api.timeout.ms = 60000
enable.auto.commit = false
exclude.internal.topics = true
fetch.max.bytes = 52428800
fetch.max.wait.ms = 500
fetch.min.bytes = 1
group.id =
heartbeat.interval.ms = 3000
interceptor.classes = []
internal.leave.group.on.close = true
isolation.level = read_uncommitted
key.deserializer = class
org.apache.kafka.common.serialization.ByteArrayDeserializer
max.partition.fetch.bytes = 1048576
max.poll.interval.ms = 300000
max.poll.records = 500
metadata.max.age.ms = 300000
metric.reporters = []
metrics.num.samples = 2
metrics.recording.level = INFO
metrics.sample.window.ms = 30000
partition.assignment.strategy = [class
org.apache.kafka.clients.consumer.RangeAssignor]
receive.buffer.bytes = 65536
reconnect.backoff.max.ms = 1000
reconnect.backoff.ms = 50
request.timeout.ms = 30000
retry.backoff.ms = 100
sasl.client.callback.handler.class = null
sasl.jaas.config = null
sasl.kerberos.kinit.cmd = /usr/bin/kinit
sasl.kerberos.min.time.before.relogin = 60000
sasl.kerberos.service.name = null
sasl.kerberos.ticket.renew.jitter = 0.05
sasl.kerberos.ticket.renew.window.factor = 0.8
sasl.login.callback.handler.class = null
sasl.login.class = null
sasl.login.refresh.buffer.seconds = 300
sasl.login.refresh.min.period.seconds = 60
sasl.login.refresh.window.factor = 0.8
sasl.login.refresh.window.jitter = 0.05
sasl.mechanism = GSSAPI
security.protocol = PLAINTEXT
send.buffer.bytes = 131072
session.timeout.ms = 20000
ssl.cipher.suites = null
ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1]
ssl.endpoint.identification.algorithm = https
ssl.key.password = null
ssl.keymanager.algorithm = SunX509
ssl.keystore.location = null
ssl.keystore.password = null
ssl.keystore.type = JKS
ssl.protocol = TLS
ssl.provider = null
ssl.secure.random.implementation = null
ssl.trustmanager.algorithm = PKIX
ssl.truststore.location = null
ssl.truststore.password = null
ssl.truststore.type = JKS
value.deserializer = class
org.apache.kafka.common.serialization.ByteArrayDeserializer
2020-08-09 23:59:11,790 INFO [main] utils.AppInfoParser:109 : Kafka version
: 2.1.1
2020-08-09 23:59:11,790 INFO [main] utils.AppInfoParser:110 : Kafka
commitId : 21234bee31165527
2020-08-09 23:59:11,792 INFO [main] kafka.KafkaSource:192 :
localConsumeStats from local checkpoint null, remoteConsumeStats from remote
checkpoint
{"0":363171,"1":363198,"2":363249,"3":363171,"4":363199,"5":363250,"6":363170,"7":363196,"8":363250,"9":363170}
2020-08-09 23:59:11,793 INFO [main] kafka.KafkaSource:239 : start kafka
connector from specified position:KafkaPosition{partitionOffsetMap={0=363171,
1=363198, 2=363249, 3=363171, 4=363199, 5=363250, 6=363170, 7=363196, 8=363250,
9=363170}}
2020-08-09 23:59:11,798 WARN [main] consumer.KafkaConnector:91 : Assume
useraction_xxyu-10 is a new added partition, let's seek it to 0
2020-08-09 23:59:11,799 WARN [main] consumer.KafkaConnector:91 : Assume
useraction_xxyu-15 is a new added partition, let's seek it to 0
2020-08-09 23:59:11,799 WARN [main] consumer.KafkaConnector:91 : Assume
useraction_xxyu-13 is a new added partition, let's seek it to 0
2020-08-09 23:59:11,799 WARN [main] consumer.KafkaConnector:91 : Assume
useraction_xxyu-12 is a new added partition, let's seek it to 0
2020-08-09 23:59:11,799 WARN [main] consumer.KafkaConnector:91 : Assume
useraction_xxyu-14 is a new added partition, let's seek it to 0
2020-08-09 23:59:11,799 WARN [main] consumer.KafkaConnector:91 : Assume
useraction_xxyu-11 is a new added partition, let's seek it to 0
2020-08-09 23:59:11,801 INFO [main] metrics.StreamingMetrics:76 : Skip
streaming metricRegistry because it is empty.
2020-08-09 23:59:11,819 INFO [main] util.log:169 : Logging initialized
@3474ms to org.eclipse.jetty.util.log.Slf4jLog
2020-08-09 23:59:11,946 INFO [UserAnalysisCube_channel]
clients.Metadata:285 : Cluster ID: W1YX_q9GRd-R0qwcwFJ9rg
2020-08-09 23:59:12,163 INFO [UserAnalysisCube_channel]
storage.StreamingSegmentManager:189 : Create new segment:StreamingCubeSegment
[cubeName=UserAnalysisCube, segmentName=20200809210000_20200809220000]
2020-08-09 23:59:12,826 INFO [UserAnalysisCube_channel]
storage.StreamingSegmentManager:189 : Create new segment:StreamingCubeSegment
[cubeName=UserAnalysisCube, segmentName=20200809230000_20200810000000]
2020-08-09 23:59:13,226 INFO [main] server.Server:359 :
jetty-9.4.26.v20200117; built: 2020-01-17T12:35:33.676Z; git:
7b38981d25d14afb4a12ff1f2596756144edf695; jvm 1.8.0_171-b11
2020-08-09 23:59:13,260 INFO [main] ContextHandler.kylin:2223 :
Initializing Spring FrameworkServlet
'org.springframework.web.servlet.DispatcherServlet-d2e6b0b'
2020-08-09 23:59:13,278 INFO [main] handler.ContextHandler:825 : Started
o.e.j.s.ServletContextHandler@708eb310{/kylin,null,AVAILABLE}
2020-08-09 23:59:13,279 INFO [main] handler.ContextHandler:825 : Started
o.e.j.s.h.ContextHandler@6ebc9573{/kylin/logs,null,AVAILABLE}
2020-08-09 23:59:13,294 INFO [main] server.AbstractConnector:330 : Started
ServerConnector@369c9bb{HTTP/1.1,[http/1.1]}{0.0.0.0:7514}
2020-08-09 23:59:13,294 INFO [main] server.Server:399 : Started @4949ms
2020-08-09 23:59:15,572 INFO [UserAnalysisCube_channel]
columnar.ColumnarMemoryStorePersister:106 : Start persist memory store for
cube:UserAnalysisCube, segment:20200809210000_20200809220000, rowCnt:50000
2020-08-09 23:59:15,635 DEBUG [UserAnalysisCube_channel]
columnar.ColumnarMemoryStorePersister:202 : cuboid-8191 transform to columnar,
take 58 ms
2020-08-09 23:59:15,655 INFO [UserAnalysisCube_channel]
dict.TrieDictionaryForestBuilder:90 : values not in ascending order, previous
'00000000000000003640', current '00000000000000003638'
2020-08-09 23:59:15,719 DEBUG [UserAnalysisCube_channel]
dict.DictionaryGenerator:91 : Dictionary cardinality: 15962
2020-08-09 23:59:15,720 DEBUG [UserAnalysisCube_channel]
dict.DictionaryGenerator:92 : Dictionary builder class:
org.apache.kylin.dict.DictionaryGenerator$NumberTrieDictForestBuilder
2020-08-09 23:59:15,720 DEBUG [UserAnalysisCube_channel]
dict.DictionaryGenerator:93 : Dictionary class:
org.apache.kylin.dict.TrieDictionaryForest
2020-08-09 23:59:15,725 DEBUG [UserAnalysisCube_channel]
dict.DictionaryGenerator:102 : Dictionary value samples: 3640=>1219,
3638=>1218, 3630=>1212, 3631=>1213, 3632=>1214
2020-08-09 23:59:15,725 DEBUG [UserAnalysisCube_channel]
columnar.ColumnarMemoryStorePersister:403 : BuildDictionary for column : UID
took : 88 ms
2020-08-09 23:59:15,738 INFO [UserAnalysisCube_channel]
dict.TrieDictionaryForestBuilder:90 : values not in ascending order, previous
'page_19', current 'page_13'
2020-08-09 23:59:15,739 DEBUG [UserAnalysisCube_channel]
dict.DictionaryGenerator:91 : Dictionary cardinality: 50
2020-08-09 23:59:15,739 DEBUG [UserAnalysisCube_channel]
dict.DictionaryGenerator:92 : Dictionary builder class:
org.apache.kylin.dict.DictionaryGenerator$StringTrieDictForestBuilder
2020-08-09 23:59:15,740 DEBUG [UserAnalysisCube_channel]
dict.DictionaryGenerator:93 : Dictionary class:
org.apache.kylin.dict.TrieDictionaryForest
2020-08-09 23:59:15,740 DEBUG [UserAnalysisCube_channel]
dict.DictionaryGenerator:102 : Dictionary value samples: page_19=>11,
page_13=>5, page_14=>6, page_11=>3, page_12=>4
2020-08-09 23:59:15,740 DEBUG [UserAnalysisCube_channel]
columnar.ColumnarMemoryStorePersister:403 : BuildDictionary for column :
PAGE_ID took : 13 ms
2020-08-09 23:59:15,751 INFO [UserAnalysisCube_channel]
dict.TrieDictionaryForestBuilder:90 : values not in ascending order, previous
'stop', current 'pause'
2020-08-09 23:59:15,752 DEBUG [UserAnalysisCube_channel]
dict.DictionaryGenerator:91 : Dictionary cardinality: 10
2020-08-09 23:59:15,752 DEBUG [UserAnalysisCube_channel]
dict.DictionaryGenerator:92 : Dictionary builder class:
org.apache.kylin.dict.DictionaryGenerator$StringTrieDictForestBuilder
2020-08-09 23:59:15,752 DEBUG [UserAnalysisCube_channel]
dict.DictionaryGenerator:93 : Dictionary class:
org.apache.kylin.dict.TrieDictionaryForest
2020-08-09 23:59:15,752 DEBUG [UserAnalysisCube_channel]
dict.DictionaryGenerator:102 : Dictionary value samples: stop=>9, pause=>5,
play=>6, like=>4, start=>8
2020-08-09 23:59:15,752 DEBUG [UserAnalysisCube_channel]
columnar.ColumnarMemoryStorePersister:403 : BuildDictionary for column :
ACT_TYPE took : 11 ms
2020-08-09 23:59:15,763 DEBUG [UserAnalysisCube_channel]
dict.DictionaryGenerator:91 : Dictionary cardinality: 34
2020-08-09 23:59:15,763 DEBUG [UserAnalysisCube_channel]
dict.DictionaryGenerator:92 : Dictionary builder class:
org.apache.kylin.dict.DictionaryGenerator$NumberTrieDictForestBuilder
2020-08-09 23:59:15,763 DEBUG [UserAnalysisCube_channel]
dict.DictionaryGenerator:93 : Dictionary class:
org.apache.kylin.dict.TrieDictionaryForest
2020-08-09 23:59:15,764 DEBUG [UserAnalysisCube_channel]
dict.DictionaryGenerator:102 : Dictionary value samples: 0=>0, 1=>1, 2=>2,
3=>3, 4=>4
2020-08-09 23:59:15,764 DEBUG [UserAnalysisCube_channel]
columnar.ColumnarMemoryStorePersister:403 : BuildDictionary for column :
DEVIDE_TYPE took : 10 ms
2020-08-09 23:59:15,774 INFO [UserAnalysisCube_channel]
dict.TrieDictionaryForestBuilder:90 : values not in ascending order, previous
'hongkong', current 'hangzhou'
2020-08-09 23:59:15,774 DEBUG [UserAnalysisCube_channel]
dict.DictionaryGenerator:91 : Dictionary cardinality: 7
2020-08-09 23:59:15,774 DEBUG [UserAnalysisCube_channel]
dict.DictionaryGenerator:92 : Dictionary builder class:
org.apache.kylin.dict.DictionaryGenerator$StringTrieDictForestBuilder
2020-08-09 23:59:15,775 DEBUG [UserAnalysisCube_channel]
dict.DictionaryGenerator:93 : Dictionary class:
org.apache.kylin.dict.TrieDictionaryForest
2020-08-09 23:59:15,775 DEBUG [UserAnalysisCube_channel]
dict.DictionaryGenerator:102 : Dictionary value samples: hongkong=>3,
hangzhou=>2, shenzhen=>6, berlin=>1, nanjing=>4
2020-08-09 23:59:15,775 DEBUG [UserAnalysisCube_channel]
columnar.ColumnarMemoryStorePersister:403 : BuildDictionary for column :
LOCATION_CITY took : 10 ms
2020-08-09 23:59:15,785 DEBUG [UserAnalysisCube_channel]
dict.DictionaryGenerator:91 : Dictionary cardinality: 4
2020-08-09 23:59:15,785 DEBUG [UserAnalysisCube_channel]
dict.DictionaryGenerator:92 : Dictionary builder class:
org.apache.kylin.dict.DictionaryGenerator$NumberTrieDictForestBuilder
2020-08-09 23:59:15,785 DEBUG [UserAnalysisCube_channel]
dict.DictionaryGenerator:93 : Dictionary class:
org.apache.kylin.dict.TrieDictionaryForest
2020-08-09 23:59:15,785 DEBUG [UserAnalysisCube_channel]
dict.DictionaryGenerator:102 : Dictionary value samples: 2=>0, 3=>1, 4=>2, 5=>3
2020-08-09 23:59:15,785 DEBUG [UserAnalysisCube_channel]
columnar.ColumnarMemoryStorePersister:403 : BuildDictionary for column :
NETWORK_TYPE took : 10 ms
2020-08-09 23:59:15,794 INFO [UserAnalysisCube_channel]
dict.TrieDictionaryForestBuilder:90 : values not in ascending order, previous
'xiaomi', current 'vivo'
2020-08-09 23:59:15,795 DEBUG [UserAnalysisCube_channel]
dict.DictionaryGenerator:91 : Dictionary cardinality: 5
2020-08-09 23:59:15,795 DEBUG [UserAnalysisCube_channel]
dict.DictionaryGenerator:92 : Dictionary builder class:
org.apache.kylin.dict.DictionaryGenerator$StringTrieDictForestBuilder
2020-08-09 23:59:15,795 DEBUG [UserAnalysisCube_channel]
dict.DictionaryGenerator:93 : Dictionary class:
org.apache.kylin.dict.TrieDictionaryForest
2020-08-09 23:59:15,795 DEBUG [UserAnalysisCube_channel]
dict.DictionaryGenerator:102 : Dictionary value samples: huawei=>0, xiaomi=>4,
vivo=>3, meizu=>2, iPhone=>1
2020-08-09 23:59:15,795 DEBUG [UserAnalysisCube_channel]
columnar.ColumnarMemoryStorePersister:403 : BuildDictionary for column :
DEVICE_BRAND took : 9 ms
2020-08-09 23:59:15,806 INFO [UserAnalysisCube_channel]
dict.TrieDictionaryForestBuilder:90 : values not in ascending order, previous
'21_14_59', current '21_14_50'
2020-08-09 23:59:15,810 DEBUG [UserAnalysisCube_channel]
dict.DictionaryGenerator:91 : Dictionary cardinality: 2032
2020-08-09 23:59:15,810 DEBUG [UserAnalysisCube_channel]
dict.DictionaryGenerator:92 : Dictionary builder class:
org.apache.kylin.dict.DictionaryGenerator$StringTrieDictForestBuilder
2020-08-09 23:59:15,810 DEBUG [UserAnalysisCube_channel]
dict.DictionaryGenerator:93 : Dictionary class:
org.apache.kylin.dict.TrieDictionaryForest
2020-08-09 23:59:15,810 DEBUG [UserAnalysisCube_channel]
dict.DictionaryGenerator:102 : Dictionary value samples: 21_14_59=>355,
21_14_50=>346, 21_14_55=>351, 21_14_56=>352, 21_14_57=>353
2020-08-09 23:59:15,810 DEBUG [UserAnalysisCube_channel]
columnar.ColumnarMemoryStorePersister:403 : BuildDictionary for column :
STR_MINUTE_SECOND took : 15 ms
2020-08-09 23:59:16,088 DEBUG [UserAnalysisCube_channel]
columnar.ColumnarMemoryStorePersister:326 : cuboid-8191 saved dimension:UID,
took: 276ms
2020-08-09 23:59:16,171 DEBUG [UserAnalysisCube_channel]
columnar.ColumnarMemoryStorePersister:326 : cuboid-8191 saved
dimension:PAGE_ID, took: 83ms
2020-08-09 23:59:16,254 DEBUG [UserAnalysisCube_channel]
columnar.ColumnarMemoryStorePersister:326 : cuboid-8191 saved
dimension:ACT_TYPE, took: 82ms
2020-08-09 23:59:16,316 DEBUG [UserAnalysisCube_channel]
columnar.ColumnarMemoryStorePersister:326 : cuboid-8191 saved
dimension:DEVIDE_TYPE, took: 61ms
2020-08-09 23:59:16,391 DEBUG [UserAnalysisCube_channel]
columnar.ColumnarMemoryStorePersister:326 : cuboid-8191 saved
dimension:LOCATION_CITY, took: 75ms
2020-08-09 23:59:16,461 DEBUG [UserAnalysisCube_channel]
columnar.ColumnarMemoryStorePersister:326 : cuboid-8191 saved
dimension:NETWORK_TYPE, took: 69ms
2020-08-09 23:59:16,532 DEBUG [UserAnalysisCube_channel]
columnar.ColumnarMemoryStorePersister:326 : cuboid-8191 saved
dimension:DEVICE_BRAND, took: 70ms
2020-08-09 23:59:16,608 DEBUG [UserAnalysisCube_channel]
columnar.ColumnarMemoryStorePersister:326 : cuboid-8191 saved
dimension:STR_MINUTE_SECOND, took: 75ms
2020-08-09 23:59:16,646 DEBUG [UserAnalysisCube_channel]
columnar.ColumnarMemoryStorePersister:326 : cuboid-8191 saved
dimension:MONTH_START, took: 38ms
2020-08-09 23:59:16,673 DEBUG [UserAnalysisCube_channel]
columnar.ColumnarMemoryStorePersister:326 : cuboid-8191 saved
dimension:WEEK_START, took: 26ms
2020-08-09 23:59:16,703 DEBUG [UserAnalysisCube_channel]
columnar.ColumnarMemoryStorePersister:326 : cuboid-8191 saved
dimension:DAY_START, took: 29ms
2020-08-09 23:59:16,736 DEBUG [UserAnalysisCube_channel]
columnar.ColumnarMemoryStorePersister:326 : cuboid-8191 saved
dimension:HOUR_START, took: 33ms
2020-08-09 23:59:16,779 DEBUG [UserAnalysisCube_channel]
columnar.ColumnarMemoryStorePersister:326 : cuboid-8191 saved
dimension:MINUTE_START, took: 41ms
2020-08-09 23:59:16,808 DEBUG [UserAnalysisCube_channel]
columnar.ColumnarMemoryStorePersister:379 : cuboid-8191 saved measure:_COUNT_,
took: 29ms
2020-08-09 23:59:16,871 DEBUG [UserAnalysisCube_channel]
columnar.ColumnarMemoryStorePersister:379 : cuboid-8191 saved
measure:SUM_PLAY_TIMES, took: 63ms
2020-08-09 23:59:16,907 DEBUG [UserAnalysisCube_channel]
columnar.ColumnarMemoryStorePersister:379 : cuboid-8191 saved
measure:SUM_ACTIVE_MINUTES, took: 34ms
2020-08-09 23:59:16,932 DEBUG [UserAnalysisCube_channel]
columnar.ColumnarMemoryStorePersister:379 : cuboid-8191 saved
measure:SUM_PLAY_DURATION, took: 25ms
2020-08-09 23:59:16,993 DEBUG [UserAnalysisCube_channel]
columnar.ColumnarMemoryStorePersister:379 : cuboid-8191 saved
measure:MAX_PLAY_TIMES, took: 60ms
2020-08-09 23:59:17,027 DEBUG [UserAnalysisCube_channel]
columnar.ColumnarMemoryStorePersister:379 : cuboid-8191 saved
measure:MAX_ACTIVE_MINUTES, took: 33ms
2020-08-09 23:59:17,049 DEBUG [UserAnalysisCube_channel]
columnar.ColumnarMemoryStorePersister:379 : cuboid-8191 saved
measure:MAX_PLAY_DURATION, took: 22ms
2020-08-09 23:59:17,081 DEBUG [UserAnalysisCube_channel]
columnar.ColumnarMemoryStorePersister:379 : cuboid-8191 saved measure:PV_STAT,
took: 31ms
2020-08-09 23:59:17,089 INFO [UserAnalysisCube_channel]
columnar.ColumnarMemoryStorePersister:110 : Finish persist memory store for
cube:UserAnalysisCube segment:20200809210000_20200809220000, take: 1517ms
2020-08-09 23:59:18,954 INFO [UserAnalysisCube_channel]
columnar.ColumnarMemoryStorePersister:106 : Start persist memory store for
cube:UserAnalysisCube, segment:20200809210000_20200809220000, rowCnt:50000
2020-08-09 23:59:18,980 DEBUG [UserAnalysisCube_channel]
columnar.ColumnarMemoryStorePersister:202 : cuboid-8191 transform to columnar,
take 26 ms
2020-08-09 23:59:18,989 INFO [UserAnalysisCube_channel]
dict.TrieDictionaryForestBuilder:90 : values not in ascending order, previous
'00000000000000003640', current '00000000000000003638'
2020-08-09 23:59:19,008 DEBUG [UserAnalysisCube_channel]
dict.DictionaryGenerator:91 : Dictionary cardinality: 16046
2020-08-09 23:59:19,008 DEBUG [UserAnalysisCube_channel]
dict.DictionaryGenerator:92 : Dictionary builder class:
org.apache.kylin.dict.DictionaryGenerator$NumberTrieDictForestBuilder
2020-08-09 23:59:19,008 DEBUG [UserAnalysisCube_channel]
dict.DictionaryGenerator:93 : Dictionary class:
org.apache.kylin.dict.TrieDictionaryForest
2020-08-09 23:59:19,008 DEBUG [UserAnalysisCube_channel]
dict.DictionaryGenerator:102 : Dictionary value samples: 3640=>1240,
3638=>1238, 3639=>1239, 3630=>1231, 3631=>1232
2020-08-09 23:59:19,008 DEBUG [UserAnalysisCube_channel]
columnar.ColumnarMemoryStorePersister:403 : BuildDictionary for column : UID
took : 28 ms
2020-08-09 23:59:19,017 INFO [UserAnalysisCube_channel]
dict.TrieDictionaryForestBuilder:90 : values not in ascending order, previous
'page_19', current 'page_13'
2020-08-09 23:59:19,017 DEBUG [UserAnalysisCube_channel]
dict.DictionaryGenerator:91 : Dictionary cardinality: 50
2020-08-09 23:59:19,017 DEBUG [UserAnalysisCube_channel]
dict.DictionaryGenerator:92 : Dictionary builder class:
org.apache.kylin.dict.DictionaryGenerator$StringTrieDictForestBuilder
2020-08-09 23:59:19,017 DEBUG [UserAnalysisCube_channel]
dict.DictionaryGenerator:93 : Dictionary class:
org.apache.kylin.dict.TrieDictionaryForest
2020-08-09 23:59:19,018 DEBUG [UserAnalysisCube_channel]
dict.DictionaryGenerator:102 : Dictionary value samples: page_19=>11,
page_13=>5, page_14=>6, page_11=>3, page_12=>4
2020-08-09 23:59:19,018 DEBUG [UserAnalysisCube_channel]
columnar.ColumnarMemoryStorePersister:403 : BuildDictionary for column :
PAGE_ID took : 8 ms
2020-08-09 23:59:19,025 INFO [UserAnalysisCube_channel]
dict.TrieDictionaryForestBuilder:90 : values not in ascending order, previous
'stop', current 'pause'
2020-08-09 23:59:19,026 DEBUG [UserAnalysisCube_channel]
dict.DictionaryGenerator:91 : Dictionary cardinality: 10
2020-08-09 23:59:19,026 DEBUG [UserAnalysisCube_channel]
dict.DictionaryGenerator:92 : Dictionary builder class:
org.apache.kylin.dict.DictionaryGenerator$StringTrieDictForestBuilder
2020-08-09 23:59:19,026 DEBUG [UserAnalysisCube_channel]
dict.DictionaryGenerator:93 : Dictionary class:
org.apache.kylin.dict.TrieDictionaryForest
2020-08-09 23:59:19,026 DEBUG [UserAnalysisCube_channel]
dict.DictionaryGenerator:102 : Dictionary value samples: stop=>9, pause=>5,
play=>6, like=>4, start=>8
2020-08-09 23:59:19,026 DEBUG [UserAnalysisCube_channel]
columnar.ColumnarMemoryStorePersister:403 : BuildDictionary for column :
ACT_TYPE took : 8 ms
2020-08-09 23:59:19,035 DEBUG [UserAnalysisCube_channel]
dict.DictionaryGenerator:91 : Dictionary cardinality: 44
2020-08-09 23:59:19,035 DEBUG [UserAnalysisCube_channel]
dict.DictionaryGenerator:92 : Dictionary builder class:
org.apache.kylin.dict.DictionaryGenerator$NumberTrieDictForestBuilder
2020-08-09 23:59:19,035 DEBUG [UserAnalysisCube_channel]
dict.DictionaryGenerator:93 : Dictionary class:
org.apache.kylin.dict.TrieDictionaryForest
2020-08-09 23:59:19,035 DEBUG [UserAnalysisCube_channel]
dict.DictionaryGenerator:102 : Dictionary value samples: 16=>0, 17=>1, 18=>2,
19=>3, 20=>4
2020-08-09 23:59:19,035 DEBUG [UserAnalysisCube_channel]
columnar.ColumnarMemoryStorePersister:403 : BuildDictionary for column :
DEVIDE_TYPE took : 8 ms
2020-08-09 23:59:19,043 INFO [UserAnalysisCube_channel]
dict.TrieDictionaryForestBuilder:90 : values not in ascending order, previous
'hongkong', current 'hangzhou'
2020-08-09 23:59:19,043 DEBUG [UserAnalysisCube_channel]
dict.DictionaryGenerator:91 : Dictionary cardinality: 7
2020-08-09 23:59:19,043 DEBUG [UserAnalysisCube_channel]
dict.DictionaryGenerator:92 : Dictionary builder class:
org.apache.kylin.dict.DictionaryGenerator$StringTrieDictForestBuilder
2020-08-09 23:59:19,043 DEBUG [UserAnalysisCube_channel]
dict.DictionaryGenerator:93 : Dictionary class:
org.apache.kylin.dict.TrieDictionaryForest
2020-08-09 23:59:19,043 DEBUG [UserAnalysisCube_channel]
dict.DictionaryGenerator:102 : Dictionary value samples: hongkong=>3,
hangzhou=>2, shenzhen=>6, berlin=>1, nanjing=>4
2020-08-09 23:59:19,044 DEBUG [UserAnalysisCube_channel]
columnar.ColumnarMemoryStorePersister:403 : BuildDictionary for column :
LOCATION_CITY took : 8 ms
2020-08-09 23:59:19,052 DEBUG [UserAnalysisCube_channel]
dict.DictionaryGenerator:91 : Dictionary cardinality: 4
2020-08-09 23:59:19,052 DEBUG [UserAnalysisCube_channel]
dict.DictionaryGenerator:92 : Dictionary builder class:
org.apache.kylin.dict.DictionaryGenerator$NumberTrieDictForestBuilder
2020-08-09 23:59:19,052 DEBUG [UserAnalysisCube_channel]
dict.DictionaryGenerator:93 : Dictionary class:
org.apache.kylin.dict.TrieDictionaryForest
2020-08-09 23:59:19,052 DEBUG [UserAnalysisCube_channel]
dict.DictionaryGenerator:102 : Dictionary value samples: 2=>0, 3=>1, 4=>2, 5=>3
2020-08-09 23:59:19,052 DEBUG [UserAnalysisCube_channel]
columnar.ColumnarMemoryStorePersister:403 : BuildDictionary for column :
NETWORK_TYPE took : 8 ms
2020-08-09 23:59:19,059 INFO [UserAnalysisCube_channel]
dict.TrieDictionaryForestBuilder:90 : values not in ascending order, previous
'xiaomi', current 'vivo'
2020-08-09 23:59:19,060 DEBUG [UserAnalysisCube_channel]
dict.DictionaryGenerator:91 : Dictionary cardinality: 5
2020-08-09 23:59:19,060 DEBUG [UserAnalysisCube_channel]
dict.DictionaryGenerator:92 : Dictionary builder class:
org.apache.kylin.dict.DictionaryGenerator$StringTrieDictForestBuilder
2020-08-09 23:59:19,060 DEBUG [UserAnalysisCube_channel]
dict.DictionaryGenerator:93 : Dictionary class:
org.apache.kylin.dict.TrieDictionaryForest
2020-08-09 23:59:19,060 DEBUG [UserAnalysisCube_channel]
dict.DictionaryGenerator:102 : Dictionary value samples: huawei=>0, xiaomi=>4,
vivo=>3, meizu=>2, iPhone=>1
2020-08-09 23:59:19,060 DEBUG [UserAnalysisCube_channel]
columnar.ColumnarMemoryStorePersister:403 : BuildDictionary for column :
DEVICE_BRAND took : 7 ms
2020-08-09 23:59:19,069 INFO [UserAnalysisCube_channel]
dict.TrieDictionaryForestBuilder:90 : values not in ascending order, previous
'21_55_9', current '21_55_0'
2020-08-09 23:59:19,077 DEBUG [UserAnalysisCube_channel]
dict.DictionaryGenerator:91 : Dictionary cardinality: 2536
2020-08-09 23:59:19,077 DEBUG [UserAnalysisCube_channel]
dict.DictionaryGenerator:92 : Dictionary builder class:
org.apache.kylin.dict.DictionaryGenerator$StringTrieDictForestBuilder
2020-08-09 23:59:19,077 DEBUG [UserAnalysisCube_channel]
dict.DictionaryGenerator:93 : Dictionary class:
org.apache.kylin.dict.TrieDictionaryForest
2020-08-09 23:59:19,077 DEBUG [UserAnalysisCube_channel]
dict.DictionaryGenerator:102 : Dictionary value samples: 21_55_7=>2341,
21_55_8=>2342, 21_55_9=>2343, 21_55_0=>2284, 21_55_1=>2285
2020-08-09 23:59:19,077 DEBUG [UserAnalysisCube_channel]
columnar.ColumnarMemoryStorePersister:403 : BuildDictionary for column :
STR_MINUTE_SECOND took : 17 ms
2020-08-09 23:59:19,185 DEBUG [UserAnalysisCube_channel]
columnar.ColumnarMemoryStorePersister:326 : cuboid-8191 saved dimension:UID,
took: 107ms
2020-08-09 23:59:19,235 DEBUG [UserAnalysisCube_channel]
columnar.ColumnarMemoryStorePersister:326 : cuboid-8191 saved
dimension:PAGE_ID, took: 49ms
2020-08-09 23:59:19,284 DEBUG [UserAnalysisCube_channel]
columnar.ColumnarMemoryStorePersister:326 : cuboid-8191 saved
dimension:ACT_TYPE, took: 48ms
2020-08-09 23:59:19,335 DEBUG [UserAnalysisCube_channel]
columnar.ColumnarMemoryStorePersister:326 : cuboid-8191 saved
dimension:DEVIDE_TYPE, took: 50ms
2020-08-09 23:59:19,396 DEBUG [UserAnalysisCube_channel]
columnar.ColumnarMemoryStorePersister:326 : cuboid-8191 saved
dimension:LOCATION_CITY, took: 60ms
2020-08-09 23:59:19,455 DEBUG [UserAnalysisCube_channel]
columnar.ColumnarMemoryStorePersister:326 : cuboid-8191 saved
dimension:NETWORK_TYPE, took: 59ms
2020-08-09 23:59:19,522 DEBUG [UserAnalysisCube_channel]
columnar.ColumnarMemoryStorePersister:326 : cuboid-8191 saved
dimension:DEVICE_BRAND, took: 66ms
2020-08-09 23:59:19,592 DEBUG [UserAnalysisCube_channel]
columnar.ColumnarMemoryStorePersister:326 : cuboid-8191 saved
dimension:STR_MINUTE_SECOND, took: 69ms
2020-08-09 23:59:19,610 DEBUG [UserAnalysisCube_channel]
columnar.ColumnarMemoryStorePersister:326 : cuboid-8191 saved
dimension:MONTH_START, took: 17ms
2020-08-09 23:59:19,626 DEBUG [UserAnalysisCube_channel]
columnar.ColumnarMemoryStorePersister:326 : cuboid-8191 saved
dimension:WEEK_START, took: 16ms
2020-08-09 23:59:19,642 DEBUG [UserAnalysisCube_channel]
columnar.ColumnarMemoryStorePersister:326 : cuboid-8191 saved
dimension:DAY_START, took: 15ms
2020-08-09 23:59:19,662 DEBUG [UserAnalysisCube_channel]
columnar.ColumnarMemoryStorePersister:326 : cuboid-8191 saved
dimension:HOUR_START, took: 19ms
2020-08-09 23:59:19,691 DEBUG [UserAnalysisCube_channel]
columnar.ColumnarMemoryStorePersister:326 : cuboid-8191 saved
dimension:MINUTE_START, took: 28ms
2020-08-09 23:59:19,708 DEBUG [UserAnalysisCube_channel]
columnar.ColumnarMemoryStorePersister:379 : cuboid-8191 saved measure:_COUNT_,
took: 16ms
2020-08-09 23:59:19,766 DEBUG [UserAnalysisCube_channel]
columnar.ColumnarMemoryStorePersister:379 : cuboid-8191 saved
measure:SUM_PLAY_TIMES, took: 57ms
2020-08-09 23:59:19,783 DEBUG [UserAnalysisCube_channel]
columnar.ColumnarMemoryStorePersister:379 : cuboid-8191 saved
measure:SUM_ACTIVE_MINUTES, took: 17ms
2020-08-09 23:59:19,801 DEBUG [UserAnalysisCube_channel]
columnar.ColumnarMemoryStorePersister:379 : cuboid-8191 saved
measure:SUM_PLAY_DURATION, took: 17ms
2020-08-09 23:59:19,856 DEBUG [UserAnalysisCube_channel]
columnar.ColumnarMemoryStorePersister:379 : cuboid-8191 saved
measure:MAX_PLAY_TIMES, took: 54ms
2020-08-09 23:59:19,873 DEBUG [UserAnalysisCube_channel]
columnar.ColumnarMemoryStorePersister:379 : cuboid-8191 saved
measure:MAX_ACTIVE_MINUTES, took: 17ms
2020-08-09 23:59:19,890 DEBUG [UserAnalysisCube_channel]
columnar.ColumnarMemoryStorePersister:379 : cuboid-8191 saved
measure:MAX_PLAY_DURATION, took: 16ms
2020-08-09 23:59:19,908 DEBUG [UserAnalysisCube_channel]
columnar.ColumnarMemoryStorePersister:379 : cuboid-8191 saved measure:PV_STAT,
took: 18ms
2020-08-09 23:59:19,913 INFO [UserAnalysisCube_channel]
columnar.ColumnarMemoryStorePersister:110 : Finish persist memory store for
cube:UserAnalysisCube segment:20200809210000_20200809220000, take: 959ms
2020-08-09 23:59:20,175 INFO [UserAnalysisCube_channel]
storage.StreamingSegmentManager:189 : Create new segment:StreamingCubeSegment
[cubeName=UserAnalysisCube, segmentName=20200809220000_20200809230000]
2020-08-09 23:59:22,201 INFO [UserAnalysisCube_channel]
columnar.ColumnarMemoryStorePersister:106 : Start persist memory store for
cube:UserAnalysisCube, segment:20200809220000_20200809230000, rowCnt:50000
2020-08-09 23:59:22,235 DEBUG [UserAnalysisCube_channel]
columnar.ColumnarMemoryStorePersister:202 : cuboid-8191 transform to columnar,
take 33 ms
2020-08-09 23:59:22,245 INFO [UserAnalysisCube_channel]
dict.TrieDictionaryForestBuilder:90 : values not in ascending order, previous
'00000000000000003640', current '00000000000000003638'
2020-08-09 23:59:22,258 DEBUG [UserAnalysisCube_channel]
dict.DictionaryGenerator:91 : Dictionary cardinality: 15977
2020-08-09 23:59:22,258 DEBUG [UserAnalysisCube_channel]
dict.DictionaryGenerator:92 : Dictionary builder class:
org.apache.kylin.dict.DictionaryGenerator$NumberTrieDictForestBuilder
2020-08-09 23:59:22,258 DEBUG [UserAnalysisCube_channel]
dict.DictionaryGenerator:93 : Dictionary class:
org.apache.kylin.dict.TrieDictionaryForest
2020-08-09 23:59:22,258 DEBUG [UserAnalysisCube_channel]
dict.DictionaryGenerator:102 : Dictionary value samples: 3640=>1166,
3638=>1165, 3630=>1158, 3631=>1159, 3632=>1160
2020-08-09 23:59:22,258 DEBUG [UserAnalysisCube_channel]
columnar.ColumnarMemoryStorePersister:403 : BuildDictionary for column : UID
took : 23 ms
2020-08-09 23:59:22,267 INFO [UserAnalysisCube_channel]
dict.TrieDictionaryForestBuilder:90 : values not in ascending order, previous
'page_19', current 'page_13'
2020-08-09 23:59:22,268 DEBUG [UserAnalysisCube_channel]
dict.DictionaryGenerator:91 : Dictionary cardinality: 50
2020-08-09 23:59:22,268 DEBUG [UserAnalysisCube_channel]
dict.DictionaryGenerator:92 : Dictionary builder class:
org.apache.kylin.dict.DictionaryGenerator$StringTrieDictForestBuilder
2020-08-09 23:59:22,268 DEBUG [UserAnalysisCube_channel]
dict.DictionaryGenerator:93 : Dictionary class:
org.apache.kylin.dict.TrieDictionaryForest
2020-08-09 23:59:22,268 DEBUG [UserAnalysisCube_channel]
dict.DictionaryGenerator:102 : Dictionary value samples: page_19=>11,
page_13=>5, page_14=>6, page_11=>3, page_12=>4
2020-08-09 23:59:22,268 DEBUG [UserAnalysisCube_channel]
columnar.ColumnarMemoryStorePersister:403 : BuildDictionary for column :
PAGE_ID took : 9 ms
2020-08-09 23:59:22,277 INFO [UserAnalysisCube_channel]
dict.TrieDictionaryForestBuilder:90 : values not in ascending order, previous
'stop', current 'pause'
2020-08-09 23:59:22,277 DEBUG [UserAnalysisCube_channel]
dict.DictionaryGenerator:91 : Dictionary cardinality: 10
2020-08-09 23:59:22,277 DEBUG [UserAnalysisCube_channel]
dict.DictionaryGenerator:92 : Dictionary builder class:
org.apache.kylin.dict.DictionaryGenerator$StringTrieDictForestBuilder
2020-08-09 23:59:22,277 DEBUG [UserAnalysisCube_channel]
dict.DictionaryGenerator:93 : Dictionary class:
org.apache.kylin.dict.TrieDictionaryForest
2020-08-09 23:59:22,277 DEBUG [UserAnalysisCube_channel]
dict.DictionaryGenerator:102 : Dictionary value samples: stop=>9, pause=>5,
play=>6, like=>4, start=>8
2020-08-09 23:59:22,277 DEBUG [UserAnalysisCube_channel]
columnar.ColumnarMemoryStorePersister:403 : BuildDictionary for column :
ACT_TYPE took : 9 ms
2020-08-09 23:59:22,285 DEBUG [UserAnalysisCube_channel]
dict.DictionaryGenerator:91 : Dictionary cardinality: 34
2020-08-09 23:59:22,285 DEBUG [UserAnalysisCube_channel]
dict.DictionaryGenerator:92 : Dictionary builder class:
org.apache.kylin.dict.DictionaryGenerator$NumberTrieDictForestBuilder
2020-08-09 23:59:22,286 DEBUG [UserAnalysisCube_channel]
dict.DictionaryGenerator:93 : Dictionary class:
org.apache.kylin.dict.TrieDictionaryForest
2020-08-09 23:59:22,286 DEBUG [UserAnalysisCube_channel]
dict.DictionaryGenerator:102 : Dictionary value samples: 0=>0, 1=>1, 2=>2,
3=>3, 4=>4
2020-08-09 23:59:22,286 DEBUG [UserAnalysisCube_channel]
columnar.ColumnarMemoryStorePersister:403 : BuildDictionary for column :
DEVIDE_TYPE took : 8 ms
2020-08-09 23:59:22,294 INFO [UserAnalysisCube_channel]
dict.TrieDictionaryForestBuilder:90 : values not in ascending order, previous
'hongkong', current 'hangzhou'
2020-08-09 23:59:22,294 DEBUG [UserAnalysisCube_channel]
dict.DictionaryGenerator:91 : Dictionary cardinality: 7
2020-08-09 23:59:22,294 DEBUG [UserAnalysisCube_channel]
dict.DictionaryGenerator:92 : Dictionary builder class:
org.apache.kylin.dict.DictionaryGenerator$StringTrieDictForestBuilder
2020-08-09 23:59:22,295 DEBUG [UserAnalysisCube_channel]
dict.DictionaryGenerator:93 : Dictionary class:
org.apache.kylin.dict.TrieDictionaryForest
2020-08-09 23:59:22,295 DEBUG [UserAnalysisCube_channel]
dict.DictionaryGenerator:102 : Dictionary value samples: hongkong=>3,
hangzhou=>2, shenzhen=>6, berlin=>1, nanjing=>4
2020-08-09 23:59:22,295 DEBUG [UserAnalysisCube_channel]
columnar.ColumnarMemoryStorePersister:403 : BuildDictionary for column :
LOCATION_CITY took : 8 ms
2020-08-09 23:59:22,303 DEBUG [UserAnalysisCube_channel]
dict.DictionaryGenerator:91 : Dictionary cardinality: 4
2020-08-09 23:59:22,303 DEBUG [UserAnalysisCube_channel]
dict.DictionaryGenerator:92 : Dictionary builder class:
org.apache.kylin.dict.DictionaryGenerator$NumberTrieDictForestBuilder
2020-08-09 23:59:22,303 DEBUG [UserAnalysisCube_channel]
dict.DictionaryGenerator:93 : Dictionary class:
org.apache.kylin.dict.TrieDictionaryForest
2020-08-09 23:59:22,303 DEBUG [UserAnalysisCube_channel]
dict.DictionaryGenerator:102 : Dictionary value samples: 2=>0, 3=>1, 4=>2, 5=>3
2020-08-09 23:59:22,303 DEBUG [UserAnalysisCube_channel]
columnar.ColumnarMemoryStorePersister:403 : BuildDictionary for column :
NETWORK_TYPE took : 8 ms
2020-08-09 23:59:22,311 INFO [UserAnalysisCube_channel]
dict.TrieDictionaryForestBuilder:90 : values not in ascending order, previous
'xiaomi', current 'vivo'
2020-08-09 23:59:22,311 DEBUG [UserAnalysisCube_channel]
dict.DictionaryGenerator:91 : Dictionary cardinality: 5
2020-08-09 23:59:22,311 DEBUG [UserAnalysisCube_channel]
dict.DictionaryGenerator:92 : Dictionary builder class:
org.apache.kylin.dict.DictionaryGenerator$StringTrieDictForestBuilder
2020-08-09 23:59:22,311 DEBUG [UserAnalysisCube_channel]
dict.DictionaryGenerator:93 : Dictionary class:
org.apache.kylin.dict.TrieDictionaryForest
2020-08-09 23:59:22,311 DEBUG [UserAnalysisCube_channel]
dict.DictionaryGenerator:102 : Dictionary value samples: huawei=>0, xiaomi=>4,
vivo=>3, meizu=>2, iPhone=>1
2020-08-09 23:59:22,311 DEBUG [UserAnalysisCube_channel]
columnar.ColumnarMemoryStorePersister:403 : BuildDictionary for column :
DEVICE_BRAND took : 8 ms
2020-08-09 23:59:22,321 INFO [UserAnalysisCube_channel]
dict.TrieDictionaryForestBuilder:90 : values not in ascending order, previous
'22_14_59', current '22_14_41'
2020-08-09 23:59:22,323 DEBUG [UserAnalysisCube_channel]
dict.DictionaryGenerator:91 : Dictionary cardinality: 1985
2020-08-09 23:59:22,324 DEBUG [UserAnalysisCube_channel]
dict.DictionaryGenerator:92 : Dictionary builder class:
org.apache.kylin.dict.DictionaryGenerator$StringTrieDictForestBuilder
2020-08-09 23:59:22,324 DEBUG [UserAnalysisCube_channel]
dict.DictionaryGenerator:93 : Dictionary class:
org.apache.kylin.dict.TrieDictionaryForest
2020-08-09 23:59:22,324 DEBUG [UserAnalysisCube_channel]
dict.DictionaryGenerator:102 : Dictionary value samples: 22_14_58=>354,
22_14_59=>355, 22_14_41=>336, 22_14_42=>337, 22_14_40=>335
2020-08-09 23:59:22,324 DEBUG [UserAnalysisCube_channel]
columnar.ColumnarMemoryStorePersister:403 : BuildDictionary for column :
STR_MINUTE_SECOND took : 12 ms
2020-08-09 23:59:22,426 DEBUG [UserAnalysisCube_channel]
columnar.ColumnarMemoryStorePersister:326 : cuboid-8191 saved dimension:UID,
took: 102ms
2020-08-09 23:59:22,477 DEBUG [UserAnalysisCube_channel]
columnar.ColumnarMemoryStorePersister:326 : cuboid-8191 saved
dimension:PAGE_ID, took: 50ms
2020-08-09 23:59:22,526 DEBUG [UserAnalysisCube_channel]
columnar.ColumnarMemoryStorePersister:326 : cuboid-8191 saved
dimension:ACT_TYPE, took: 49ms
2020-08-09 23:59:22,574 DEBUG [UserAnalysisCube_channel]
columnar.ColumnarMemoryStorePersister:326 : cuboid-8191 saved
dimension:DEVIDE_TYPE, took: 47ms
2020-08-09 23:59:22,706 DEBUG [UserAnalysisCube_channel]
columnar.ColumnarMemoryStorePersister:326 : cuboid-8191 saved
dimension:LOCATION_CITY, took: 61ms
2020-08-09 23:59:22,768 DEBUG [UserAnalysisCube_channel]
columnar.ColumnarMemoryStorePersister:326 : cuboid-8191 saved
dimension:NETWORK_TYPE, took: 61ms
2020-08-09 23:59:22,834 DEBUG [UserAnalysisCube_channel]
columnar.ColumnarMemoryStorePersister:326 : cuboid-8191 saved
dimension:DEVICE_BRAND, took: 66ms
2020-08-09 23:59:22,903 DEBUG [UserAnalysisCube_channel]
columnar.ColumnarMemoryStorePersister:326 : cuboid-8191 saved
dimension:STR_MINUTE_SECOND, took: 68ms
2020-08-09 23:59:22,920 DEBUG [UserAnalysisCube_channel]
columnar.ColumnarMemoryStorePersister:326 : cuboid-8191 saved
dimension:MONTH_START, took: 16ms
2020-08-09 23:59:22,937 DEBUG [UserAnalysisCube_channel]
columnar.ColumnarMemoryStorePersister:326 : cuboid-8191 saved
dimension:WEEK_START, took: 16ms
2020-08-09 23:59:22,953 DEBUG [UserAnalysisCube_channel]
columnar.ColumnarMemoryStorePersister:326 : cuboid-8191 saved
dimension:DAY_START, took: 16ms
2020-08-09 23:59:22,971 DEBUG [UserAnalysisCube_channel]
columnar.ColumnarMemoryStorePersister:326 : cuboid-8191 saved
dimension:HOUR_START, took: 17ms
2020-08-09 23:59:22,995 DEBUG [UserAnalysisCube_channel]
columnar.ColumnarMemoryStorePersister:326 : cuboid-8191 saved
dimension:MINUTE_START, took: 24ms
2020-08-09 23:59:23,008 DEBUG [UserAnalysisCube_channel]
columnar.ColumnarMemoryStorePersister:379 : cuboid-8191 saved measure:_COUNT_,
took: 12ms
2020-08-09 23:59:23,064 DEBUG [UserAnalysisCube_channel]
columnar.ColumnarMemoryStorePersister:379 : cuboid-8191 saved
measure:SUM_PLAY_TIMES, took: 55ms
2020-08-09 23:59:23,082 DEBUG [UserAnalysisCube_channel]
columnar.ColumnarMemoryStorePersister:379 : cuboid-8191 saved
measure:SUM_ACTIVE_MINUTES, took: 17ms
2020-08-09 23:59:23,100 DEBUG [UserAnalysisCube_channel]
columnar.ColumnarMemoryStorePersister:379 : cuboid-8191 saved
measure:SUM_PLAY_DURATION, took: 17ms
2020-08-09 23:59:23,157 DEBUG [UserAnalysisCube_channel]
columnar.ColumnarMemoryStorePersister:379 : cuboid-8191 saved
measure:MAX_PLAY_TIMES, took: 56ms
2020-08-09 23:59:23,175 DEBUG [UserAnalysisCube_channel]
columnar.ColumnarMemoryStorePersister:379 : cuboid-8191 saved
measure:MAX_ACTIVE_MINUTES, took: 17ms
2020-08-09 23:59:23,193 DEBUG [UserAnalysisCube_channel]
columnar.ColumnarMemoryStorePersister:379 : cuboid-8191 saved
measure:MAX_PLAY_DURATION, took: 18ms
2020-08-09 23:59:23,213 DEBUG [UserAnalysisCube_channel]
columnar.ColumnarMemoryStorePersister:379 : cuboid-8191 saved measure:PV_STAT,
took: 19ms
2020-08-09 23:59:23,214 INFO [UserAnalysisCube_channel]
columnar.ColumnarMemoryStorePersister:110 : Finish persist memory store for
cube:UserAnalysisCube segment:20200809220000_20200809230000, take: 1012ms
2020-08-09 23:59:25,467 INFO [UserAnalysisCube_channel]
columnar.ColumnarMemoryStorePersister:106 : Start persist memory store for
cube:UserAnalysisCube, segment:20200809230000_20200810000000, rowCnt:50000
2020-08-09 23:59:25,495 DEBUG [UserAnalysisCube_channel]
columnar.ColumnarMemoryStorePersister:202 : cuboid-8191 transform to columnar,
take 26 ms
2020-08-09 23:59:25,504 INFO [UserAnalysisCube_channel]
dict.TrieDictionaryForestBuilder:90 : values not in ascending order, previous
'00000000000000003639', current '00000000000000003630'
2020-08-09 23:59:25,519 DEBUG [UserAnalysisCube_channel]
dict.DictionaryGenerator:91 : Dictionary cardinality: 15936
2020-08-09 23:59:25,519 DEBUG [UserAnalysisCube_channel]
dict.DictionaryGenerator:92 : Dictionary builder class:
org.apache.kylin.dict.DictionaryGenerator$NumberTrieDictForestBuilder
2020-08-09 23:59:25,519 DEBUG [UserAnalysisCube_channel]
dict.DictionaryGenerator:93 : Dictionary class:
org.apache.kylin.dict.TrieDictionaryForest
2020-08-09 23:59:25,519 DEBUG [UserAnalysisCube_channel]
dict.DictionaryGenerator:102 : Dictionary value samples: 3638=>1084,
3639=>1085, 3630=>1077, 3631=>1078, 3632=>1079
2020-08-09 23:59:25,519 DEBUG [UserAnalysisCube_channel]
columnar.ColumnarMemoryStorePersister:403 : BuildDictionary for column : UID
took : 24 ms
2020-08-09 23:59:25,532 INFO [UserAnalysisCube_channel]
dict.TrieDictionaryForestBuilder:90 : values not in ascending order, previous
'page_19', current 'page_13'
2020-08-09 23:59:25,533 DEBUG [UserAnalysisCube_channel]
dict.DictionaryGenerator:91 : Dictionary cardinality: 50
2020-08-09 23:59:25,533 DEBUG [UserAnalysisCube_channel]
dict.DictionaryGenerator:92 : Dictionary builder class:
org.apache.kylin.dict.DictionaryGenerator$StringTrieDictForestBuilder
2020-08-09 23:59:25,533 DEBUG [UserAnalysisCube_channel]
dict.DictionaryGenerator:93 : Dictionary class:
org.apache.kylin.dict.TrieDictionaryForest
2020-08-09 23:59:25,533 DEBUG [UserAnalysisCube_channel]
dict.DictionaryGenerator:102 : Dictionary value samples: page_19=>11,
page_13=>5, page_14=>6, page_11=>3, page_12=>4
2020-08-09 23:59:25,533 DEBUG [UserAnalysisCube_channel]
columnar.ColumnarMemoryStorePersister:403 : BuildDictionary for column :
PAGE_ID took : 13 ms
2020-08-09 23:59:25,542 INFO [UserAnalysisCube_channel]
dict.TrieDictionaryForestBuilder:90 : values not in ascending order, previous
'stop', current 'pause'
2020-08-09 23:59:25,543 DEBUG [UserAnalysisCube_channel]
dict.DictionaryGenerator:91 : Dictionary cardinality: 10
2020-08-09 23:59:25,543 DEBUG [UserAnalysisCube_channel]
dict.DictionaryGenerator:92 : Dictionary builder class:
org.apache.kylin.dict.DictionaryGenerator$StringTrieDictForestBuilder
2020-08-09 23:59:25,543 DEBUG [UserAnalysisCube_channel]
dict.DictionaryGenerator:93 : Dictionary class:
org.apache.kylin.dict.TrieDictionaryForest
2020-08-09 23:59:25,543 DEBUG [UserAnalysisCube_channel]
dict.DictionaryGenerator:102 : Dictionary value samples: stop=>9, pause=>5,
play=>6, like=>4, start=>8
2020-08-09 23:59:25,543 DEBUG [UserAnalysisCube_channel]
columnar.ColumnarMemoryStorePersister:403 : BuildDictionary for column :
ACT_TYPE took : 9 ms
2020-08-09 23:59:25,553 DEBUG [UserAnalysisCube_channel]
dict.DictionaryGenerator:91 : Dictionary cardinality: 41
2020-08-09 23:59:25,553 DEBUG [UserAnalysisCube_channel]
dict.DictionaryGenerator:92 : Dictionary builder class:
org.apache.kylin.dict.DictionaryGenerator$NumberTrieDictForestBuilder
2020-08-09 23:59:25,553 DEBUG [UserAnalysisCube_channel]
dict.DictionaryGenerator:93 : Dictionary class:
org.apache.kylin.dict.TrieDictionaryForest
2020-08-09 23:59:25,553 DEBUG [UserAnalysisCube_channel]
dict.DictionaryGenerator:102 : Dictionary value samples: 19=>0, 20=>1, 21=>2,
22=>3, 23=>4
2020-08-09 23:59:25,553 DEBUG [UserAnalysisCube_channel]
columnar.ColumnarMemoryStorePersister:403 : BuildDictionary for column :
DEVIDE_TYPE took : 10 ms
2020-08-09 23:59:25,562 INFO [UserAnalysisCube_channel]
dict.TrieDictionaryForestBuilder:90 : values not in ascending order, previous
'hongkong', current 'hangzhou'
2020-08-09 23:59:25,563 DEBUG [UserAnalysisCube_channel]
dict.DictionaryGenerator:91 : Dictionary cardinality: 7
2020-08-09 23:59:25,563 DEBUG [UserAnalysisCube_channel]
dict.DictionaryGenerator:92 : Dictionary builder class:
org.apache.kylin.dict.DictionaryGenerator$StringTrieDictForestBuilder
2020-08-09 23:59:25,563 DEBUG [UserAnalysisCube_channel]
dict.DictionaryGenerator:93 : Dictionary class:
org.apache.kylin.dict.TrieDictionaryForest
2020-08-09 23:59:25,563 DEBUG [UserAnalysisCube_channel]
dict.DictionaryGenerator:102 : Dictionary value samples: hongkong=>3,
hangzhou=>2, shenzhen=>6, berlin=>1, nanjing=>4
2020-08-09 23:59:25,563 DEBUG [UserAnalysisCube_channel]
columnar.ColumnarMemoryStorePersister:403 : BuildDictionary for column :
LOCATION_CITY took : 9 ms
2020-08-09 23:59:25,572 DEBUG [UserAnalysisCube_channel]
dict.DictionaryGenerator:91 : Dictionary cardinality: 4
2020-08-09 23:59:25,573 DEBUG [UserAnalysisCube_channel]
dict.DictionaryGenerator:92 : Dictionary builder class:
org.apache.kylin.dict.DictionaryGenerator$NumberTrieDictForestBuilder
2020-08-09 23:59:25,573 DEBUG [UserAnalysisCube_channel]
dict.DictionaryGenerator:93 : Dictionary class:
org.apache.kylin.dict.TrieDictionaryForest
2020-08-09 23:59:25,573 DEBUG [UserAnalysisCube_channel]
dict.DictionaryGenerator:102 : Dictionary value samples: 2=>0, 3=>1, 4=>2, 5=>3
2020-08-09 23:59:25,573 DEBUG [UserAnalysisCube_channel]
columnar.ColumnarMemoryStorePersister:403 : BuildDictionary for column :
NETWORK_TYPE took : 9 ms
2020-08-09 23:59:25,582 INFO [UserAnalysisCube_channel]
dict.TrieDictionaryForestBuilder:90 : values not in ascending order, previous
'xiaomi', current 'vivo'
2020-08-09 23:59:25,583 DEBUG [UserAnalysisCube_channel]
dict.DictionaryGenerator:91 : Dictionary cardinality: 5
2020-08-09 23:59:25,583 DEBUG [UserAnalysisCube_channel]
dict.DictionaryGenerator:92 : Dictionary builder class:
org.apache.kylin.dict.DictionaryGenerator$StringTrieDictForestBuilder
2020-08-09 23:59:25,583 DEBUG [UserAnalysisCube_channel]
dict.DictionaryGenerator:93 : Dictionary class:
org.apache.kylin.dict.TrieDictionaryForest
2020-08-09 23:59:25,583 DEBUG [UserAnalysisCube_channel]
dict.DictionaryGenerator:102 : Dictionary value samples: huawei=>0, xiaomi=>4,
vivo=>3, meizu=>2, iPhone=>1
2020-08-09 23:59:25,583 DEBUG [UserAnalysisCube_channel]
columnar.ColumnarMemoryStorePersister:403 : BuildDictionary for column :
DEVICE_BRAND took : 10 ms
2020-08-09 23:59:25,594 INFO [UserAnalysisCube_channel]
dict.TrieDictionaryForestBuilder:90 : values not in ascending order, previous
'23_53_19', current '23_53_10'
2020-08-09 23:59:25,596 DEBUG [UserAnalysisCube_channel]
dict.DictionaryGenerator:91 : Dictionary cardinality: 2412
2020-08-09 23:59:25,596 DEBUG [UserAnalysisCube_channel]
dict.DictionaryGenerator:92 : Dictionary builder class:
org.apache.kylin.dict.DictionaryGenerator$StringTrieDictForestBuilder
2020-08-09 23:59:25,596 DEBUG [UserAnalysisCube_channel]
dict.DictionaryGenerator:93 : Dictionary class:
org.apache.kylin.dict.TrieDictionaryForest
2020-08-09 23:59:25,596 DEBUG [UserAnalysisCube_channel]
dict.DictionaryGenerator:102 : Dictionary value samples: 23_53_16=>2035,
23_53_17=>2036, 23_53_18=>2037, 23_53_19=>2038, 23_53_10=>2029
2020-08-09 23:59:25,596 DEBUG [UserAnalysisCube_channel]
columnar.ColumnarMemoryStorePersister:403 : BuildDictionary for column :
STR_MINUTE_SECOND took : 12 ms
2020-08-09 23:59:25,702 DEBUG [UserAnalysisCube_channel]
columnar.ColumnarMemoryStorePersister:326 : cuboid-8191 saved dimension:UID,
took: 105ms
2020-08-09 23:59:25,757 DEBUG [UserAnalysisCube_channel]
columnar.ColumnarMemoryStorePersister:326 : cuboid-8191 saved
dimension:PAGE_ID, took: 54ms
2020-08-09 23:59:25,812 DEBUG [UserAnalysisCube_channel]
columnar.ColumnarMemoryStorePersister:326 : cuboid-8191 saved
dimension:ACT_TYPE, took: 54ms
2020-08-09 23:59:25,866 DEBUG [UserAnalysisCube_channel]
columnar.ColumnarMemoryStorePersister:326 : cuboid-8191 saved
dimension:DEVIDE_TYPE, took: 54ms
2020-08-09 23:59:25,932 DEBUG [UserAnalysisCube_channel]
columnar.ColumnarMemoryStorePersister:326 : cuboid-8191 saved
dimension:LOCATION_CITY, took: 65ms
2020-08-09 23:59:25,996 DEBUG [UserAnalysisCube_channel]
columnar.ColumnarMemoryStorePersister:326 : cuboid-8191 saved
dimension:NETWORK_TYPE, took: 63ms
2020-08-09 23:59:26,065 DEBUG [UserAnalysisCube_channel]
columnar.ColumnarMemoryStorePersister:326 : cuboid-8191 saved
dimension:DEVICE_BRAND, took: 68ms
2020-08-09 23:59:26,138 DEBUG [UserAnalysisCube_channel]
columnar.ColumnarMemoryStorePersister:326 : cuboid-8191 saved
dimension:STR_MINUTE_SECOND, took: 73ms
2020-08-09 23:59:26,157 DEBUG [UserAnalysisCube_channel]
columnar.ColumnarMemoryStorePersister:326 : cuboid-8191 saved
dimension:MONTH_START, took: 18ms
2020-08-09 23:59:26,175 DEBUG [UserAnalysisCube_channel]
columnar.ColumnarMemoryStorePersister:326 : cuboid-8191 saved
dimension:WEEK_START, took: 18ms
2020-08-09 23:59:26,193 DEBUG [UserAnalysisCube_channel]
columnar.ColumnarMemoryStorePersister:326 : cuboid-8191 saved
dimension:DAY_START, took: 17ms
2020-08-09 23:59:26,210 DEBUG [UserAnalysisCube_channel]
columnar.ColumnarMemoryStorePersister:326 : cuboid-8191 saved
dimension:HOUR_START, took: 17ms
2020-08-09 23:59:26,235 DEBUG [UserAnalysisCube_channel]
columnar.ColumnarMemoryStorePersister:326 : cuboid-8191 saved
dimension:MINUTE_START, took: 24ms
2020-08-09 23:59:26,256 DEBUG [UserAnalysisCube_channel]
columnar.ColumnarMemoryStorePersister:379 : cuboid-8191 saved measure:_COUNT_,
took: 20ms
2020-08-09 23:59:26,316 DEBUG [UserAnalysisCube_channel]
columnar.ColumnarMemoryStorePersister:379 : cuboid-8191 saved
measure:SUM_PLAY_TIMES, took: 60ms
2020-08-09 23:59:26,336 DEBUG [UserAnalysisCube_channel]
columnar.ColumnarMemoryStorePersister:379 : cuboid-8191 saved
measure:SUM_ACTIVE_MINUTES, took: 19ms
2020-08-09 23:59:26,355 DEBUG [UserAnalysisCube_channel]
columnar.ColumnarMemoryStorePersister:379 : cuboid-8191 saved
measure:SUM_PLAY_DURATION, took: 18ms
2020-08-09 23:59:26,412 DEBUG [UserAnalysisCube_channel]
columnar.ColumnarMemoryStorePersister:379 : cuboid-8191 saved
measure:MAX_PLAY_TIMES, took: 57ms
2020-08-09 23:59:26,431 DEBUG [UserAnalysisCube_channel]
columnar.ColumnarMemoryStorePersister:379 : cuboid-8191 saved
measure:MAX_ACTIVE_MINUTES, took: 18ms
2020-08-09 23:59:26,450 DEBUG [UserAnalysisCube_channel]
columnar.ColumnarMemoryStorePersister:379 : cuboid-8191 saved
measure:MAX_PLAY_DURATION, took: 18ms
2020-08-09 23:59:26,470 DEBUG [UserAnalysisCube_channel]
columnar.ColumnarMemoryStorePersister:379 : cuboid-8191 saved measure:PV_STAT,
took: 20ms
2020-08-09 23:59:26,471 INFO [UserAnalysisCube_channel]
columnar.ColumnarMemoryStorePersister:110 : Finish persist memory store for
cube:UserAnalysisCube segment:20200809230000_20200810000000, take: 1003ms
```
The key part is the folowing:
```
2020-08-09 23:59:11,792 INFO [main] kafka.KafkaSource:192 :
localConsumeStats from local checkpoint null, remoteConsumeStats from remote
checkpoint
{"0":363171,"1":363198,"2":363249,"3":363171,"4":363199,"5":363250,"6":363170,"7":363196,"8":363250,"9":363170}
2020-08-09 23:59:11,793 INFO [main] kafka.KafkaSource:239 : start kafka
connector from specified position:KafkaPosition{partitionOffsetMap={0=363171,
1=363198, 2=363249, 3=363171, 4=363199, 5=363250, 6=363170, 7=363196, 8=363250,
9=363170}}
2020-08-09 23:59:11,798 WARN [main] consumer.KafkaConnector:91 : Assume
useraction_xxyu-10 is a new added partition, let's seek it to 0
2020-08-09 23:59:11,799 WARN [main] consumer.KafkaConnector:91 : Assume
useraction_xxyu-15 is a new added partition, let's seek it to 0
2020-08-09 23:59:11,799 WARN [main] consumer.KafkaConnector:91 : Assume
useraction_xxyu-13 is a new added partition, let's seek it to 0
2020-08-09 23:59:11,799 WARN [main] consumer.KafkaConnector:91 : Assume
useraction_xxyu-12 is a new added partition, let's seek it to 0
2020-08-09 23:59:11,799 WARN [main] consumer.KafkaConnector:91 : Assume
useraction_xxyu-14 is a new added partition, let's seek it to 0
2020-08-09 23:59:11,799 WARN [main] consumer.KafkaConnector:91 : Assume
useraction_xxyu-11 is a new added partition, let's seek it to 0
```
It indicated taht receiver consume from position, which shoulf be the
correct way:
```json
{
"0":363171,
"1":363198,
"2":363249,
"3":363171,
"4":363199,
"5":363250,
"6":363170,
"7":363196,
"8":363250,
"9":363170,
"10":0,
"11":0,
"12":0,
"13":0,
"14":0,
"15":0
}
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
![https://user-images.githubusercontent.com/14030549/89735825-bd5b1000-da97-11ea-9944-ae54331002aa.png"](https://user-images.githubusercontent.com/14030549/89735825-bd5b1000-da97-11ea-9944-ae54331002aa.png"){kind=link}