Hélder Hugo Ferreira created KYLIN-4866:
-------------------------------------------
Summary: Streaming from Kafka with nested timestamp column
Key: KYLIN-4866
URL: https://issues.apache.org/jira/browse/KYLIN-4866
Project: Kylin
Issue Type: Bug
Components: Real-time Streaming
Affects Versions: v3.1.0
Environment: Docker setup
Reporter: Hélder Hugo Ferreira
Attachments: image-2021-01-08-16-16-51-615.png
We have an IOT Data Platform which saves in Kafka Json messages with the
following format:
{code:json}
{
"SysProperties": {
"EventId": "1610120564321723276",
"EnqueueTime": "2021-01-08T15:42:44.045Z",
"UserName": "dummyuser"
},
"AppProperties": {
"EventDefinition": "EquipmentData",
"EventTime": "2021-01-08T15:42:43.26+00:00",
"ApplicationName": "ConnectIoTSimulator",
"ApplicationContext": null
},
"Data": {
"Equipment": "DUMMY-01",
"Material": "DUMMYZA9983884",
"MeasurementData": {
"Temp": 57,
"Pressure": 48.379,
"Energy": 47,
"Humidity": 31.727
}
}
{code}
>From these messages, we are trying to build a cube in which the
>"AppProperties.EventTime" shall be the timestamp column (using format
>yyyy-MM-dd'T'HH:mm:ss.SSSZZ), "Data.MeasurementData.Temp",
>"Data.MeasurementData.Pressure", "Data.MeasurementData.Energy" and
>"Data.MeasurementData.Humidity" shall be the metrics and all other fields
>dimensions.
We can setup everything fine in the kylin portal like in the tutorial, however
on runtime we have the following exceptions logged into the
kylin_streaming_receiver.log:
!image-2021-01-08-16-16-51-615.png!
After some investigation, we figured out it only works if the timestamp field
within the Json messages is not nested inside inner objects, like in the
following example:
{code:json}
{
"timestamp": 1610038228492,
"nestedData": {
"Equipment": "EPFTES-01",
"Material": "ZA9983884",
"Temp": 10,
"Pressure": 20.459,
"Energy": 30,
"Humidity": 80.967
}
}{code}
Therefore, we believe there is a bug handling the timestamp column whenever it
is nested within Json messages.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
