anshu-makkar commented on issue #18390:
URL: https://github.com/apache/druid/issues/18390#issuecomment-3218843357
I see another issue in MSQ for Tuple Sketch vs Index_Parallel Job.
For the same data set, via index_parallel it takes 2 hrs to ingest the data(
data set is hourly and has 24 hours of data) , with MSQ it goes on for ever
like more than 5 hours, but it still keeps on going on.
On comparing the Ingestion Spec, difference is how the rollups( metrics vs
aggregations) is defined
IN MSQ below is how the spec has metrics
```
"aggregations": [
{
"type": "arrayOfDoublesSketch",
"name": "a0",
"fieldName": "user_ifa",
"nominalEntries": 16384,
"metricColumns": [
"req_count"
],
"numberOfValues": 1
},
{
"type": "arrayOfDoublesSketch",
"name": "a1",
"fieldName": "user_ifa",
"nominalEntries": 16384,
"metricColumns": [
"win_rate_prob"
],
"numberOfValues": 1
},
{
"type": "longSum",
"name": "a2",
"fieldName": "req_count"
},
{
"type": "doubleSum",
"name": "a3",
"fieldName": "win_rate_prob"
}
]
```
In case of `index_parallel` job
```
"metricsSpec": [
{
"type": "arrayOfDoublesSketch",
"name": "user_tuple_sketch_count",
"fieldName": "user_ifa",
"nominalEntries": 16384,
"metricColumns": [
"req_count"
],
"numberOfValues": 1
},
{
"type": "arrayOfDoublesSketch",
"name": "user_tuple_sketch_probability",
"fieldName": "user_ifa",
"nominalEntries": 16384,
"metricColumns": [
"win_rate_prob"
],
"numberOfValues": 1
},
{
"type": "longSum",
"name": "total_req_count",
"fieldName": "psid_count"
},
{
"type": "doubleSum",
"name": "sum_win_probability",
"fieldName": "win_rate_prob"
}
],
```
There is a stark difference between the time it is taken via MSQ.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
