apoorvmittal10 opened a new issue #6013:
URL: https://github.com/apache/incubator-pinot/issues/6013
Detection job gets triggered multiple times at same time because multiple
detection tasks get scheduled. The problem is with detection task creation
which doesn't factor in concurrent insertions. Scheduler can call same
detection job creation concurrently depending on free scheduler threads.
Though after taking lock on scheduler job (`@DisallowConcurrentExecution`)
the problem will persist when multiple scheduler nodes exists in cluster.
`DetectionPipelineJob` `taskDao` insertion need to have some distributed lock
semantics.
```
INFO [2020-09-13 18:18:27,548] org.quartz.core.QuartzScheduler: Scheduler
meta-data: Quartz Scheduler (v2.2.1) 'DefaultQuartzScheduler' with instanceId
'NON_CLUSTERED'
Scheduler class: 'org.quartz.core.QuartzScheduler' - running locally.
NOT STARTED.
Currently in standby mode.
Number of jobs executed: 0
Using thread pool 'org.quartz.simpl.SimpleThreadPool' - with 10 threads.
Using job-store 'org.quartz.simpl.RAMJobStore' - which does not support
persistence. and is not clustered.
.
.
.
.
INFO [2020-09-13 17:30:07,425]
org.apache.pinot.thirdeye.detection.alert.DetectionAlertJob: Created
DETECTION_ALERT task 299 with settings
TaskDTO[taskType=DETECTION_ALERT,workerId=<null>,jobId=<null>,jobName=DETECTION_ALERT_24,status=WAITING,startTime=0,endTime=0,taskInfo={"detectionAlertConfigId":24},message=<null>,lastModified=<null>,id=299,version=0,updateTime=<null>,createdBy=<null>,updatedBy=<null>]
INFO [2020-09-13 17:30:07,928]
org.apache.pinot.thirdeye.detection.alert.DetectionAlertJob: Created
DETECTION_ALERT task 300 with settings
TaskDTO[taskType=DETECTION_ALERT,workerId=<null>,jobId=<null>,jobName=DETECTION_ALERT_24,status=WAITING,startTime=0,endTime=0,taskInfo={"detectionAlertConfigId":24},message=<null>,lastModified=<null>,id=300,version=0,updateTime=<null>,createdBy=<null>,updatedBy=<null>]
INFO [2020-09-13 17:30:08,647]
org.apache.pinot.thirdeye.detection.DetectionPipelineJob: Created DETECTION
task
TaskDTO[taskType=DETECTION,workerId=<null>,jobId=<null>,jobName=DETECTION_23,status=WAITING,startTime=0,endTime=0,taskInfo={"configId":23,"start":1600018092445,"end":1600018201552,"online":false,"detectionConfigBean":null},message=<null>,lastModified=<null>,id=301,version=0,updateTime=<null>,createdBy=<null>,updatedBy=<null>]
with taskId 301
INFO [2020-09-13 17:30:08,813]
org.apache.pinot.thirdeye.detection.DetectionPipelineJob: Created DETECTION
task
TaskDTO[taskType=DETECTION,workerId=<null>,jobId=<null>,jobName=DETECTION_23,status=WAITING,startTime=0,endTime=0,taskInfo={"configId":23,"start":1600018092445,"end":1600018201980,"online":false,"detectionConfigBean":null},message=<null>,lastModified=<null>,id=302,version=0,updateTime=<null>,createdBy=<null>,updatedBy=<null>]
with taskId 302
INFO [2020-09-13 17:30:08,969]
org.apache.pinot.thirdeye.detection.alert.DetectionAlertJob: Created
DETECTION_ALERT task 303 with settings
TaskDTO[taskType=DETECTION_ALERT,workerId=<null>,jobId=<null>,jobName=DETECTION_ALERT_24,status=WAITING,startTime=0,endTime=0,taskInfo={"detectionAlertConfigId":24},message=<null>,lastModified=<null>,id=303,version=0,updateTime=<null>,createdBy=<null>,updatedBy=<null>]
INFO [2020-09-13 17:30:10,384]
org.apache.pinot.thirdeye.detection.DetectionPipelineJob: Created DETECTION
task
TaskDTO[taskType=DETECTION,workerId=<null>,jobId=<null>,jobName=DETECTION_23,status=WAITING,startTime=0,endTime=0,taskInfo={"configId":23,"start":1600018092445,"end":1600018203484,"online":false,"detectionConfigBean":null},message=<null>,lastModified=<null>,id=304,version=0,updateTime=<null>,createdBy=<null>,updatedBy=<null>]
with taskId 304
INFO [2020-09-13 17:30:10,844]
org.apache.pinot.thirdeye.detection.DetectionPipelineJob: Skip scheduling
detection task for DETECTION_23 with start time 1600018092445. Task is already
in the queue.
INFO [2020-09-13 17:30:10,942]
org.apache.pinot.thirdeye.detection.alert.DetectionAlertJob: Created
DETECTION_ALERT task 305 with settings
TaskDTO[taskType=DETECTION_ALERT,workerId=<null>,jobId=<null>,jobName=DETECTION_ALERT_24,status=WAITING,startTime=0,endTime=0,taskInfo={"detectionAlertConfigId":24},message=<null>,lastModified=<null>,id=305,version=0,updateTime=<null>,createdBy=<null>,updatedBy=<null>]
INFO [2020-09-13 17:30:11,083]
org.apache.pinot.thirdeye.detection.alert.DetectionAlertJob: Created
DETECTION_ALERT task 307 with settings
TaskDTO[taskType=DETECTION_ALERT,workerId=<null>,jobId=<null>,jobName=DETECTION_ALERT_24,status=WAITING,startTime=0,endTime=0,taskInfo={"detectionAlertConfigId":24},message=<null>,lastModified=<null>,id=307,version=0,updateTime=<null>,createdBy=<null>,updatedBy=<null>]
INFO [2020-09-13 17:30:11,092]
org.apache.pinot.thirdeye.detection.DetectionPipelineJob: Created DETECTION
task
TaskDTO[taskType=DETECTION,workerId=<null>,jobId=<null>,jobName=DETECTION_23,status=WAITING,startTime=0,endTime=0,taskInfo={"configId":23,"start":1600018092445,"end":1600018204022,"online":false,"detectionConfigBean":null},message=<null>,lastModified=<null>,id=306,version=0,updateTime=<null>,createdBy=<null>,updatedBy=<null>]
with taskId 306
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
