Reamer commented on code in PR #4582:
URL: https://github.com/apache/zeppelin/pull/4582#discussion_r1176399171
##########
zeppelin-zengine/src/main/java/org/apache/zeppelin/interpreter/remote/RemoteInterpreter.java:
##########
@@ -342,17 +341,14 @@ public Scheduler getScheduler() {
// running under the scheduler of this session will be aborted.
String executionMode = getProperty(".execution.mode", "paragraph");
if (executionMode.equals("paragraph")) {
- Scheduler s = new RemoteScheduler(
- RemoteInterpreter.class.getSimpleName() + "-" +
getInterpreterGroup().getId() + "-" + sessionId,
- SchedulerFactory.singleton().getExecutor(),
Review Comment:
There are several threads involved.
1) main - JUnit thread
2) RemoteScheduler thread - Receives jobs and submits the jobs to its own
executor.
https://github.com/apache/zeppelin/blob/ce1c3e57a4432c520ceaa29ffdde617f6eb654a4/zeppelin-interpreter/src/main/java/org/apache/zeppelin/scheduler/AbstractScheduler.java#L84-L97
Previously, the same executor pool from the SchedulerFactory was used
https://github.com/apache/zeppelin/blob/ce1c3e57a4432c520ceaa29ffdde617f6eb654a4/zeppelin-interpreter/src/main/java/org/apache/zeppelin/scheduler/SchedulerFactory.java#L52-L58
https://github.com/apache/zeppelin/blob/ce1c3e57a4432c520ceaa29ffdde617f6eb654a4/zeppelin-interpreter/src/main/java/org/apache/zeppelin/scheduler/SchedulerFactory.java#L118-L120
3) RemoteScheduler Executor - Starts the JobStatusPoller thread and submits
the job to the remote interpreter, now waits until the
evaluation/interpretation remotely is done.
https://github.com/apache/zeppelin/blob/ce1c3e57a4432c520ceaa29ffdde617f6eb654a4/zeppelin-zengine/src/main/java/org/apache/zeppelin/scheduler/RemoteScheduler.java#L179-L191
4) JobStatusPoller - Queries the remote interpreter at regular intervals
about the status of the remote job.
5) main from Remote-Interpreter - starts the Remote-Interpreter and also a
Thrift-Thread Pool, which received jobs from zeppelin server.
Because the "RemoteScheduler Executor" was a pool of threads, several
requests were made to the remote interpreter at the same time.
The task execution of jobs was done instantaneously.
https://github.com/apache/zeppelin/blob/ce1c3e57a4432c520ceaa29ffdde617f6eb654a4/zeppelin-zengine/src/test/java/org/apache/zeppelin/scheduler/RemoteSchedulerTest.java#L241-L242
The cancellation of jobs that are pending only works if the job has not yet
been forwarded to the Remote-Interpreter and is in the queue of the
RemoteScheduler. If the RemoteScheduler Executor now wants to process a job
that has already been cancelled, then it is skipped and not forwarded to the
Remote-Interpreter at all.
https://github.com/apache/zeppelin/blob/ce1c3e57a4432c520ceaa29ffdde617f6eb654a4/zeppelin-interpreter/src/main/java/org/apache/zeppelin/scheduler/AbstractScheduler.java#L119-L125
The thread behaviour is quite complex. I hope you can understand the
explanation, if not then just ask.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
