[GitHub] [incubator-livy] huianyi commented on a change in pull request #238: [LIVY-689] Deliver stage process message to the end user using thriftserver
huianyi commented on a change in pull request #238: [LIVY-689] Deliver stage
process message to the end user using thriftserver
URL: https://github.com/apache/incubator-livy/pull/238#discussion_r341545096
##
File path:
thriftserver/server/src/main/scala/org/apache/livy/thriftserver/LivyThriftSessionManager.scala
##
@@ -252,6 +259,8 @@ class LivyThriftSessionManager(val server:
LivyThriftServer, val livyConf: LivyC
}
}
initSession(sessionHandle, livySession, initStatements)
+operationMessages.foreach(
+ _.offer(s"tracking URL:
${livySession.appInfo.sparkUiUrl.orNull}"))
Review comment:
This trackingURL normally will not be changed, so print it only once is
enough?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-livy] huianyi commented on a change in pull request #238: [LIVY-689] Deliver stage process message to the end user using thriftserver
huianyi commented on a change in pull request #238: [LIVY-689] Deliver stage
process message to the end user using thriftserver
URL: https://github.com/apache/incubator-livy/pull/238#discussion_r341545096
##
File path:
thriftserver/server/src/main/scala/org/apache/livy/thriftserver/LivyThriftSessionManager.scala
##
@@ -252,6 +259,8 @@ class LivyThriftSessionManager(val server:
LivyThriftServer, val livyConf: LivyC
}
}
initSession(sessionHandle, livySession, initStatements)
+operationMessages.foreach(
+ _.offer(s"tracking URL:
${livySession.appInfo.sparkUiUrl.orNull}"))
Review comment:
This trackingURL normally will not changed, so print it only once is enough?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-livy] huianyi commented on a change in pull request #238: [LIVY-689] Deliver stage process message to the end user using thriftserver
huianyi commented on a change in pull request #238: [LIVY-689] Deliver stage
process message to the end user using thriftserver
URL: https://github.com/apache/incubator-livy/pull/238#discussion_r340951898
##
File path: rsc/src/main/java/org/apache/livy/rsc/driver/RSCDriver.java
##
@@ -212,6 +212,41 @@ public void onSaslComplete(Rpc client) {
setupIdleTimeout();
}
+ /**
+ * Calculate each stage's process and broadcast message to the end user
+ *
+ * @param sqlJobStatementId
+ */
+ void handleProcessMessage(String sqlJobStatementId) {
+SparkStatusTracker sparkStatusTracker;
+synchronized (jcLock) {
+ if (jc == null) {
+return;
+ }
+ sparkStatusTracker = jc.sc().sc().statusTracker();
+}
+int[] jobIdsForGroup =
sparkStatusTracker.getJobIdsForGroup(sqlJobStatementId);
Review comment:
SqlJob will use statementId to set sparkJobGroup,
[SqlJob.java#L60](https://github.com/apache/incubator-livy/blob/c7313c5648c63089413464b47f33c9749dbabb20/thriftserver/session/src/main/java/org/apache/livy/thriftserver/session/SqlJob.java#L60)
Let Sql job extends abstract class SqlJobStatement, so we can get its
statementId.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-livy] huianyi commented on a change in pull request #238: [LIVY-689] Deliver stage process message to the end user using thriftserver
huianyi commented on a change in pull request #238: [LIVY-689] Deliver stage
process message to the end user using thriftserver
URL: https://github.com/apache/incubator-livy/pull/238#discussion_r337496485
##
File path: rsc/src/main/java/org/apache/livy/rsc/RSCClient.java
##
@@ -330,6 +332,14 @@ public long getReplLastActivity() {
private class ClientProtocol extends BaseProtocol {
+private final String PROCESS_FORMAT = "stage %3s: percentage: %3d%%
active: %s " +
+"completed: %s failed: %s all: %s";
+private ConcurrentBoundedLinkedQueue operationMessages;
Review comment:
Yes, this could be an issue, gonna change it.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-livy] huianyi commented on a change in pull request #238: [LIVY-689] Deliver stage process message to the end user using thriftserver
huianyi commented on a change in pull request #238: [LIVY-689] Deliver stage
process message to the end user using thriftserver
URL: https://github.com/apache/incubator-livy/pull/238#discussion_r337496046
##
File path: rsc/src/main/java/org/apache/livy/rsc/driver/JobWrapper.java
##
@@ -42,10 +45,17 @@
private Future future;
+ private final long firstDelayMSec = 500L;
+ private final long updatePeriodMSec;
+
+ private Timer timer = new Timer("refresh progress", true);
Review comment:
sure.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-livy] huianyi commented on a change in pull request #238: [LIVY-689] Deliver stage process message to the end user using thriftserver
huianyi commented on a change in pull request #238: [LIVY-689] Deliver stage
process message to the end user using thriftserver
URL: https://github.com/apache/incubator-livy/pull/238#discussion_r337495478
##
File path: rsc/src/main/java/org/apache/livy/rsc/driver/JobWrapper.java
##
@@ -60,6 +70,14 @@ public Void call() throws Exception {
}
}
+ timer.schedule(new TimerTask() {
Review comment:
Maybe Creating this time instance in `RSCDriver` is a good way? Notify this
timer when job submit, other times would be wait.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-livy] huianyi commented on a change in pull request #238: [LIVY-689] Deliver stage process message to the end user using thriftserver
huianyi commented on a change in pull request #238: [LIVY-689] Deliver stage
process message to the end user using thriftserver
URL: https://github.com/apache/incubator-livy/pull/238#discussion_r334220873
##
File path: rsc/src/main/java/org/apache/livy/rsc/driver/RSCDriver.java
##
@@ -212,6 +215,33 @@ public void onSaslComplete(Rpc client) {
setupIdleTimeout();
}
+ /**
+ * calculate each stage's process and broadcast message to the end user
+ * @param jobId
+ */
+ void handleProcessMessage(String jobId){
+SparkStatusTracker sparkStatusTracker;
+synchronized (jcLock){
+ if (jc == null){
+return;
+ }
+ sparkStatusTracker = jc.sc().sc().statusTracker();
+}
+int[] activeStageIds = sparkStatusTracker.getActiveStageIds();
Review comment:
: ), ok, I'm gonna fix other problems soon, very thanks for your review.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-livy] huianyi commented on a change in pull request #238: [LIVY-689] Deliver stage process message to the end user using thriftserver
huianyi commented on a change in pull request #238: [LIVY-689] Deliver stage
process message to the end user using thriftserver
URL: https://github.com/apache/incubator-livy/pull/238#discussion_r334217847
##
File path: rsc/src/main/java/org/apache/livy/rsc/driver/RSCDriver.java
##
@@ -212,6 +215,33 @@ public void onSaslComplete(Rpc client) {
setupIdleTimeout();
}
+ /**
+ * calculate each stage's process and broadcast message to the end user
+ * @param jobId
+ */
+ void handleProcessMessage(String jobId){
+SparkStatusTracker sparkStatusTracker;
+synchronized (jcLock){
+ if (jc == null){
+return;
+ }
+ sparkStatusTracker = jc.sc().sc().statusTracker();
+}
+int[] activeStageIds = sparkStatusTracker.getActiveStageIds();
Review comment:
> just remembered that Livy cannot submit one job a time
Do you mean one session should only run a livy job a time? if this is true,
current implementation should not be a problem?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-livy] huianyi commented on a change in pull request #238: [LIVY-689] Deliver stage process message to the end user using thriftserver
huianyi commented on a change in pull request #238: [LIVY-689] Deliver stage
process message to the end user using thriftserver
URL: https://github.com/apache/incubator-livy/pull/238#discussion_r334217660
##
File path: rsc/src/main/java/org/apache/livy/rsc/driver/RSCDriver.java
##
@@ -212,6 +215,33 @@ public void onSaslComplete(Rpc client) {
setupIdleTimeout();
}
+ /**
+ * calculate each stage's process and broadcast message to the end user
+ * @param jobId
+ */
+ void handleProcessMessage(String jobId){
+SparkStatusTracker sparkStatusTracker;
+synchronized (jcLock){
+ if (jc == null){
+return;
+ }
+ sparkStatusTracker = jc.sc().sc().statusTracker();
+}
+int[] activeStageIds = sparkStatusTracker.getActiveStageIds();
Review comment:
> NVM, just remembered that Livy cannot submit one job a time. But there's
another issue, some queries may split into multiple jobs, then the measurement
here may not be accurate.
If Livy cannot submit one job a time, stage id in one query is unique, I
already log stage process message with stage id, so this should not be a
problem?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-livy] huianyi commented on a change in pull request #238: [LIVY-689] Deliver stage process message to the end user using thriftserver
huianyi commented on a change in pull request #238: [LIVY-689] Deliver stage
process message to the end user using thriftserver
URL: https://github.com/apache/incubator-livy/pull/238#discussion_r334216550
##
File path: rsc/src/main/java/org/apache/livy/rsc/driver/RSCDriver.java
##
@@ -212,6 +215,33 @@ public void onSaslComplete(Rpc client) {
setupIdleTimeout();
}
+ /**
+ * calculate each stage's process and broadcast message to the end user
+ * @param jobId
+ */
+ void handleProcessMessage(String jobId){
+SparkStatusTracker sparkStatusTracker;
+synchronized (jcLock){
+ if (jc == null){
+return;
+ }
+ sparkStatusTracker = jc.sc().sc().statusTracker();
+}
+int[] activeStageIds = sparkStatusTracker.getActiveStageIds();
Review comment:
Yes, I find another way by using jobGroupId(which is livy jobId) may fix
this problem, gonna test it and pull a new commit if it would work.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-livy] huianyi commented on a change in pull request #238: [LIVY-689] Deliver stage process message to the end user using thriftserver
huianyi commented on a change in pull request #238: [LIVY-689] Deliver stage
process message to the end user using thriftserver
URL: https://github.com/apache/incubator-livy/pull/238#discussion_r334216550
##
File path: rsc/src/main/java/org/apache/livy/rsc/driver/RSCDriver.java
##
@@ -212,6 +215,33 @@ public void onSaslComplete(Rpc client) {
setupIdleTimeout();
}
+ /**
+ * calculate each stage's process and broadcast message to the end user
+ * @param jobId
+ */
+ void handleProcessMessage(String jobId){
+SparkStatusTracker sparkStatusTracker;
+synchronized (jcLock){
+ if (jc == null){
+return;
+ }
+ sparkStatusTracker = jc.sc().sc().statusTracker();
+}
+int[] activeStageIds = sparkStatusTracker.getActiveStageIds();
Review comment:
Yes, I find another way by using jobGroupId(which is livy jobId) may fix
this problem, gonna test it and pull a new commit if it would work.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-livy] huianyi commented on a change in pull request #238: [LIVY-689] Deliver stage process message to the end user using thriftserver
huianyi commented on a change in pull request #238: [LIVY-689] Deliver stage
process message to the end user using thriftserver
URL: https://github.com/apache/incubator-livy/pull/238#discussion_r334214800
##
File path:
thriftserver/server/src/main/scala/org/apache/livy/thriftserver/LivyThriftSessionManager.scala
##
@@ -252,6 +259,8 @@ class LivyThriftSessionManager(val server:
LivyThriftServer, val livyConf: LivyC
}
}
initSession(sessionHandle, livySession, initStatements)
+operationMessages.foreach(
+ _.offer(s"tracking URL:
${livySession.appInfo.sparkUiUrl.orNull}"))
Review comment:
I can get the application URL in my environment, which should be set after
`initSession` method, could you please show me how your produce this issue? May
`initSession` method not run right?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-livy] huianyi commented on a change in pull request #238: [LIVY-689] Deliver stage process message to the end user using thriftserver
huianyi commented on a change in pull request #238: [LIVY-689] Deliver stage
process message to the end user using thriftserver
URL: https://github.com/apache/incubator-livy/pull/238#discussion_r333841848
##
File path: rsc/src/main/java/org/apache/livy/rsc/BaseProtocol.java
##
@@ -126,6 +126,28 @@ public JobResult() {
}
+ protected static class JobProcessMessage {
+public final String id;
+public final int stageId;
+public final int completed;
+public final int all;
+public final int active;
+public final int failed;
+public JobProcessMessage(String id, int stageId, int completed, int active,
Review comment:
Looks like all java code writes all parameter one line, should I change this?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-livy] huianyi commented on a change in pull request #238: [LIVY-689] Deliver stage process message to the end user using thriftserver
huianyi commented on a change in pull request #238: [LIVY-689] Deliver stage
process message to the end user using thriftserver
URL: https://github.com/apache/incubator-livy/pull/238#discussion_r333412835
##
File path: core/src/main/scala/org/apache/livy/OperationMessageManager.scala
##
@@ -0,0 +1,37 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.livy
+
+object OperationMessageManager extends Logging{
+
+ def offer(message: Any): Unit = {
+if (threadLocalLogQueue.get() != null){
+ threadLocalLogQueue.get().offer(message.toString)
+}
+ }
+
+ def get(): ConcurrentBoundedLinkedQueue[String] = {
+threadLocalLogQueue.get()
+ }
+
+ def set(threadLocalLogQueue: ConcurrentBoundedLinkedQueue[String]){
+this.threadLocalLogQueue.set(threadLocalLogQueue)
+ }
+ private val threadLocalLogQueue = new
ThreadLocal[ConcurrentBoundedLinkedQueue[String]]
Review comment:
I think this gives us another more convenient way to get operationMessage
object in other classes if they are in the same thread(just like
`Sessionstate.get()` method in hive), for those classes are not in the same
thread, we need deliver this object to them, like Class
`LivyExecuteStatementOperation`.
Actually, if we do not use this threadLocal variable, for class
`ClientProtocol` needing operationMessage to deliver process message, we should
first deliver operationMessage to RSCClient, then RSCClient deliver it to
ClientProtocol, which needs lots of set and get method.
Would it be aIright if I add detailed comment in `OperationMessageManager`
to call developers' attention to the constraint?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-livy] huianyi commented on a change in pull request #238: [LIVY-689] Deliver stage process message to the end user using thriftserver
huianyi commented on a change in pull request #238: [LIVY-689] Deliver stage
process message to the end user using thriftserver
URL: https://github.com/apache/incubator-livy/pull/238#discussion_r68731
##
File path: core/src/main/scala/org/apache/livy/OperationMessageManager.scala
##
@@ -0,0 +1,37 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.livy
+
+object OperationMessageManager extends Logging{
+
+ def offer(message: Any): Unit = {
+if (threadLocalLogQueue.get() != null){
+ threadLocalLogQueue.get().offer(message.toString)
+}
+ }
+
+ def get(): ConcurrentBoundedLinkedQueue[String] = {
+threadLocalLogQueue.get()
+ }
+
+ def set(threadLocalLogQueue: ConcurrentBoundedLinkedQueue[String]){
+this.threadLocalLogQueue.set(threadLocalLogQueue)
+ }
+ private val threadLocalLogQueue = new
ThreadLocal[ConcurrentBoundedLinkedQueue[String]]
Review comment:
Using this threadLocal variable gives us an easy way to get operationMessage
object in other classes, we do not need transfer it in other classes. For
example, Class ClientProtocol need operationMessage to deliver process message
to the end user, we can just call `OperationMessageManager.get()` to get it.
```java
private final ConcurrentBoundedLinkedQueue operationMessages =
OperationMessageManager.get();
```
I don't think this might be reused for another rpc client,
ConcurrentBoundedLinkedQueue would be initialized in
`LivyThriftSessionManager.openSession` method, for every new session, we will
create only one RSCClient for it, and both of them are in the same thread, so
for every RSCClient, they will have the right ConcurrentBoundedLinkedQueue.
Please correct me if I miss something.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-livy] huianyi commented on a change in pull request #238: [LIVY-689] Deliver stage process message to the end user using thriftserver
huianyi commented on a change in pull request #238: [LIVY-689] Deliver stage
process message to the end user using thriftserver
URL: https://github.com/apache/incubator-livy/pull/238#discussion_r68731
##
File path: core/src/main/scala/org/apache/livy/OperationMessageManager.scala
##
@@ -0,0 +1,37 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.livy
+
+object OperationMessageManager extends Logging{
+
+ def offer(message: Any): Unit = {
+if (threadLocalLogQueue.get() != null){
+ threadLocalLogQueue.get().offer(message.toString)
+}
+ }
+
+ def get(): ConcurrentBoundedLinkedQueue[String] = {
+threadLocalLogQueue.get()
+ }
+
+ def set(threadLocalLogQueue: ConcurrentBoundedLinkedQueue[String]){
+this.threadLocalLogQueue.set(threadLocalLogQueue)
+ }
+ private val threadLocalLogQueue = new
ThreadLocal[ConcurrentBoundedLinkedQueue[String]]
Review comment:
Using this threadLocal variable gives us an easy way to get operationMessage
object in other classes, we do not need transfer it in other classes. For
example, Class ClientProtocol need operationMessage to deliver process message
to the end user, we can just call `OperationMessageManager.get()` to get it.
```java
private final ConcurrentBoundedLinkedQueue operationMessages =
OperationMessageManager.get();
```
I don't think this might be reused of another rpc client,
ConcurrentBoundedLinkedQueue would be initialized in
`LivyThriftSessionManager.openSession` method, for every new session, we will
create only one RSCClient for it, and both of them are in the same thread, so
for every RSCClient, they will have the right ConcurrentBoundedLinkedQueue.
Please correct me if I miss something.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-livy] huianyi commented on a change in pull request #238: [LIVY-689] Deliver stage process message to the end user using thriftserver
huianyi commented on a change in pull request #238: [LIVY-689] Deliver stage
process message to the end user using thriftserver
URL: https://github.com/apache/incubator-livy/pull/238#discussion_r329540012
##
File path: rsc/src/main/java/org/apache/livy/rsc/RSCClient.java
##
@@ -413,6 +426,28 @@ private void handle(ChannelHandlerContext ctx, JobResult
msg) {
}
}
+private void handle(ChannelHandlerContext ctx, JobProcessMessage msg){
Review comment:
gonna change this format soon.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-livy] huianyi commented on a change in pull request #238: [LIVY-689] Deliver stage process message to the end user using thriftserver
huianyi commented on a change in pull request #238: [LIVY-689] Deliver stage
process message to the end user using thriftserver
URL: https://github.com/apache/incubator-livy/pull/238#discussion_r328917853
##
File path: rsc/src/main/java/org/apache/livy/rsc/driver/RSCDriver.java
##
@@ -212,6 +215,33 @@ public void onSaslComplete(Rpc client) {
setupIdleTimeout();
}
+ /**
+ * calculate each stage's process and broadcast message to the end user
+ * @param jobId
+ */
+ void handleProcessMessage(String jobId){
+SparkStatusTracker sparkStatusTracker;
+synchronized (jcLock){
+ if (jc == null){
+return;
+ }
+ sparkStatusTracker = jc.sc().sc().statusTracker();
+}
+int[] activeStageIds = sparkStatusTracker.getActiveStageIds();
Review comment:
We can only submit one job using beeline at a time, so this may not happen?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
