[GitHub] [incubator-livy] huianyi edited a comment on issue #238: [LIVY-689] Deliver stage process message to the end user using thriftserver
huianyi edited a comment on issue #238: [LIVY-689] Deliver stage process message to the end user using thriftserver URL: https://github.com/apache/incubator-livy/pull/238#issuecomment-560985424 @mgaido91 The previous link you pointed actually is another feature we can implement(process bar), you can check it here, [HiveStatement.java#L381](https://github.com/apache/hive/blob/209096a71b25a3d75dca1930f825c8c10b99d2b9/jdbc/src/java/org/apache/hive/jdbc/HiveStatement.java#L381) , [BeelineInPlaceUpdateStream.java#L39](https://github.com/apache/hive/blob/209096a71b25a3d75dca1930f825c8c10b99d2b9/beeline/src/java/org/apache/hive/beeline/logs/BeelineInPlaceUpdateStream.java#L39) Like I previous said, I can support this in a new PR. > > > Beeline from Hive 2.2 has in-place progress bar hive.server2.in.place.progress > > > https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-hive.server2.in.place.progress > > > (I think it works only with tez - would be nice if this would work with Livy too ) > > > > > > Would be better to support this if it is not a big effort. > > After reading hive related code, this needs write some new classes to implement this feature, which might cause this PR large, if you don not mind, I'd love to open a new PR to support this. ProcessBar is great in real-time interaction environment, but it can not work well if we do scheduling jobs, which we need print the job's status in seperate lines. > In this way, we need to store nothing on Livy server memory. By default setting, we can keep each session keep not more than 100 rows, actually under normal circumstances, client will get these message immediately, this might not put much memory pressure on Livy server side. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [incubator-livy] huianyi edited a comment on issue #238: [LIVY-689] Deliver stage process message to the end user using thriftserver
huianyi edited a comment on issue #238: [LIVY-689] Deliver stage process message to the end user using thriftserver URL: https://github.com/apache/incubator-livy/pull/238#issuecomment-560985424 @mgaido91 The previous link you pointed actually is another feature we can implement(process bar), you can check it here, [HiveStatement.java#L381](https://github.com/apache/hive/blob/209096a71b25a3d75dca1930f825c8c10b99d2b9/jdbc/src/java/org/apache/hive/jdbc/HiveStatement.java#L381) , [BeelineInPlaceUpdateStream.java#L39](https://github.com/apache/hive/blob/209096a71b25a3d75dca1930f825c8c10b99d2b9/beeline/src/java/org/apache/hive/beeline/logs/BeelineInPlaceUpdateStream.java#L39) Like I previous said, I can support this in a new PR. > > > Beeline from Hive 2.2 has in-place progress bar hive.server2.in.place.progress > > > https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-hive.server2.in.place.progress > > > (I think it works only with tez - would be nice if this would work with Livy too ) > > > > > > Would be better to support this if it is not a big effort. > > After reading hive related code, this needs write some new classes to implement this feature, which might cause this PR large, if you don not mind, I'd love to open a new PR to support this. ProcessBar is great in real-time interaction environment, but it can not work well if we do scheduling jobs, which we need print the job's status by individual lines. > In this way, we need to store nothing on Livy server memory. By default setting, we can keep each session keep not more than 100 rows, actually under normal circumstances, client will get these message immediately, this might not put much memory pressure on Livy server side. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [incubator-livy] huianyi edited a comment on issue #238: [LIVY-689] Deliver stage process message to the end user using thriftserver
huianyi edited a comment on issue #238: [LIVY-689] Deliver stage process message to the end user using thriftserver URL: https://github.com/apache/incubator-livy/pull/238#issuecomment-560985424 @mgaido91 The previous link you pointed actually is another feature we can implement(process bar), you can check it here, [HiveStatement.java#L381](https://github.com/apache/hive/blob/209096a71b25a3d75dca1930f825c8c10b99d2b9/jdbc/src/java/org/apache/hive/jdbc/HiveStatement.java#L381) , [BeelineInPlaceUpdateStream.java#L39](https://github.com/apache/hive/blob/209096a71b25a3d75dca1930f825c8c10b99d2b9/beeline/src/java/org/apache/hive/beeline/logs/BeelineInPlaceUpdateStream.java#L39) Like I previous said, I can support this in a new PR. > > > Beeline from Hive 2.2 has in-place progress bar hive.server2.in.place.progress > > > https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-hive.server2.in.place.progress > > > (I think it works only with tez - would be nice if this would work with Livy too ) > > > > > > Would be better to support this if it is not a big effort. > > After reading hive related code, this needs write some new classes to implement this feature, which might cause this PR large, if you don not mind, I'd love to open a new PR to support this. ProcessBar is great in real-time interaction environment, but it can not work well if we do off-line scheduling jobs, which we need print the job's status by individual lines. > In this way, we need to store nothing on Livy server memory. By default setting, we can keep each session keep not more than 100 rows, actually under normal circumstances, client will get these message immediately, this might not put much memory pressure on Livy server side. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [incubator-livy] huianyi edited a comment on issue #238: [LIVY-689] Deliver stage process message to the end user using thriftserver
huianyi edited a comment on issue #238: [LIVY-689] Deliver stage process message to the end user using thriftserver URL: https://github.com/apache/incubator-livy/pull/238#issuecomment-545715699 > > Rethink about the way this ticket doing, I think current way may not be the best way. > > > > 1. `jobId` may not exactly match the currently running jobs, there's no issue for beeline, but could be an issue for multi-thread usage. Currently when getting active jobs, there's no direct relation to jobId. > > 2. There's already a way to calculate job progress in ReplDriver for statement, we should extend the current way to also apply to job API. > > 3. Using LivyServer to store the session/job status may not be fault tolerant. It would better to extend the job API to get job progress synchronously. > > Thanks for your reply, if I have understood your meaning correctly, actually I can use `jobId`(which is spark jobs' group id delivered by livy) in `handleProcessMessage` method of RSCDriver, then get related stages using this `jobId`, to handle these circumstances above. May this be a good way? @jerryshao After a deep dive into this implement, I find it is difficult to get job group id from RSCDriver, at first I thought `jobId` might be the spark job group id(because in JobWrapper class, it calls setting JobGroup with `jobId`), actually I'm wrong. Spark JobGroup is set as SqlJob's statementId, and if I want to get this statementId in JobWrapper or RscDriver, I must convert JobWrapper's Job to SqlJob, this needs SqlJob's definition, as SqlJob is in thriftserver module, importing its definition in rsc module may not be a good way. Could you please give me some advice? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [incubator-livy] huianyi edited a comment on issue #238: [LIVY-689] Deliver stage process message to the end user using thriftserver
huianyi edited a comment on issue #238: [LIVY-689] Deliver stage process message to the end user using thriftserver URL: https://github.com/apache/incubator-livy/pull/238#issuecomment-544522301 > Rethink about the way this ticket doing, I think current way may not be the best way. > > 1. `jobId` may not exactly match the currently running jobs, there's no issue for beeline, but could be an issue for multi-thread usage. Currently when getting active jobs, there's no direct relation to jobId. > 2. There's already a way to calculate job progress in ReplDriver for statement, we should extend the current way to also apply to job API. > 3. Using LivyServer to store the session/job status may not be fault tolerant. It would better to extend the job API to get job progress synchronously. Thanks for your reply, if I have understood your meaning correctly, actually I can use `jobId`(which is spark jobs' group id delivered by livy) in `handleProcessMessage` method of RSCDriver, then get related stages using this `jobId`, to handle these circumstances above. May this be a good way? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [incubator-livy] huianyi edited a comment on issue #238: [LIVY-689] Deliver stage process message to the end user using thriftserver
huianyi edited a comment on issue #238: [LIVY-689] Deliver stage process
message to the end user using thriftserver
URL: https://github.com/apache/incubator-livy/pull/238#issuecomment-536483452
> Hi @huianyi ，I have saw the ERROR in your uts. The root cause of your UT
failure is:
> Can't find class ConcurrentBoundedLinkedQueue in thriftserver .
>
> You can avoid this error by adding the following code in
thriftserver/session/pom.xml：
> ` org.apache.livy
livy-core_${scala.binary.version}
${project.version} provided `
Thanks for your very kind support, gonna push a new commit.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-livy] huianyi edited a comment on issue #238: [LIVY-689] Deliver stage process message to the end user using thriftserver
huianyi edited a comment on issue #238: [LIVY-689] Deliver stage process message to the end user using thriftserver URL: https://github.com/apache/incubator-livy/pull/238#issuecomment-535766621 Currently I use operationMessage in LivyThriftSeesionManager to achieve this feature, but maybe take out operationMessage as a separate class is a good way? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
