[GitHub] zeppelin issue #1587: [ZEPPELIN-1609] using pyspark(python3) with livy inter...
Github user felixcheung commented on the issue: https://github.com/apache/zeppelin/pull/1587 all green! merging if no more comment --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin issue #1587: [ZEPPELIN-1609] using pyspark(python3) with livy inter...
Github user purechoc commented on the issue: https://github.com/apache/zeppelin/pull/1587 i think CI failure cause is not this PR. some other reason in other testcase. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin issue #1587: [ZEPPELIN-1609] using pyspark(python3) with livy inter...
Github user felixcheung commented on the issue: https://github.com/apache/zeppelin/pull/1587 LGTM merging if no more comment --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin issue #1587: [ZEPPELIN-1609] using pyspark(python3) with livy inter...
Github user purechoc commented on the issue: https://github.com/apache/zeppelin/pull/1587 thank to address. update code completed --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin issue #1587: [ZEPPELIN-1609] using pyspark(python3) with livy inter...
Github user felixcheung commented on the issue: https://github.com/apache/zeppelin/pull/1587 shouldn't this be listed here: https://github.com/apache/zeppelin/blob/master/conf/zeppelin-site.xml.template#L193 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin issue #1587: [ZEPPELIN-1609] using pyspark(python3) with livy inter...
Github user purechoc commented on the issue: https://github.com/apache/zeppelin/pull/1587 @felixcheung @Leemoonsoo please review this. thanks --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin issue #1587: [ZEPPELIN-1609] using pyspark(python3) with livy inter...
Github user zjffdu commented on the issue: https://github.com/apache/zeppelin/pull/1587 lgtm --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin issue #1587: [ZEPPELIN-1609] using pyspark(python3) with livy inter...
Github user purechoc commented on the issue: https://github.com/apache/zeppelin/pull/1587 refector code based on #1612 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin issue #1587: [ZEPPELIN-1609] using pyspark(python3) with livy inter...
Github user purechoc commented on the issue: https://github.com/apache/zeppelin/pull/1587 @zjffdu thanks to comment. i am wait #1612 #1612 is LGTM --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin issue #1587: [ZEPPELIN-1609] using pyspark(python3) with livy inter...
Github user zjffdu commented on the issue: https://github.com/apache/zeppelin/pull/1587 I prefer to use `%livy.pyspark` & `$livy.pyspark3` instead of `zeppelin.livy.kind`. Because if user change `zeppelin.livy.kind` some paragraphs may not work again, this is very confusing. So explicitly specifying python version is better IMO. And could you wait for #1612 which is refactoring livy interpreter ? After #1612 , it should be pretty easy to add `%livy.pyspark3` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin issue #1587: [ZEPPELIN-1609] using pyspark(python3) with livy inter...
Github user purechoc commented on the issue: https://github.com/apache/zeppelin/pull/1587 @zjffdu @felixcheung @Leemoonsoo could you merge about this? or need to fix something? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin issue #1587: [ZEPPELIN-1609] using pyspark(python3) with livy inter...
Github user felixcheung commented on the issue: https://github.com/apache/zeppelin/pull/1587 Ah thanks for pointing out. I will look into what Livy is doing shortly --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin issue #1587: [ZEPPELIN-1609] using pyspark(python3) with livy inter...
Github user Leemoonsoo commented on the issue: https://github.com/apache/zeppelin/pull/1587 I see what you mean. Thanks for the explanation. Make sense. LGTM --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin issue #1587: [ZEPPELIN-1609] using pyspark(python3) with livy inter...
Github user purechoc commented on the issue: https://github.com/apache/zeppelin/pull/1587 @Leemoonsoo . thanks to comment. you mean like this? if user add property `zeppelin.livy.kind` = pyspark, "%livy.pyspark" working on `python2`. if user add property `zeppelin.livy.kind` = pyspark3, "%livy.pyspark" working on `python3`. like a set `zeppelin.python` = python or python3 ``pyspark `interpreter` but if user want to `python2 `and `python3 `same time `python` interpreter. user need to create new interperter in the GUI. ``` interperter name = python3 interperter group = python zeppelin.python = python3 %python3 2/3 = 1.5 ``` if live using this approch, user using livy interperter like this. ``` # run python2 %livy.pyspark # run python3 %livy_python3.pyspark ``` add property `zeppelin.livy.kind` and set python version if user want to run. (each time. not same time) is this right? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin issue #1587: [ZEPPELIN-1609] using pyspark(python3) with livy inter...
Github user Leemoonsoo commented on the issue: https://github.com/apache/zeppelin/pull/1587 Thanks for explain. but i'm little bit confused. If you are creating `LivyPySpark3Interpreter` in addition to `LivyPySparkInterpreter`, there will be no need to add property `zeppelin.livy.kind`. But by adding a property, `LivyPySparkInterpreter` interpreter can read property and change it's behavior which version of python to use, in runtime. and no need to create `LivyPySpark3Interpreter`. In this case, to use both python2 and python3 at the same time, user can create multiple interpreter settings in the GUI and use them. I have no strong opinion on any approaches, but at least `python`, `pyspark` interpreter handle python version using property instead of having separate interpreter class. @purechoc @zjffdu @prabhjyotsingh @felixcheung What do you think? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin issue #1587: [ZEPPELIN-1609] using pyspark(python3) with livy inter...
Github user purechoc commented on the issue:
https://github.com/apache/zeppelin/pull/1587
@Leemoonsoo
thanks to review.
first. i try like that.
```
# add properties
{
"group": "livy",
"name": "pyspark3",
"className": "org.apache.zeppelin.livy.LivyPySparkInterpreter",
"properties": {
"zeppelin.livy.kind": {
"propertyName": "zeppelin.livy.kind",
"defaultValue": "pyspark3"
},
},
"editor": {
"language": "python",
"editOnDblClick": false
}
},
```
it is work. but that property assign both "livy.pyspark" and
"livy.pyspark3".
if user want to python2 and python3 both,
user need to make new interpreter base on livy group and set property
zeppelin.livy.kind=pyspark3.
if, this is better than adding new interpreter class, i fixed code and
commit soon.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin issue #1587: [ZEPPELIN-1609] using pyspark(python3) with livy inter...
Github user purechoc commented on the issue: https://github.com/apache/zeppelin/pull/1587 can't do that, %livy.pyspark support python2 only. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
