[GitHub] zeppelin issue #3206: [WIP][ZEPPELIN-3810] Support Spark 2.4

[zjffdu]

Github user zjffdu commented on the issue:

https://github.com/apache/zeppelin/pull/3206
  
Awesome @HyukjinKwon Let's wait for spark 2.4 release


---

[GitHub] zeppelin issue #3206: [WIP][ZEPPELIN-3810] Support Spark 2.4

[HyukjinKwon]

Github user HyukjinKwon commented on the issue:

https://github.com/apache/zeppelin/pull/3206
  
I also tested this patch against Spark RC5.


---

[GitHub] zeppelin issue #3206: [WIP][ZEPPELIN-3810] Support Spark 2.4

[HyukjinKwon]

Github user HyukjinKwon commented on the issue:

https://github.com/apache/zeppelin/pull/3206
  
BTW, I tested this via my Travis CI - 
https://travis-ci.org/HyukjinKwon/zeppelin/builds/448215776. Tests seems got 
passed.


---

[GitHub] zeppelin pull request #3206: [WIP][ZEPPELIN-3810] Support Spark 2.4

[HyukjinKwon]

GitHub user HyukjinKwon reopened a pull request:

https://github.com/apache/zeppelin/pull/3206

[WIP][ZEPPELIN-3810] Support Spark 2.4

### What is this PR for?

Spark 2.4 changed it's Scala version from 2.11.8 to 2.11.12 (see 
SPARK-24418).

There are two problems for this upgrade at Zeppelin side:

1.. Some methods that are used in private by reflection, for instance, 
`loopPostInit` became inaccessible.

See:
 - 
https://github.com/scala/scala/blob/v2.11.8/src/repl/scala/tools/nsc/interpreter/ILoop.scala
 - 
https://github.com/scala/scala/blob/v2.11.12/src/repl/scala/tools/nsc/interpreter/ILoop.scala

To work around this, I manually ported `loopPostInit` at 2.11.8 to retain 
the behaviour. Some functions that are commonly existing at both Scala 2.11.8 
and Scala 2.11.12 are used inside of the new `loopPostInit` by reflection.


2.. Upgrade from 2.11.8 to 2.11.12 requires `jline.version` upgrade. 
Otherwise, we will hit:
```
Caused by: java.lang.NoSuchMethodError: 

jline.console.completer.CandidateListCompletionHandler.setPrintSpaceAfterFullCompletion(Z)V
  at 
scala.tools.nsc.interpreter.jline.JLineConsoleReader.initCompletion(JLineReader.scala:139)
```

To work around this, I tweaked this by upgrading jline from `2.12.1` to 
`2.14.3`.


### What type of PR is it?
[Improvement]

### Todos
* [ ] - Wait until Spark 2.4.0 is officially released.

### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-3810

### How should this be tested?

Verified manually against Spark 2.4.0 RC3

### Questions:
* Does the licenses files need update? Yes
* Is there breaking changes for older versions? No
* Does this needs documentation? No


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/HyukjinKwon/zeppelin ZEPPELIN-3810

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/zeppelin/pull/3206.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3206


commit 9ac1797a7e175a73350d61a0e62b3e4f89b29b8f
Author: hyukjinkwon 
Date:   2018-10-17T14:41:29Z

Support Spark 2.4




---

[GitHub] zeppelin pull request #3206: [WIP][ZEPPELIN-3810] Support Spark 2.4

[HyukjinKwon]

Github user HyukjinKwon closed the pull request at:

https://github.com/apache/zeppelin/pull/3206


---

[GitHub] zeppelin pull request #3212: [ZEPPELIN-3837] update how_to_build.md

[Leemoonsoo]

GitHub user Leemoonsoo opened a pull request:

https://github.com/apache/zeppelin/pull/3212

[ZEPPELIN-3837] update how_to_build.md

### What is this PR for?
how_to_build.md includes some build profiles that has removed/changed.
This PR makes how_to_build.md up to dated.

### What type of PR is it?
Documentation

### What is the Jira issue?
https://issues.apache.org/jira/browse/ZEPPELIN-3837

### Questions:
* Does the licenses files need update? no
* Is there breaking changes for older versions? no
* Does this needs documentation? no


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/Leemoonsoo/zeppelin ZEPPELIN-3837

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/zeppelin/pull/3212.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3212


commit 63a97373129955e1003fa961f4a8967091bc9f5e
Author: Lee moon soo 
Date:   2018-10-30T23:27:46Z

update how_to_build.md




---

[jira] [Created] (ZEPPELIN-3837) Make 'how to build' up to date.

[Lee moon soo (JIRA)]

Lee moon soo created ZEPPELIN-3837:
--

 Summary: Make 'how to build' up to date.
 Key: ZEPPELIN-3837
 URL: https://issues.apache.org/jira/browse/ZEPPELIN-3837
 Project: Zeppelin
  Issue Type: Task
Reporter: Lee moon soo
Assignee: Lee moon soo
 Fix For: 0.9.0


docs/setup/basics/how_to_build.md is bit outdated.

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] zeppelin pull request #3211: [ZEPPELIN-3836] update website url rewrite rule...

[Leemoonsoo]

GitHub user Leemoonsoo opened a pull request:

https://github.com/apache/zeppelin/pull/3211

[ZEPPELIN-3836] update website url rewrite rule for 0.8.0 release

### What is this PR for?
`https://zeppelin.apache.org/docs/latest` supposed to point latest release 
version documentation. But it currently points 0.7.3 version doc, instead of 
0.8.0.


### What type of PR is it?
Fix

### What is the Jira issue?
https://issues.apache.org/jira/browse/ZEPPELIN-3836


### Questions:
* Does the licenses files need update? no
* Is there breaking changes for older versions? no
* Does this needs documentation? no


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/Leemoonsoo/zeppelin 
gh-pages-rewrite_rule_for_0.8

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/zeppelin/pull/3211.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3211


commit 9f01b45b56fb53c066119b82a85952b0139b8f0e
Author: Lee moon soo 
Date:   2018-10-30T22:53:01Z

update website url rewrite rule for 0.8.0 doc




---

[jira] [Created] (ZEPPELIN-3836) zeppelin.apache.org/docs/latest points 0.7.3, instead of 0.8.0

[Lee moon soo (JIRA)]

Lee moon soo created ZEPPELIN-3836:
--

 Summary: zeppelin.apache.org/docs/latest points 0.7.3, instead of 
0.8.0
 Key: ZEPPELIN-3836
 URL: https://issues.apache.org/jira/browse/ZEPPELIN-3836
 Project: Zeppelin
  Issue Type: Task
Reporter: Lee moon soo
Assignee: Lee moon soo


[https://zeppelin.apache.org/docs/latest] points 0.7.3 version doc, instead of 
0.8.0



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] zeppelin pull request #3210: [ZEPPELIN-3835] Update website for 0.8.0 releas...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/zeppelin/pull/3210


---

[GitHub] zeppelin issue #3210: [ZEPPELIN-3835] Update website for 0.8.0 release

[Leemoonsoo]

Github user Leemoonsoo commented on the issue:

https://github.com/apache/zeppelin/pull/3210
  
Thanks @zjffdu for a review.


---

Re: Zeppelin add hadoop submarine(machine learning framework) interpreter

[liuxun]

HI,


I updated the design document and submitted it to the hadoop community.

https://docs.google.com/document/d/16YN8Kjmxt1Ym3clx5pDnGNXGajUT36hzQxjaik1cP4A/edit#
 

 
If you have any questions, you can ask them in the email or directly in 
the document. Thank you!

I developed the submarine interpreter module in zeppelin. Below is the 
interface for the system to run.

zeppelin submarine interpreter properties

System administrators can configure multiple interpreters, and different 
interpreters can have different resource configurations.
Allow different users to use different interpreters for resource allocation and 
management.
zeppelin submarine Tensorflow interpreter

Users use their own interpreter, First through the %submarine.tensorflow 
paragraph, To write python code for tensorflow.
After the user writes the python code of tensorflow, Click the [RUN] button and 
zeppelin will upload the python code to the specified HDFS directory. The 
submarine is loaded into the docker container when it is running.

zeppelin submarine interpreter

The user sets the call parameter value of the tensorflow python code, Then 
enter the job run command.
The zeppelin submarine interpreter first checks that all parameters of the 
submarine run are set completely. After the check is passed, the implementation 
of YARN is performed via submarine.jar.
The progress and log information of the submarine will be displayed in 
zeppelin's note.
You can enter submarine's other commands in the submarine interpreter, view all 
the jobs in the submarine, etc.



> 在 2018年10月21日，上午6:26，Felix Cheung  写道：
> 
> Very cool!
> 
> 
> 
> From: Jeff Zhang 
> Sent: Friday, October 19, 2018 7:14 AM
> To: dev@zeppelin.apache.org
> Subject: Re: Zeppelin add hadoop submarine(machine learning framework) 
> interpreter
> 
> Thanks xun. This would be a great addon for zeppelin to support deep
> learning. I will check the design later.
> 
> 
> 
> liu xun 于2018年10月19日周五 下午3:56写道：
> 
>> Hi,
>> Hadoop Submarine is the latest machine learning framework subproject in
>> the Hadoop 3.2 release. It allows Hadoop to support Tensorflow,
>> MXNet,Caffe, Spark, etc. A variety of deep learning frameworks provide a
>> full-featured system framework for machine learning algorithm development,
>> distributed model training, model management, and model publishing,
>> combined with hadoop's intrinsic data storage and data processing
>> capabilities to enable data scientists to Good mining and the value of the
>> data.
>> 
>> 
>> I was involved in the development of the hadoop submarine project. So I
>> plan to add the interpreter module of hadoop submarine to zeppelin.
>> Let zeppeline increase the development of deep learning. This is my design
>> document, let's see if there is any opinion, you can put it directly in the
>> document, thank you!
>> 
>> 
>> https://docs.google.com/document/d/16YN8Kjmxt1Ym3clx5pDnGNXGajUT36hzQxjaik1cP4A/edit?ts=5bc6bfdd
>> 
>> 
>>