[GitHub] zeppelin pull request #1369: [ZEPPELIN-1376]

[doanduyhai]

GitHub user doanduyhai reopened a pull request:

https://github.com/apache/zeppelin/pull/1369

[ZEPPELIN-1376]

### What is this PR for?
When using Zeppelin behind corporate firewall, sometimes the dependencies 
download just fails silently. This PR has 2 objectives:

* add proxy credentials information for dependencies repo
* raise clear error message in case of dependencies download failure

There are 3 commits.

The first one add extra inputs in the form for adding new repository


![add_repo](https://cloud.githubusercontent.com/assets/1532977/18017489/0b486fda-6bd2-11e6-90c7-ceda18c53575.png)

The second commit fixes some issues and display a clear and explicit error 
message when download of dependencies fail.

Before that, when the download fails, we can see the below behaviour


![irrelevant_double_error_message](https://cloud.githubusercontent.com/assets/1532977/18017541/3cf0de1e-6bd2-11e6-8285-af03f222e8d2.gif)

* the error message is displayed twice because the call twice the method 
`checkDownloadingDependencies();`. One in the success callback of:

```javascript
 $scope.updateInterpreterSetting = function(form, settingId) {
  ...
$http.put(baseUrlSrv.getRestApiBase() + '/interpreter/setting/' 
+ settingId, request)
  .success(function(data, status, headers, config) {
$scope.interpreterSettings[index] = data.body;
removeTMPSettings(index);
thisConfirm.close();
checkDownloadingDependencies();
$route.reload();
  })
  .error(function(data, status, headers, config) {
 ...
};
```

Another call is inside success callback of `getInterpreterSettings()`

```javascript
var getInterpreterSettings = function() {
  $http.get(baseUrlSrv.getRestApiBase() + '/interpreter/setting')
  .success(function(data, status, headers, config) {
$scope.interpreterSettings = data.body;
checkDownloadingDependencies();
  }).error(function(data, status, headers, config) {
  
```

The problem is that `$route.reload();` in the success callback of 
`updateInterpreterSetting()` will trigger `init()` then 
`getInterpreterSettings()` so `checkDownloadingDependencies()` is called twice.

I remove the call to `checkDownloadingDependencies()` from success callback 
of `updateInterpreterSetting()` 

The second modification is on class `DependencyResolver`. In the screen 
capture above, we get a **cryptic** NullPointerException coming from 
`DefaultRepositorySystem`. I now catch this NPE to wrap it into a more sensible 
and clearer exception:

```java

  public List getArtifactsWithDep(String dependency,
Collection excludes) throws RepositoryException {
Artifact artifact = new DefaultArtifact(dependency);
DependencyFilter classpathFilter = 
DependencyFilterUtils.classpathFilter(JavaScopes.COMPILE);
PatternExclusionsDependencyFilter exclusionFilter =
new PatternExclusionsDependencyFilter(excludes);

CollectRequest collectRequest = new CollectRequest();
collectRequest.setRoot(new Dependency(artifact, JavaScopes.COMPILE));

synchronized (repos) {
  for (RemoteRepository repo : repos) {
collectRequest.addRepository(repo);
  }
}
DependencyRequest dependencyRequest = new 
DependencyRequest(collectRequest,
DependencyFilterUtils.andFilter(exclusionFilter, 
classpathFilter));

 //Catch NPE thrown by aether and give a proper error message 
try {
  return system.resolveDependencies(session, 
dependencyRequest).getArtifactResults();
} catch (NullPointerException ex) {
  throw new RepositoryException(String.format("Cannot fetch 
dependencies for %s", dependency));
}
  }
```

The result is much more cleaner


![dependencies_download_error_popup](https://cloud.githubusercontent.com/assets/1532977/18033855/1be5fe9a-6d2e-11e6-91f9-2f5ea66cab26.gif)



The last commit is just doc update


![updated_docs](https://cloud.githubusercontent.com/assets/1532977/18017797/97302f14-6bd3-11e6-97cc-77bd52f25cde.png)


### What type of PR is it?
[Improvement]

### Todos
* [ ] - Code Review
* [ ] - Simple test with no Internet connection
* [ ] - Test within a corporate firewall env with a third-party dependency, 
requiring download

### What is the Jira issue?
**[ZEPPELIN-1376]**

### How should this be tested?

# Simple test
* `git fetch origin pull/1369/head:WebProxy`
* `git checkout WebProxy`
* `mvn clean package -DskipTests`
* 

[GitHub] zeppelin pull request #746: [ZEPPELIN-699] Add new synchronous paragraph run...

[doanduyhai]

GitHub user doanduyhai reopened a pull request:

https://github.com/apache/zeppelin/pull/746

[ZEPPELIN-699] Add new synchronous paragraph run REST API

### What is this PR for?
Right now, when calling the REST API 
`http://:/api/notebook/job//` Zeppelin always 
returns **OK** as shown by this source code: 
https://github.com/apache/incubator-zeppelin/blob/master/zeppelin-server/src/main/java/org/apache/zeppelin/rest/NotebookRestApi.java#L477

This ticket will update the behavior so that Zeppelin also return the 
result of the paragraph execution

### What type of PR is it?
[Improvement]

### Todos
* [ ] - Code Review
* [ ] - Simple Test

### Is there a relevant Jira issue?
**[ZEPPELIN-699]**

### How should this be tested?
* `git fetch origin pull/746/head:ParagraphExecutionRESTAPI`
* `git checkout ParagraphExecutionRESTAPI`
* `mvn clean package -DskipTests`
* `bin/zeppelin-daemon.sh restart`
* Create a new note
* In the first paragraph, put the following code

```scala
%sh

echo "Current time = "`date +"%T"
```

* Retrieve the current note id in the URL
* Retrieve the current paragraph id
* Use a REST Client like **[POSTman]** to create a HTTP POST query 
`http://:/api/notebook/run//` 
* You should receive something similar as follow for answer

```
{
"status": "OK",
"body": {
"code": "SUCCESS",
"type": "TEXT",
"msg": "Current time = 16:14:18\n"
}
}
```

### Screenshots (if appropriate)

![zeppelin_synchronous_rest_api](https://cloud.githubusercontent.com/assets/1532977/15748069/b4a26a46-28dd-11e6-8f51-aa13ddba3f1c.gif)

API Documentation update

**Existing asynchronous API**

![image](https://cloud.githubusercontent.com/assets/1532977/15773274/5b508cae-2976-11e6-9e52-14d8b7e7828e.png)

**New synchronous API**

![image](https://cloud.githubusercontent.com/assets/1532977/15773309/84965a94-2976-11e6-9719-81d8b555c3c4.png)


### Questions:
* Does the licenses files need update? --> **No**
* Is there breaking changes for older versions? --> **No**
* Does this needs documentation? --> **Yes**

[ZEPPELIN-699]: https://issues.apache.org/jira/browse/ZEPPELIN-699
[POSTman]: https://www.getpostman.com/

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/doanduyhai/incubator-zeppelin ZEPPELIN-699

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/zeppelin/pull/746.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #746


commit 8367acfc0526965a884f58eea8f8ddda70d011d7
Author: DuyHai DOAN 
Date:   2016-06-02T14:15:46Z

[ZEPPELIN-699] Add new synchronous paragraph run REST API

commit fb0570ce70c47d3648ce4b9cc8265ebe51f6bdc8
Author: DuyHai DOAN 
Date:   2016-06-02T14:24:38Z

[ZEPPELIN-699] Update Notebook REST API documentation




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] zeppelin issue #1376: [ZEPPELIN-449] Enhance log messages when interpreter i...

[doanduyhai]

Github user doanduyhai commented on the issue:

https://github.com/apache/zeppelin/pull/1376
  
@bzz 

I don't know why the tests fail on this error message, because this is 
indeed **a test which expects a failure** because `system.complex_table` does 
not exist, see below:

```java
@Test
public void should_error_describing_non_existing_table() throws 
Exception {
//Given
String query = "USE system;\n" +
"DESCRIBE TABLE complex_table;";

//When
final InterpreterResult actual = interpreter.interpret(query, 
intrContext);

//Then
assertThat(actual.code()).isEqualTo(Code.ERROR);
assertThat(actual.message()).contains("Cannot find table 
system.complex_table");
}

```


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] zeppelin issue #1378: [ZEPPELIN-1358] Add support to display Pandas DataFram...

[bustios]

Github user bustios commented on the issue:

https://github.com/apache/zeppelin/pull/1378
  
Sure @bzz, it can be optional. I just thought it was more appropriate, 
since a dataframe index is always printed, but I can add a parameter like 
`show_index=False`  to make it optional.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] zeppelin pull request #1367: [ZEPPELIN-1040] Show the time when the result i...

[rajarajan-g]

GitHub user rajarajan-g reopened a pull request:

https://github.com/apache/zeppelin/pull/1367

 [ZEPPELIN-1040] Show the time when the result is updated

### What is this PR for?
As per existing usage, the time shown in end of each paragraph is the time 
the paragraph is updated not when the paragraph is actually executed/run.

_" Took 10 sec. Last updated by anonymous at **August 26 2016, 1:52:01 
PM.** "_

PR is aimed at changing the existing usage to show when the paragraph is 
last executed as this gives  clarification to users about the executed time of 
paragraph


### What type of PR is it?
 Improvement 

### Todos
* [ ] - Task

### What is the Jira issue?
https://issues.apache.org/jira/browse/ZEPPELIN-1040

### How should this be tested?
1. Start the server and create a new note book
2. create a new paragraph and execute the paragraph
3. Now rerun the paragraph, the time should get updated now inline with 
execution of the paragraph

### Screenshots (if appropriate)

### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/rajarajan-g/zeppelin ZEPPELIN-1040

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/zeppelin/pull/1367.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1367


commit d30f1b28e3a7c3a3160a3eab7e6e30aa1f53083c
Author: rajarajan-g 
Date:   2016-08-26T08:30:10Z

code changed for showing last run time after execution of paragraph




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] zeppelin pull request #1367: [ZEPPELIN-1040] Show the time when the result i...

[rajarajan-g]

Github user rajarajan-g closed the pull request at:

https://github.com/apache/zeppelin/pull/1367


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] zeppelin issue #1377: ZEPPELIN-1384. Spark interpreter binary compatibility ...

[bzz]

Github user bzz commented on the issue:

https://github.com/apache/zeppelin/pull/1377
  
LGTM.

CI is green as well.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] zeppelin issue #1378: [ZEPPELIN-1358] Add support to display Pandas DataFram...

[bzz]

Github user bzz commented on the issue:

https://github.com/apache/zeppelin/pull/1378
  
@bustios thank you for improvements! 

Could you please explain a bit the benefits of having print index behaviour 
enabled by default? Do you think it should not be optional? 

It's nice advanced feature but I also see the value in having default 
behaviour consistent between interpreters. What do you think?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] zeppelin issue #1358: [ZEPPELIN-1365] Error of Zeppelin Application in devel...

[bzz]

Github user bzz commented on the issue:

https://github.com/apache/zeppelin/pull/1358
  
Looks great to me!

Merging to master if there is no further discussion.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] zeppelin pull request #1378: [ZEPPELIN-1358] Add support to display Pandas D...

[bustios]

GitHub user bustios opened a pull request:

https://github.com/apache/zeppelin/pull/1378

[ZEPPELIN-1358] Add support to display Pandas DataFrame index using z.show()

### What is this PR for?
Add support to display Pandas DataFrame index using z.show() in python 
interpreter

### What type of PR is it?
Improvement

### What is the Jira issue?
[ZEPPELIN-1358](https://issues.apache.org/jira/browse/ZEPPELIN-1358)

### How should this be tested?
```
mvn -Dpython.test.exclude='' test -pl python -am
```

### Screenshots (if appropriate)
![screenshot from 2016-08-28 
22-47-14](https://cloud.githubusercontent.com/assets/7907284/18039966/4bd179fe-6d7e-11e6-8d39-9d7df884221f.png)

### Questions:
* Does the licenses files need update? no
* Is there breaking changes for older versions? no
* Does this needs documentation? no

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/bustios/zeppelin ZEPPELIN-1358

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/zeppelin/pull/1378.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1378


commit 6fe315b09308677a27ea481652601aa816a0bbd6
Author: Paul Bustios 
Date:   2016-08-29T03:06:06Z

Add support to display Pandas DataFrame index using z.show() and modifies 
test.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] zeppelin issue #1370: [ZEPPELIN-1379] Flink interpreter is missing scala lib...

[bzz]

Github user bzz commented on the issue:

https://github.com/apache/zeppelin/pull/1370
  
Looks good to me, thank you @lresende 

BTW, from my experience, postin CI failure helps to speed up a review 
process.

CI fails on Spark 2.0 profile \w `ZeppelinSparkClusterTest.zRunTest` which 
looks like a flaky test, not relevant to the changes.
```
[INFO] Zeppelin: Server ... FAILURE [04:11 
min]

Failed tests: 
  ZeppelinSparkClusterTest.zRunTest:204 expected: but was:

00:32:54,193  INFO org.apache.zeppelin.interpreter.InterpreterFactory:665 - 
Create interpreter instance md for note 2BW956553
00:32:54,193  INFO org.apache.zeppelin.interpreter.InterpreterFactory:696 - 
Interpreter org.apache.zeppelin.markdown.Markdown 578260863 created
00:32:54,194  INFO org.apache.zeppelin.notebook.Paragraph:295 - run 
paragraph 20160828-003254_783795023 using md 
org.apache.zeppelin.interpreter.LazyOpenInterpreter@22778f7f
00:32:54,195  INFO 
org.apache.zeppelin.interpreter.remote.RemoteInterpreterManagedProcess:116 - 
Run interpreter process [..//bin/interpreter.sh, -d, ../interpreter/md, -p, 
46521, -l, ..//local-repo/2BUN9SMBJ]
00:32:54,195  INFO org.apache.zeppelin.socket.NotebookServer:115 - New 
connection from null : 0
00:32:54,196  INFO org.apache.zeppelin.socket.NotebookServer:115 - New 
connection from null : 0
00:32:54,198  INFO org.apache.zeppelin.socket.NotebookServer:529 - New 
operation from null : 0 : anonymous : GET_NOTE : 2BW956553
00:32:54,209  INFO org.apache.zeppelin.socket.NotebookServer:529 - New 
operation from null : 0 : anonymous : GET_NOTE : 2BW956553
00:32:54,210 ERROR 
org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess:234 - Remote 
interpreter process not started
java.lang.NullPointerException
at 
org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess.updateRemoteAngularObject(RemoteInterpreterProcess.java:229)
at 
org.apache.zeppelin.interpreter.remote.RemoteAngularObject.set(RemoteAngularObject.java:48)
at 
org.apache.zeppelin.interpreter.remote.RemoteAngularObject.set(RemoteAngularObject.java:40)
at 
org.apache.zeppelin.socket.NotebookServer.angularObjectUpdated(NotebookServer.java:841)
at 
org.apache.zeppelin.socket.NotebookServer.onMessage(NotebookServer.java:221)
at 
org.apache.zeppelin.socket.NotebookServerTest.testMakeSureNoAngularObjectBroadcastToWebsocketWhoFireTheEvent(NotebookServerTest.java:135)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

00:32:54,216 ERROR 
org.apache.zeppelin.interpreter.remote.RemoteAngularObjectRegistry:111 - Error
java.lang.NullPointerException
at 
org.apache.zeppelin.interpreter.remote.RemoteAngularObjectRegistry.removeAndNotifyRemoteProcess(RemoteAngularObjectRegistry.java:105)
at 
org.apache.zeppelin.interpreter.remote.RemoteAngularObjectRegistry.removeAllAndNotifyRemoteProcess(RemoteAngularObjectRegistry.java:123)
at org.apache.zeppelin.notebook.Notebook.removeNote(Notebook.java:326)
at 
org.apache.zeppelin.socket.NotebookServerTest.testMakeSureNoAngularObjectBroadcastToWebsocketWhoFireTheEvent(NotebookServerTest.java:147)
```

Merging to _master_ and _branch-0.6_ if there is no further discussion.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] zeppelin issue #1333: [ZEPPELIN-1334] Environment variable defined in interp...

[bzz]

Github user bzz commented on the issue:

https://github.com/apache/zeppelin/pull/1333
  
@zjffdu Looks great to me, modulo minor style issue above, thank you and 
:+1: for tests!

One more thing - it's a bit strange ZEPPELIN-1334 has fix-for version ONLY 
0.6.2 which implies merging only to _branch-0.6_ but this PR is to _master_.

Let's merge it, as soon as those 2 issues are addressed!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] zeppelin issue #1363: [ZEPPELIN-1371]add text/numeric conversion support to ...

[Peilin-Yang]

Github user Peilin-Yang commented on the issue:

https://github.com/apache/zeppelin/pull/1363
  
@bzz I agree


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] zeppelin issue #1363: [ZEPPELIN-1371]add text/numeric conversion support to ...

[bzz]

Github user bzz commented on the issue:

https://github.com/apache/zeppelin/pull/1363
  
Let's merge #1371 first, and then it makes perfect sense to adjust this one 
to match user expectations of sorting changes according to column type (so it 
lets user override auto-detection). 

@corneadoug what do you think?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] zeppelin issue #1371: [ZEPPELIN-1372]Automatically Detect the data type in t...

[echarles]

Github user echarles commented on the issue:

https://github.com/apache/zeppelin/pull/1371
  
Upon numeric values, the automatic detection of dates... would be great.

Detection at client (javascript) side could be misleading as you only get a 
subset of the data.

On the other side, detection on server (scala) side may be more time 
consuming for hughe resultset. Still this could be a real added value as you 
could expose the types, units... via the API.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] zeppelin issue #1371: [ZEPPELIN-1372]Automatically Detect the data type in t...

[bzz]

Github user bzz commented on the issue:

https://github.com/apache/zeppelin/pull/1371
  
Thank you @Peilin-Yang ! 

It looks like it's `O(N)` from the table size - do you think there might be 
a performance implication here? Just curious, but if it's a table - should not 
analysing just a few rows be enough?  

@arunsoman that sounds interesting, but I think dev@ mailing list is much 
better place to discuss such changes, while keeping this PR to the scope of 
ZEPPELIN-1372.

\cc @corneadoug for review


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] zeppelin issue #1376: [ZEPPELIN-449] Enhance log messages when interpreter i...

[zjffdu]

Github user zjffdu commented on the issue:

https://github.com/apache/zeppelin/pull/1376
  
@lresende maybe ZEPPELIN-1187 is useful for you. You can edit 
log4j.properties to add the followign line. This is helpful for me to debug the 
remote interpreter process. 
```
log4j.logger.org.apache.zeppelin.interpreter.remote=DEBUG
```


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] zeppelin issue #1371: [ZEPPELIN-1372]Automatically Detect the data type in t...

[Peilin-Yang]

Github user Peilin-Yang commented on the issue:

https://github.com/apache/zeppelin/pull/1371
  
@arunsoman Yes. in this PR the code can automatically detect the type of 
the cells, but restricted to strings and numbers. Dates can be added either and 
this is a good idea.
In my other PR #1363 the drop down menu intents to make user aware of the 
data type of each column.
Please let me know if you have better idea (better with detailed 
suggestions) 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] zeppelin issue #1358: [ZEPPELIN-1365] Error of Zeppelin Application in devel...

[AhyoungRyu]

Github user AhyoungRyu commented on the issue:

https://github.com/apache/zeppelin/pull/1358
  
@astroshim I rebuild with `mvn clean package -DskipTests -Pexamples` and it 
works. I should've attached `-Pexamples`. Now I can see the Helium icon :) 

Anyway I checked `ArrayIndexOutOfBoundsException ` is gone with this patch. 
LGTM 👍 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] zeppelin pull request #1351: [ZEPPELIN/1356] The graph legend truncates at t...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/zeppelin/pull/1351


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] zeppelin pull request #1377: ZEPPELIN-1384. Spark interpreter binary compati...

[zjffdu]

GitHub user zjffdu opened a pull request:

https://github.com/apache/zeppelin/pull/1377

ZEPPELIN-1384. Spark interpreter binary compatibility to scala 2.10 / 2.11 
broken

### What is this PR for?

As described in ZEPPELIN-1384,  we may hit the following error when running 
zeppelin on spark 1.6 if we build zeppelin this way
```
dev/change_scala_version.sh 2.11
mvn -DskipTests -Drat.skip=true -Pscala-2.11 -Pspark-2.0 
-Dspark.version=2.0.0 -Phadoop-2.6 -Pyarn -Ppyspark -Psparkr clean package
```
```
java.lang.NoSuchMethodError: 
scala.runtime.VolatileByteRef.create(B)Lscala/runtime/VolatileByteRef;
at scala.xml.MetaData$.iterate$1(MetaData.scala:39)
at scala.xml.MetaData$.normalize(MetaData.scala:45)
at scala.xml.Elem.(Elem.scala:99)
at 
org.apache.spark.ui.jobs.StagePage$$anonfun$26.apply(StagePage.scala:57)
at 
org.apache.spark.ui.jobs.StagePage$$anonfun$26.apply(StagePage.scala:55)
at 
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
at 
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
at scala.collection.immutable.List.foreach(List.scala:318)
at scala.collection.TraversableLike$class.map(TraversableLike.scala:244)
at scala.collection.AbstractTraversable.map(Traversable.scala:105)
at org.apache.spark.ui.jobs.StagePage.(StagePage.scala:55)
at org.apache.spark.ui.jobs.StagesTab.(StagesTab.scala:34)
at org.apache.spark.ui.SparkUI.(SparkUI.scala:57)
at org.apache.spark.ui.SparkUI$.create(SparkUI.scala:195)
at org.apache.spark.ui.SparkUI$.createLiveUI(SparkUI.scala:146)
at org.apache.spark.SparkContext.(SparkContext.scala:473)
at 
org.apache.zeppelin.spark.SparkInterpreter.createSparkContext_1(SparkInterpreter.java:440)
at 
org.apache.zeppelin.spark.SparkInterpreter.createSparkContext(SparkInterpreter.java:354)
at 
org.apache.zeppelin.spark.SparkInterpreter.getSparkContext(SparkInterpreter.java:137)
at 
org.apache.zeppelin.spark.SparkInterpreter.open(SparkInterpreter.java:743)
at 
org.apache.zeppelin.interpreter.LazyOpenInterpreter.open(LazyOpenInterpreter.java:69)
at 
org.apache.zeppelin.interpreter.LazyOpenInterpreter.interpret(LazyOpenInterpreter.java:93)
```

The root cause is that scala-xml is removed from scala 2.11 to a separate 
library, so here we have class conflict of scala-xml api. In this PR, I make 
the scope of scala-xml to be provided and also make the scope of scala-library 
to be provided although it will be override in `ZEPPELIN_HOME/spark/pom.xml`

### What type of PR is it?
[Bug Fix]

### Todos
* [ ] - Task

### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-1384

### How should this be tested?
Tested manually. 
Use the following command to build zeppelin and then run it on spark-1.6.2
```
dev/change_scala_version.sh 2.11
mvn -DskipTests -Drat.skip=true -Pscala-2.11 -Pspark-2.0 
-Dspark.version=2.0.0 -Phadoop-2.6 -Pyarn -Ppyspark -Psparkr clean package
```

### Screenshots (if appropriate)

### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/zjffdu/zeppelin ZEPPELIN-1384

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/zeppelin/pull/1377.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1377


commit eb075352c7ba1157154b9a35f11bfca4d7d2927b
Author: Jeff Zhang 
Date:   2016-08-29T01:58:29Z

ZEPPELIN-1384. Spark interpreter binary compatibility to scala 2.10 / 2.11 
broken




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] zeppelin issue #1358: [ZEPPELIN-1365] Error of Zeppelin Application in devel...

[AhyoungRyu]

Github user AhyoungRyu commented on the issue:

https://github.com/apache/zeppelin/pull/1358
  
I think it's not an issue related with this PR but Helium itself.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] zeppelin pull request #1376: [ZEPPELIN-449] Enhance log messages when interp...

[lresende]

GitHub user lresende opened a pull request:

https://github.com/apache/zeppelin/pull/1376

[ZEPPELIN-449] Enhance log messages when interpreter is initializing

### What is this PR for?
Enhance log messages when interpreter is initializing to avoid user 
confusion.

### What type of PR is it?
[Bug Fix]

### What is the Jira issue?
* [ZEPPELIN-449](https://issues.apache.org/jira/browse/ZEPPELIN-449)


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/lresende/incubator-zeppelin zeppelin-449

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/zeppelin/pull/1376.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1376


commit 1011ca72c6b4e733722a93d8f1318c9217b719ef
Author: Luciano Resende 
Date:   2016-08-29T01:06:59Z

[ZEPPELIN-449] Enhance log messages when intepreter is initializing

Remove the "connection refused" stack trace and provide a better
error message when the interpreter is not accessible, which might
indicate that it is still initializing.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] zeppelin pull request #1347: ZEPPELIN-1284. Unable to run paragraph with def...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/zeppelin/pull/1347


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] zeppelin issue #1347: ZEPPELIN-1284. Unable to run paragraph with default in...

[lresende]

Github user lresende commented on the issue:

https://github.com/apache/zeppelin/pull/1347
  
LGTM, Do we need to wait on anything else to get this merged ? 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] zeppelin pull request #1373: [ZEPPELIN-1385] Avoid NPE when create SparkSess...

[lresende]

Github user lresende closed the pull request at:

https://github.com/apache/zeppelin/pull/1373


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] zeppelin issue #1373: [ZEPPELIN-1385] Avoid NPE when create SparkSession

[Leemoonsoo]

Github user Leemoonsoo commented on the issue:

https://github.com/apache/zeppelin/pull/1373
  
Thanks @lresende for the fix and verification. But i think we already have 
exactly the same fix at #1347.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] zeppelin pull request #1366: ZEPPELIN-1342. Adding dependencies via SPARK_SU...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/zeppelin/pull/1366


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] zeppelin pull request #1374: [MINOR] Add new line before logging paragraph c...

[lresende]

GitHub user lresende opened a pull request:

https://github.com/apache/zeppelin/pull/1374

[MINOR] Add new line before logging paragraph content

### What is this PR for?
Add new line before logging paragraph content

### What type of PR is it?
[Improvement]

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/lresende/incubator-zeppelin debug

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/zeppelin/pull/1374.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1374


commit d4e80d0b78260e20676c2a3a68b47f621a423dcb
Author: Luciano Resende 
Date:   2016-08-28T23:49:02Z

[MINOR] Add new line before logging paragraph content




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] zeppelin pull request #746: [ZEPPELIN-699] Add new synchronous paragraph run...

[doanduyhai]

GitHub user doanduyhai reopened a pull request:

https://github.com/apache/zeppelin/pull/746

[ZEPPELIN-699] Add new synchronous paragraph run REST API

### What is this PR for?
Right now, when calling the REST API 
`http://:/api/notebook/job//` Zeppelin always 
returns **OK** as shown by this source code: 
https://github.com/apache/incubator-zeppelin/blob/master/zeppelin-server/src/main/java/org/apache/zeppelin/rest/NotebookRestApi.java#L477

This ticket will update the behavior so that Zeppelin also return the 
result of the paragraph execution

### What type of PR is it?
[Improvement]

### Todos
* [ ] - Code Review
* [ ] - Simple Test

### Is there a relevant Jira issue?
**[ZEPPELIN-699]**

### How should this be tested?
* `git fetch origin pull/746/head:ParagraphExecutionRESTAPI`
* `git checkout ParagraphExecutionRESTAPI`
* `mvn clean package -DskipTests`
* `bin/zeppelin-daemon.sh restart`
* Create a new note
* In the first paragraph, put the following code

```scala
%sh

echo "Current time = "`date +"%T"
```

* Retrieve the current note id in the URL
* Retrieve the current paragraph id
* Use a REST Client like **[POSTman]** to create a HTTP POST query 
`http://:/api/notebook/run//` 
* You should receive something similar as follow for answer

```
{
"status": "OK",
"body": {
"code": "SUCCESS",
"type": "TEXT",
"msg": "Current time = 16:14:18\n"
}
}
```

### Screenshots (if appropriate)

![zeppelin_synchronous_rest_api](https://cloud.githubusercontent.com/assets/1532977/15748069/b4a26a46-28dd-11e6-8f51-aa13ddba3f1c.gif)

API Documentation update

**Existing asynchronous API**

![image](https://cloud.githubusercontent.com/assets/1532977/15773274/5b508cae-2976-11e6-9e52-14d8b7e7828e.png)

**New synchronous API**

![image](https://cloud.githubusercontent.com/assets/1532977/15773309/84965a94-2976-11e6-9719-81d8b555c3c4.png)


### Questions:
* Does the licenses files need update? --> **No**
* Is there breaking changes for older versions? --> **No**
* Does this needs documentation? --> **Yes**

[ZEPPELIN-699]: https://issues.apache.org/jira/browse/ZEPPELIN-699
[POSTman]: https://www.getpostman.com/

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/doanduyhai/incubator-zeppelin ZEPPELIN-699

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/zeppelin/pull/746.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #746


commit 8367acfc0526965a884f58eea8f8ddda70d011d7
Author: DuyHai DOAN 
Date:   2016-06-02T14:15:46Z

[ZEPPELIN-699] Add new synchronous paragraph run REST API

commit fb0570ce70c47d3648ce4b9cc8265ebe51f6bdc8
Author: DuyHai DOAN 
Date:   2016-06-02T14:24:38Z

[ZEPPELIN-699] Update Notebook REST API documentation




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] zeppelin pull request #746: [ZEPPELIN-699] Add new synchronous paragraph run...

[doanduyhai]

Github user doanduyhai closed the pull request at:

https://github.com/apache/zeppelin/pull/746


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] zeppelin issue #1334: [ZEPPELIN-682] New interpreter for Apache Beam (incuba...

[mfelgamal]

Github user mfelgamal commented on the issue:

https://github.com/apache/zeppelin/pull/1334
  
@bzz Gentle reminder!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] zeppelin issue #1373: [ZEPPELIN-1385] Avoid NPE when create SparkSession

[lresende]

Github user lresende commented on the issue:

https://github.com/apache/zeppelin/pull/1373
  
After this fix is applied, have verified that can connect and execute a job 
on a remote Spark cluster.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] zeppelin pull request #746: [ZEPPELIN-699] Add new synchronous paragraph run...

[doanduyhai]

GitHub user doanduyhai reopened a pull request:

https://github.com/apache/zeppelin/pull/746

[ZEPPELIN-699] Add new synchronous paragraph run REST API

### What is this PR for?
Right now, when calling the REST API 
`http://:/api/notebook/job//` Zeppelin always 
returns **OK** as shown by this source code: 
https://github.com/apache/incubator-zeppelin/blob/master/zeppelin-server/src/main/java/org/apache/zeppelin/rest/NotebookRestApi.java#L477

This ticket will update the behavior so that Zeppelin also return the 
result of the paragraph execution

### What type of PR is it?
[Improvement]

### Todos
* [ ] - Code Review
* [ ] - Simple Test

### Is there a relevant Jira issue?
**[ZEPPELIN-699]**

### How should this be tested?
* `git fetch origin pull/746/head:ParagraphExecutionRESTAPI`
* `git checkout ParagraphExecutionRESTAPI`
* `mvn clean package -DskipTests`
* `bin/zeppelin-daemon.sh restart`
* Create a new note
* In the first paragraph, put the following code

```scala
%sh

echo "Current time = "`date +"%T"
```

* Retrieve the current note id in the URL
* Retrieve the current paragraph id
* Use a REST Client like **[POSTman]** to create a HTTP POST query 
`http://:/api/notebook/run//` 
* You should receive something similar as follow for answer

```
{
"status": "OK",
"body": {
"code": "SUCCESS",
"type": "TEXT",
"msg": "Current time = 16:14:18\n"
}
}
```

### Screenshots (if appropriate)

![zeppelin_synchronous_rest_api](https://cloud.githubusercontent.com/assets/1532977/15748069/b4a26a46-28dd-11e6-8f51-aa13ddba3f1c.gif)

API Documentation update

**Existing asynchronous API**

![image](https://cloud.githubusercontent.com/assets/1532977/15773274/5b508cae-2976-11e6-9e52-14d8b7e7828e.png)

**New synchronous API**

![image](https://cloud.githubusercontent.com/assets/1532977/15773309/84965a94-2976-11e6-9719-81d8b555c3c4.png)


### Questions:
* Does the licenses files need update? --> **No**
* Is there breaking changes for older versions? --> **No**
* Does this needs documentation? --> **Yes**

[ZEPPELIN-699]: https://issues.apache.org/jira/browse/ZEPPELIN-699
[POSTman]: https://www.getpostman.com/

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/doanduyhai/incubator-zeppelin ZEPPELIN-699

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/zeppelin/pull/746.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #746


commit 8367acfc0526965a884f58eea8f8ddda70d011d7
Author: DuyHai DOAN 
Date:   2016-06-02T14:15:46Z

[ZEPPELIN-699] Add new synchronous paragraph run REST API

commit fb0570ce70c47d3648ce4b9cc8265ebe51f6bdc8
Author: DuyHai DOAN 
Date:   2016-06-02T14:24:38Z

[ZEPPELIN-699] Update Notebook REST API documentation




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] zeppelin issue #1358: [ZEPPELIN-1365] Error of Zeppelin Application in devel...

[astroshim]

Github user astroshim commented on the issue:

https://github.com/apache/zeppelin/pull/1358
  
@AhyoungRyu Thank you for reviewing!!
Did you clear your browser cache?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] zeppelin issue #1302: [ZEPPELIN-1306] Interpreter restarts on a note.

[astroshim]

Github user astroshim commented on the issue:

https://github.com/apache/zeppelin/pull/1302
  
I just changed to 'icon'.
Thank you.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] zeppelin issue #1358: [ZEPPELIN-1365] Error of Zeppelin Application in devel...

[AhyoungRyu]

Github user AhyoungRyu commented on the issue:

https://github.com/apache/zeppelin/pull/1358
  
@astroshim Definitely in the latest master branch, I faced the below 
exception when I tried to run 
`org.apache.zeppelin.interpreter.dev.ZeppelinApplicationDevServer` in my IDE.
```
Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException: 0
at 
org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer.main(RemoteInterpreterServer.java:136)
```

Maybe that's why you created this patch. 

So I tried to test this PR with below steps
1. Build again with this patch: `mvn clean package -DskipTests`
2. Restart Zeppelin: `./bin/zeppelin-daemon.sh restart`
3. Run `org.apache.zeppelin.interpreter.dev.ZeppelinApplicationDevServer` 
in my IntelliJ -> Successfully started
4. Run `%spark new java.util.Date` -> Nothing happened. I can't see the 
above Helium icons. Only Spark repl result was printed like `res0: 
java.util.Date = Sun Aug 28 22:31:32 KST 2016`. (I'm using `Spark 2.0`.)

Did I miss something? ㅠ_ㅠ


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] zeppelin issue #1302: [ZEPPELIN-1306] Interpreter restarts on a note.

[AhyoungRyu]

Github user AhyoungRyu commented on the issue:

https://github.com/apache/zeppelin/pull/1302
  
@astroshim Great! Personally I would prefer "just icon" like the reload 
notes icon in Zeppelin home. 
But it's just my personal opinion. Final decision is up to you :)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] zeppelin pull request #746: [ZEPPELIN-699] Add new synchronous paragraph run...

[doanduyhai]

Github user doanduyhai closed the pull request at:

https://github.com/apache/zeppelin/pull/746


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] zeppelin issue #1369: [ZEPPELIN-1376]

[doanduyhai]

Github user doanduyhai commented on the issue:

https://github.com/apache/zeppelin/pull/1369
  
@felixcheung 

That's the main issue I'm facing now, how to **automate** test with 
corporate firewall ... I'm going to ask someone to test the this PR at his 
place (he has a corporate firewall)

But apart from manual testing the firewall proxy, I don't see an **easy** 
way to create integration test for this feature, unless we build a Docker image 
of Zeppelin with a Docker image of a HTTP Proxy server ...



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---