[GitHub] zeppelin issue #1215: ZEPPELIN-1225. Errors before the last shell command ar...

[zjffdu]

Github user zjffdu commented on the issue:

https://github.com/apache/zeppelin/pull/1215
  
@Leemoonsoo, mind to take look at it ? Thanks


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] zeppelin issue #1189: ZEPPELIN-1185. ZEPPELIN_INTP_JAVA_OPTS should not use ...

[zjffdu]

Github user zjffdu commented on the issue:

https://github.com/apache/zeppelin/pull/1189
  
@Leemoonsoo  I pushed another commit to add migration section in 
upgrade.md, please help review. Thanks.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] zeppelin pull request #1228: ZEPPELIN-1175. AM log is not available for yarn...

[zjffdu]

GitHub user zjffdu opened a pull request:

https://github.com/apache/zeppelin/pull/1228

ZEPPELIN-1175. AM log is not available for yarn-client mode

### What is this PR for?
For now, we share the same class path for zeppelin server and remote 
interpreter process. The cause the issue that AM log is not available for 
yarn-client mode because the yarn app also use the 
`ZEPPELIN_HOME/conf/log4j.properties` which is only for zeppelin server. So 
this PR just distinguish the CLASSPATH of zeppelin server and remote 
interpreter process. I use `ZEPPELIN_INTP_CLASSPATH` to represent the classpath 
of remote interpreter process and won't include 
`ZEPPELIN_HOME/conf/log4j.properties` in `ZEPPELIN_INTP_CLASSPATH`. 


### What type of PR is it?
[Improvement]

### Todos
* [ ] - Task

### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-1175

### How should this be tested?
Tested manually. 

### Screenshots (if appropriate)

### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? Yes, if user put custom 
config file (hive-site.xml) under ZEPPELIN_HOME/conf, it won't take effect 
after this PR 
* Does this needs documentation? Yes



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/zjffdu/zeppelin ZEPPELIN-1175

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/zeppelin/pull/1228.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1228


commit 0973477dcfc84a5121a1e83f6a3f835bfab6e7ea
Author: Jeff Zhang <zjf...@apache.org>
Date:   2016-07-26T02:41:50Z

ZEPPELIN-1175. AM log is not available for yarn-client mode




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] zeppelin issue #1189: ZEPPELIN-1185. ZEPPELIN_INTP_JAVA_OPTS should not use ...

[zjffdu]

Github user zjffdu commented on the issue:

https://github.com/apache/zeppelin/pull/1189
  
BTW, I will do the windows related change after we make consensus on this 
approach. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] zeppelin issue #1189: ZEPPELIN-1185. ZEPPELIN_INTP_JAVA_OPTS should not use ...

[zjffdu]

Github user zjffdu commented on the issue:

https://github.com/apache/zeppelin/pull/1189
  
Thanks @AhyoungRyu , I have updated the doc.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] zeppelin issue #1228: ZEPPELIN-1175. AM log is not available for yarn-client...

[zjffdu]

Github user zjffdu commented on the issue:

https://github.com/apache/zeppelin/pull/1228
  
@Leemoonsoo  
* `$ZEPPELIN_HOME/interpreter/` is not in the classpath, 
actually the jars under this folder are in the classpath. Although I can put 
all the files under this folder on the classpath to allow user to put custom 
config file there, I just feel it might not be the best approach as it would 
introduce other issues.
 - what config file we should use if user also specify `hive-site.xml` 
under `$SPARK_CONF_DIR` ? 
 - what property we should use if there's conflict between these 2 
`hive-site.xml` ? Do we allow override ?

So in this PR, I don't allow user to put custom config file under 
`$ZEPPELIN_HOME/interpreter/`, I would suggest user to put it 
outside ZEPPELIN ($SPARK_CONF_DIR or $SPARK_HOME/conf), seems we use this 
approach for hbase. 
* Windows related change and Migration doc are not done yet, will do that 
after we make consensus on this approach. 



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] zeppelin issue #1221: [MINOR] Change url in pom.xml files

[zjffdu]

Github user zjffdu commented on the issue:

https://github.com/apache/zeppelin/pull/1221
  
lgtm


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] zeppelin issue #1228: ZEPPELIN-1175. AM log is not available for yarn-client...

[zjffdu]

Github user zjffdu commented on the issue:

https://github.com/apache/zeppelin/pull/1228
  
Build fails due to some network issue. 
```
[ERROR] Failed to execute goal 
com.github.eirslett:frontend-maven-plugin:0.0.25:npm (npm install) on project 
zeppelin-web: Failed to run task: 'npm install --color=false' failed. (error 
code 1) -> [Help 1]
[ERROR] 
```


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] zeppelin pull request #1232: ZEPPELIN-1197. Should print output directly wit...

[zjffdu]

GitHub user zjffdu opened a pull request:

https://github.com/apache/zeppelin/pull/1232

ZEPPELIN-1197. Should print output directly without invoking function print 
in pyspark interpreter

### What is this PR for?
For now, user need to invoke print to make the output displayed on the 
notebook. This behavior is not natural and consistent with other notebooks. 
This PR is to make the pyspark interpreter in zeppelin behave the same as other 
notebook. 2 main changes 
* use single mode to compile the last statement, so that the evaluation 
result of the last statement will be printed to stdout, this is consistent with 
other notebooks (like jupyter)
* Make SparkOutputStream extends LogOutputStream so that we can see the 
output of inner process (Python/R), it is helpful for diagnosing. 

### What type of PR is it?
[Bug Fix]

### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-1197

### How should this be tested?
Tested it manually. Input the following text in pyspark paragraph,
```
1+1
sc.version
```
And get the following output
```
u'1.6.1'
```

### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? User don't need to call 
print explicitly. 
* Does this needs documentation? Yes



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/zjffdu/zeppelin ZEPPELIN-1197

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/zeppelin/pull/1232.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1232


commit 10182e695014911fd2bbdbbc9e59fc960d9c4b72
Author: Jeff Zhang <zjf...@apache.org>
Date:   2016-07-27T05:29:17Z

ZEPPELIN-1197. Should print output directly without invoking function print 
in pyspark interpreter




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] zeppelin issue #1206: ZEPPELIN-1199. Need to login using keytab and principa...

[zjffdu]

Github user zjffdu commented on the issue:

https://github.com/apache/zeppelin/pull/1206
  
@prabhjyotsingh Conflicts resolved. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] zeppelin issue #1237: ZEPPELIN-1242. Should set property SPARK_YARN_MODE and...

[zjffdu]

Github user zjffdu commented on the issue:

https://github.com/apache/zeppelin/pull/1237
  
@prabhjyotsingh  I close #1206, and merge it here. Please help review, 
thanks


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] zeppelin issue #1206: ZEPPELIN-1199. Need to login using keytab and principa...

[zjffdu]

Github user zjffdu commented on the issue:

https://github.com/apache/zeppelin/pull/1206
  
Close it and merge it to #1237


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] zeppelin issue #1237: ZEPPELIN-1242. Should set property SPARK_YARN_MODE and...

[zjffdu]

Github user zjffdu commented on the issue:

https://github.com/apache/zeppelin/pull/1237
  
Build fails due to network issue.
```
[INFO] 

[ERROR] Failed to execute goal 
org.apache.maven.plugins:maven-remote-resources-plugin:1.4:process (default) on 
project zeppelin: Error downloading resources archive. Could not transfer 
artifact 
org.apache.apache.resources:apache-jar-resource-bundle:jar:1.5-SNAPSHOT from/to 
codehaus-snapshots (https://nexus.codehaus.org/snapshots/): nexus.codehaus.org: 
Name or service not known
[ERROR] 
org.apache.apache.resources:apache-jar-resource-bundle:jar:1.5-SNAPSHOT
[ERROR] 
[ERROR] from the specified remote repositories:
[ERROR] central (http://repo.maven.apache.org/maven2, releases=true, 
snapshots=false),
[ERROR] sonatype (https://oss.sonatype.org/content/repositories/releases/, 
releases=true, snapshots=false),
[ERROR] sonatype-snapshots 
(https://oss.sonatype.org/content/repositories/snapshots/, releases=false, 
snapshots=true),
[ERROR] sonatype-apache (https://repository.apache.org/releases/, 
releases=true, snapshots=false),
[ERROR] apache-snapshots (https://repository.apache.org/snapshots/, 
releases=false, snapshots=true),
[ERROR] codehaus-snapshots (https://nexus.codehaus.org/snapshots/, 
releases=false, snapshots=true),
[ERROR] apache.snapshots (http://repository.apache.org/snapshots, 
releases=false, snapshots=true): Unknown host nexus.codehaus.org: Name or 
service not known
```


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] zeppelin issue #1163: [ZEPPELIN-1149] %sh interpreter kerberos support

[zjffdu]

Github user zjffdu commented on the issue:

https://github.com/apache/zeppelin/pull/1163
  
agree with @felixcheung, use the zeppelin service keytab/principal might 
bring some other potential security problems. I think user A would expect to 
run shell script as himself, but actually it run as user zeppelin. And seems 
all the users are sharing the same principal if I understand correctly. 

Besides that I have one question about shiro of zeppelin, does shiro will 
map the username to hadoop side ? Say I login zeppelin as user A, would that 
guaranteed there should also be a user A in the hadoop ecosystem?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] zeppelin issue #1163: [ZEPPELIN-1149] %sh interpreter kerberos support

[zjffdu]

Github user zjffdu commented on the issue:

https://github.com/apache/zeppelin/pull/1163
  
Sorry, I mistake it as hdfs interpreter. For shell interpreter, do we must 
have kerberos support ? I think here kinit is just for the hadoop command, so 
actually it is for running hdfs command in secured cluster rather than running 
general shell command in secured cluster. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] zeppelin issue #1163: [ZEPPELIN-1149] %sh interpreter kerberos support

[zjffdu]

Github user zjffdu commented on the issue:

https://github.com/apache/zeppelin/pull/1163
  
@prabhjyotsingh  Thanks for clarification. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] zeppelin issue #1215: ZEPPELIN-1225. Errors before the last shell command ar...

[zjffdu]

Github user zjffdu commented on the issue:

https://github.com/apache/zeppelin/pull/1215
  
Travis build fails due to network issue I think, 
```
[INFO] bower@1.7.2 node_modules/bower
[INFO] └── semver-utils@1.1.1
[ERROR] npm ERR! Linux 3.13.0-40-generic
[ERROR] npm ERR! argv 
"/home/travis/build/apache/zeppelin/zeppelin-web/node/node" 
"/home/travis/build/apache/zeppelin/zeppelin-web/node/node_modules/npm/bin/npm-cli.js"
 "install" "--color=false"
[ERROR] npm ERR! node v0.12.13
[ERROR] npm ERR! npm  v2.15.0
[ERROR] 
[ERROR] npm ERR! Callback called more than once.
[ERROR] npm ERR! 
[ERROR] npm ERR! If you need help, you may report this error at:
[ERROR] npm ERR! <https://github.com/npm/npm/issues>
[ERROR] 
[ERROR] npm ERR! Please include the following file with any support request:
[ERROR] npm ERR! 
/home/travis/build/apache/zeppelin/zeppelin-web/npm-debug.log
```


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] zeppelin issue #1215: ZEPPELIN-1225. Errors before the last shell command ar...

[zjffdu]

Github user zjffdu commented on the issue:

https://github.com/apache/zeppelin/pull/1215
  
After checking ShellInterpreter.java, I found other 2 issues. 
* Should remove entry from executors after the shell execution is done 
otherwise it would accumulate more and more. 
* executors should be ConcurrentHashMap as it would be accessed in 2 
threads (interpret & cancel will be in 2 threads if I understand correctly) 

I push another commit to fix these 2 issues, let me know if it is better to 
do it another ticket. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] zeppelin pull request #1215: ZEPPELIN-1225. Errors before the last shell com...

[zjffdu]

GitHub user zjffdu opened a pull request:

https://github.com/apache/zeppelin/pull/1215

ZEPPELIN-1225. Errors before the last shell command are ignored

### What is this PR for?
The problem is that command "bash -c " will always return 0 
as long as the last line of shell script run correctly. e.g the following 
command will run correctly without any error message.
```
hello
pwd
``` 
This PR will redirect stderr and stdout to the same place, and will display 
both the stderr and stdout to frontend just like what user see in the native 
shell terminal. So the output of above command will be as following
```
bash: hello: command not found
/Users/jzhang/github/zeppelin
```

### What type of PR is it?
[Bug Fix]


### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-1225

### How should this be tested?
Unit test is added and also manually verify it on zeppelin notebook. 

### Screenshots (if appropriate)

### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/zjffdu/incubator-zeppelin ZEPPELIN-1225

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/zeppelin/pull/1215.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1215


commit 0266c719c2481a30817b36a1508eab62f868263f
Author: Jeff Zhang <zjf...@apache.org>
Date:   2016-07-22T05:02:30Z

ZEPPELIN-1225. Errors before the last shell command are ignored




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] zeppelin issue #1232: ZEPPELIN-1197. Should print output directly without in...

[zjffdu]

Github user zjffdu commented on the issue:

https://github.com/apache/zeppelin/pull/1232
  
@Leemoonsoo  Thanks for the careful checking. I compare it with jupyter, 
only the second case is different. Let me investigate how to fix it. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] zeppelin issue #1232: ZEPPELIN-1197. Should print output directly without in...

[zjffdu]

Github user zjffdu commented on the issue:

https://github.com/apache/zeppelin/pull/1232
  
I also compare it with native python repl, the second case is consistent. 
So I think this behvior is fine, although it is different from jyputer. 

![2016-07-29_0746](https://cloud.githubusercontent.com/assets/164491/17233269/06ebec6e-5561-11e6-8d18-41b69e54fa44.png)





---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] zeppelin issue #1245: ZEPPELIN-1225. Errors before the last shell command ar...

[zjffdu]

Github user zjffdu commented on the issue:

https://github.com/apache/zeppelin/pull/1245
  
@Leemoonsoo this is for branch-0.6, the test fail seems irrelevant. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] zeppelin issue #1241: [ZEPPELIN-1217] Remove horizontal scrollbar in Zeppeli...

[zjffdu]

Github user zjffdu commented on the issue:

https://github.com/apache/zeppelin/pull/1241
  
yes, I run it under `ZEPPELIN_HOME/docs`


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] zeppelin issue #1241: [ZEPPELIN-1217] Remove horizontal scrollbar in Zeppeli...

[zjffdu]

Github user zjffdu commented on the issue:

https://github.com/apache/zeppelin/pull/1241
  
@AhyoungRyu I follow the instruction here to build the doc 
https://github.com/apache/zeppelin/blob/master/docs/README.md#build-documentation,
 but fails. Here's the error after `bundle install`
```
Using bundler 1.12.5
An error occurred while installing RedCloth (4.2.9), and Bundler cannot 
continue.
Make sure that `gem install RedCloth -v '4.2.9'` succeeds before bundling.
```

I ran it on mac os 10.11, and also did `xcode-select --install`,  do you 
know how can I fix this ? Thanks


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] zeppelin issue #1241: [ZEPPELIN-1217] Remove horizontal scrollbar in Zeppeli...

[zjffdu]

Github user zjffdu commented on the issue:

https://github.com/apache/zeppelin/pull/1241
  
Thanks @AhyoungRyu , the new UI lgtm.  BTW, this link helps me in case 
someone else need this,
http://pragmaticstudio.com/blog/2010/9/23/install-rails-ruby-mac


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] zeppelin issue #1255: Adding extra libraries to livy

[zjffdu]

Github user zjffdu commented on the issue:

https://github.com/apache/zeppelin/pull/1255
  
I would not recommend user to do that to add extra library. Because most of 
time zeppelin user don't know the cluster details like which extra library are 
located in the cluster machines. I would suggest livy to support dynamic 
library loading as native spark interpreter.  


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] zeppelin pull request #1260: ZEPPELIN-1267. PySparkInterpreter doesn't work ...

[zjffdu]

GitHub user zjffdu opened a pull request:

https://github.com/apache/zeppelin/pull/1260

ZEPPELIN-1267. PySparkInterpreter doesn't work in spark 2.0

### What is this PR for?
PySparkInterpreter doesn't work in spark 2.0 because pyspark and py4j is 
not distributed to executors.  This PR extract the setup staff for pyspark 
interpreter into method setupConfForPySpark and use it for both spark1 and 
spark2. But this is just a short term solution, as I think this should be 
handled by spark rather than zeppelin, here zeppelin duplicate part of spark's 
work. In the long term, I'd like to resolve it in `ZEPPELIN-1263`.


### What type of PR is it?
[Bug Fix]

### Todos
* https://issues.apache.org/jira/browse/ZEPPELIN-1263

### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-1267

### How should this be tested?
Verify it manually. 

### Screenshots (if appropriate)

### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/zjffdu/zeppelin ZEPPELIN-1267

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/zeppelin/pull/1260.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1260


commit 81d1d56c94934dd6c4b32144d810626356ce3c0e
Author: Jeff Zhang <zjf...@apache.org>
Date:   2016-08-02T09:41:21Z

ZEPPELIN-1267. PySparkInterpreter doesn't work in spark 2.0




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] zeppelin issue #1255: Adding extra libraries to livy

[zjffdu]

Github user zjffdu commented on the issue:

https://github.com/apache/zeppelin/pull/1255
  
You can use --conf spark.jars= and --conf 
spark.jars.packages= instead 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] zeppelin issue #1268: minor doc fix for r.md

[zjffdu]

Github user zjffdu commented on the issue:

https://github.com/apache/zeppelin/pull/1268
  
@AhyoungRyu  Please help review. Thanks


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] zeppelin issue #1269: ZEPPELIN-1270. Remove getting SQLContext from SparkSes...

[zjffdu]

Github user zjffdu commented on the issue:

https://github.com/apache/zeppelin/pull/1269
  
\cc @Leemoonsoo  Please help review, thanks


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] zeppelin pull request #1269: ZEPPELIN-1270. Remove getting SQLContext from S...

[zjffdu]

GitHub user zjffdu opened a pull request:

https://github.com/apache/zeppelin/pull/1269

ZEPPELIN-1270. Remove getting SQLContext from SparkSession.wrapped()

### What is this PR for?
SparkSession.wrapped is only for spark 2.0 preview, it is not supported in 
spark 2.0 release. So I think we can remove that piece of code. Otherwise we 
will get the following error in log which might be a little confusing.
```
08:05:44,946 - Thread(pool-2-thread-3) - (Utils.java:40) - 
org.apache.spark.sql.SparkSession.wrapped()
java.lang.NoSuchMethodException: org.apache.spark.sql.SparkSession.wrapped()
at java.lang.Class.getMethod(Class.java:1786)
at org.apache.zeppelin.spark.Utils.invokeMethod(Utils.java:38)
at org.apache.zeppelin.spark.Utils.invokeMethod(Utils.java:33)
```


### What type of PR is it?
[Improvement]

### Todos
* [ ] - Task

### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-1270

### How should this be tested?
Check the log and the above error is gone. 

### Screenshots (if appropriate)

### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/zjffdu/zeppelin ZEPPELIN-1270

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/zeppelin/pull/1269.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1269


commit 4401561ca852654fecc0affa5617816517402641
Author: Jeff Zhang <zjf...@apache.org>
Date:   2016-08-03T00:20:03Z

ZEPPELIN-1270. Remove getting SQLContext from SparkSession.wrapped()




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] zeppelin issue #1255: Adding extra libraries to livy

[zjffdu]

Github user zjffdu commented on the issue:

https://github.com/apache/zeppelin/pull/1255
  
@jimdowling  Right, there's no ZeppelinContext in livy right now. Would 
allowing user to specify jars/packages when creating livy session more general 
? (like --jars/--packages in spark-submit)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] zeppelin issue #1265: [ZEPPELIN-1210] Run interpreter per user

[zjffdu]

Github user zjffdu commented on the issue:

https://github.com/apache/zeppelin/pull/1265
  
@jongyoul  Does that mean the login user is the process owner ? Because it 
matters for security reason. 
e.g. 
* For spark interpreter, it would be better to launch the yarn app as the 
user who login in zeppelin.  
* For shell interpreter, it would be better to launch the shell as the user 
who login in zeppelin. Otherwise it is pretty dangerous. The user can delete 
all the files owner by others. 
 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] zeppelin issue #1316: [ZEPPELIN-1118] Migrate Zeppelin Live Demo feature fro...

[zjffdu]

Github user zjffdu commented on the issue:

https://github.com/apache/zeppelin/pull/1316
  
One suggestion is asking the 3rd party to specify their zeppelin version 
for this demo. If they update the demo, they should also update the version 
accordingly. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] zeppelin pull request #1278: ZEPPELIN-1287. No need to call print to display...

[zjffdu]

GitHub user zjffdu opened a pull request:

https://github.com/apache/zeppelin/pull/1278

ZEPPELIN-1287. No need to call print to display output in PythonInter…

### What is this PR for?
It is not necessary to call print to display output in PythonInterpreter. 2 
main changes:
* the root cause is the displayhook in bootstrap.py
* also did some code refactoring on PythonInterpreter


### What type of PR is it?
[Bug Fix]

### Todos
* [ ] - Task

### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-1287

### How should this be tested?
Verify it manually

### Screenshots (if appropriate)

![2016-08-04_1404](https://cloud.githubusercontent.com/assets/164491/17392006/090279d2-5a4d-11e6-840b-4cddb595a42e.png)

### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No

…preter

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/zjffdu/zeppelin ZEPPELIN-1287

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/zeppelin/pull/1278.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1278


commit f8e52b47b01f7021abe11d6dec65073a60129061
Author: Jeff Zhang <zjf...@apache.org>
Date:   2016-08-04T06:05:22Z

ZEPPELIN-1287. No need to call print to display output in PythonInterpreter




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] zeppelin issue #1278: ZEPPELIN-1287. No need to call print to display output...

[zjffdu]

Github user zjffdu commented on the issue:

https://github.com/apache/zeppelin/pull/1278
  
\\cc @bzz @Leemoonsoo  ready for review,  not sure whether the test fail is 
relevant. CI is very unstable recently, we might need to fix it asap :smiley:. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] zeppelin issue #1260: ZEPPELIN-1267. PySparkInterpreter doesn't work in spar...

[zjffdu]

Github user zjffdu commented on the issue:

https://github.com/apache/zeppelin/pull/1260
  
I also use the latest master and build zeppelin with this command: 
```mvn clean package -Pspark-2.0 -Ppyspark -Psparkr -DskipTests
```, and then export `SPARK_HOME` to where spark-2.0 is located and run 
pyspark interpreter using `yarn-client` mode.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] zeppelin issue #1189: ZEPPELIN-1185. ZEPPELIN_INTP_JAVA_OPTS should not use ...

[zjffdu]

Github user zjffdu commented on the issue:

https://github.com/apache/zeppelin/pull/1189
  
\\cc @Leemoonsoo  Ready for review. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] zeppelin issue #1301: [MINOR] Removed unused profiles from spark/pom.xml

[zjffdu]

Github user zjffdu commented on the issue:

https://github.com/apache/zeppelin/pull/1301
  
Definitely should remove profile yarn. I got classpath issue when enabling 
yarn profile. 

And we need to update the docs accordingly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] zeppelin pull request #1212: ZEPPELIN-1222. ClassNotFoundException of SparkJ...

[zjffdu]

GitHub user zjffdu opened a pull request:

https://github.com/apache/zeppelin/pull/1212

ZEPPELIN-1222. ClassNotFoundException of SparkJLineCompletion in Spark 
Interpreter

### What is this PR for?

ClassNotFoundException happens because not fully qualified class name is 
specified. Specify the fully qualified class name in this PR, and remove method 
findClass in SparkInterpter, use findClass in Utils instead. 

### What type of PR is it?
[Bug Fix]

### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-1222

### How should this be tested?
Manually verified, restart zeppelin server and spark interpreter, this 
issue is gone. 

### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/zjffdu/incubator-zeppelin SPARK-1222

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/zeppelin/pull/1212.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1212


commit bbe42a6a3c57c5d608f9954a551f351cbffa6f64
Author: Jeff Zhang <zjf...@apache.org>
Date:   2016-07-21T03:17:06Z

ZEPPELIN-1222. ClassNotFoundException of SparkJLineCompletion in Spark 
Interpreter




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] zeppelin issue #1212: ZEPPELIN-1222. ClassNotFoundException of SparkJLineCom...

[zjffdu]

Github user zjffdu commented on the issue:

https://github.com/apache/zeppelin/pull/1212
  
@Leemoonsoo  Could you help take a look at it ? Thanks


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] zeppelin pull request #1206: ZEPPELIN-1215. Need to login using keytab and p...

[zjffdu]

GitHub user zjffdu opened a pull request:

https://github.com/apache/zeppelin/pull/1206

ZEPPELIN-1215. Need to login using keytab and principal before creating 
SparkContext in secured cluster

### What is this PR for?
Need to login using keytab and principal before creating SparkContext in 
secured cluster. 


### What type of PR is it?
[Bug Fix]


### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-1215

### How should this be tested?
Manually verified in a secured cluster. 

### Screenshots (if appropriate)

### Questions:
* Does the licenses files need update? no
* Is there breaking changes for older versions? no
* Does this needs documentation? no


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/zjffdu/incubator-zeppelin ZEPPELIN-1215

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/zeppelin/pull/1206.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1206


commit bd4a3d7f657f0b2b46f9f7a0e52c24aa0395be20
Author: Jeff Zhang <zjf...@apache.org>
Date:   2016-07-20T04:18:30Z

ZEPPELIN-1215. Need to login using keytab and principal before creating 
SparkContext in secured cluster




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] zeppelin issue #1206: ZEPPELIN-1199. Need to login using keytab and principa...

[zjffdu]

Github user zjffdu commented on the issue:

https://github.com/apache/zeppelin/pull/1206
  
pyspark and sprakr both create SparkContext, so this fix also works in 
pyspark & sparkr


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] zeppelin issue #1206: ZEPPELIN-1215. Need to login using keytab and principa...

[zjffdu]

Github user zjffdu commented on the issue:

https://github.com/apache/zeppelin/pull/1206
  
Might need to check whether it works in pyspark/sparkr. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] zeppelin issue #1184: [ZEPPELIN-1159] Livy interpreter gets "404 not found" ...

[zjffdu]

Github user zjffdu commented on the issue:

https://github.com/apache/zeppelin/pull/1184
  
@prabhjyotsingh  
```this is result in proper error message.```
What does this mean ? Is the behavior livy interpreter give proper error 
message to client or just recreate a new session ?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] zeppelin pull request #1189: ZEPPELIN-1185. ZEPPELIN_INTP_JAVA_OPTS should n...

[zjffdu]

GitHub user zjffdu opened a pull request:

https://github.com/apache/zeppelin/pull/1189

ZEPPELIN-1185. ZEPPELIN_INTP_JAVA_OPTS should not use ZEPPELIN_JAVA_OPTS

### What is this PR for?

Don't use ZEPPELIN_JAVA_OPTS as the default value of ZEPPELIN_INTP_JAVA_OPTS

### What type of PR is it?
Improvement

### Todos
* [ ] - Task

### What is the Jira issue?
https://issues.apache.org/jira/browse/ZEPPELIN-1185

### How should this be tested?
Tested manually.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/zjffdu/incubator-zeppelin ZEPPELIN-1185

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/zeppelin/pull/1189.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1189


commit a54d843979a7a7c843394d56b1678c0380ffa048
Author: Jeff Zhang <zjf...@apache.org>
Date:   2016-07-15T06:21:21Z

ZEPPELIN-1185. ZEPPELIN_INTP_JAVA_OPTS should not use ZEPPELIN_JAVA_OPTS as 
default value




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] zeppelin pull request #1190: ZEPPELIN-1187. Redirect output of the process r...

[zjffdu]

GitHub user zjffdu opened a pull request:

https://github.com/apache/zeppelin/pull/1190

ZEPPELIN-1187. Redirect output of the process running interpreter.sh to 
log4j

### What is this PR for?
For now the output of process running interpter.sh is lost, it would be 
nice to redirect it to log4j for debugging, especially debugging related with 
interpreter.sh

### What type of PR is it?
Improvement


### What is the Jira issue?
https://issues.apache.org/jira/browse/ZEPPELIN-1187

### How should this be tested?
Tested manully, here's the output in the log file of the process running 
interpreter.sh

```
INFO [2016-07-15 15:30:10,537] ({Exec Stream Pumper} 
RemoteInterpreterManagedProcess.java[processLine]:179) - Spark Command: 
/Library/Java/JavaVirtualMachines/jdk1.8.0_45.jdk/Contents/Home/bin/java -cp 
/Users/jzhang/github/zeppelin/interpreter/spark/*:/Users/jzhang/github/zeppelin/zeppelin-interpreter/target/lib/*:/Users/jzhang/github/zeppelin/conf/:/Users/jzhang/github/zeppelin/conf/:/Users/jzhang/github/zeppelin/zeppelin-interpreter/target/classes/Users/jzhang/github/zeppelin/interpreter/spark/zeppelin-spark-0.7.0-SNAPSHOT.jar:/Users/jzhang/Java/lib/spark-1.6.1/conf/:/Users/jzhang/Java/lib/spark-1.6.1/assembly/target/scala-2.10/spark-assembly-1.6.1-hadoop2.6.0.jar:/Users/jzhang/Java/lib/spark-1.6.1/lib_managed/jars/datanucleus-api-jdo-3.2.6.jar:/Users/jzhang/Java/lib/spark-1.6.1/lib_managed/jars/datanucleus-core-3.2.10.jar:/Users/jzhang/Java/lib/spark-1.6.1/lib_managed/jars/datanucleus-rdbms-3.2.9.jar:/Users/jzhang/Java/lib/hadoop-2.7.2/etc/hadoop/
 -Xms1g -Xmx1g -Dfile.encoding
 =UTF-8 
-Dlog4j.configuration=file:///Users/jzhang/github/zeppelin/conf/log4j.properties
 
-Dzeppelin.log.file=/Users/jzhang/github/zeppelin/logs/zeppelin-interpreter-spark-jzhang-jzhangMBPr.local.log
 org.apache.spark.deploy.SparkSubmit --conf 
spark.driver.extraClassPath=::/Users/jzhang/github/zeppelin/interpreter/spark/*:/Users/jzhang/github/zeppelin/zeppelin-interpreter/target/lib/*::/Users/jzhang/github/zeppelin/conf:/Users/jzhang/github/zeppelin/conf:/Users/jzhang/github/zeppelin/zeppelin-interpreter/target/classes/Users/jzhang/github/zeppelin/interpreter/spark/zeppelin-spark-0.7.0-SNAPSHOT.jar
 --conf spark.driver.extraJavaOptions= -Dfile.encoding=UTF-8 
-Dlog4j.configuration=file:///Users/jzhang/github/zeppelin/conf/log4j.properties
 
-Dzeppelin.log.file=/Users/jzhang/github/zeppelin/logs/zeppelin-interpreter-spark-jzhang-jzhangMBPr.local.log
 --class org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer 
/Users/jzhang/github/zeppelin/interpreter/spark/zeppelin-spark-0.7.0-SNA
 PSHOT.jar 55592
 INFO [2016-07-15 15:30:10,537] ({Exec Stream Pumper} 
RemoteInterpreterManagedProcess.java[processLine]:179) - 

 INFO [2016-07-15 15:30:11,058] ({Exec Stream Pumper} 
RemoteInterpreterManagedProcess.java[processLine]:179) - SLF4J: Class path 
contains multiple SLF4J bindings.
```


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/zjffdu/incubator-zeppelin ZEPPELIN-1187

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/zeppelin/pull/1190.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1190


commit 977f42610f1dd0e556b4e3e63d1ccf3a6387dd89
Author: Jeff Zhang <zjf...@apache.org>
Date:   2016-07-15T07:36:22Z

ZEPPELIN-1187. Redirect output of the process running interpreter.sh to 
log4j




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] zeppelin issue #1190: ZEPPELIN-1187. Redirect output of the process running ...

[zjffdu]

Github user zjffdu commented on the issue:

https://github.com/apache/zeppelin/pull/1190
  
\cc @Leemoonsoo  Could you help take look at this PR ? Since you are the 
original author of this file. Thanks.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] zeppelin issue #1116: minor doc fix

[zjffdu]

Github user zjffdu commented on the issue:

https://github.com/apache/zeppelin/pull/1116
  
right, I miss the newbie label. Add it in the next commit.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] zeppelin pull request #1189: ZEPPELIN-1185. ZEPPELIN_INTP_JAVA_OPTS should n...

[zjffdu]

GitHub user zjffdu reopened a pull request:

https://github.com/apache/zeppelin/pull/1189

ZEPPELIN-1185. ZEPPELIN_INTP_JAVA_OPTS should not use ZEPPELIN_JAVA_OPTS

### What is this PR for?

Don't use ZEPPELIN_JAVA_OPTS as the default value of ZEPPELIN_INTP_JAVA_OPTS

### What type of PR is it?
Improvement

### What is the Jira issue?
https://issues.apache.org/jira/browse/ZEPPELIN-1185

### How should this be tested?
Tested manually. By exporting the following variable, I can debug zeppelin 
server correctly and remote interpreter process can ran successfully. (Before 
this PR, the remote  interpreter process will fail to launch because it would 
also listen the same debug port)
```
export 
ZEPPELIN_JAVA_OPTS="-agentlib:jdwp=transport=dt_socket,server=y,suspend=y,address=5005"
```

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/zjffdu/zeppelin ZEPPELIN-1185

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/zeppelin/pull/1189.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1189


commit ef5a36097e0b43bb08b0e4a3659e41d849039f34
Author: Jeff Zhang <zjf...@apache.org>
Date:   2016-07-15T06:21:21Z

ZEPPELIN-1185. ZEPPELIN_INTP_JAVA_OPTS should not use ZEPPELIN_JAVA_OPTS as 
default value

commit e82d88986e2ff7cddec0cb2c28f0f874dd07ad83
Author: Jeff Zhang <zjf...@apache.org>
Date:   2016-07-26T01:14:24Z

add migration doc

commit 3ff5561e4fde109e40e28faee41711e960803586
Author: Jeff Zhang <zjf...@apache.org>
Date:   2016-07-26T03:07:49Z

update doc format

commit 9e48ad75c64b1efa4ad310e44dbfa48e225efab0
Author: Jeff Zhang <zjf...@apache.org>
Date:   2016-08-04T10:16:56Z

change for windows




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] zeppelin issue #1269: ZEPPELIN-1270. Remove getting SQLContext from SparkSes...

[zjffdu]

Github user zjffdu commented on the issue:

https://github.com/apache/zeppelin/pull/1269
  
@Leemoonsoo  PR rebased.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] zeppelin pull request #1189: ZEPPELIN-1185. ZEPPELIN_INTP_JAVA_OPTS should n...

[zjffdu]

Github user zjffdu closed the pull request at:

https://github.com/apache/zeppelin/pull/1189


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] zeppelin issue #1300: ZEPPELIN-1305. Fix bug of ZEPPELIN-1215

[zjffdu]

Github user zjffdu commented on the issue:

https://github.com/apache/zeppelin/pull/1300
  
\cc @prabhjyotsingh 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] zeppelin issue #1307: ZEPPELIN-1311. Typo in ZEPPELIN-1197

[zjffdu]

Github user zjffdu commented on the issue:

https://github.com/apache/zeppelin/pull/1307
  
\cc @Leemoonsoo @minahlee Is it possible to merge it into 0.6.1 ?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] zeppelin issue #1307: ZEPPELIN-1311. Typo in ZEPPELIN-1197

[zjffdu]

Github user zjffdu commented on the issue:

https://github.com/apache/zeppelin/pull/1307
  
Thanks @minahlee, the CI fail should be irrelevant, it is due to network 
issue.
```
[INFO] └── semver-utils@1.1.1
[ERROR] npm ERR! Linux 3.13.0-40-generic
[ERROR] npm ERR! argv 
"/home/travis/build/apache/zeppelin/zeppelin-web/node/node" 
"/home/travis/build/apache/zeppelin/zeppelin-web/node/node_modules/npm/bin/npm-cli.js"
 "install" "--color=false"
[ERROR] npm ERR! node v0.12.13
[ERROR] npm ERR! npm  v2.15.0
[ERROR] 
[ERROR] npm ERR! Callback called more than once.
[ERROR] npm ERR! 
[ERROR] npm ERR! If you need help, you may report this error at:
[ERROR] npm ERR! <https://github.com/npm/npm/issues>
[ERROR] 
[ERROR] npm ERR! Please include the following file with any support request:
[ERROR] npm ERR! 
/home/travis/build/apache/zeppelin/zeppelin-web/npm-debug.log
```


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] zeppelin issue #1307: ZEPPELIN-1311. Typo in ZEPPELIN-1197

[zjffdu]

Github user zjffdu commented on the issue:

https://github.com/apache/zeppelin/pull/1307
  
@minahlee  Please hold the commit, I will have another improvement soon.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] zeppelin issue #1307: ZEPPELIN-1311. Typo in ZEPPELIN-1197

[zjffdu]

Github user zjffdu commented on the issue:

https://github.com/apache/zeppelin/pull/1307
  
oh, sorry, I didn't realize `ZEPPELIN-1197` is not in 0.6. Then let's merge 
this to master only. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] zeppelin pull request #1189: ZEPPELIN-1185. ZEPPELIN_INTP_JAVA_OPTS should n...

[zjffdu]

Github user zjffdu closed the pull request at:

https://github.com/apache/zeppelin/pull/1189


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] zeppelin pull request #1189: ZEPPELIN-1185. ZEPPELIN_INTP_JAVA_OPTS should n...

[zjffdu]

GitHub user zjffdu reopened a pull request:

https://github.com/apache/zeppelin/pull/1189

ZEPPELIN-1185. ZEPPELIN_INTP_JAVA_OPTS should not use ZEPPELIN_JAVA_OPTS

### What is this PR for?

Don't use ZEPPELIN_JAVA_OPTS as the default value of ZEPPELIN_INTP_JAVA_OPTS

### What type of PR is it?
Improvement

### What is the Jira issue?
https://issues.apache.org/jira/browse/ZEPPELIN-1185

### How should this be tested?
Tested manually. By exporting the following variable, I can debug zeppelin 
server correctly and remote interpreter process can ran successfully. (Before 
this PR, the remote  interpreter process will fail to launch because it would 
also listen the same debug port)
```
export 
ZEPPELIN_JAVA_OPTS="-agentlib:jdwp=transport=dt_socket,server=y,suspend=y,address=5005"
```

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/zjffdu/zeppelin ZEPPELIN-1185

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/zeppelin/pull/1189.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1189


commit ef5a36097e0b43bb08b0e4a3659e41d849039f34
Author: Jeff Zhang <zjf...@apache.org>
Date:   2016-07-15T06:21:21Z

ZEPPELIN-1185. ZEPPELIN_INTP_JAVA_OPTS should not use ZEPPELIN_JAVA_OPTS as 
default value

commit e82d88986e2ff7cddec0cb2c28f0f874dd07ad83
Author: Jeff Zhang <zjf...@apache.org>
Date:   2016-07-26T01:14:24Z

add migration doc

commit 3ff5561e4fde109e40e28faee41711e960803586
Author: Jeff Zhang <zjf...@apache.org>
Date:   2016-07-26T03:07:49Z

update doc format

commit 9e48ad75c64b1efa4ad310e44dbfa48e225efab0
Author: Jeff Zhang <zjf...@apache.org>
Date:   2016-08-04T10:16:56Z

change for windows




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] zeppelin pull request #1278: ZEPPELIN-1287. No need to call print to display...

[zjffdu]

GitHub user zjffdu reopened a pull request:

https://github.com/apache/zeppelin/pull/1278

ZEPPELIN-1287. No need to call print to display output in PythonInterpreter

### What is this PR for?
It is not necessary to call print to display output in PythonInterpreter. 2 
main changes:
* the root cause is the displayhook in bootstrap.py
* also did some code refactoring on PythonInterpreter


### What type of PR is it?
[Bug Fix]

### Todos
* [ ] - Task

### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-1287

### How should this be tested?
Verify it manually

### Screenshots (if appropriate)

![2016-08-04_1404](https://cloud.githubusercontent.com/assets/164491/17392006/090279d2-5a4d-11e6-840b-4cddb595a42e.png)

### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/zjffdu/zeppelin ZEPPELIN-1287

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/zeppelin/pull/1278.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1278


commit 0eade713f1609c77518e19bec2c8750bc3447367
Author: Jeff Zhang <zjf...@apache.org>
Date:   2016-08-04T06:05:22Z

ZEPPELIN-1287. No need to call print to display output in PythonInterpreter

commit 3e9f1699ee77b1a3b38ee5fef168d82d470831af
Author: Jeff Zhang <zjf...@apache.org>
Date:   2016-08-04T12:19:41Z

address comments

commit b48b56fbe68dfd3e3a20a71babac72ada52c02d8
Author: Jeff Zhang <zjf...@apache.org>
Date:   2016-08-08T00:40:46Z

fix unit test fail




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] zeppelin pull request #1278: ZEPPELIN-1287. No need to call print to display...

[zjffdu]

Github user zjffdu closed the pull request at:

https://github.com/apache/zeppelin/pull/1278


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] zeppelin pull request #1189: ZEPPELIN-1185. ZEPPELIN_INTP_JAVA_OPTS should n...

[zjffdu]

GitHub user zjffdu reopened a pull request:

https://github.com/apache/zeppelin/pull/1189

ZEPPELIN-1185. ZEPPELIN_INTP_JAVA_OPTS should not use ZEPPELIN_JAVA_OPTS

### What is this PR for?

Don't use ZEPPELIN_JAVA_OPTS as the default value of ZEPPELIN_INTP_JAVA_OPTS

### What type of PR is it?
Improvement

### What is the Jira issue?
https://issues.apache.org/jira/browse/ZEPPELIN-1185

### How should this be tested?
Tested manually. By exporting the following variable, I can debug zeppelin 
server correctly and remote interpreter process can ran successfully. (Before 
this PR, the remote  interpreter process will fail to launch because it would 
also listen the same debug port)
```
export 
ZEPPELIN_JAVA_OPTS="-agentlib:jdwp=transport=dt_socket,server=y,suspend=y,address=5005"
```

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/zjffdu/zeppelin ZEPPELIN-1185

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/zeppelin/pull/1189.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1189


commit ef5a36097e0b43bb08b0e4a3659e41d849039f34
Author: Jeff Zhang <zjf...@apache.org>
Date:   2016-07-15T06:21:21Z

ZEPPELIN-1185. ZEPPELIN_INTP_JAVA_OPTS should not use ZEPPELIN_JAVA_OPTS as 
default value

commit e82d88986e2ff7cddec0cb2c28f0f874dd07ad83
Author: Jeff Zhang <zjf...@apache.org>
Date:   2016-07-26T01:14:24Z

add migration doc

commit 3ff5561e4fde109e40e28faee41711e960803586
Author: Jeff Zhang <zjf...@apache.org>
Date:   2016-07-26T03:07:49Z

update doc format

commit 9e48ad75c64b1efa4ad310e44dbfa48e225efab0
Author: Jeff Zhang <zjf...@apache.org>
Date:   2016-08-04T10:16:56Z

change for windows




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] zeppelin pull request #1189: ZEPPELIN-1185. ZEPPELIN_INTP_JAVA_OPTS should n...

[zjffdu]

Github user zjffdu closed the pull request at:

https://github.com/apache/zeppelin/pull/1189


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] zeppelin issue #673: [Zeppelin-630] Introduce new way of dependency loading ...

[zjffdu]

Github user zjffdu commented on the issue:

https://github.com/apache/zeppelin/pull/673
  
Thanks @minahlee, it works. But any reason to make it hidden by default ? 
But it is not intuitive for me that I can click the gear button to display it. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] zeppelin issue #1255: Adding extra libraries to livy

[zjffdu]

Github user zjffdu commented on the issue:

https://github.com/apache/zeppelin/pull/1255
  
No, I mean add interpreter property to allow user to specify jars/packages. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] zeppelin issue #1189: ZEPPELIN-1185. ZEPPELIN_INTP_JAVA_OPTS should not use ...

[zjffdu]

Github user zjffdu commented on the issue:

https://github.com/apache/zeppelin/pull/1189
  
ping @Leemoonsoo  in case you miss it. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] zeppelin issue #1265: [ZEPPELIN-1210] Run interpreter per user

[zjffdu]

Github user zjffdu commented on the issue:

https://github.com/apache/zeppelin/pull/1265
  
@sourav-mazumder 's suggestion is about another point of multiple user 
support, this is might not be a trivial task to implement in this PR.  I think 
we have a discussion about the multiple-user support for zeppelin in the mail 
list recently.  There's lots of works to do, I will create a umbrella ticket 
for that so that we can have a more clear whole picture of that. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] zeppelin issue #1318: [ZEPPELIN-1280][Spark on Yarn] Documents for running z...

[zjffdu]

Github user zjffdu commented on the issue:

https://github.com/apache/zeppelin/pull/1318
  
@astroshim I can use spark 2.0 and hadoop 2.7 successfully. I hit this 
issue when building zeppelin with profile yarn enabled. So please don't enable 
yarn profile otherwise you will get hadoop version mismatch. I have left a 
comment in #1301 to remove yarn profile.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] zeppelin issue #1265: [ZEPPELIN-1210] Run interpreter per user

[zjffdu]

Github user zjffdu commented on the issue:

https://github.com/apache/zeppelin/pull/1265
  
Create umbrella ticket for multiple user support.
https://issues.apache.org/jira/browse/ZEPPELIN-1337


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] zeppelin issue #1960: [ZEPPELIN-2030] Select statement doesn't work in LivyS...

[zjffdu]

Github user zjffdu commented on the issue:

https://github.com/apache/zeppelin/pull/1960
  
Yes, the PR is ready. CI failure is not relevant.  Please help include it 
in 0.7


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] zeppelin issue #1970: ZEPPELIN-2045. Pass interpreter properties with "spark...

[zjffdu]

Github user zjffdu commented on the issue:

https://github.com/apache/zeppelin/pull/1970
  
@Leemoonsoo @felixcheung Please help review. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] zeppelin issue #1960: [ZEPPELIN-2030] Select statement doesn't work in LivyS...

[zjffdu]

Github user zjffdu commented on the issue:

https://github.com/apache/zeppelin/pull/1960
  
@del680202 Can you try this on `%spark.sql` as well ? I guess `Output 
exceeds...` may also cause issues in displaying table format data in 
`%spark.sql`


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] zeppelin issue #1960: [ZEPPELIN-2030] Select statement doesn't work in LivyS...

[zjffdu]

Github user zjffdu commented on the issue:

https://github.com/apache/zeppelin/pull/1960
  
@del680202 Can you paste the full stacktrace ?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] zeppelin issue #1960: [ZEPPELIN-2030] Select statement doesn't work in LivyS...

[zjffdu]

Github user zjffdu commented on the issue:

https://github.com/apache/zeppelin/pull/1960
  
I am not sure whether that would help, because `Output exceeds ...` is 
generated by zeppelin, not by livy or spark. 
https://github.com/apache/zeppelin/blob/master/zeppelin-interpreter/src/main/java/org/apache/zeppelin/interpreter/InterpreterOutput.java#L189


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] zeppelin issue #1960: [ZEPPELIN-2030] Select statement doesn't work in LivyS...

[zjffdu]

Github user zjffdu commented on the issue:

https://github.com/apache/zeppelin/pull/1960
  
hmm, right. `LivySparkSQLInterpreter` delegate interpret to 
`LivySparkInterpreter`. So it is possible to get `Output exceeds...` in  
`LivySparkSQLInterpreter`. Thanks @del680202 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] zeppelin issue #1960: [ZEPPELIN-2030] Select statement doesn't work in LivyS...

[zjffdu]

Github user zjffdu commented on the issue:

https://github.com/apache/zeppelin/pull/1960
  
@del680202 Thanks for checking. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] zeppelin issue #1981: [ZEPPELIN-2067] SparkInterpreter prints unnecessary ne...

[zjffdu]

Github user zjffdu commented on the issue:

https://github.com/apache/zeppelin/pull/1981
  
LGTM


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] zeppelin issue #1574: ZEPPELIN-1595. Make ZeppelinContext extensible

[zjffdu]

Github user zjffdu commented on the issue:

https://github.com/apache/zeppelin/pull/1574
  
Thanks @wesm , I updated this PR yesterday, but didn't use `git merge`. I 
use `git rebase` and `git commit --amend`, does these also the issue as well ?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] zeppelin pull request #1970: ZEPPELIN-2045. Pass interpreter properties with...

[zjffdu]

GitHub user zjffdu opened a pull request:

https://github.com/apache/zeppelin/pull/1970

ZEPPELIN-2045. Pass interpreter properties with "spark." as prefix to 
SparkConf

### What is this PR for?
Minor change to only pass interpreter properties with "spark." as prefix to 
SparkConf. Other properties is used by zeppelin interpreter process, so don't 
need to be passed to SparkConf.


### What type of PR is it?
[Bug Fix]

### Todos
* [ ] - Task

### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-2045

### How should this be tested?
Tested manually, this is the log after this PR
```
 INFO [2017-02-03 09:05:33,664] ({pool-2-thread-2} 
SparkInterpreter.java[createSparkContext_1]:384) - -- Create new 
SparkContext yarn-client ---
DEBUG [2017-02-03 09:05:33,668] ({pool-2-thread-2} 
SparkInterpreter.java[createSparkContext_1]:467) - SparkConf: key = 
[spark.cores.max], value = [2]
DEBUG [2017-02-03 09:05:33,668] ({pool-2-thread-2} 
SparkInterpreter.java[createSparkContext_1]:467) - SparkConf: key = 
[spark.app.name], value = [Zeppelin]
```

### Screenshots (if appropriate)

### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/zjffdu/zeppelin ZEPPELIN-2045

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/zeppelin/pull/1970.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1970


commit 3a146d3e0022f70576f34634910f3bd39c911ac3
Author: Jeff Zhang <zjf...@apache.org>
Date:   2017-02-03T01:01:09Z

ZEPPELIN-2045. Pass interpreter properties with "spark." as prefix to 
SparkConf




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] zeppelin issue #1970: ZEPPELIN-2045. Pass interpreter properties with "spark...

[zjffdu]

Github user zjffdu commented on the issue:

https://github.com/apache/zeppelin/pull/1970
  
This doesn't conform with spark. Spark don't allow property that doesn't 
start with `spark.` [1]
If user want to use any properties in driver side, they should use 
properties of Interpreter rather than SparkConf. 

[1] 
https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/deploy/SparkSubmitArguments.scala#L132



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] zeppelin issue #1975: [ZEPPELIN-2046] The final output is a little inconsist...

[zjffdu]

Github user zjffdu commented on the issue:

https://github.com/apache/zeppelin/pull/1975
  
@Leemoonsoo  I still on extra blank between outputs which is inconsistent 
with zeppelin 0.6


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] zeppelin issue #1574: ZEPPELIN-1595. Make ZeppelinContext extensible

[zjffdu]

Github user zjffdu commented on the issue:

https://github.com/apache/zeppelin/pull/1574
  
@Leemoonsoo @felixcheung I update this PR, please help help review that. My 
purpose is to make ZeppelinContext extensible is in 2 perspective. 
* All the interpreter can has its own ZeppelinContext. e.g. My next PR 
would to implement FlinkZeppelinContext which would display flink DataSet/Table 
data in zeppelin's table format
* Interpreters can also add customized features for its own 
ZeppelinContext. e.g. SparkZeppelinContext can add one extention point to allow 
user to visualize mllib model in zeppelin.

Communication between interpreter group mentioned by @felixcheung above is 
not implemented in this PR as I think this would bring much large change and 
need more sophisticated design. But we could put it in `BaseZeppelinContext` in 
future when we want to implement it. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] zeppelin issue #1975: [ZEPPELIN-2046] The final output is a little inconsist...

[zjffdu]

Github user zjffdu commented on the issue:

https://github.com/apache/zeppelin/pull/1975
  
@Leemoonsoo I don't think the blank line is caused by scala. Here's what I 
see in zeppelin 0.6.2 and zeppelin 0.7, both use spark 1.6.2

Zeppelin 0.7

![image](https://cloud.githubusercontent.com/assets/164491/22634310/694c847e-ec63-11e6-89c8-491c6247da4a.png)

Zeppelin 0.6.2

![image](https://cloud.githubusercontent.com/assets/164491/22634319/7a7fcb98-ec63-11e6-88a8-96c7fee29c37.png)



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] zeppelin issue #1975: [ZEPPELIN-2046] The final output is a little inconsist...

[zjffdu]

Github user zjffdu commented on the issue:

https://github.com/apache/zeppelin/pull/1975
  
Thanks @Leemoonsoo , it make sense to fix it in another PR. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] zeppelin pull request #1933: HOTFIX: Add the missing quotes for livy cancel

[zjffdu]

GitHub user zjffdu opened a pull request:

https://github.com/apache/zeppelin/pull/1933

HOTFIX: Add the missing quotes for livy cancel

### What is this PR for?
Seems I miss the quote when rebase. 


### What type of PR is it?
[Hot Fix |]

### Todos
* [ ] - Task

### What is the Jira issue?
No jira created, just minor change for hotfix

### How should this be tested?
Tested manually 

### Screenshots (if appropriate)

![image](https://cloud.githubusercontent.com/assets/164491/22193778/b8f960dc-e178-11e6-8595-b21f01ac62f9.png)

### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/zjffdu/zeppelin cancel_livy

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/zeppelin/pull/1933.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1933


commit d68b6cfded936d20e6c642e9e2d3010481689799
Author: Jeff Zhang <zjf...@apache.org>
Date:   2017-01-23T06:30:44Z

HOTFIX: Add the missing quotes for livy cancel




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] zeppelin issue #1933: HOTFIX: Add the missing quotes for livy cancel

[zjffdu]

Github user zjffdu commented on the issue:

https://github.com/apache/zeppelin/pull/1933
  
@minahlee Could you merge it into 0.7 if you plan for another RC ? Very 
appreciated. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] zeppelin issue #1950: ZEPPELIN-2015 Improve parsing logic of livy sql output

[zjffdu]

Github user zjffdu commented on the issue:

https://github.com/apache/zeppelin/pull/1950
  
@felixcheung @prabhjyotsingh Please help review and try to get it into 0.7. 
 And I may not reponse in the next few days due to Chinese Spring Festival  


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] zeppelin pull request #1950: ZEPPELIN-2015 Improve parsing logic of livy sql...

[zjffdu]

GitHub user zjffdu opened a pull request:

https://github.com/apache/zeppelin/pull/1950

ZEPPELIN-2015 Improve parsing logic of livy sql output

### What is this PR for?
This PR is trying to resolve the table display issue related with #1942. 
But when I do this PR, I find other 2 issues in livy interpreter.
* livy integration is never run due to refactoring of travis script in 
#1786.
* pyspark integration would fail


### What type of PR is it?
[Bug Fix | Improvement]

### Todos
* [ ] - Task

### What is the Jira issue?
https://issues.apache.org/jira/browse/ZEPPELIN-2015


### Screenshots (if appropriate)

### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/zjffdu/zeppelin ZEPPELIN-2015

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/zeppelin/pull/1950.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1950


commit ba6aab2c391ca72ca97e1ccfa79dc368b1ad96d2
Author: Chin Tzulin <jp20316@w022341412910m.local>
Date:   2017-01-25T05:34:28Z

[ZEPPELIN-1982] When using the 'Select * ...' statement doesn't show the 
response In %sql interpreter

commit 4667eef79ecb16b93d2a187dfdf3b4004dc5a77c
Author: Jeff Zhang <zjf...@apache.org>
Date:   2017-01-26T09:30:49Z

ZEPPELIN-2015. Improve parsing logic of livy sql output

commit 98b1d96e4e031d438268834840ab2761e5012c6a
Author: Jeff Zhang <zjf...@apache.org>
Date:   2017-01-26T14:47:56Z

Fix appId and webui extraction for pyspark




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] zeppelin issue #1960: [ZEPPELIN-2030] Select statement doesn't work in LivyS...

[zjffdu]

Github user zjffdu commented on the issue:

https://github.com/apache/zeppelin/pull/1960
  
lgtm


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] zeppelin issue #1960: [ZEPPELIN-2030] Select statement doesn't work in LivyS...

[zjffdu]

Github user zjffdu commented on the issue:

https://github.com/apache/zeppelin/pull/1960
  
Thanks @del680202 Could you add unit test in `LivySQLInterpreterTest.java`?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] zeppelin issue #1942: [ZEPPELIN-1982] When using the 'Select * ...' statemen...

[zjffdu]

Github user zjffdu commented on the issue:

https://github.com/apache/zeppelin/pull/1942
  
@soralee I can see the same error in chrome. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] zeppelin issue #1942: [ZEPPELIN-1982] When using the 'Select * ...' statemen...

[zjffdu]

Github user zjffdu commented on the issue:

https://github.com/apache/zeppelin/pull/1942
  
Jdbc interpreter may be effected. Livy interpreter would return a list of 
InterpreterResult so it is fine for livy. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] zeppelin issue #1942: [ZEPPELIN-1982] When using the 'Select * ...' statemen...

[zjffdu]

Github user zjffdu commented on the issue:

https://github.com/apache/zeppelin/pull/1942
  
@del680202  I create ZEPPELIN-2015 and will try to fix that. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] zeppelin issue #1915: ZEPPELIN-1985 Remove user from pig tutorial note

[zjffdu]

Github user zjffdu commented on the issue:

https://github.com/apache/zeppelin/pull/1915
  
Thanks @AhyoungRyu for noticing that. I have added the missing part. BTW, 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] zeppelin issue #1950: ZEPPELIN-2015 Improve parsing logic of livy sql output

[zjffdu]

Github user zjffdu commented on the issue:

https://github.com/apache/zeppelin/pull/1950
  
Failure CI is not relevant.

```
Tests in error: 
  NotebookTest.testRunAll:323 NullPointer
```


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] zeppelin issue #1950: ZEPPELIN-2015 Improve parsing logic of livy sql output

[zjffdu]

Github user zjffdu commented on the issue:

https://github.com/apache/zeppelin/pull/1950
  
I think it is flaky test as this PR only change the code of livy 
interpreter, didn't touch other components of zeppelin. And I believe I see 
this flaky test before. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] zeppelin issue #1598: [ZEPPELIN-1618] [WIP] Support ElasticSearch 5.0

[zjffdu]

Github user zjffdu commented on the issue:

https://github.com/apache/zeppelin/pull/1598
  
\cc @1ambda 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] zeppelin issue #1495: ZEPPELIN-1325: WIP - Python, advanced interpreter impl

[zjffdu]

Github user zjffdu commented on the issue:

https://github.com/apache/zeppelin/pull/1495
  
@bzz What the status of this PR ? Do you plan to put it in 0.8 ?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] zeppelin issue #2034: [ZEPPELIN-2133] All interpreters sometimes throw rando...

[zjffdu]

Github user zjffdu commented on the issue:

https://github.com/apache/zeppelin/pull/2034
  
@prabhjyotsingh Would this violate the behavior of 
[ZEPPELIN-1770](https://issues.apache.org/jira/browse/ZEPPELIN-1770) ? 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] zeppelin issue #2021: [ZEPPELIN-2116]. livy interpreter needs to restart aft...

[zjffdu]

Github user zjffdu commented on the issue:

https://github.com/apache/zeppelin/pull/2021
  
@prabhjyotsingh @felixcheung Please help review, CI failure is not 
relevant. 

```
TypeError: namedtuple() missing 3 required keyword-only arguments: 
'verbose', 'rename', and 'module'
, %text pyspark is not responding] expected: but was:
  PySparkInterpreterMatplotlibTest.showPlot:161 [%text pyspark is not 
responding] expected: but was:
  PySparkInterpreterMatplotlibTest.testAngular:230 [%text pyspark is not 
responding] expected: but was:
  PySparkInterpreterMatplotlibTest.testClose:184 expected:<0> but was:<1>
  PySparkInterpreterMatplotlibTest.testNoClose:211 [%text pyspark is not 
responding] expected: but was:
  PySparkInterpreterTest.testBasicIntp:111 expected: but 
was:
  PySparkInterpreterTest.testCompletion:120 null
Tests run: 7, Failures: 7, Errors: 0, Skipped: 0
```


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---