[GitHub] zeppelin pull request #2544: [ZEPPELIN-2724] Another Attempt to solve the de...

2017-11-29 Thread sadhen 
Github user sadhen closed the pull request at:

https://github.com/apache/zeppelin/pull/2544


---

[GitHub] zeppelin pull request #2539: [ZEPPELIN-2863] change notebook-authorization.j...

2017-11-29 Thread sadhen 
Github user sadhen closed the pull request at:

https://github.com/apache/zeppelin/pull/2539


---

[GitHub] zeppelin issue #2544: [ZEPPELIN-2724] Another Attempt to solve the dead lock...

2017-08-24 Thread sadhen 
Github user sadhen commented on the issue:

https://github.com/apache/zeppelin/pull/2544
  
No, this PR is for the master, we recently upgraded from 0.6.0 to master.

Still to be confirmed and verified. However, we are running my PR in 
production for over 50 analysts (we mainly use Zeppelin as a platform for 
analysts to write and run Spark SQL query). It works well.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] zeppelin issue #2539: [ZEPPELIN-2863] change notebook-authorization.json and...

2017-08-23 Thread sadhen 
Github user sadhen commented on the issue:

https://github.com/apache/zeppelin/pull/2539
  
@zjffdu 
It won't satisfy my need.

You may understand my concerns when you start dockerizing Zeppelin.

And these days, I encounter another dockerization problem. If I put 
interpreter and server processes all in one container, the interpreter process 
will run out of memory. My final solution is deploying interpreters elsewhere.

```
ERROR [2017-08-24 00:12:17,391] ({Thread-0} 
TThreadPoolServer.java[serve]:194) - ExecutorService threw error: 
java.lang.OutOfMemoryError: unable to create new native thread
java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:714)
at 
java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:950)
at 
java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1368)
at 
org.apache.thrift.server.TThreadPoolServer.serve(TThreadPoolServer.java:168)
at 
org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer.run(RemoteInterpreterServer.java:104)
 INFO [2017-08-24 00:24:30,979] ({Thread-0} 
RemoteInterpreterServer.java[run]:103) - Starting remote interpreter server on 
port 40623
```


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] zeppelin pull request #2544: [ZEPPELIN-2724] Another Attempt to solve the de...

2017-08-21 Thread sadhen 
GitHub user sadhen opened a pull request:

https://github.com/apache/zeppelin/pull/2544

[ZEPPELIN-2724] Another Attempt to solve the dead lock problem

### What is this PR for?
https://github.com/apache/zeppelin/pull/2465

This is my attempt under the help with my colleague. We originally wrote 
the patch in Java 8 based on Zeppelin 0.6.0. Today, I rewrite this part in Java 
7 based on Zeppelin master. I confirm the code I wrote for Zeppelin 0.6.0 
solved the dead lock problem. We have over 80 websocket connections for several 
months and our Zeppelin 0.6.0 works well and didn't crash for websocket reason 
since we apply the patch.

I will explain the patch in detail if necessary.

### What type of PR is it?
[ Improvement ]

### Todos
* [ ] - To be verified

### How should this be tested?
Our QA wrote a python script, I will post it here later.

### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/wacai/zeppelin websocket

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/zeppelin/pull/2544.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2544


commit 82ec4e0efc49daa60aed9f54bf42ff991a7873f9
Author: 忍冬 <rend...@wacai.com>
Date:   2017-08-15T13:44:47Z

change notebook-authorization.json and interpreter.json to NotebookDir for 
dockerization

commit be8877cb4323d8a4e76f4a15ec9675f00396a5f6
Author: 忍冬 <rend...@wacai.com>
Date:   2017-08-16T03:04:14Z

update corresponding comments and configs

commit 9e21f0e2518f1899fbd8c891b89d74ee03b1f9e8
Author: 忍冬 <rend...@wacai.com>
Date:   2017-08-16T03:22:55Z

fix

commit d7a6d71d9af011f9fd2ebf2be813a60b8118924b
Author: 忍冬 <rend...@wacai.com>
Date:   2017-08-16T06:07:39Z

fix unit test

commit 6afb733515620bf0bd4a6b43abe6344ad24bdc16
Author: 忍冬 <rend...@wacai.com>
Date:   2017-08-16T06:29:27Z

fix unit test  on NoteInterpreterLoaderTest

commit 58013e34cf298c7ba343a21742c809be3c95850a
Author: 忍冬 <rend...@wacai.com>
Date:   2017-08-21T07:45:40Z

some logging and make broadcast concurrent

commit 3c017eca125cebb5a4dc1cf8406852d83420aee9
Author: 忍冬 <rend...@wacai.com>
Date:   2017-08-21T09:33:04Z

refactor on noteSocketMap




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] zeppelin issue #2539: [ZEPPELIN-2863] change notebook-authorization.json and...

2017-08-20 Thread sadhen 
Github user sadhen commented on the issue:

https://github.com/apache/zeppelin/pull/2539
  
@zjffdu 

Your solution didn't solve the problem in the quoted:

`The critical part is that files under conf/ such as shiro.ini is part of 
the docker image and should be immutable.`

Imaging the following use case, I have a base docker image, and two other 
end-point docker images. The base docker image consists of jars and a default 
configuration. And the end-point docker images just config the base image, that 
means for example image0 use CAS auth in `shiro.ini` and image1 use LDAP in 
`shiro.ini`. The immutable part of a deployment identifies a docker image. That 
is the charming part of docker. We should not modify the conf files in a 
mounted volume. If you keep this files in a mounted volume,  you will never 
correctly start up zeppelin with ill configurations.

This is my understanding of dockerization. If you don't agree with me, 
please downvote and I will not waste my time completing this PR.

-

TODOs:
1. provide an environment variable to tell Zeppelin where is the notebook 
dir. The variable should be prior to configs in `zeppelin-site.xml` (regardless 
of this pr, this would be nice feature for dockerization)
2. add some logic in `bin/*sh` for compat.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] zeppelin issue #2539: [ZEPPELIN-2863] change notebook-authorization.json and...

2017-08-16 Thread sadhen 
Github user sadhen commented on the issue:

https://github.com/apache/zeppelin/pull/2539
  
We deployed Zeppelin in K8S cluster, and are using volumes for persistence 
as you stated.

A1. I will provide a solution later. A possible solution: mv the files in 
`conf/` if exists in `zeppelin.sh`.

A2. Although `notebook` dir is version controlled, it is not relevant to 
this particular issue. Untracked files also lives well under version(git) 
controlled dir.

-

The critical part is that files under `conf/` such as `shiro.ini` is part 
of the docker image and should be immutable. But `notebook-authorization.json` 
and `interpreter.json` is mutable and is changed in user activities. They are 
different in nature.

The differences:
1. `shiro.ini` is provided by admin and these two json is generated by 
Zeppelin itself
2. `shiro.ini` should be part of the docker image, two docker images with 
totally different `shiro.ini` behaves in a totally different way. It is part of 
the deployment but the json files are record of user activities.

Why should we keep these in different dirs?
Since volumes are dirs. And the two json files should be kept in a volume.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] zeppelin pull request #2539: [ZEPPELIN-2863] change notebook-authorization.j...

2017-08-16 Thread sadhen 
GitHub user sadhen reopened a pull request:

https://github.com/apache/zeppelin/pull/2539

[ZEPPELIN-2863] change notebook-authorization.json and interpreter.json to 
NotebookDir

### TODO
+ [ ] Fix travis

### What is this PR for?
When dockerizing Zeppelin, we need to persist the mutable part of Zeppelin 
on storage out of docker (for example NFS).

For this reason, notebook-authorization.json and interpreter.json should be 
persisted in Notebook Dir.


### What type of PR is it?
[Improvement]

### What is the Jira issue?
https://issues.apache.org/jira/browse/ZEPPELIN-2863

### How should this be tested?
Minor change, no need to test it.

### Screenshots (if appropriate)

### Questions:
* Does the licenses files need update?
No
* Is there breaking changes for older versions?
No
* Does this needs documentation?
Yes


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/wacai/zeppelin ZEPPELIN-2863

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/zeppelin/pull/2539.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2539


commit 82ec4e0efc49daa60aed9f54bf42ff991a7873f9
Author: 忍冬 <rend...@wacai.com>
Date:   2017-08-15T13:44:47Z

change notebook-authorization.json and interpreter.json to NotebookDir for 
dockerization

commit be8877cb4323d8a4e76f4a15ec9675f00396a5f6
Author: 忍冬 <rend...@wacai.com>
Date:   2017-08-16T03:04:14Z

update corresponding comments and configs

commit 9e21f0e2518f1899fbd8c891b89d74ee03b1f9e8
Author: 忍冬 <rend...@wacai.com>
Date:   2017-08-16T03:22:55Z

fix

commit d7a6d71d9af011f9fd2ebf2be813a60b8118924b
Author: 忍冬 <rend...@wacai.com>
Date:   2017-08-16T06:07:39Z

fix unit test

commit c802f2c176a961abc4820dd093e4593de906f117
Author: 忍冬 <rend...@wacai.com>
Date:   2017-08-16T06:29:27Z

fix unit test on NoteInterpreterLoaderTest




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] zeppelin pull request #2539: [ZEPPELIN-2863] change notebook-authorization.j...

2017-08-16 Thread sadhen 
Github user sadhen closed the pull request at:

https://github.com/apache/zeppelin/pull/2539


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] zeppelin issue #2539: [ZEPPELIN-2863] change notebook-authorization.json and...

2017-08-15 Thread sadhen 
Github user sadhen commented on the issue:

https://github.com/apache/zeppelin/pull/2539
  
@1ambda  `interpreter.json` and `notebook-authorization.json` will be 
changed during the use of Zeppelin. If the docker instance is killed and 
restarted, the updated json files will be lost. 

And logically the two json files are related to the note.json, and should 
be kept along side with them.

Currently, this files are located under conf/ ,but other files under conf 
will not be changed thus we do not have to put them into the mounted NFS.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] zeppelin pull request #2539: [ZEPPELIN-2863] change notebook-authorization.j...

2017-08-15 Thread sadhen 
GitHub user sadhen reopened a pull request:

https://github.com/apache/zeppelin/pull/2539

[ZEPPELIN-2863] change notebook-authorization.json and interpreter.json to 
NotebookDir

### What is this PR for?
When dockerizing Zeppelin, we need to persist the mutable part of Zeppelin 
on storage out of docker (for example NFS).

For this reason, notebook-authorization.json and interpreter.json should be 
persisted in Notebook Dir.


### What type of PR is it?
[Improvement]

### What is the Jira issue?
https://issues.apache.org/jira/browse/ZEPPELIN-2863

### How should this be tested?
Minor change, no need to test it.

### Screenshots (if appropriate)

### Questions:
* Does the licenses files need update?
No
* Is there breaking changes for older versions?
No
* Does this needs documentation?
Yes


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/wacai/zeppelin ZEPPELIN-2863

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/zeppelin/pull/2539.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2539


commit 82ec4e0efc49daa60aed9f54bf42ff991a7873f9
Author: 忍冬 <rend...@wacai.com>
Date:   2017-08-15T13:44:47Z

change notebook-authorization.json and interpreter.json to NotebookDir for 
dockerization




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] zeppelin pull request #2539: [ZEPPELIN-2863] change notebook-authorization.j...

2017-08-15 Thread sadhen 
Github user sadhen closed the pull request at:

https://github.com/apache/zeppelin/pull/2539


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] zeppelin pull request #2539: [ZEPPELIN-533] change notebook-authorization.js...

2017-08-15 Thread sadhen 
GitHub user sadhen opened a pull request:

https://github.com/apache/zeppelin/pull/2539

[ZEPPELIN-533] change notebook-authorization.json and interpreter.json to 
NotebookDir

### What is this PR for?
When dockerizing Zeppelin, we need to persist the mutable part of Zeppelin 
on storage out of docker (for example NFS).

For this reason, notebook-authorization.json and interpreter.json should be 
persisted in Notebook Dir.


### What type of PR is it?
[Improvement]

### What is the Jira issue?
https://issues.apache.org/jira/browse/ZEPPELIN-2863

### How should this be tested?
Minor change, no need to test it.

### Screenshots (if appropriate)

### Questions:
* Does the licenses files need update?
No
* Is there breaking changes for older versions?
No
* Does this needs documentation?
Yes


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/wacai/zeppelin ZEPPELIN-2863

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/zeppelin/pull/2539.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2539


commit 82ec4e0efc49daa60aed9f54bf42ff991a7873f9
Author: 忍冬 <rend...@wacai.com>
Date:   2017-08-15T13:44:47Z

change notebook-authorization.json and interpreter.json to NotebookDir for 
dockerization




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] zeppelin issue #319: ZEPPELIN-312: use threadpoll to do broadcast in zeppeli...

2017-03-23 Thread sadhen 
Github user sadhen commented on the issue:

https://github.com/apache/zeppelin/pull/319
  
@Leemoonsoo  the delivery orders do not matter. Since the call of broadcast 
on message 1 and the call of broadcast on message 2 would be processed one by 
one since `noteSocketMap` is synchronized. (in the latest code)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---