Re: What is the user case for remote of interpreter option is false

[Jeff Zhang]

Although it can bring some benefits, but there are 2 disadvantages for
remote=false as I can think of

1. The interpreter log would mix with zeppelin server log. This may cause
diagnosing difficult, markdown might be OK, but I am afraid it would be a
problem for jdbc interpreter.
2. Extra memory consumption. This might be a problem for jdbc interpreter
when it gets large amount of data from database.



moon soo Lee 于2017年5月8日周一 下午5:27写道：

> The option is legacy and being used in few unit tests as far as i remember.
>
> I think we can either try completely remove this code (to keep code base
> simple) or we can try re-introduce this feature (to optimize resource
> usage, less restriction on data sharing between interpreters).
>
> I think interpreters like markdown, jdbc will have no big problem with
> remote=false while Spark interpreter require separate JVM and remote=true
> is required.
>
> remote=false will provide some chances to save some system resource because
> it does not run separate JVM process for interpreter.
>
> Also communication between interpreter (through ResourcePool) can be much
> faster and have less restricted on object serialization.
>
> So remote=false option definitely provides some advantages.
> However, keeping code base for remote=false and remote=true introduces more
> work.
>
> What do you think?
>
> Thanks,
> moon
>
> On Mon, May 8, 2017 at 8:13 PM Jeff Zhang  wrote:
>
> > I see all the interpreter setting set remote of interpreter option as
> true,
> > and didn't find any false case in interpreter-setting.json, does anyone
> > know what is the use case of remote of interpreter option is false. Is
> this
> > a legacy mode that should be removed ?
> >
> >
> >
> https://github.com/apache/zeppelin/blob/master/zeppelin-zengine/src/main/java/org/apache/zeppelin/interpreter/InterpreterSettingManager.java#L183
> >
>

Re: What is the user case for remote of interpreter option is false

[moon soo Lee]

The option is legacy and being used in few unit tests as far as i remember.

I think we can either try completely remove this code (to keep code base
simple) or we can try re-introduce this feature (to optimize resource
usage, less restriction on data sharing between interpreters).

I think interpreters like markdown, jdbc will have no big problem with
remote=false while Spark interpreter require separate JVM and remote=true
is required.

remote=false will provide some chances to save some system resource because
it does not run separate JVM process for interpreter.

Also communication between interpreter (through ResourcePool) can be much
faster and have less restricted on object serialization.

So remote=false option definitely provides some advantages.
However, keeping code base for remote=false and remote=true introduces more
work.

What do you think?

Thanks,
moon

On Mon, May 8, 2017 at 8:13 PM Jeff Zhang  wrote:

> I see all the interpreter setting set remote of interpreter option as true,
> and didn't find any false case in interpreter-setting.json, does anyone
> know what is the use case of remote of interpreter option is false. Is this
> a legacy mode that should be removed ?
>
>
> https://github.com/apache/zeppelin/blob/master/zeppelin-zengine/src/main/java/org/apache/zeppelin/interpreter/InterpreterSettingManager.java#L183
>

[jira] [Created] (ZEPPELIN-2515) spark.r interpreter becomes unresponsive after some time and R process quits silently

[Pietro Pugni (JIRA)]

Pietro Pugni created ZEPPELIN-2515:
--

 Summary: spark.r interpreter becomes unresponsive after some time 
and R process quits silently
 Key: ZEPPELIN-2515
 URL: https://issues.apache.org/jira/browse/ZEPPELIN-2515
 Project: Zeppelin
  Issue Type: Bug
  Components: Interpreters, r-interpreter
Affects Versions: 0.7.1, 0.7.0
 Environment: Ubuntu 16.04.2 LTS Server
Reporter: Pietro Pugni


On Zeppelin 0.7.1 R process quits/crash/gets killed but Spark session remain 
active, so %spark.r interpreter needs to be restarted. On 0.7.0 sometimes 
restarting the interpreter doesn't work, so Zeppelin requires a full restart. 

These are the steps I followed to produce this behaviour:

1) Enable log4j debug properties.

2) Start a brand new instance of zeppelin issuing:
service zeppelin start

3) Open an existing notebook or create a new one and execute this block of code:

%spark.r
2+2

4) Wait for 3) to finish and close the browser. The zeppelin log should report 
something like this:

INFO [2017-05-08 12:26:15,879] ({qtp423031029-60} 
NotebookServer.java[onClose]:363) - Closed connection to 127.0.0.1 : 33798. 
(1001) null

5) Wait several minutes without using zeppelin and without reconnecting via 
browser and at a certain point R will quit. The interpreter log file should 
contain something similar to these lines:

DEBUG [2017-05-08 13:08:00,187] ({Exec Stream Pumper} 
InterpreterOutputStream.java[processLine]:72) - Interpreter output:Error in 
handleErrors(returnStatus, conn) : 
DEBUG [2017-05-08 13:08:00,188] ({Exec Stream Pumper} 
InterpreterOutputStream.java[processLine]:72) - Interpreter output:  No status 
is returned. Java SparkR backend might have failed.
DEBUG [2017-05-08 13:08:00,188] ({Exec Stream Pumper} 
InterpreterOutputStream.java[processLine]:72) - Interpreter output:Calls: 
 -> invokeJava -> handleErrors
DEBUG [2017-05-08 13:08:00,188] ({Exec Stream Pumper} 
InterpreterOutputStream.java[processLine]:72) - Interpreter output:Execution 
halted

This issue has been discussed here: 
https://lists.apache.org/thread.html/17d8bbea0b755a9cf866a6d78909493d63b09bfa92a477bc0e28bc1c@%3Cusers.zeppelin.apache.org%3E



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] zeppelin issue #932: Replace CXF with Jersey2 [ZEPPELIN-903]

[sohaibiftikhar]

Github user sohaibiftikhar commented on the issue:

https://github.com/apache/zeppelin/pull/932
  
@jongyoul I'd be happy to look into it but would be better for me if you 
can provide a starting point to start looking at.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (ZEPPELIN-2514) Command + F doesn't work in Chrome

[Jeff Zhang (JIRA)]

Jeff Zhang created ZEPPELIN-2514:


 Summary: Command + F doesn't work in Chrome
 Key: ZEPPELIN-2514
 URL: https://issues.apache.org/jira/browse/ZEPPELIN-2514
 Project: Zeppelin
  Issue Type: Bug
Affects Versions: 0.7.1
Reporter: Jeff Zhang


The search function still works, but I didn't see the searchbox



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (ZEPPELIN-2513) Provide REST API to bind a notebook with an interpreter

[Danish Khan (JIRA)]

Danish Khan created ZEPPELIN-2513:
-

 Summary: Provide REST API to bind a notebook with an interpreter 
 Key: ZEPPELIN-2513
 URL: https://issues.apache.org/jira/browse/ZEPPELIN-2513
 Project: Zeppelin
  Issue Type: Bug
  Components: rest-api
Affects Versions: 0.7.2
Reporter: Danish Khan


Notebook creation and update APIs should have the ability to accept an 
interpreter, and associate this interpreter with the notebook. I should also be 
able to provide the priority of this interpreter through the REST API, like we 
can do from the notebook by clicking on interpreter binding. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: Output of Spark interpreter dramatically different from spark-shell

[Alexander Bezzubov]

Thanks for update on yarn-client.

I get the difference between driver and executor, but why is there a
difference between output in spark local mode running in spark-shell and
Zeppelin on the same machine though?

As a user, I would expect output to be the same, in case when executor runs
on the same host as driver, on both systems.

--
Alex

On Mon, May 8, 2017, 19:10 Jeff Zhang  wrote:

> This is expected. And I believe you are using local mode. You should be
> able to get the same output in yarn-client mode.
> println function is invoked on executor side, while IMain of spark repl
> only capture the output of driver. The reason you see the output of println
> in spark-shell is that the executor runs in the same host of driver. So it
> mixes with output of driver.
>
>
> Alexander Bezzubov 于2017年5月8日周一 上午7:18写道：
>
> > Hey guys,
> >
> > Introducing Apache Zeppelin to a new org, I have recently noticed that on
> > quite a simple but important use case the output of Zeppelin is *very
> > different* from spark-shell
> >
> > I can print partitions of the RDD in spark-shell
> >
> > ```
> > scala> val data = sc.parallelize(List((1, 2), (1, 1), (2, 3), (2, 1), (1,
> > 4), (3, 5)), 2)
> > scala> data.mapPartitions { _.map { println(_) } } collect
> > [Stage 0:>  (0 +
> > 0) / 2]
> > (2,1)
> > (1,4)
> > (3,5)
> > (1,2)
> > (1,1)
> > (2,3)
> > res0: Array[Unit] = Array((), (), (), (), (), ())
> > ```
> >
> > But the same code in Zeppelin does not include output of the print
> > statement at all :/ Tried both, 0.7.1 and master.
> >
> > ```
> > data.mapPartitions { _.map { println(_) } } collect
> > res2: Array[Unit] = Array((), (), (), (), (), ())
> > ```
> > [image: Inline image 1]
> >
> > Is that expected or did I miss something? Please let me know if you have
> > any ideas.
> >
> > --
> > Alex
> >
>

Re: Output of Spark interpreter dramatically different from spark-shell

[Jeff Zhang]

This is expected. And I believe you are using local mode. You should be
able to get the same output in yarn-client mode.
println function is invoked on executor side, while IMain of spark repl
only capture the output of driver. The reason you see the output of println
in spark-shell is that the executor runs in the same host of driver. So it
mixes with output of driver.


Alexander Bezzubov 于2017年5月8日周一 上午7:18写道：

> Hey guys,
>
> Introducing Apache Zeppelin to a new org, I have recently noticed that on
> quite a simple but important use case the output of Zeppelin is *very
> different* from spark-shell
>
> I can print partitions of the RDD in spark-shell
>
> ```
> scala> val data = sc.parallelize(List((1, 2), (1, 1), (2, 3), (2, 1), (1,
> 4), (3, 5)), 2)
> scala> data.mapPartitions { _.map { println(_) } } collect
> [Stage 0:>  (0 +
> 0) / 2]
> (2,1)
> (1,4)
> (3,5)
> (1,2)
> (1,1)
> (2,3)
> res0: Array[Unit] = Array((), (), (), (), (), ())
> ```
>
> But the same code in Zeppelin does not include output of the print
> statement at all :/ Tried both, 0.7.1 and master.
>
> ```
> data.mapPartitions { _.map { println(_) } } collect
> res2: Array[Unit] = Array((), (), (), (), (), ())
> ```
> [image: Inline image 1]
>
> Is that expected or did I miss something? Please let me know if you have
> any ideas.
>
> --
> Alex
>

[GitHub] zeppelin pull request #2324: [ZEPPELIN-2512] Precent slow note, interpreter ...

[1ambda]

GitHub user 1ambda opened a pull request:

https://github.com/apache/zeppelin/pull/2324

[ZEPPELIN-2512] Precent slow note, interpreter search. (master, branch-0.7)

### What is this PR for?

Use debounce in search input to avoid the slow search.

See more:


![image](https://cloud.githubusercontent.com/assets/4968473/25814653/3be8890e-3459-11e7-9be2-6a440ab41861.png)

### What type of PR is it?
[Improvement]

### Todos

NONE

### What is the Jira issue?

[ZEPPELIN-2512](https://issues.apache.org/jira/browse/ZEPPELIN-2512)

### How should this be tested?

1. Create many notes with different names. (50+)
2. Search using note name filter in home and navbar.
3. Do the same thing for interpreters in the interpreter page

### Screenshots (if appropriate)

NONE

### Questions:
* Does the licenses files need update? - NO
* Is there breaking changes for older versions? - NO
* Does this needs documentation? - NO


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/1ambda/zeppelin 
ZEPPELIN-2512/add-debounce-for-search-inputs

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/zeppelin/pull/2324.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2324


commit 41fef1d13b9564279a7a791f5b030a9ff0548ca4
Author: 1ambda <1am...@gmail.com>
Date:   2017-05-08T16:32:55Z

feat: Add debounce for note name filter

commit 33e0942d516c7d39ad5a4d49753134e28b7e99e0
Author: 1ambda <1am...@gmail.com>
Date:   2017-05-08T16:35:38Z

feat: Add debounce for interpreter search input




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (ZEPPELIN-2512) Add debounce for search inputs to buffer input texts.

[Hoon Park (JIRA)]

Hoon Park created ZEPPELIN-2512:
---

 Summary: Add debounce for search inputs to buffer input texts.
 Key: ZEPPELIN-2512
 URL: https://issues.apache.org/jira/browse/ZEPPELIN-2512
 Project: Zeppelin
  Issue Type: Improvement
  Components: front-end
Affects Versions: 0.7.2, 0.8.0
Reporter: Hoon Park


Add debounce for search inputs to buffer input texts.
Otherwise, it will slow when there are a lot of search targets.

See also: https://docs.angularjs.org/api/ng/directive/ngModelOptions



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] zeppelin issue #1210: [ZEPPELIN-1220] Add geographical map as visualization ...

[volumeint]

Github user volumeint commented on the issue:

https://github.com/apache/zeppelin/pull/1210
  
Just closing the loop.  Pull Request #2313 contained the webpack.config.js 
change.  It has been merged into master.  


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] zeppelin pull request #2267: [ZEPPELIN-2435](WIP) Add logging level facility...

[1ambda]

Github user 1ambda closed the pull request at:

https://github.com/apache/zeppelin/pull/2267


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] zeppelin issue #2267: [ZEPPELIN-2435](WIP) Add logging level facility to fro...

[1ambda]

Github user 1ambda commented on the issue:

https://github.com/apache/zeppelin/pull/2267
  
It's better to use `console.debug`, ... and so on.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] zeppelin pull request #2323: [ZEPPELIN-2411] Improve Table

[1ambda]

GitHub user 1ambda reopened a pull request:

https://github.com/apache/zeppelin/pull/2323

[ZEPPELIN-2411] Improve Table

### What is this PR for?

**Improve Table** using 
[ui-grid](http://ui-grid.info/docs/#/tutorial/401_AllFeatures).

- persist column type (**custom feature** which is not provided by 
[ui-grid](https://github.com/angular-ui/ui-grid))
- persist column state: order, hide/show, **sorting**
- **column filter**
- **pagination**
- **configurable table**: using `setting` menu.
- **selectable rows** + export only selected rows to CSV
- **group by** + **aggregation**
- **pinning**: left / right

And **all these things are persisted and synchronized among web socket 
clients**

See the screenshot section for more detail.

### What type of PR is it?
[Improvement]

### Todos
* [x] - Remove handsontable dependencies
* [x] - Use npm packaged moment* packages.
* [x] - Apply ui-grid
* [x] - Add setting menu
* [x] - Fix some issues
* [x] - Persist column type

### What is the Jira issue?

[ZEPPELIN-2411](https://issues.apache.org/jira/browse/ZEPPELIN-2411)

### How should this be tested?

1. Build: `mvn clean package -DskipTests; ./bin/zeppelin-daemon.sh restart`
2. Open a note and create tables. If you don't have proper paragraphs, use 
this snippet.

```scala
%spark

import org.apache.commons.io.IOUtils
import java.net.URL
import java.nio.charset.Charset

val bankText = sc.parallelize(
IOUtils.toString(
new 
URL("https://s3.amazonaws.com/apache-zeppelin/tutorial/bank/bank.csv;),
Charset.forName("utf8")).split("\n"))

case class Bank(
age: Integer, 
job: String, 
marital: String, 
education: String, 
balance: Integer,
housing: Boolean,
loan: Boolean,
contact: String,
day: Int,
month: String,
duration: Int,
y: Boolean
)

val bank = bankText.map(s => s.split(";")).filter(s => s(0) != 
"\"age\"").map(
s => Bank(s(0).toInt, 
s(1).replaceAll("\"", ""),
s(2).replaceAll("\"", ""),
s(3).replaceAll("\"", ""),
s(5).replaceAll("\"", "").toInt,
if (s(6).replaceAll("\"", "") == "yes") true else false,
if (s(7).replaceAll("\"", "") == "yes") true else false,
s(8).replaceAll("\"", ""),
s(9).replaceAll("\"", "").toInt,
s(10).replaceAll("\"", ""),
s(11).replaceAll("\"", "").toInt,
if (s(16).replaceAll("\"", "") == "yes") true else false
)
).toDF()
bank.registerTempTable("bank")
```

```sql
select age, education, job, balance from bank limit 1000
```

### Screenshots (if appropriate)

 Before


![image](https://cloud.githubusercontent.com/assets/4968473/25803644/d0d81524-3432-11e7-8cf6-dde16465a447.png)

 After: column filter


![image](https://cloud.githubusercontent.com/assets/4968473/25805689/8972d7a6-343b-11e7-96b3-918292dfacd9.png)

 After: group by + aggregation


![image](https://cloud.githubusercontent.com/assets/4968473/25805892/50dc99e4-343c-11e7-82e2-c8e5b2609fd8.png)

 After: pagination


![image](https://cloud.githubusercontent.com/assets/4968473/25805872/4138e808-343c-11e7-8140-963357340708.png)

 After: selectable rows which can be exported into CSV


![image](https://cloud.githubusercontent.com/assets/4968473/25805970/a16dbbe0-343c-11e7-9232-321ed4fff9e8.png)

 After: setting menu for table


![image](https://cloud.githubusercontent.com/assets/4968473/25806504/5dba8d9a-343e-11e7-8afa-9ea0ff84c214.png)

 After: change order, hide / show, pinning

![Uploading 2411_change_order.gif…]()

 After: persisted column type + sorting state


![image](https://cloud.githubusercontent.com/assets/4968473/25805972/a4251d74-343c-11e7-86f0-135b8cbf6b2b.png)

 synchronized

![Uploading 2411_sync.gif…]()


### Questions:
* Does the licenses files need update? - YES, updated
* Is there breaking changes for older versions? - NO
* Does this needs documentation? - NO


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/1ambda/zeppelin ZEPPELIN-2411/prettify-table

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/zeppelin/pull/2323.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2323


commit ed7f7782bc724ca6a78bad6663f21c5949c68279
Author: 1ambda 

[GitHub] zeppelin pull request #2323: [ZEPPELIN-2411] Improve Table

[1ambda]

Github user 1ambda closed the pull request at:

https://github.com/apache/zeppelin/pull/2323


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] zeppelin issue #2322: [ZEPPELIN-2502] RemoteInterpreterServer hang forever d...

[andreaTP]

Github user andreaTP commented on the issue:

https://github.com/apache/zeppelin/pull/2322
  
Looking at the code it looks quite harmful also the pattern used 
[here](https://github.com/apache/zeppelin/blob/master/zeppelin-interpreter/src/main/java/org/apache/zeppelin/interpreter/remote/RemoteInterpreterEventClient.java#L138),
 
[here](https://github.com/apache/zeppelin/blob/master/zeppelin-interpreter/src/main/java/org/apache/zeppelin/interpreter/remote/RemoteInterpreterEventClient.java#L191)
 and 
[here](https://github.com/apache/zeppelin/blob/master/zeppelin-interpreter/src/main/java/org/apache/zeppelin/interpreter/remote/RemoteInterpreterEventClient.java#L245)

if anything goes wrong the Thread can stay in `wait` status potentially 
forever and operations performed there are all but not transactional.

Could I ask if somebody can at least provide some kind of deadline also for 
those?
Better will be a little refactoring using something like 
[ConcurrentHashMap](https://docs.oracle.com/javase/7/docs/api/java/util/concurrent/ConcurrentHashMap.html)
 for operations.

I can even do the refactoring if I have positive feedbacks, anyhow I 
believe this PR is ok as is and I open a new issue and new PR in case.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] zeppelin issue #932: Replace CXF with Jersey2 [ZEPPELIN-903]

[jongyoul]

Github user jongyoul commented on the issue:

https://github.com/apache/zeppelin/pull/932
  
I also had a similar problem. We need to handle it.

On Mon, May 8, 2017 at 11:31 PM, Sohaib Iftikhar 
wrote:

> @echarles  I tested with both simple and
> LdapGroupRealm and it was working. It is only with LdapRealm that I am
> facing an issue. I will post a sample shiro.ini for your reference. As I
> said I am still not sure why /login is being called internally. I tried
> debugging with the source code but due to my unfamiliarity with the
> codebase it has not been of much help.
>
> [main]
>  ldapRealm = org.apache.zeppelin.realm.LdapRealm
>  ldapRealm.userDnTemplate = uid={0},ou=people,dc=my-company,dc=net
>  ldapRealm.searchBase = dc=my-company,dc=net
>  ldapRealm.userSearchBase = ou=people,dc=my-company,dc=net
>  ldapRealm.groupSearchBase = ou=groups,dc=my-company,dc=net
>  ldapRealm.contextFactory.url = ldaps://auth.mycompany.net:636
>  ldapRealm.contextFactory.authenticationMechanism = simple
>  ldapRealm.userObjectClass = posixAccount
>  ldapRealm.groupObjectClass = posixGroup
>  ldapRealm.authorizationEnabled = true
>  ldapRealm.memberAttribute = memberUid
>  
ldapRealm.memberAttributeValueTemplate=uid={0},ou=people,dc=my-company,dc=net
>  ldapRealm.rolesByGroup = USERS:admin
>  ldapRealm.userSearchAttributeName = uid
>  securityManager.realms = $ldapRealm
>  sessionManager = 
org.apache.shiro.web.session.mgt.DefaultWebSessionManager
>  shiro.loginUrl = /api/login
>
>  [roles]
>  admin = *
>
>  [urls]
>  /api/version = anon
>  /** = authc
>
> Eventually I will need group based permissions but this is just a
> simplified ini. Here is a screenshot of the network activity on calling
> api/notebooks.
>
> https://ibb.co/jLw9vk 
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> , or 
mute
> the thread
> 

> .
>



-- 
이종열, Jongyoul Lee, 李宗烈
http://madeng.net



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] zeppelin issue #932: Replace CXF with Jersey2 [ZEPPELIN-903]

[sohaibiftikhar]

Github user sohaibiftikhar commented on the issue:

https://github.com/apache/zeppelin/pull/932
  
@echarles I tested with both simple and LdapGroupRealm and it was working. 
It is only with LdapRealm that I am facing an issue. I will post a sample 
`shiro.ini` for your reference. As I said I am still not sure why /login is 
being called internally. I tried debugging with the source code but due to my 
unfamiliarity with the codebase it has not been of much help.

```
[main]
 ldapRealm = org.apache.zeppelin.realm.LdapRealm
 ldapRealm.userDnTemplate = uid={0},ou=people,dc=my-company,dc=net
 ldapRealm.searchBase = dc=my-company,dc=net
 ldapRealm.userSearchBase = ou=people,dc=my-company,dc=net
 ldapRealm.groupSearchBase = ou=groups,dc=my-company,dc=net
 ldapRealm.contextFactory.url = ldaps://auth.mycompany.net:636
 ldapRealm.contextFactory.authenticationMechanism = simple
 ldapRealm.userObjectClass = posixAccount
 ldapRealm.groupObjectClass = posixGroup
 ldapRealm.authorizationEnabled = true
 ldapRealm.memberAttribute = memberUid
 
ldapRealm.memberAttributeValueTemplate=uid={0},ou=people,dc=my-company,dc=net
 ldapRealm.rolesByGroup = USERS:admin
 ldapRealm.userSearchAttributeName = uid
 securityManager.realms = $ldapRealm
 sessionManager = org.apache.shiro.web.session.mgt.DefaultWebSessionManager
 shiro.loginUrl = /api/login

 [roles]
 admin = *

 [urls]
 /api/version = anon
 /** = authc
```

Eventually I will need group based permissions but this is just a 
simplified ini. Here is a screenshot of the network activity on calling 
api/notebooks.

[https://ibb.co/jLw9vk](url)



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Output of Spark interpreter dramatically different from spark-shell

[Alexander Bezzubov]

Hey guys,

Introducing Apache Zeppelin to a new org, I have recently noticed that on
quite a simple but important use case the output of Zeppelin is *very
different* from spark-shell

I can print partitions of the RDD in spark-shell

```
scala> val data = sc.parallelize(List((1, 2), (1, 1), (2, 3), (2, 1), (1,
4), (3, 5)), 2)
scala> data.mapPartitions { _.map { println(_) } } collect
[Stage 0:>  (0 + 0)
/ 2]
(2,1)
(1,4)
(3,5)
(1,2)
(1,1)
(2,3)
res0: Array[Unit] = Array((), (), (), (), (), ())
```

But the same code in Zeppelin does not include output of the print
statement at all :/ Tried both, 0.7.1 and master.

```
data.mapPartitions { _.map { println(_) } } collect
res2: Array[Unit] = Array((), (), (), (), (), ())
```
[image: Inline image 1]

Is that expected or did I miss something? Please let me know if you have
any ideas.

--
Alex

[GitHub] zeppelin pull request #2323: [ZEPPELIN-2411] Improve Table

[1ambda]

GitHub user 1ambda opened a pull request:

https://github.com/apache/zeppelin/pull/2323

[ZEPPELIN-2411] Improve Table

### What is this PR for?

**Improve Table** using 
[ui-grid](http://ui-grid.info/docs/#/tutorial/401_AllFeatures).

- persist column type (**custom feature** which is not provided by 
[ui-grid](https://github.com/angular-ui/ui-grid))
- persist column state: order, hide/show, **sorting**
- **column filter**
- **pagination**
- **configurable table**: using `setting` menu.
- **selectable rows** + export only selected rows to CSV
- **group by** + **aggregation**
- **pinning**: left / right

And **all these things are persisted and synchronized among web socket 
clients**

See the screenshot section for more detail.

### What type of PR is it?
[Improvement]

### Todos
* [x] - Remove handsontable dependencies
* [x] - Use npm packaged moment* packages.
* [x] - Apply ui-grid
* [x] - Add setting menu
* [x] - Fix some issues
* [x] - Persist column type

### What is the Jira issue?

[ZEPPELIN-2411](https://issues.apache.org/jira/browse/ZEPPELIN-2411)

### How should this be tested?

1. Build: `mvn clean package -DskipTests; ./bin/zeppelin-daemon.sh restart`
2. Open a note and create tables. If you don't have proper paragraphs, use 
this snippet.

```scala
%spark

import org.apache.commons.io.IOUtils
import java.net.URL
import java.nio.charset.Charset

val bankText = sc.parallelize(
IOUtils.toString(
new 
URL("https://s3.amazonaws.com/apache-zeppelin/tutorial/bank/bank.csv;),
Charset.forName("utf8")).split("\n"))

case class Bank(
age: Integer, 
job: String, 
marital: String, 
education: String, 
balance: Integer,
housing: Boolean,
loan: Boolean,
contact: String,
day: Int,
month: String,
duration: Int,
y: Boolean
)

val bank = bankText.map(s => s.split(";")).filter(s => s(0) != 
"\"age\"").map(
s => Bank(s(0).toInt, 
s(1).replaceAll("\"", ""),
s(2).replaceAll("\"", ""),
s(3).replaceAll("\"", ""),
s(5).replaceAll("\"", "").toInt,
if (s(6).replaceAll("\"", "") == "yes") true else false,
if (s(7).replaceAll("\"", "") == "yes") true else false,
s(8).replaceAll("\"", ""),
s(9).replaceAll("\"", "").toInt,
s(10).replaceAll("\"", ""),
s(11).replaceAll("\"", "").toInt,
if (s(16).replaceAll("\"", "") == "yes") true else false
)
).toDF()
bank.registerTempTable("bank")
```

```sql
select age, education, job, balance from bank limit 1000
```

### Screenshots (if appropriate)

 Before


![image](https://cloud.githubusercontent.com/assets/4968473/25803644/d0d81524-3432-11e7-8cf6-dde16465a447.png)

 After: column filter


![image](https://cloud.githubusercontent.com/assets/4968473/25805689/8972d7a6-343b-11e7-96b3-918292dfacd9.png)

 After: group by + aggregation


![image](https://cloud.githubusercontent.com/assets/4968473/25805892/50dc99e4-343c-11e7-82e2-c8e5b2609fd8.png)

 After: pagination


![image](https://cloud.githubusercontent.com/assets/4968473/25805872/4138e808-343c-11e7-8140-963357340708.png)

 After: selectable rows which can be exported into CSV


![image](https://cloud.githubusercontent.com/assets/4968473/25805970/a16dbbe0-343c-11e7-9232-321ed4fff9e8.png)

 After: setting menu for table


![image](https://cloud.githubusercontent.com/assets/4968473/25805971/a16ff644-343c-11e7-81fd-783892799818.png)

 After: change order, hide / show, pinning

![Uploading 2411_change_order.gif…]()

 After: persisted column type + sorting state


![image](https://cloud.githubusercontent.com/assets/4968473/25805972/a4251d74-343c-11e7-86f0-135b8cbf6b2b.png)

 synchronized

![Uploading 2411_sync.gif…]()


### Questions:
* Does the licenses files need update? - YES, updated
* Is there breaking changes for older versions? - NO
* Does this needs documentation? - NO


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/1ambda/zeppelin ZEPPELIN-2411/prettify-table

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/zeppelin/pull/2323.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2323


commit ed7f7782bc724ca6a78bad6663f21c5949c68279
Author: 1ambda 

[GitHub] zeppelin pull request #2271: [ZEPPELIN-2414] Memory leak under scoped mode o...

[hammertank]

GitHub user hammertank reopened a pull request:

https://github.com/apache/zeppelin/pull/2271

[ZEPPELIN-2414] Memory leak under scoped mode of SparkInterpreter caused by 
inapproprately setting Thread.contextClassLoader

### What is this PR for?
Fix memory leak under scoped mode of SparkInterpreter which is caused by 
inapproprately setting Thread.contextClassLoader.

### What type of PR is it?
Bug Fix

### Todos
Rest Thread.contextClassLoader in Methods:
* [x] - Wraps interpreters with ClassLoader with ClassLoaderInterpreter in 
RemoteInterpreterServer.createInterpreter

### What is the Jira issue?
https://issues.apache.org/jira/browse/ZEPPELIN-2414

### How should this be tested?

### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/hammertank/zeppelin ZEPPELIN-2414

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/zeppelin/pull/2271.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2271


commit 2b67899ffe71658ca99f802005e1c5e20ffa0db7
Author: z0621 
Date:   2017-05-04T05:57:30Z

Wraps interpreters with ClassloaderInterpreters




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] zeppelin pull request #2271: [ZEPPELIN-2414] Memory leak under scoped mode o...

[hammertank]

Github user hammertank closed the pull request at:

https://github.com/apache/zeppelin/pull/2271


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] zeppelin issue #2294: [ZEPPELIN-2438]: Add local winutils.exe in build step ...

[cfries]

Github user cfries commented on the issue:

https://github.com/apache/zeppelin/pull/2294
  
(close and reopen to kick Jenkins to recognise Travis)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] zeppelin pull request #2294: [ZEPPELIN-2438]: Add local winutils.exe in buil...

[cfries]

GitHub user cfries reopened a pull request:

https://github.com/apache/zeppelin/pull/2294

[ZEPPELIN-2438]: Add local winutils.exe in build step and use if required 
…

### What is this PR for?

Add local winutils.exe in build and use as temporary HADOOP_HOME if 
required on run (windows only).

The build of the spark dependencies now downloads winutils.exe and installs 
it in interpreter/spark/winutils.
The zeppelin-env.cmd (only used if we run on windows) then checks if 
HADOOP_HOME exists, if not, it will set it to point to winutils. This is 
required to have the spark interpreter work in Windows.

In order to have this active, the zeppelin-env.cmd had to be removed from 
the .gitignore.

### What type of PR is it?
[Bug Fix | Improvement]

Without this fix, the binary distribution or a local build of Zeppelin will 
have the spark interpreter (default interpreter) fail on a Windows platform 
(unless the user follows additional / manual installation steps or already has 
a local HADOOP). This will frustrate users trying to checkout Zeppelin.

For users having HADOOP or macOS/Linux users, the chance will have no 
effect.

### Todos

### What is the Jira issue?
https://issues.apache.org/jira/browse/ZEPPELIN-2438

### How should this be tested?
- Non-Regression Test
- Standard Unit Test
- Run Zeppelin without additional installations (except Java) on Windows

### Screenshots (if appropriate)
N/A

### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/cfries/zeppelin windows-compatiblity

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/zeppelin/pull/2294.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2294


commit 3635ed8a717108ba82753413bd0500519ffd4add
Author: Christian Fries 
Date:   2017-04-08T20:00:33Z

[ZEPPELIN-2375]: Avoid modification of CLASSPATH variable.

The script previously modified the CLASSPATH variable. This may lead to
an undesired side effect, where the zeppelin server classpath is
exported to the zeppelin interpreter classpath, see JIRA issue
ZEPPELIN-2375 for details. Instead of modifying classpath we now work
the other way around and set ZEPPELIN_CLASSPATH or
ZEPPELIN_INT_CLASSPATH to include CLASSPATH and then use
ZEPPELIN_CLASSPATH or ZEPPELIN_INT_CLASSPATH in the corresponding exec
(runner), respectively.

See also https://issues.apache.org/jira/browse/ZEPPELIN-2375

commit 096fed6cfde37051b95018cedd455db3084aa1c9
Author: Christian Fries 
Date:   2017-04-27T16:38:53Z

Merge branch 'master' into apache/master

commit 303511f15dc95b0b75b224607f79ef00229392ba
Author: Christian Fries 
Date:   2017-04-27T17:19:28Z

[ZEPPELIN-2438]: Add local winutils.exe in build and use if required on run.

The build of the spark dependencies now downloads winutils.exe and
installs it in interpreter/spark/winutils.
The zeppelin-env.cmd (only used if we run on windows) then checks if
HADOOP_HOME exists, if not, it will set it to point to winutils. This
is required to have the spark interpreter work in Windows.

In order to have this active, the zeppelin-env.cmd had to be removed
from the .gitignore.

commit 18e226be7dfc11493edd17bf039019bb0da93e15
Author: Christian Fries 
Date:   2017-04-30T00:11:21Z

Added zeppelin-env.cmd to distribution.

commit 4ba7f564841cb09072969f3538460a2ff285fbcf
Author: Christian Fries 
Date:   2017-05-01T10:43:17Z

Moved fallback setting HADOOP_HOME to common.cmd.

commit 97f97850d2fb1e2056d99530c78d754f02841a67
Author: Christian Fries 
Date:   2017-05-01T10:45:31Z

Cosmetic change. Added newline to end (effectively reverting previous 
commit).

commit 93017a036cba2b49e00763f8ac65d977c9ccab26
Author: Christian Fries 
Date:   2017-05-01T11:23:42Z

Moved fallback setting HADOOP_HOME to common.cmd.

Fixed syntax and path.

commit f59e39eedda069c29f0c1945f39e6e8feb259afc
Author: Christian Fries 
Date:   2017-05-01T14:13:57Z

Merge branch 'apache/master' into windows-compatiblity

commit b4babef9add9c1d6dfda39b3fdec032a76d9d26c
Author: Christian Fries 
Date:   2017-05-05T17:36:45Z

Added remark on additional Zepplin Spark on Windows requirements.

Zeppelin Spark on Windows requires winutils.exe which requires the
Visual Studio C++ 

[GitHub] zeppelin pull request #2294: [ZEPPELIN-2438]: Add local winutils.exe in buil...

[cfries]

Github user cfries closed the pull request at:

https://github.com/apache/zeppelin/pull/2294


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] zeppelin pull request #2314: [minor] Fix JDBC doc after #2229

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/zeppelin/pull/2314


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] zeppelin pull request #2314: [minor] Fix JDBC doc after #2229

[prabhjyotsingh]

Github user prabhjyotsingh closed the pull request at:

https://github.com/apache/zeppelin/pull/2314


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] zeppelin issue #932: Replace CXF with Jersey2 [ZEPPELIN-903]

[echarles]

Github user echarles commented on the issue:

https://github.com/apache/zeppelin/pull/932
  
It is working here with shiro-enabled (simple username password in 
shiro,ini, not the ldaprealm).
I would need to test with ldaprealm...
@sohaibiftikhar  is it working for your with the simple username/password 
in shiro.ini ?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---