[GitHub] zeppelin pull request #2592: ZEPPELIN-2685. Improvement on Interpreter class

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/zeppelin/pull/2592


---

[GitHub] zeppelin issue #2542: ZEPPELIN-277 Add Tab as Autocomplete for Notebook non-...

[zjffdu]

Github user zjffdu commented on the issue:

https://github.com/apache/zeppelin/pull/2542
  
ping @malayhm 


---

[GitHub] zeppelin pull request #2621: [Minor] Remove hardcoded key in zeppelinhub.

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/zeppelin/pull/2621


---

[jira] [Created] (ZEPPELIN-2996) How to set roles and restrict users in Zeppelin

[dhanu (JIRA)]

dhanu created ZEPPELIN-2996:
---

 Summary: How to set roles and restrict users in Zeppelin
 Key: ZEPPELIN-2996
 URL: https://issues.apache.org/jira/browse/ZEPPELIN-2996
 Project: Zeppelin
  Issue Type: Task
  Components: security
Affects Versions: 0.7.2, 0.7.1
Reporter: dhanu
Priority: Blocker


Hi ,

Hope you are doing great.

I am wondering how to set roles and permissions for users in shiro.ini .Can you 
explain in a detailed picture about shiro.ini how to set restrictions to 
perform specific tasks (read/write) for  users .

Regards,
Dhanu.T



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Hello

[Alvin Raj]

Hi Dev,

Wanted to introduce myself -- I'm working with a small team to explore 
embedding Zeppelin into our stack. We'd like to connect Zeppelin to 
Oracle's Business Intelligence server, and also be able to host a 
Zeppelin instance on our WebLogic app server.


One thing we noticed was that there seems to be a pretty tight coupling 
with Jetty in some parts of the code (using Jetty apis for websockets, 
for example). Are there any plans to decouple from Jetty (and use 
JSR-356 for websockets which Jetty also supports now)?


Thanks,

~Alvin Raj

[GitHub] zeppelin issue #2618: [WIP] ZEPPELIN-2986. Add hadoop-aws and hadoop-azure t...

[steveloughran]

Github user steveloughran commented on the issue:

https://github.com/apache/zeppelin/pull/2618
  
bq. Can this be just a list of steps people can follow?

It's a very brittle list of steps. It's easier at build time as you already 
have your hadoop version fixed, the hadoop-aws and hadoop-azure poms give you 
the library versions they need. All you need is to add them *and evict all 
conflict with the later stuff Spark has chosen*. It's really hard to get this 
right.

The spark work adds a new optional module and profile to set this up. I'd 
recommend doing the same thing for now, using the code in spark's POMs to tell 
you what to exclude


---

[GitHub] zeppelin issue #2614: Add support for minio storage over s3 api

[felixcheung]

Github user felixcheung commented on the issue:

https://github.com/apache/zeppelin/pull/2614
  
also probably would be a good idea to open a JIRA


---

[GitHub] zeppelin issue #2618: [WIP] ZEPPELIN-2986. Add hadoop-aws and hadoop-azure t...

[felixcheung]

Github user felixcheung commented on the issue:

https://github.com/apache/zeppelin/pull/2618
  
I agree this is going to be a huge moving target to include in the release 
of the project
Can this be just a list of steps people can follow? Dockerfile or actual 
Docker images people can use, instead?


---

[GitHub] zeppelin issue #2618: [WIP] ZEPPELIN-2986. Add hadoop-aws and hadoop-azure t...

[steveloughran]

Github user steveloughran commented on the issue:

https://github.com/apache/zeppelin/pull/2618
  
# Spark 2.3 has a spark-hadoop-cloud module which adds a set of 
dependencies on the hadoop cloud modules, throwing out all the transitive 
conflicting modules. Either pick that up or lift all the exclusion logic there 
to keep out things like: jackson, guava, joda-time which will only cause pain. 
Hadoop 2.9+ will move to a shaded aws JAR here.

Do be aware, hadoop-aws is very fussy about aws-SDK versions: that library 
is a moving target. You can't expect to change versions and not expect to have 
the code link, let alone run.


---

[GitHub] zeppelin issue #1302: [ZEPPELIN-1306] Interpreter restarts on a note.

[Leemoonsoo]

Github user Leemoonsoo commented on the issue:

https://github.com/apache/zeppelin/pull/1302
  
Yeah, i think we can make "auto-restart interpreter on cron execution" 
checkbox restart specific interpreter instance to the notebook.


---

[GitHub] zeppelin issue #1302: [ZEPPELIN-1306] Interpreter restarts on a note.

[toughrogrammer]

Github user toughrogrammer commented on the issue:

https://github.com/apache/zeppelin/pull/1302
  
But isn't this original issue also important? I usually want to restart 
specific notebook. Because spark executors aren't returned automatically, I 
must kill zeppelin application from YARN web UI. And then, zeppelin notebook 
lost spark context and I need to restart spark interpreter. 



---

[GitHub] zeppelin issue #1302: [ZEPPELIN-1306] Interpreter restarts on a note.

[Leemoonsoo]

Github user Leemoonsoo commented on the issue:

https://github.com/apache/zeppelin/pull/1302
  
Ah i see. That definitely looks like a problem. I filed an issue 
https://issues.apache.org/jira/browse/ZEPPELIN-2995.


---

[GitHub] zeppelin issue #1302: [ZEPPELIN-1306] Interpreter restarts on a note.

[toughrogrammer]

Github user toughrogrammer commented on the issue:

https://github.com/apache/zeppelin/pull/1302
  
https://user-images.githubusercontent.com/1473538/31558707-001fa9e6-b089-11e7-9ca6-48aabcc60839.png;>

I'm using spark interpreter per note isolated mode and I wrote a notebook 
for batch job with cron execution. (Please refer to the attached image.)
When this notebook is executed by cron, interpreter is restarted. But 
problem is other notebook's spark context also to be expired. Every hour I need 
to re-run all paragraph and read large data set from storage. 




---

[GitHub] zeppelin issue #1302: [ZEPPELIN-1306] Interpreter restarts on a note.

[Leemoonsoo]

Github user Leemoonsoo commented on the issue:

https://github.com/apache/zeppelin/pull/1302
  
@toughrogrammer Could you explain little more about the problem? How cron 
execution is related with restart interpreter?


---

[GitHub] zeppelin issue #2598: [ZEPPELIN-2950] Support Ceph as a notebook storage

[Leemoonsoo]

Github user Leemoonsoo commented on the issue:

https://github.com/apache/zeppelin/pull/2598
  
CI failure looks unrelated to this.

LGTM and merge to master if no further review.
Thanks again @kjmrknsn for the contribution.


---

[GitHub] zeppelin issue #1302: [ZEPPELIN-1306] Interpreter restarts on a note.

[toughrogrammer]

Github user toughrogrammer commented on the issue:

https://github.com/apache/zeppelin/pull/1302
  
How it's going on? I want to restart *note specific interpreter* when cron 
is executed or refreshing spark context, but current version(0.7.2), it's 
impossible. It's really unproductive experience.


---

Re: Disable cron runs

[moon soo Lee]

Hi,

Currently, there's no easy way to disable cron.
I think it's okay to add configuration parameter that disables all cron
run, while the default value of the parameter enable the cron (for backward
compatibility).

Let me know if you need any help on this.

Thanks,
moon

On Wed, Oct 4, 2017 at 2:35 AM Belousov Maksim Eduardovich <
m.belou...@tinkoff.ru> wrote:

> Hello, dev!
>
> Our team have about 600 notes and 35 of them are scheduled. Some scheduled
> notes create tables in DB and send e-mails.
> We want to hide testing activities when we test notes on another server.
> Now we remove cron parameters in json files. It is not cool.
>
> Are there any workarounds to disable running of notes with cron?
> Can it be a good decision to add a new server parameter that will disable
> all cron runs?
>
>
> Thanks,
>
> Maksim Belousov
>
>

[jira] [Created] (ZEPPELIN-2994) Websocket/REST API to access ResourcePool

[Lee moon soo (JIRA)]

Lee moon soo created ZEPPELIN-2994:
--

 Summary: Websocket/REST API to access ResourcePool
 Key: ZEPPELIN-2994
 URL: https://issues.apache.org/jira/browse/ZEPPELIN-2994
 Project: Zeppelin
  Issue Type: New Feature
Reporter: Lee moon soo


h2. Motivation

[ResourcePool|https://github.com/apache/zeppelin/blob/master/zeppelin-interpreter/src/main/java/org/apache/zeppelin/resource/ResourcePool.java]
 is general purpose, distributed map across interpreter processes.

ResourcePool currently provides

* Put / Get any object
* List objects
* Invoke method of object in the ResourcePool

Because of ResourcePool is (logically) distributed across Interpreters, 
interpreter can use it to communicate to other interpreters. Pass interpreter 
result to other interpreter, let other interpreter call directly call one of 
the method of SparkContext, etc.

h2. Websocket/REST API to ResourcePool

However, currently ResourcePool API is only accessible in Interpreter. If 
front-end can have Websocket/Rest API to the ResourcePool, we'll have usages 
like

* Interpreter store 
[TableData|https://github.com/apache/zeppelin/blob/master/zeppelin-interpreter/src/main/java/org/apache/zeppelin/tabledata/TableData.java]
 into the ResourcePool.
* Front-end read TableData directly from ResourcePool. (and visualize data)

This will open possibilities of

* Move pivot routine to the backend (because current pivot works in front-end 
with partial data)
* Realtime visualization update on front-end as data changes




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (ZEPPELIN-2993) Job manager should only display user's jobs

[Maziyar PANAHI (JIRA)]

Maziyar PANAHI created ZEPPELIN-2993:


 Summary: Job manager should only display user's jobs
 Key: ZEPPELIN-2993
 URL: https://issues.apache.org/jira/browse/ZEPPELIN-2993
 Project: Zeppelin
  Issue Type: Bug
Affects Versions: 0.7.3
Reporter: Maziyar PANAHI


Hi,

Job manager displays all the jobs of all the users to each user. This may cause 
some issues:

1- Privacy: Although it is not possible to stop, run nor view someone else's 
notebook from Job manager, users still can see the names of those notebooks. 
One can argue if the notebooks are not accessible by some users then there is 
no reason for them to see their status in Job manager.
2- Management: It is much easier to only deal with your own jobs rather than 
all the users. There is a search bar to filter, but what if the names are the 
same? Imagine having 10 notebooks by 10 users all named "Test". Then the only 
way is to try and fail until you find your own job.
3- Use case: The only situation that listing all the jobs will help is as if 
you are an admin. Which this takes me to my question:

How do you set permission for Job manager? How does someone sets permission the 
way that only people in [admin] can see all the jobs but the rest only seeing 
their own jobs?
Is this possible in Shiro? 

Many thanks,




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (ZEPPELIN-2992) Table can not be displayed correctly when there's delimiter in field

[Jeff Zhang (JIRA)]

Jeff Zhang created ZEPPELIN-2992:


 Summary: Table can not be displayed correctly when there's 
delimiter in field
 Key: ZEPPELIN-2992
 URL: https://issues.apache.org/jira/browse/ZEPPELIN-2992
 Project: Zeppelin
  Issue Type: Improvement
Reporter: Jeff Zhang


{code}
%spark.ipyspark

df = spark.createDataFrame([("hello\nworld", 1), ("helloworld",2 )])
df.createOrReplaceTempView("df");

{code}
%spark.sql

select * from df
{code}

{code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (ZEPPELIN-2991) Content in table is not selectable

[Jeff Zhang (JIRA)]

Jeff Zhang created ZEPPELIN-2991:


 Summary: Content in table is not selectable 
 Key: ZEPPELIN-2991
 URL: https://issues.apache.org/jira/browse/ZEPPELIN-2991
 Project: Zeppelin
  Issue Type: Improvement
Reporter: Jeff Zhang


The table output in frontend is not selectable. Sometimes I need to copy the 
content in table. It would be better for it to be selectable. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] zeppelin pull request #2622: ZEPPELIN-2990. Matplotlib sometimes fails in IP...

[zjffdu]

GitHub user zjffdu opened a pull request:

https://github.com/apache/zeppelin/pull/2622

ZEPPELIN-2990. Matplotlib sometimes fails in IPythonInterpreter

### What is this PR for?
Trivial fix for matplotlib fail in `IPythonInterpreter`. Besides that, I 
also make some minor changes on zeppelin code to improve logging. 

### What type of PR is it?
[Bug Fix]

### Todos
* [ ] - Task

### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-2990


### How should this be tested?
UT is added.

### Screenshots (if appropriate)

### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/zjffdu/zeppelin ZEPPELIN-2990

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/zeppelin/pull/2622.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2622


commit 8f696cd4274cc15b700e8b86879e48747e227637
Author: Jeff Zhang 
Date:   2017-10-13T07:41:01Z

ZEPPELIN-2990. Matplotlib sometimes fails in IPythonInterpreter




---

[jira] [Created] (ZEPPELIN-2990) Matplotlib sometimes fails in IPythonInterpreter

[Jeff Zhang (JIRA)]

Jeff Zhang created ZEPPELIN-2990:


 Summary: Matplotlib sometimes fails in IPythonInterpreter
 Key: ZEPPELIN-2990
 URL: https://issues.apache.org/jira/browse/ZEPPELIN-2990
 Project: Zeppelin
  Issue Type: Bug
Reporter: Jeff Zhang






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] zeppelin issue #2619: Run all paragraphs sequentially

[zjffdu]

Github user zjffdu commented on the issue:

https://github.com/apache/zeppelin/pull/2619
  
@namanmishra91 I don't understand why it would invoke frontend. Backend 
know more context about paragraph status and backend could control the workflow 
of paragraph running easily. 


---

[GitHub] zeppelin issue #2605: [ZEPPELIN-2963] Fix paragraph aborting on next run aft...

[namanmishra91]

Github user namanmishra91 commented on the issue:

https://github.com/apache/zeppelin/pull/2605
  
Agreed that should be the ideal solution but considering how tightly it is 
integrated with the core execution workflow, this will be a risky change and 
needs to be done with utmost care to avoid any side-effects.


---

[GitHub] zeppelin issue #2619: Run all paragraphs sequentially

[namanmishra91]

Github user namanmishra91 commented on the issue:

https://github.com/apache/zeppelin/pull/2619
  
@zjffdu Yeah, I can change the implementation to make 
`persistAndExecuteSingleParagraph` paragraph blocking. Then we won't need UI 
side changes for separate event handling etc. However, we will still need other 
UI changes for paragraph state management because the execution workflow is 
changing. Just to make sure that we are on the same page, let me take a moment 
to explain why those will be needed:

Unless we want to change the zeppelin architecture of separate interpreter 
queues, we need to hold paragraphs to prevent them from being executed. 
Currently all paragraphs get submitted immediately and their status changes to 
PENDING. With the above approach, only one paragraph will be in any 
interpreter's queue at any given point of time; hence the status of _yet-to-be 
executed_ paragraphs will not get updated and the communication to the user 
that those paragraphs will run eventually will be lost. This will be a major 
change in the UX. We need to discuss and think about what should be the best 
way to handle this but as a preliminary implementation, we need to prevent 
explicit runs of the paragraphs that are not running yet. This will require 
maintaining state about whether a _run all_ is in progress and passing this to 
front-end.

Let me know your thoughts on the above.


---

[jira] [Created] (ZEPPELIN-2989) Anonymou not allowed exception when using zeppelinhub

[Jeff Zhang (JIRA)]

Jeff Zhang created ZEPPELIN-2989:


 Summary: Anonymou not allowed exception when using zeppelinhub
 Key: ZEPPELIN-2989
 URL: https://issues.apache.org/jira/browse/ZEPPELIN-2989
 Project: Zeppelin
  Issue Type: Bug
Reporter: Jeff Zhang


Although I can still use zeppelin hub correctly, but the following exception in 
log a annoying and suspicious.  

{code}
DEBUG [2017-10-13 14:44:44,202] ({pool-4-thread-9} 
ZeppelinHeartbeat.java[run]:41) - Sending PING to Zeppelin Websocket Server
ERROR [2017-10-13 14:44:44,203] ({qtp523691575-69} 
NotebookServer.java[onMessage]:360) - Can't handle message: 
{"op":"PING","data":{},"ticket":"anonymous","principal":"anonymous","roles":""}
java.lang.Exception: Anonymous access not allowed
at 
org.apache.zeppelin.socket.NotebookServer.onMessage(NotebookServer.java:194)
at 
org.apache.zeppelin.socket.NotebookSocket.onWebSocketText(NotebookSocket.java:59)
at 
org.eclipse.jetty.websocket.common.events.JettyListenerEventDriver.onTextMessage(JettyListenerEventDriver.java:128)
at 
org.eclipse.jetty.websocket.common.message.SimpleTextMessage.messageComplete(SimpleTextMessage.java:69)
at 
org.eclipse.jetty.websocket.common.events.AbstractEventDriver.appendMessage(AbstractEventDriver.java:65)
at 
org.eclipse.jetty.websocket.common.events.JettyListenerEventDriver.onTextFrame(JettyListenerEventDriver.java:122)
at 
org.eclipse.jetty.websocket.common.events.AbstractEventDriver.incomingFrame(AbstractEventDriver.java:161)
at 
org.eclipse.jetty.websocket.common.WebSocketSession.incomingFrame(WebSocketSession.java:309)
at 
org.eclipse.jetty.websocket.common.extensions.ExtensionStack.incomingFrame(ExtensionStack.java:214)
at org.eclipse.jetty.websocket.common.Parser.notifyFrame(Parser.java:220)
at org.eclipse.jetty.websocket.common.Parser.parse(Parser.java:258)
at 
org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.readParse(AbstractWebSocketConnection.java:632)
at 
org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.onFillable(AbstractWebSocketConnection.java:480)
at 
org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544)
at 
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
at 
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
at java.lang.Thread.run(Thread.java:745)
{code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] zeppelin issue #2592: ZEPPELIN-2685. Improvement on Interpreter class

[zjffdu]

Github user zjffdu commented on the issue:

https://github.com/apache/zeppelin/pull/2592
  
Will merge it if no more discussion. 


---

[GitHub] zeppelin issue #2612: [MINOR] remove the duplicate property username in Remo...

[wary]

Github user wary commented on the issue:

https://github.com/apache/zeppelin/pull/2612
  
@zjffdu done！thx for code review


---

[GitHub] zeppelin pull request #2585: [ZEPPELIN-2647] Bypass auth logic when a user l...

[yu74n]

GitHub user yu74n reopened a pull request:

https://github.com/apache/zeppelin/pull/2585

[ZEPPELIN-2647] Bypass auth logic when a user logins as admin role

### What is this PR for?
For administrator, make new admin role that assigned user can see all 
notebooks.

### What type of PR is it?
Improvement

### What is the Jira issue?
https://issues.apache.org/jira/browse/ZEPPELIN-2647

### How should this be tested?
1. Set role name to use as admin through ZEPPELIN_OWNER_ROLE =  
or zeppelin.owner.role = .
Default role name is admin
2. Login as user who is not assigned as admin and create notebook.
3. Logout the user and login another user who is assigned as admin, open 
the created notebook.

### Questions:
* Does the licenses files need update? N
* Is there breaking changes for older versions? Y/N
* Does this needs documentation? Y


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/yu74n/zeppelin bypass-auth-logic

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/zeppelin/pull/2585.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2585


commit d3e9dcab1bc2efc355140857b91a08bb2035bb69
Author: Yuta Hongo 
Date:   2017-08-10T01:38:01Z

Make admin role to bypass auth logic

commit cfcea0dc5f7d16aecc7e9a8249a01bcb63c38952
Author: Yuta Hongo 
Date:   2017-09-26T05:04:40Z

Rename property name




---

[GitHub] zeppelin pull request #2585: [ZEPPELIN-2647] Bypass auth logic when a user l...

[yu74n]

Github user yu74n closed the pull request at:

https://github.com/apache/zeppelin/pull/2585


---