[GitHub] zeppelin pull request #2411: [ZEPPELIN-2650] fix: DON'T use uppercase for th...
GitHub user 1ambda opened a pull request: https://github.com/apache/zeppelin/pull/2411 [ZEPPELIN-2650] fix: DON'T use uppercase for the first letter of each column name ### What is this PR for? Fixed not to use uppercase for the first letter of each column name ### What type of PR is it? [Bug Fix] ### Todos DONE ### What is the Jira issue? [ZEPPELIN-2650](https://issues.apache.org/jira/browse/ZEPPELIN-2650) ### How should this be tested? 1. Create a table result 2. Check the column names ### Screenshots (if appropriate) Before ![image](https://user-images.githubusercontent.com/4968473/27167154-51c6fdf2-51da-11e7-990c-ee8c3b8db10b.png) After ![image](https://user-images.githubusercontent.com/4968473/27167147-44962cd4-51da-11e7-8339-8c14eee305ef.png) ### Questions: * Does the licenses files need update? - NO * Is there breaking changes for older versions? - NO * Does this needs documentation? - NO You can merge this pull request into a Git repository by running: $ git pull https://github.com/1ambda/zeppelin ZEPPELIN-2650/do-not-apply-uppercase-for-column-name Alternatively you can review and apply these changes as the patch at: https://github.com/apache/zeppelin/pull/2411.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2411 commit b5afb5ae3922c65d7287c92a63fdc74c10eb8ca4 Author: 1ambda <1am...@gmail.com> Date: 2017-06-15T05:47:26Z fix: Set displayName to colDef to use original col name --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin pull request #2405: [ZEPPELIN-2640] Roles are not getting honored f...
Github user asfgit closed the pull request at: https://github.com/apache/zeppelin/pull/2405 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Created] (ZEPPELIN-2653) Apply virtual rendering for paragraphs to fix slow rendering
Hoon Park created ZEPPELIN-2653: --- Summary: Apply virtual rendering for paragraphs to fix slow rendering Key: ZEPPELIN-2653 URL: https://issues.apache.org/jira/browse/ZEPPELIN-2653 Project: Zeppelin Issue Type: Improvement Components: front-end Reporter: Hoon Park When a note has lots of paragraphs, note rendering is going to be slow. We can resolve this problem by applying virtual rendering. Here is a library we might use. - https://github.com/kamilkp/angular-vs-repeat -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[GitHub] zeppelin issue #2409: Revert layout change made by pr2356
Github user Leemoonsoo commented on the issue: https://github.com/apache/zeppelin/pull/2409 Thanks @1ambda for the review. Merge to master if no further comment. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin issue #2374: [ZEPPELIN-2593] Add storage settings to persist on com...
Github user khalidhuseynov commented on the issue: https://github.com/apache/zeppelin/pull/2374 yes that's true, i'll look into saving when stopping zeppelin --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin issue #2374: [ZEPPELIN-2593] Add storage settings to persist on com...
Github user 1ambda commented on the issue: https://github.com/apache/zeppelin/pull/2374 Can we make *stopping zeppelin* also trigger *persist action*? - usually, people expect *stopping zeppelin* will save their changes regardless of whatever mode they are using. they might think *Zeppelin lost my changes!* - I guess, the original intention is that saving external storage everytime is a burden. But **stopping zeppelin** occurs only once. It's not a burden. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin issue #2374: [ZEPPELIN-2593] Add storage settings to persist on com...
Github user khalidhuseynov commented on the issue: https://github.com/apache/zeppelin/pull/2374 @1ambda thanks for testing. yes that's the purpose of `save & commit`, the changes are applied in memory only unless it's explicitly saved and commit (persisted into file) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Created] (ZEPPELIN-2652) Can't open spark tutorial note in 0.8.0-SNAPSHOT
Hoon Park created ZEPPELIN-2652: --- Summary: Can't open spark tutorial note in 0.8.0-SNAPSHOT Key: ZEPPELIN-2652 URL: https://issues.apache.org/jira/browse/ZEPPELIN-2652 Project: Zeppelin Issue Type: Bug Affects Versions: 0.8.0 Environment: 0.8.0-SNAPSHOT Reporter: Hoon Park Fix For: 0.8.0 Can't open the spark tutorial note. - Build Zeppelin - Open `Zeppelin Tutorial/Basic Features (Spark)` I attached a screenshot and error log. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[GitHub] zeppelin issue #2374: [ZEPPELIN-2593] Add storage settings to persist on com...
Github user 1ambda commented on the issue: https://github.com/apache/zeppelin/pull/2374 @khalidhuseynov I tested and works well in usual cases. However, 1. Set `Persist on note commits` 2. Execute paragraph 3. Stop Zeppelin without committing. **In this case, The paragraph which was executed is not saved.** But users normally expect all their changes will be saved. What do you think of? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin pull request #2371: [ZEPPELIN-2596] Improving documentation page
Github user 1ambda closed the pull request at: https://github.com/apache/zeppelin/pull/2371 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin issue #2371: [ZEPPELIN-2596] Improving documentation page
Github user 1ambda commented on the issue: https://github.com/apache/zeppelin/pull/2371 Ready to be reviewed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin pull request #2371: [ZEPPELIN-2596] Improving documentation page
GitHub user 1ambda reopened a pull request: https://github.com/apache/zeppelin/pull/2371 [ZEPPELIN-2596] Improving documentation page ### What is this PR for? Improving documentation page. Please check *TODO* and *Screenshots* sections for detail. The motivation is described in [the JIRA ticket](https://issues.apache.org/jira/browse/ZEPPELIN-2583) and discussion is ongoing on the mailing list. ### What type of PR is it? [Improvement | Documentation] ### Todos * [x] - improved the navbar style * [x] - improved the main page * [x] - re-organized content structure * [x] - added tutorial pages: `spark_with_zeppelin.md`, `python_with_zeppelin.md`, `sql_with_zeppelin.md` for overview * [x] - added `multi_user_support.md` page to provide overview * [x] - added the empty `interpreter_binding_mode` page. This will be handed in the different issue: [ZEPPELIN-2582](https://issues.apache.org/jira/browse/ZEPPELIN-2582) * [x] - added the empty `trouble_shooting` page. This can be filled in the following PRs. * [x] - added the empty `useful_developer_tools` page. This can be filled in the following PRs. ### What is the Jira issue? [ZEPPELIN-2596](https://issues.apache.org/jira/browse/ZEPPELIN-2596) ### How should this be tested? 1. checkout 2. `cd docs` 3. `bundle install` (make sure that you have ruby 2.1.0+ and bundle) 4. `bundle exec jekyll serve --watch` 5. open `localhost:4000` ### Screenshots (if appropriate) better navbar: before ![2596_before_nav](https://cloud.githubusercontent.com/assets/4968473/26542353/89004e7a-4494-11e7-89c0-28d608f5f375.gif) better navbar: after ![2596_after_nav](https://cloud.githubusercontent.com/assets/4968473/26542356/8bfb7b90-4494-11e7-9979-0bcaef8ba97b.gif) improved main page: before ![2596_before_main](https://cloud.githubusercontent.com/assets/4968473/26542358/8f35b0be-4494-11e7-8a6c-e74ec52fc384.gif) improved main page: after ![2596_after_main](https://cloud.githubusercontent.com/assets/4968473/26542366/93b333c8-4494-11e7-981f-3f7b4545868f.gif) organized content structure: before ![2596_before_content](https://cloud.githubusercontent.com/assets/4968473/26542398/ad81ac26-4494-11e7-9a17-70dff41396fb.gif) organized content structure: after ![2596_after_content](https://cloud.githubusercontent.com/assets/4968473/26542403/b0a42ad2-4494-11e7-8bd3-8a5bd194c6af.gif) ### Questions: * Does the licenses files need update? - NO * Is there breaking changes for older versions? - NO * Does this needs documentation? - related with docs You can merge this pull request into a Git repository by running: $ git pull https://github.com/1ambda/zeppelin updating-version-doc Alternatively you can review and apply these changes as the patch at: https://github.com/apache/zeppelin/pull/2371.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2371 commit 3a1ad5e11510d9071dc33975a17eb178dd651725 Author: 1ambda <1am...@gmail.com> Date: 2017-05-25T02:22:51Z commit commit 43c51cfa25ee38f25368adbaa40d277318871a9d Author: 1ambda <1am...@gmail.com> Date: 2017-05-26T01:21:42Z feat: Style collapsed nav for mobile browser commit ce746740903171c0caf3697aa205255c927b9b5c Author: 1ambda <1am...@gmail.com> Date: 2017-05-26T01:24:52Z fix: Add RL padding for mobile browser commit 646f696ed374c668e4ad9388fb894961745ad451 Author: 1ambda <1am...@gmail.com> Date: 2017-05-26T01:29:02Z fix: Better image display in index.md commit 16af1429b888b194e7e0deed128fab8dedd5e55d Author: 1ambda <1am...@gmail.com> Date: 2017-05-26T05:13:44Z feat: Hide version in navbar when md commit 9da8ab780653764d0a32a27cba8224bcbf015981 Author: 1ambda <1am...@gmail.com> Date: 2017-05-26T05:40:20Z feat: Update doc section names commit 61b3fb2b360f57db5597fc5cc7d992f9b47131c6 Author: 1ambda <1am...@gmail.com> Date: 2017-05-26T06:15:03Z fix: doens't display navbar version in small commit 63b18f0d6555bc59a83c5119cc56c7ae893226b9 Author: 1ambda <1am...@gmail.com> Date: 2017-05-26T08:26:01Z feat: Change URLs in nav, index commit 65606d0a8d7a3e548735c9537ce0dee580ba1c4f Author: 1ambda <1am...@gmail.com> Date: 2017-05-26T08:46:43Z fix: Add missing links commit f0f96f7be9e8909f96432a8385b31cdcf9900d70 Author: 1ambda <1am...@gmail.com> Date: 2017-05-26T09:02:25Z feat: rename /quickstart commit c7f97b2e7a3da735a2ba5b738bbfc84d6ee9a128 Author: 1ambda <1am...@gmail.com> Date: 2017-05-26T13:02:20Z feat: Add docs/usage commit 0f0842a1bccd645a38b7310976bb3f23dda2dca8 Author: 1ambda <1am...@gmail.com> Date:
[GitHub] zeppelin issue #2309: [ZEPPELIN-2482] added font size for paragraph
Github user 1ambda commented on the issue: https://github.com/apache/zeppelin/pull/2309 LGTM --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Created] (ZEPPELIN-2651) I with that notebooks can be stored on Google cloud storage
MIN-FU YANG created ZEPPELIN-2651: - Summary: I with that notebooks can be stored on Google cloud storage Key: ZEPPELIN-2651 URL: https://issues.apache.org/jira/browse/ZEPPELIN-2651 Project: Zeppelin Issue Type: Wish Components: Core Reporter: MIN-FU YANG Priority: Minor -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (ZEPPELIN-2650) Column names from JDBC Interpretations don't respect capitalization.
Spencer Smith created ZEPPELIN-2650: --- Summary: Column names from JDBC Interpretations don't respect capitalization. Key: ZEPPELIN-2650 URL: https://issues.apache.org/jira/browse/ZEPPELIN-2650 Project: Zeppelin Issue Type: Bug Components: front-end, Interpreters Environment: Running .8 on CentOS7 Reporter: Spencer Smith Priority: Minor The names of columns in both tables and graphs do not respect the capitalization from the SQL query. For example: "select YTD_Revenue" would return a column named either "Ytd_Revenue" or "ytd_revenue" depending on if you're viewing the data in table or graph format. You can see example screenshots here: https://dl.dropboxusercontent.com/1/view/ywp0d5pxrgpe37o/Apps/Shutter/Selection_025.png https://dl.dropboxusercontent.com/1/view/52n3tz16er44atz/Apps/Shutter/Selection_026.png -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (ZEPPELIN-2649) Presto JDBC Driver Slow
Charles Pritchard created ZEPPELIN-2649: --- Summary: Presto JDBC Driver Slow Key: ZEPPELIN-2649 URL: https://issues.apache.org/jira/browse/ZEPPELIN-2649 Project: Zeppelin Issue Type: Bug Reporter: Charles Pritchard Using the latest PrestoDB JDBC driver as an interpreter, it's taking 40 seconds to run a simple "show tables" statement -- this returns nearly instantly using the CLI and other methods. No idea what the slow-down is caused by. com.facebook.presto.jdbc.PrestoDriver -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[GitHub] zeppelin issue #2404: fix bugs for: when interpreter setting restart of one ...
Github user wary commented on the issue: https://github.com/apache/zeppelin/pull/2404 @jongyoul thx for testing my code, any question, please feel free to let me know, thank you --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin pull request #2407: [ZEPPELIN-1907] Shell Interpreter does not rene...
GitHub user prabhjyotsingh reopened a pull request: https://github.com/apache/zeppelin/pull/2407 [ZEPPELIN-1907] Shell Interpreter does not renew ticket on secure cluster ### What is this PR for? Kerberos ticket and renew lifetime are set to 1 hour. On accessing secure Hadoop from shell interpreter, it does kinit and returns result successfully but after 1 hour, the ticket gets expired and Hadoop list fails with below exception. ``` %sh hadoop fs -ls / 17/01/05 09:29:45 WARN ipc.Client: Exception encountered while connecting to the server : javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)] at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:211) at org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:413) at org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:595) at org.apache.hadoop.ipc.Client$Connection.access$2000(Client.java:397) at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:762) at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:758) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724) at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:757) at org.apache.hadoop.ipc.Client$Connection.access$3200(Client.java:397) at org.apache.hadoop.ipc.Client.getConnection(Client.java:1618) at org.apache.hadoop.ipc.Client.call(Client.java:1449) at org.apache.hadoop.ipc.Client.call(Client.java:1396) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:233) at com.sun.proxy.$Proxy10.getFileInfo(Unknown Source) ls: Failed on local exception: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]; Host Details : local host is: "zeppelin1.hwxblr.com/10.0.1.57"; destination host is: "zeppelin1.hwxblr.com":8020; ExitValue: 1 ``` ### What type of PR is it? [Bug Fix] ### What is the Jira issue? * [ZEPPELIN-1907](https://issues.apache.org/jira/browse/ZEPPELIN-1907) ### How should this be tested? On a Kerberos enabled cluster, run this paragraph ``` %sh hdfs dfs -ls /user/zeppelin/ ``` Wait for key-tab to expire (or run `kdestroy`), and re-run the same paragraph. ### Screenshots (if appropriate) Before: https://user-images.githubusercontent.com/674497/27078184-511ed810-5050-11e7-8afa-90247f33047a.png;> After: https://user-images.githubusercontent.com/674497/27078183-5109d690-5050-11e7-82e4-d79a5e98295f.png;> ### Questions: * Does the licenses files need update? * Is there breaking changes for older versions? * Does this needs documentation? You can merge this pull request into a Git repository by running: $ git pull https://github.com/prabhjyotsingh/zeppelin ZEPPELIN-1907 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/zeppelin/pull/2407.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2407 commit ab823d3ee9373c38f5f083d3471adf351c9177a2 Author: Prabhjyot SinghDate: 2017-06-13T10:18:44Z relogin using keytab, and append message for the same commit 7c539ef2eb943b30befc97bd5a15120e8d9ee42e Author: Prabhjyot Singh Date: 2017-06-13T14:11:11Z add null check commit ee741e4835b94104baa530dcdc8933d6d8ae Author: Prabhjyot Singh Date: 2017-06-14T11:26:19Z @zjffdu review comments --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin pull request #2407: [ZEPPELIN-1907] Shell Interpreter does not rene...
Github user prabhjyotsingh closed the pull request at: https://github.com/apache/zeppelin/pull/2407 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
Re: [DISCUSSION] Extending TableData API
>>> But not sure about how other interpreters can do the same thing. (e.g trivial, but let’s think about shell interpreter which keeps it’s tabledata on memory) The approach I proposed is general for all the interpreters. What we need do is to add one method in RemoteInterpreterProcess for other interpreters to fetch resources. >>> Some people might wonder why we do not use external storages to persist (large) table resources instead of keeping them in memory of ZeppelinServer. It is fine to use memory for now. But we should leave an interface there for other storages. For now we could just have MemoryStorage, could have other implementations in future. Park Hoon <1am...@gmail.com>于2017年6月14日周三 下午10:22写道: > @Jeff, Thanks for sharing your opinions and important questions. > > > > Q1. What does the resource registration mean? IIUC, currently it means > it would cache the data in Interpreter Process. Then it might be a memory > issue when more and more resources are registered. Maybe we could introduce > resource retention mechanism or cache the data in other formats (just like > the spark table cache policy, user can specify how to cache the data, like > memory, disk and etc. > > A1. It depends on an implementation of TableData for each interpreter. > For example, > > If JDBC interpreter only keeps the SQL in a paragraph to reproduce the > table, we don’t need to persist the whole table data in memory or file > system or an external storage. That’s what the section 3.2 describes. > > [image: Inline image 2] > > > > > > Q2. The scope of resource sharing. For now, it seems it is globally > shared. But I think user level sharing might be more common. Then we need > to create a namespace for each user. That means the same resource name > could exist in different user namespace. > > A2. Regarding the namespace concept, the proposal only describes what the > table resource name should be? (Section 5.3) not about namespaces. > > The namespace can be the name of a note or custom (e.g creating users’ > namespace). We can discuss this. > > Personally, +1 for having namespace because it is helpful for searching > and sharing. This might be included by `ResourceRegistry` > > > [image: Inline image 1] > > > > Q3. The data route might cause performance issue. From the diagram, If > spark interpreter needs to access a resource from jdbc interpreter. Then > first data needs to be send to zeppelin server, and then zeppelin server > send the data to spark interpreter. This kind of data route introduce a bit > more overhead to me. And zeppelin server will become a bottleneck and > require large memory when there're many resources to be shared across > users/interpreters. So I would suggest the following approach. Zeppelin > Server just control the metadata and ACL of resources. And Spark > Interpreter will fetch data from Jdbc Interpreter directly instead of > through zeppelin server. Here's the sequences >1). SparkInterpreter ask for metadata and token for the resource >2). Zeppelin Server will check whether this SparkInterprter has > permission to access this resource, if yes, then send the metadata and > token to SparkInterpreter. The metadata includes the RPC address of the > JdbcInterpreter and token is for security. >3). SparkInterpreter ask JdbcInterpreter for the resource via the > the token and metadata received in step 2 >4). JdbcInterpreter verify the token, and send the data to > SparkInterpreter. > > A3. +1 direct accessing in spark interpreter to JDBC since it’s better for > large data handling. But not sure about how other interpreters can do the > same thing. (e.g trivial, but let’s think about shell interpreter which > keeps it’s tabledata on memory) > > > -- > > Some people might wonder why we do not use external storages to persist > (large) table resources instead of keeping them in memory of ZeppelinServer. > > The authors originally discussed whether having an external storage or > not. But having external storage > > - requires additional (lots of) dependencies. (Geode? Redis? HDFS? Which > one should we use? or support all?) > - even with external storage, we might not persist 400GB, 10TB. > > Thus, the proposal was written to > > - utilize interpreter’s own storage (e.g spark cluster for spark > interpreter) > - keep the minimal things to reproduce the table result (e.g keeping the > only query) while don’t affect on external storage as well at first. > > > And now we are discussing, hope we can improve the proposal and turn it > into a reall implementation soon. :) > > > > Thanks. > > > > > On Wed, Jun 14, 2017 at 12:20 PM, Jeff Zhangwrote: > >> >> Hi Park, >> >> Thanks for the sharing, this is a very interested and innovated idea. I >> have several comments and concerns. >> >> 1. What does the resource registration mean ? >>IIUC, currently it means it would cache the data in Interpreter >> Process. Then it
Re: [DISCUSSION] Extending TableData API
@Jeff, Thanks for sharing your opinions and important questions. > Q1. What does the resource registration mean? IIUC, currently it means it would cache the data in Interpreter Process. Then it might be a memory issue when more and more resources are registered. Maybe we could introduce resource retention mechanism or cache the data in other formats (just like the spark table cache policy, user can specify how to cache the data, like memory, disk and etc. A1. It depends on an implementation of TableData for each interpreter. For example, If JDBC interpreter only keeps the SQL in a paragraph to reproduce the table, we don’t need to persist the whole table data in memory or file system or an external storage. That’s what the section 3.2 describes. [image: Inline image 2] > Q2. The scope of resource sharing. For now, it seems it is globally shared. But I think user level sharing might be more common. Then we need to create a namespace for each user. That means the same resource name could exist in different user namespace. A2. Regarding the namespace concept, the proposal only describes what the table resource name should be? (Section 5.3) not about namespaces. The namespace can be the name of a note or custom (e.g creating users’ namespace). We can discuss this. Personally, +1 for having namespace because it is helpful for searching and sharing. This might be included by `ResourceRegistry` [image: Inline image 1] > Q3. The data route might cause performance issue. From the diagram, If spark interpreter needs to access a resource from jdbc interpreter. Then first data needs to be send to zeppelin server, and then zeppelin server send the data to spark interpreter. This kind of data route introduce a bit more overhead to me. And zeppelin server will become a bottleneck and require large memory when there're many resources to be shared across users/interpreters. So I would suggest the following approach. Zeppelin Server just control the metadata and ACL of resources. And Spark Interpreter will fetch data from Jdbc Interpreter directly instead of through zeppelin server. Here's the sequences 1). SparkInterpreter ask for metadata and token for the resource 2). Zeppelin Server will check whether this SparkInterprter has permission to access this resource, if yes, then send the metadata and token to SparkInterpreter. The metadata includes the RPC address of the JdbcInterpreter and token is for security. 3). SparkInterpreter ask JdbcInterpreter for the resource via the the token and metadata received in step 2 4). JdbcInterpreter verify the token, and send the data to SparkInterpreter. A3. +1 direct accessing in spark interpreter to JDBC since it’s better for large data handling. But not sure about how other interpreters can do the same thing. (e.g trivial, but let’s think about shell interpreter which keeps it’s tabledata on memory) -- Some people might wonder why we do not use external storages to persist (large) table resources instead of keeping them in memory of ZeppelinServer. The authors originally discussed whether having an external storage or not. But having external storage - requires additional (lots of) dependencies. (Geode? Redis? HDFS? Which one should we use? or support all?) - even with external storage, we might not persist 400GB, 10TB. Thus, the proposal was written to - utilize interpreter’s own storage (e.g spark cluster for spark interpreter) - keep the minimal things to reproduce the table result (e.g keeping the only query) while don’t affect on external storage as well at first. And now we are discussing, hope we can improve the proposal and turn it into a reall implementation soon. :) Thanks. On Wed, Jun 14, 2017 at 12:20 PM, Jeff Zhangwrote: > > Hi Park, > > Thanks for the sharing, this is a very interested and innovated idea. I > have several comments and concerns. > > 1. What does the resource registration mean ? >IIUC, currently it means it would cache the data in Interpreter > Process. Then it might be a memory issue when more and more resources are > registered. Maybe we could introduce resource retention mechanism or cache > the data in other formats (just like the spark table cache policy, user can > specify how to cache the data, like memory, disk and etc.) > > 2. The scope of resource sharing >For now, it seems it is globally shared. But I think user level sharing > might be more common. Then we need to create a namespace for each user. > That means the same resource name could exist in different user namespace. > > 3. The data route might cause performance issue. >From the diagram, If spark interpreter needs to access a resource from > jdbc interpreter. Then first data needs to be send to zeppelin server, and > then zeppelin server send the data to spark interpreter. This kind of data > route introduce a bit more overhead to me. And zeppelin server will
[GitHub] zeppelin issue #2398: [ZEPPELIN-2622] Add Zeppelin Git info REST API
Github user jongyoul commented on the issue: https://github.com/apache/zeppelin/pull/2398 Merging if there's no more discussion --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin issue #2373: [ZEPPELIN-2598] Securing Zeppelin with OpenID Connect
Github user volumeint commented on the issue: https://github.com/apache/zeppelin/pull/2373 I just submitted a [pull request to buji-pac4j](https://github.com/bujiio/buji-pac4j/pull/62) to make the value of Principal.getName() configurable via shiro.ini. We just have to wait for it to be accepted and released. I will provide some documentation on integrating with one of the social OAuth providers after I clean up my zeppelin pull request. An ounce of code can save a pound of documentation. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin pull request #2410: [ZEPPELIN-2645] Adding way to register RemoteIn...
GitHub user jongyoul opened a pull request: https://github.com/apache/zeppelin/pull/2410 [ZEPPELIN-2645] Adding way to register RemoteInterpreterServer's port into InterpreterProcess ### What is this PR for? Changing the way to register interpreter to server. interpreter tells server its port of thrift server, and server connect interpreter through this information ### What type of PR is it? [Feature] ### Todos * [x] - And netty session for transferring data ### What is the Jira issue? * https://issues.apache.org/jira/browse/ZEPPELIN-2645 ### How should this be tested? 1. Set `SPARK_HOME` and `HADOOP_CONF_DIR` 1. Set 'yarn-cluster' to spark's master 1. Run a script ### Screenshots (if appropriate) ![image](https://user-images.githubusercontent.com/3612566/27132701-c9b3d708-514a-11e7-8892-f134d0423c84.png) ### Questions: * Does the licenses files need update? No * Is there breaking changes for older versions? No * Does this needs documentation? No You can merge this pull request into a Git repository by running: $ git pull https://github.com/jongyoul/zeppelin ZEPPELIN-2645 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/zeppelin/pull/2410.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2410 commit 0570de5dc3c9f4be933bff18dd6fd2aa11536768 Author: Jongyoul LeeDate: 2017-06-14T10:43:49Z Added getters --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin issue #2374: [ZEPPELIN-2593] Add storage settings to persist on com...
Github user 1ambda commented on the issue: https://github.com/apache/zeppelin/pull/2374 Thanks for the contribution. Let me test and comment soon. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin pull request #2391: [ZEPPELIN-2501] Better Job Navigation Experienc...
GitHub user 1ambda reopened a pull request: https://github.com/apache/zeppelin/pull/2391 [ZEPPELIN-2501] Better Job Navigation Experience ### What is this PR for? Improve UX for Job Navigation. Please refer the *TODOs* section and screenshots attached below for detail. FYI, **pagination** is added to improve page loading speed. Currently, it is going to be slow when there are 50+ jobs. ### What type of PR is it? [Improvement | Feature] ### Todos * [x] - improve UI * [x] - add pagination to improve page loading speed * [x] - interactive job searching without *enter key* * [x] - add date sorter * [x] - display total job count ### What is the Jira issue? [ZEPPELIN-2501](https://issues.apache.org/jira/browse/ZEPPELIN-2501) ### How should this be tested? 1. build: `mvn clean package -DskipTests; ./bin/zeppelin-daemon.sh restart` 2. open the job page: `http://localhost:8080/#/jobmanager` 3. try to search, filter, sort. ### Screenshots (if appropriate) Before ![image](https://cloud.githubusercontent.com/assets/4968473/26729918/39f51c8a-47ea-11e7-88a1-30ffa5947e3a.png) After: Larger UI components ![image](https://cloud.githubusercontent.com/assets/4968473/26729934/48ea4c74-47ea-11e7-91f0-4aa97bb93161.png) After: Interactive search without enter-key ![2501_interactive](https://cloud.githubusercontent.com/assets/4968473/2673/7f39045a-47ea-11e7-8377-fc9d12b12237.gif) After Newly added timestamp sorter ![2501_timestamp](https://cloud.githubusercontent.com/assets/4968473/26730073/c50ea656-47ea-11e7-8c5d-db61064ca745.gif) After: Newly added pagination ![2501_pagination](https://cloud.githubusercontent.com/assets/4968473/26730130/ef173c06-47ea-11e7-9e5d-818693d3220a.gif) Total Job Count ![image](https://cloud.githubusercontent.com/assets/4968473/26747803/55bf15a8-4833-11e7-9444-b0c69f4bd9b6.png) ### Questions: * Does the licenses files need update? - NO * Is there breaking changes for older versions? - NO * Does this needs documentation? - NO You can merge this pull request into a Git repository by running: $ git pull https://github.com/1ambda/zeppelin ZEPPELIN-2501/pagination-for-job-page Alternatively you can review and apply these changes as the patch at: https://github.com/apache/zeppelin/pull/2391.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2391 commit 0ae31a01bc600ef342ba850fd29f27020e5f1373 Author: 1ambda <1am...@gmail.com> Date: 2017-06-02T08:32:57Z style: Reindent jobmanager.html commit f8fab3f10bb4c49f1b807ae1bfcf24f2e9723f73 Author: 1ambda <1am...@gmail.com> Date: 2017-06-02T09:02:04Z refactor: Remove duplicated switch DOMs commit 018048af11f9f76c500f928081b07c342f66090a Author: 1ambda <1am...@gmail.com> Date: 2017-06-02T09:10:31Z fix: Simplify job names commit 084407a1eda593d6bda8145746a190894c0dc6d9 Author: 1ambda <1am...@gmail.com> Date: 2017-06-02T09:43:04Z feat: Add pagination for JOB page commit d7017f41c16e2a84745eef3196fa31031da002be Author: 1ambda <1am...@gmail.com> Date: 2017-06-02T10:02:29Z fix: Better layout for jobmanager header commit d52008e1f3352e8b6234ecd48d5ffbb30774bf27 Author: 1ambda <1am...@gmail.com> Date: 2017-06-02T10:13:43Z fix: Remove all styles in search tools commit 081b9bb4ecae2c0491fa44b7967579b4096a04b8 Author: 1ambda <1am...@gmail.com> Date: 2017-06-02T11:00:57Z feat: re-style search tools commit e20d1dbe1efa2dd89cfccfb36ddd7ad58e43c0f6 Author: 1ambda <1am...@gmail.com> Date: 2017-06-02T12:28:31Z fix: Apply interactive search commit 3dd50815560b60f460de46ed0ef608a44bfbfc2c Author: 1ambda <1am...@gmail.com> Date: 2017-06-02T14:12:26Z feat: Add date sorter commit c5ff1f5a2fae1661d40834197aac70a9a1f2aa07 Author: 1ambda <1am...@gmail.com> Date: 2017-06-02T14:27:01Z fix: Add license notation to job-status commit 53769661edd6945e8c854fa8d4725c62320e9ba2 Author: 1ambda <1am...@gmail.com> Date: 2017-06-02T23:04:14Z feat: Add total job count --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin pull request #2391: [ZEPPELIN-2501] Better Job Navigation Experienc...
Github user 1ambda closed the pull request at: https://github.com/apache/zeppelin/pull/2391 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin pull request #2309: [ZEPPELIN-2482] added font size for paragraph
GitHub user tinkoff-dwh reopened a pull request: https://github.com/apache/zeppelin/pull/2309 [ZEPPELIN-2482] added font size for paragraph ### What is this PR for? 1. Added possibilty to change font size of paragraph text 2. Fixed cloneParagraphAction (immediately after the cloning is incorrectly displayed a new paragraph) 3. Fixed paragraph settings into Firefox ### What type of PR is it? Feature ### What is the Jira issue? https://issues.apache.org/jira/browse/ZEPPELIN-2482 ### How should this be tested? 1. Open note 2. Add some text to paragraph 3. Open settings of paragraph and chenge "Font size" ### Screenshots (if appropriate) 1. Font size and clone BEFORE ![before_clone](https://cloud.githubusercontent.com/assets/25951039/25618618/21852dee-2f61-11e7-80f5-634320ffc1ce.gif) AFTER ![after_clone](https://cloud.githubusercontent.com/assets/25951039/25618637/2d8f484a-2f61-11e7-8f94-2f95026098f4.gif) 2. Fix for Firefox BEFORE ![before_float_firefox](https://cloud.githubusercontent.com/assets/25951039/25618663/44c7d19e-2f61-11e7-8513-ee74eab4e8f4.png) AFTER ![after_float_firefox](https://cloud.githubusercontent.com/assets/25951039/25618677/4bf49c90-2f61-11e7-8ade-14d501a4e239.png) ### Questions: * Does the licenses files need update? no * Is there breaking changes for older versions? no * Does this needs documentation? no You can merge this pull request into a Git repository by running: $ git pull https://github.com/tinkoff-dwh/zeppelin ZEPPELIN-2482 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/zeppelin/pull/2309.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2309 commit 5e2be167d5a532e7d9522bdedee1c5cde360be66 Author: Tinkoff DWHDate: 2017-05-02T11:18:04Z [ZEPPELIN-2482] font size for paragraph commit 794bbf1b10e7926dd5917bdd727b4245630b9255 Author: Tinkoff DWH Date: 2017-05-02T12:34:49Z [ZEPPELIN-2482] close settings commit 62ef72860dde7c7b50e6b15043fea9ef38884bd9 Author: Tinkoff DWH Date: 2017-05-02T15:32:07Z [ZEPPELIN-2482] font size test commit 42135ad078dce603de57fd17c975c1dcb3121611 Author: Tinkoff DWH Date: 2017-05-15T07:47:31Z Merge remote-tracking branch 'upstream/master' into ZEPPELIN-2482 commit fea747097c243a21f077994271ccc2318384a120 Author: Tinkoff DWH Date: 2017-06-12T10:43:50Z Merge remote-tracking branch 'upstream/master' into ZEPPELIN-2482 commit a8aa65ded60b78ff8374d47379e5e7a4c6f9d12a Author: Tinkoff DWH Date: 2017-06-12T11:13:10Z Merge remote-tracking branch 'origin/ZEPPELIN-2482' into ZEPPELIN-2482 commit 5b92b285628f838f6971a997de59ebdb12a4be9b Author: Tinkoff DWH Date: 2017-06-12T17:11:32Z [ZEPPELIN-2482] broadcast changes of fontSize commit 0197fd16bafcc7f4f202821bf95aadf26104a62d Author: Tinkoff DWH Date: 2017-06-14T09:15:36Z [ZEPPELIN-2482] fix selenium tests commit abf9e4520a309188e1c55dc63e774f4d5218f9dc Author: Tinkoff DWH Date: 2017-06-14T09:38:58Z [ZEPPELIN-2482] fix selenium tests --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin pull request #2309: [ZEPPELIN-2482] added font size for paragraph
Github user tinkoff-dwh closed the pull request at: https://github.com/apache/zeppelin/pull/2309 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin issue #2391: [ZEPPELIN-2501] Better Job Navigation Experience
Github user 1ambda commented on the issue: https://github.com/apache/zeppelin/pull/2391 @khalidhuseynov Thanks for the review. The error was fixed in a different PR. I just rebased to apply it. :) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Created] (ZEPPELIN-2648) zeppelin-spark_2.10 on maven central is not compiled based on scala-2.10
slee created ZEPPELIN-2648: -- Summary: zeppelin-spark_2.10 on maven central is not compiled based on scala-2.10 Key: ZEPPELIN-2648 URL: https://issues.apache.org/jira/browse/ZEPPELIN-2648 Project: Zeppelin Issue Type: Bug Components: Interpreters Affects Versions: 0.7.1 Reporter: slee zeppelin-spark_2.10 artifcat on maven central is not compiled based on scala-2.10. So the artifact get from the below command is not of scala-2.10. ./bin/install-interpreter.sh —name spark —artifact org.apache.zeppelin:zeppelin-spark_2.10:0.7.1. The maven link is https://mvnrepository.com/artifact/org.apache.zeppelin/zeppelin-spark_2.10/0.7.1. If you decompile the class ZeppelinRDisplay.class on the artifact and check the htmlDisplay method ```scala private RDisplay htmlDisplay(Element body, String imageWidth) { ObjectRef div = ObjectRef.create(new String()); JavaConversions..MODULE$.asScalaBuffer(body.children()).foreach(new AbstractFunction1() { public static final long serialVersionUID = 0L; public final void apply(Element element) { String eHtml = element.html(); String eOuterHtml = element.outerHtml(); eOuterHtml = eOuterHtml.replace("“%html ``` You can see it calls ObjectRef.create which method is from scala 2.11 not scala 2.10. This problem will influence those who use R with spark_2.10. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[GitHub] zeppelin issue #2401: [gh-pages] [ZEPPELIN-2597] updating navbar and main se...
Github user soralee commented on the issue: https://github.com/apache/zeppelin/pull/2401 @1ambda Thank you for all your comment. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin issue #2401: [gh-pages] [ZEPPELIN-2597] updating navbar and main se...
Github user 1ambda commented on the issue: https://github.com/apache/zeppelin/pull/2401 LGTM --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin issue #2409: Revert layout change made by pr2356
Github user 1ambda commented on the issue: https://github.com/apache/zeppelin/pull/2409 Tested and works well. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin issue #2309: [ZEPPELIN-2482] added font size for paragraph
Github user tinkoff-dwh commented on the issue: https://github.com/apache/zeppelin/pull/2309 sorry it is my fault. i added new dropdown menu (font size) so order of elements (select) is changed --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin issue #2401: [gh-pages] [ZEPPELIN-2597] updating navbar and main se...
Github user 1ambda commented on the issue: https://github.com/apache/zeppelin/pull/2401 @soralee LGTM except for one thing, h1 is hidden in mobile size browser. ![image](https://user-images.githubusercontent.com/4968473/27124501-3821ef96-512c-11e7-9007-1f6f38799ab5.png) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin issue #2401: [gh-pages] [ZEPPELIN-2597] updating navbar and main se...
Github user soralee commented on the issue: https://github.com/apache/zeppelin/pull/2401 Fixed 6.2 comments and, here is more improvement list 1. main section - changed PR phrase to `font-size : 22px`, `line-height: 1.4` - added padding-top/bottom in main section - changed button font and weight - added button margin-side - added button interval - added interval between PR phrase in main section and buttons 2. revert to original what's new section --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Created] (ZEPPELIN-2647) Set admin user as owner when user create a notebook
Yuta Hongo created ZEPPELIN-2647: Summary: Set admin user as owner when user create a notebook Key: ZEPPELIN-2647 URL: https://issues.apache.org/jira/browse/ZEPPELIN-2647 Project: Zeppelin Issue Type: Improvement Reporter: Yuta Hongo I think an administrator would want to see queries a user writes in the situation on trouble shooting. In private mode, a note book's owner will be set as the user who create the notebook by default, then an administrator cannot see the notebook. I have an idea to realise the feature. # Add a new field in configuration file to set default owner # Set the default owners in the configuration file when user creates new notebook. If there are any features to realise that things already, please let me know. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[GitHub] zeppelin issue #2407: [ZEPPELIN-1907] Shell Interpreter does not renew ticke...
Github user prabhjyotsingh commented on the issue: https://github.com/apache/zeppelin/pull/2407 @zjffdu Yes make sense I have created a JIRA (https://issues.apache.org/jira/browse/ZEPPELIN-2646) for it, I'll handle it in a different PR. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Created] (ZEPPELIN-2646) Code duplication and inconsistency between ShellInterpreter and ShellSecurityImpl.
Prabhjyot Singh created ZEPPELIN-2646: - Summary: Code duplication and inconsistency between ShellInterpreter and ShellSecurityImpl. Key: ZEPPELIN-2646 URL: https://issues.apache.org/jira/browse/ZEPPELIN-2646 Project: Zeppelin Issue Type: Improvement Reporter: Prabhjyot Singh This is based on the comment https://github.com/apache/zeppelin/pull/2407#issuecomment-308289991, where we have identified there is a code duplication and inconsistency between ShellInterpreter and ShellSecurityImpl. Along with that, there is a case where while ShellInterpreter is trying to kinit it cannot be canceled. -- This message was sent by Atlassian JIRA (v6.4.14#64029)