+1 on @rick. quality is really important... I am still encountering bugs consistently
On Tue, Mar 1, 2016 at 10:16 AM, TEJA SRIVASTAV <tejasrivas...@gmail.com> wrote: > +1 on @rick > > On Tue, Mar 1, 2016 at 11:26 PM Benjamin Kim <bbuil...@gmail.com> wrote: > >> I see in the Enterprise section that multi-tenancy will be included, will >> this have user impersonation too? In this way, the user executing will be >> the user owning the process. >> >> On Mar 1, 2016, at 12:51 AM, Shabeel Syed <shabeels...@gmail.com> wrote: >> >> +1 >> >> Hi Tamas, >> Pluggable external visualization is really a GREAT feature to have. >> I'm looking forward to this :) >> >> Regards >> Shabeel >> >> On Tue, Mar 1, 2016 at 2:16 PM, Tamas Szuromi <tamas.szur...@odigeo.com> >> wrote: >> >>> Hey, >>> >>> Really promising roadmap. >>> >>> I'd only push more visualization options. I agree built in >>> visualization is needed with limited charting options but I think we also >>> need somehow 'inject' external js visualizations also. >>> >>> >>> For scheduling Zeppelin notebooks we use >>> https://github.com/airbnb/airflow <https://github.com/airbnb/airflow> >>> through >>> the job rest api. It's an enterprise ready and very robust solution >>> right now. >>> >>> >>> *Tamas* >>> >>> On 1 March 2016 at 09:12, Eran Witkon <eranwit...@gmail.com> wrote: >>> >>>> One point to clarify, I don't want to suggest Oozie in specific, I want >>>> to think about which features we develop and which ones we integrate >>>> external, preferred Apache, technology? We don't think about building our >>>> own storage services so why build our own scheduler? >>>> Eran >>>> On Tue, 1 Mar 2016 at 09:49 moon soo Lee <m...@apache.org> wrote: >>>> >>>>> @Vinayak, @Eran, @Benjamin, @Guilherme, @Sourav, @Rick >>>>> Now I can see a lot of demands around enterprise level job scheduling. >>>>> Either external or built-in, I completely agree having enterprise level >>>>> job >>>>> scheduling support on the roadmap. >>>>> ZEPPELIN-137 <https://issues.apache.org/jira/browse/ZEPPELIN-137>, >>>>> ZEPPELIN-531 <https://issues.apache.org/jira/browse/ZEPPELIN-531> are >>>>> related issues i can find in our JIRA. >>>>> >>>>> @Vinayak >>>>> Regarding importing notebook from github, Zeppelin has pluggable >>>>> notebook storage layer (see related package >>>>> <https://github.com/apache/incubator-zeppelin/tree/master/zeppelin-zengine/src/main/java/org/apache/zeppelin/notebook/repo>). >>>>> So, github notebook sync can be implemented easily. >>>>> >>>>> @Shabeel >>>>> Right, we need better manage management to prevent such OOM. >>>>> And i think table is one of the most frequently used way of displaying >>>>> data. So definitely, we'll need more features like filter, sort, etc. >>>>> After this roadmap discussion, discussion for the next release will >>>>> follow. Then we'll get idea when those features will be available. >>>>> >>>>> @Prasad >>>>> Thanks for mentioning HA and DR. They're really important subject for >>>>> enterprise use. Definitely Zeppelin will need to address them. >>>>> And displaying meta information of notebook on top level page is good >>>>> idea. >>>>> >>>>> It's really great to hear many opinions and ideas. >>>>> And thanks @Rick for sharing valuable view to Zeppelin project. >>>>> >>>>> Thanks, >>>>> moon >>>>> >>>>> >>>>> On Mon, Feb 29, 2016 at 11:14 PM Rick Moritz <rah...@gmail.com> wrote: >>>>> >>>>>> Hi, >>>>>> >>>>>> For one, I know that there is rudimentary scheduling built into >>>>>> Zeppelin already (at least I fixed a bug in the test for a scheduling >>>>>> feature a few months ago). >>>>>> But another point is, that Zeppelin should also focus on quality, >>>>>> reproduceability and portability. >>>>>> Although this doesn't offer exciting new features, it would make >>>>>> development much easier. >>>>>> >>>>>> Cross-platform testability, Tests that pass when run sequentially, >>>>>> compatibility with Firefox, and many more open issues that make it so >>>>>> much >>>>>> harder to enhance Zeppelin and add features should be addressed soon, >>>>>> preferably before more features are added. Already Zeppelin is suffering >>>>>> - >>>>>> in my opinion - from quite a lot of feature creep, and we should avoid >>>>>> putting in the kitchen sink, at the cost of quality and maintainability. >>>>>> Instead modularity (ZEPPELIN-533 in particular) should be targeted. >>>>>> >>>>>> Oozie, in my opinion, is a dead end - it may de-facto still be in use >>>>>> on many clusters, but it's not getting the love it needs, and I wouldn't >>>>>> bet on it, when it comes to integrating scheduling. Instead, any external >>>>>> tool should be able to use the REST-API to trigger executions, if you >>>>>> want >>>>>> external scheduling. >>>>>> >>>>>> So, in conclusion, if we take Moon's list as a list of descending >>>>>> priorities, I fully agree, under the condition that code quality is >>>>>> included as a subset of enterprise-readyness. Auth* is paramount >>>>>> (Kerberos >>>>>> SPNEGO SSO support is what we really want) with user and group rights >>>>>> assignment on the notebook level. We probably also need Knox-integration >>>>>> (ODP-Members looking at integrating Zeppelin should consider contributing >>>>>> this), and integration of something like Spree ( >>>>>> https://github.com/hammerlab/spree) to be able to profile jobs. >>>>>> >>>>>> I'm hopeful that soon I can resume contributing some quality-oriented >>>>>> code, to drive this "necessary evil" forward ;) >>>>>> >>>>>> On Mon, Feb 29, 2016 at 8:27 PM, Sourav Mazumder < >>>>>> sourav.mazumde...@gmail.com> wrote: >>>>>> >>>>>>> I do agree with Vinayak. It need not be coupled with Oozie. >>>>>>> >>>>>>> Rather one should be able to call it from any scheduler typically >>>>>>> used in enterprise level. May be support for BPML. >>>>>>> >>>>>>> I believe the existing ability to call/execute a Zeppelin Notebook >>>>>>> or a specific paragraph within a notebook using REST API should take >>>>>>> care >>>>>>> of this requirement to some extent. >>>>>>> >>>>>>> Regards, >>>>>>> Sourav >>>>>>> >>>>>>> On Mon, Feb 29, 2016 at 11:23 AM, Vinayak Agrawal < >>>>>>> vinayakagrawa...@gmail.com> wrote: >>>>>>> >>>>>>>> @Eran Witkon, >>>>>>>> Thanks for the suggestion Eran. I concur with your thought. >>>>>>>> If Zepplin can be integrated with oozie, that would be wonderful. >>>>>>>> Users will also be able to leverage their Oozie skills. >>>>>>>> This would be promising for now. >>>>>>>> However, in the future Hadoop might not necessarily be installed in >>>>>>>> Spark Cluster and Oozie (since its installs with Hadoop Distribution) >>>>>>>> might >>>>>>>> not be available. >>>>>>>> So perhaps we should give a thought about this feature for the >>>>>>>> future. Should it depend on oozie or should Zeppelin have its owns >>>>>>>> scheduling? >>>>>>>> >>>>>>>> As Benjamin has iterated, Databrick notebook has this as a core >>>>>>>> notebook feature. >>>>>>>> >>>>>>>> >>>>>>>> Also, would anybody give any suggestions regarding "sync with >>>>>>>> github" feature? >>>>>>>> -Exporting notebook to Github >>>>>>>> -Importing notebook from Github >>>>>>>> >>>>>>>> Thanks >>>>>>>> Vinayak >>>>>>>> >>>>>>>> >>>>>>>> On Mon, Feb 29, 2016 at 4:17 AM, Eran Witkon <eranwit...@gmail.com> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> @Vinayak Agrawal I would suggest adding the ability to connect >>>>>>>>> zeppelin to existing scheduling tools\workflow tools such as >>>>>>>>> https://oozie.apache.org/. this requires betters hooks and status >>>>>>>>> reporting but doesn't make zeppeling and ETL\scheduler tool by itself/ >>>>>>>>> >>>>>>>>> >>>>>>>>> On Mon, Feb 29, 2016 at 10:21 AM Vinayak Agrawal < >>>>>>>>> vinayakagrawa...@gmail.com> wrote: >>>>>>>>> >>>>>>>>>> Moon, >>>>>>>>>> The new roadmap looks very promising. I am very happy to see >>>>>>>>>> security in the list. >>>>>>>>>> I have some suggestions regarding Enterprise Ready features: >>>>>>>>>> >>>>>>>>>> 1. Job Scheduler - Can this be improved? >>>>>>>>>> Currently the scheduler can be used with Cron expression or a >>>>>>>>>> pre-set time. But in an enterprise solution, a notebook might be one >>>>>>>>>> piece >>>>>>>>>> of the workflow. Can we look towards the functionality of scheduling >>>>>>>>>> notebook's based on other notebooks finishing their job successfully? >>>>>>>>>> This requirement would arise in any ETL workflow, where all the >>>>>>>>>> downstream users wait for the ETL notebook to finish successfully. >>>>>>>>>> Only >>>>>>>>>> after that, other business oriented notebooks can be executed. >>>>>>>>>> >>>>>>>>>> 2. Importing a notebook - Is there a current requirement or >>>>>>>>>> future plan to implement a feature that allows >>>>>>>>>> import-notebook-from-github? >>>>>>>>>> This would allow users to share notebooks seamlessly. >>>>>>>>>> >>>>>>>>>> Thanks >>>>>>>>>> Vinayak >>>>>>>>>> >>>>>>>>>> On Sun, Feb 28, 2016 at 11:22 PM, moon soo Lee <m...@apache.org> >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>>> Zhong Wang, >>>>>>>>>>> Right, Folder support would be quite useful. Thanks for the >>>>>>>>>>> opinion. >>>>>>>>>>> >>>>>>>>>> Hope i can finish the work pr-190 >>>>>>>>>>> <https://github.com/apache/incubator-zeppelin/pull/190>. >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>>> Sourav, >>>>>>>>>>> Regarding concurrent running, Zeppelin doesn't have limitation >>>>>>>>>>> of run paragraph/query concurrently. Interpreter can implement it's >>>>>>>>>>> own >>>>>>>>>>> scheduling policy. For example, SparkSQL interpreter and >>>>>>>>>>> ShellInterpreter >>>>>>>>>>> can already run paragraph/query concurrently. >>>>>>>>>>> >>>>>>>>>>> SparkInterpreter is implemented with FIFO scheduler considering >>>>>>>>>>> nature of scala compiler. That's why user can not run multiple >>>>>>>>>>> paragraph >>>>>>>>>>> concurrently when they work with SparkInterpreter. >>>>>>>>>>> But as Zhong Wang mentioned, pr-703 enables each notebook will >>>>>>>>>>> have separate scala compiler so paragraphs run concurrently, while >>>>>>>>>>> they're >>>>>>>>>>> in different notebooks. >>>>>>>>>>> Thanks for the feedback! >>>>>>>>>>> >>>>>>>>>>> Best, >>>>>>>>>>> moon >>>>>>>>>>> >>>>>>>>>> On Sat, Feb 27, 2016 at 8:59 PM Zhong Wang < >>>>>>>>>>> wangzhong....@gmail.com> wrote: >>>>>>>>>>> >>>>>>>>>> Sourav: I think this newly merged PR can help you >>>>>>>>>>>> https://github.com/apache/incubator-zeppelin/pull/703#issuecomment-185582537 >>>>>>>>>>>> >>>>>>>>>>>> On Sat, Feb 27, 2016 at 1:46 PM, Sourav Mazumder < >>>>>>>>>>>> sourav.mazumde...@gmail.com> wrote: >>>>>>>>>>>> >>>>>>>>>>> Hi Moon, >>>>>>>>>>>>> >>>>>>>>>>>>> This looks great. >>>>>>>>>>>>> >>>>>>>>>>>>> My only suggestion would be to include a PR/feature - Support >>>>>>>>>>>>> for Running Concurrent paragraphs/queries in Zeppelin. >>>>>>>>>>>>> >>>>>>>>>>>>> Right now if more than one user tries to run paragraphs in >>>>>>>>>>>>> multiple notebooks concurrently through a single Zeppelin >>>>>>>>>>>>> instance (and >>>>>>>>>>>>> single interpreter instance) the performance is very slow. It is >>>>>>>>>>>>> obvious >>>>>>>>>>>>> that the queue gets built up within the zeppelin process and >>>>>>>>>>>>> interpreter >>>>>>>>>>>>> process in that scenario as the time taken to move the status >>>>>>>>>>>>> from start to >>>>>>>>>>>>> pending and pending to running is very high compared to the >>>>>>>>>>>>> actual running >>>>>>>>>>>>> time of a paragraph. >>>>>>>>>>>>> >>>>>>>>>>>>> Without this the multi tenancy support would be meaningless as >>>>>>>>>>>>> no one can practically use it in a situation where multiple users >>>>>>>>>>>>> are >>>>>>>>>>>>> trying to connect to the same instance of Zeppelin (and the >>>>>>>>>>>>> related >>>>>>>>>>>>> interpreter). A possible solution would be to spawn separate >>>>>>>>>>>>> instance of >>>>>>>>>>>>> the same interpreter at every notebook/user level. >>>>>>>>>>>>> >>>>>>>>>>>>> Regards, >>>>>>>>>>>>> Sourav >>>>>>>>>>>>> >>>>>>>>>>>> On Sat, Feb 27, 2016 at 12:48 PM, moon soo Lee <m...@apache.org >>>>>>>>>>>>> > wrote: >>>>>>>>>>>>> >>>>>>>>>>>> Hi Zeppelin users and developers, >>>>>>>>>>>>>> >>>>>>>>>>>>>> The roadmap we have published at >>>>>>>>>>>>>> >>>>>>>>>>>>>> https://cwiki.apache.org/confluence/display/ZEPPELIN/Zeppelin+Roadmap >>>>>>>>>>>>>> is almost 9 month old, and it doesn't reflect where the >>>>>>>>>>>>>> community goes anymore. It's time to update. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Based on mailing list, jira issues, pullrequests, feedbacks >>>>>>>>>>>>>> from users, conferences and meetings, I could summarize the >>>>>>>>>>>>>> major interest >>>>>>>>>>>>>> of users and developers in 7 categories. Enterprise ready, >>>>>>>>>>>>>> Usability >>>>>>>>>>>>>> improvement, Pluggability, Documentation, Backend integration, >>>>>>>>>>>>>> Notebook >>>>>>>>>>>>>> storage, and Visualization. >>>>>>>>>>>>>> >>>>>>>>>>>>>> And i could list related subjects under each categories. >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>>> - Enterprise ready >>>>>>>>>>>>>> - Authentication >>>>>>>>>>>>>> - Shiro authentication ZEPPELIN-548 >>>>>>>>>>>>>> <https://issues.apache.org/jira/browse/ZEPPELIN-548> >>>>>>>>>>>>>> - Authorization >>>>>>>>>>>>>> - Notebook authorization PR-681 >>>>>>>>>>>>>> <https://github.com/apache/incubator-zeppelin/pull/681> >>>>>>>>>>>>>> - Security >>>>>>>>>>>>>> - Multi-tenancy >>>>>>>>>>>>>> - Stability >>>>>>>>>>>>>> - Usability Improvement >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> - UX improvement >>>>>>>>>>>>>> - Better Table data support >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> - Download data as csv, etc PR-725 >>>>>>>>>>>>>> <https://github.com/apache/incubator-zeppelin/pull/725> >>>>>>>>>>>>>> , PR-714 >>>>>>>>>>>>>> <https://github.com/apache/incubator-zeppelin/pull/714> >>>>>>>>>>>>>> , PR-6 >>>>>>>>>>>>>> <https://github.com/apache/incubator-zeppelin/pull/6> >>>>>>>>>>>>>> , PR-89 >>>>>>>>>>>>>> <https://github.com/apache/incubator-zeppelin/pull/89> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> - Featureful table data display (pagenation, etc) >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> - Pluggability ZEPPELIN-533 >>>>>>>>>>>>>> <https://issues.apache.org/jira/browse/ZEPPELIN-533> >>>>>>>>>>>>>> - Pluggable visualization >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> - Dynamic Interpreter, notebook, visualization loading >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> - Repository and registry for pluggable components >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> - Improve documentation >>>>>>>>>>>>>> - Improve contents and readability >>>>>>>>>>>>>> - more tutorials, examples >>>>>>>>>>>>>> - Interpreter >>>>>>>>>>>>>> - Generic JDBC Interpreter >>>>>>>>>>>>>> - (spark)R Interpreter >>>>>>>>>>>>>> - Cluster manager for interpreter (Proposal >>>>>>>>>>>>>> >>>>>>>>>>>>>> <https://cwiki.apache.org/confluence/display/ZEPPELIN/Cluster+Manager+Proposal> >>>>>>>>>>>>>> ) >>>>>>>>>>>>>> - more interpreters >>>>>>>>>>>>>> - Notebook storage >>>>>>>>>>>>>> - Versioning ZEPPELIN-540 >>>>>>>>>>>>>> <http://issues.apache.org/jira/browse/ZEPPELIN-540> >>>>>>>>>>>>>> - more notebook storages >>>>>>>>>>>>>> - Visualization >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> - More visualizations PR-152 >>>>>>>>>>>>>> <https://github.com/apache/incubator-zeppelin/pull/152> >>>>>>>>>>>>>> , PR-728 >>>>>>>>>>>>>> <https://github.com/apache/incubator-zeppelin/pull/728> >>>>>>>>>>>>>> , PR-336 >>>>>>>>>>>>>> <https://github.com/apache/incubator-zeppelin/pull/336> >>>>>>>>>>>>>> , PR-321 >>>>>>>>>>>>>> <https://github.com/apache/incubator-zeppelin/pull/321> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> - Customize graph (show/hide label, color, etc) >>>>>>>>>>>>>> >>>>>>>>>>>>>> It will help anyone quickly get overall interest of project >>>>>>>>>>>>>> and the direction. And based on this roadmap, we can discuss and >>>>>>>>>>>>>> re-define >>>>>>>>>>>>>> the next release 0.6.0 scope and it's schedule. >>>>>>>>>>>>>> >>>>>>>>>>>>>> What do you think? Any feedback would be appreciated. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>> moon >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> Vinayak Agrawal >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> "To Strive, To Seek, To Find and Not to Yield!" >>>>>>>>>> ~Lord Alfred Tennyson >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> Vinayak Agrawal >>>>>>>> Big Data Analytics >>>>>>>> IBM >>>>>>>> >>>>>>>> "To Strive, To Seek, To Find and Not to Yield!" >>>>>>>> ~Lord Alfred Tennyson >>>>>>>> >>>>>>> >>>>>>> >>>>>> >>> >> >>
