Hi All, I've submitted a design approach for Multi-tenancy and Security for Zeppelin - https://issues.apache.org/jira/browse/ZEPPELIN-773.
Look forward for the reviews and suggestions on the topic. Thanks, Rohit. On Sat, Mar 26, 2016 at 10:04 PM, moon soo Lee <m...@apache.org> wrote: > There is an discussion thread for Release Policy. > https://s.apache.org/3JCm please check this thread, too. > > Thanks, > moon > > > On Thu, Mar 24, 2016 at 12:02 PM Guilherme Silveira < > guilhermecgss...@gmail.com> wrote: > >> Is there a predefined release interval, lets say, 6 months or 1 year, >> between one version and another? >> Em 23 de mar de 2016 4:10 PM, "Joel Van Veluwen" < >> joel.vanvelu...@quantium.com.au> escreveu: >> >>> Hi Nikolay, >>> >>> >>> >>> I raised this with MapR and there doesn’t appear to be plans to add >>> Zeppelin to 5.1 >>> >>> >>> >>> https://community.mapr.com/message/40332 >>> >>> >>> >>> We are deploying it manually and everything is pretty stable – but it >>> will vary depending on your environment. >>> >>> >>> >>> Cheers, >>> >>> >>> >>> Joel Van Veluwen >>> *QUANTIUM* >>> Level 25, 8 Chifley >>> 8-12 Chifley Square >>> Sydney NSW 2000 >>> >>> T: +61 2 8224 8981 >>> M: +61 403 153 265 >>> F: +61 2 9292 6444 >>> >>> W: quantium.com.au <http://www.quantium.com.au> >>> ------------------------------ >>> >>> linkedin.com/company/quantium <http://www.linkedin.com/company/quantium> >>> facebook.com/QuantiumAustralia >>> <http://www.facebook.com/QuantiumAustralia> >>> twitter.com/QuantiumAU <http://www.twitter.com/QuantiumAU> >>> >>> The contents of this email, including attachments, may be confidential >>> information. If you are not the intended recipient, any use, disclosure or >>> copying of the information is unauthorised. If you have received this email >>> in error, we would be grateful if you would notify us immediately by email >>> reply, phone (+ 61 2 9292 6400) or fax (+ 61 2 9292 6444) and delete >>> the message from your system. >>> >>> >>> >>> *From:* Nikolay Voronchikhin [mailto:nvoronchik...@gmail.com] >>> *Sent:* Tuesday, 22 March 2016 11:39 AM >>> *To:* users@zeppelin.incubator.apache.org >>> *Subject:* Re: [DISCUSS] Update Roadmap >>> >>> >>> >>> Hi Zeppelin Users and Developers, >>> >>> >>> >>> Do you know if MapR will be adding Zeppelin to its roadmap for the next >>> version after MapR 5.1? >>> >>> >>> >>> We see in Hue 3.9 that it provides notebooks for R Shell, Python Shell, >>> PySpark, SparkR, Hive SQL, Impala SQL, and Spark SQL, but no Drill SQL >>> notebook. >>> >>> We are looking for an Apache Project that focuses on a Drill Notebook UI >>> that performs better than the Drill Web Console UI itself. >>> >>> >>> >>> Sincerely, >>> >>> *Nikolay Voronchikhin* >>> >>> *Big Data/Data Warehouse/Data Science/Data Platforms Engineer at Cisco* >>> >>> *https://www.linkedin.com/in/nvoronchikhin >>> <https://www.linkedin.com/in/nvoronchikhin>* >>> >>> *E-mail: nvoronchik...@gmail.com <nvoronchik...@gmail.com>* >>> >>> *Mobile: 951-288-2778 <951-288-2778>* >>> >>> >>> >>> >>> >>> On Mon, Mar 21, 2016 at 2:44 PM, rohit choudhary <rconl...@gmail.com> >>> wrote: >>> >>> Dear All, >>> >>> >>> >>> I think direction setting is important for Enterprise readiness. I have >>> a little bit of an overview of Ambari Views, which is very similar in >>> nature to Zeppelin. Please let me explain: >>> >>> >>> >>> Hive View - interacts with Hive >>> >>> Pig View - interacts with Pig >>> >>> Workflow Designer - interacts with Oozie >>> >>> >>> >>> We have a very similar architecture in Zeppelin where we interact with >>> these systems through Interpreters. The usage will also be similar, as both >>> with interact with Hadoop clusters or in some cases Spark with Yarn on >>> HDFS. Our priorities should include: >>> >>> >>> >>> - Design & implement for multi-tenancy >>> >>> - Auditability from Data/State and Lineage perspective >>> >>> - Ability to share Notebooks/Data/State across users, preferably through >>> SparkContext sharing >>> >>> - Security between Zeppelin and the other systems, not limited to Spark >>> through Kerberos. (@Rick +1) >>> >>> >>> >>> I will share an initial draft of the thoughts I have in mind, in the >>> next couple of days. >>> >>> >>> >>> Thanks, >>> >>> Rohit. >>> >>> >>> >>> >>> >>> >>> >>> On Thu, Mar 3, 2016 at 7:54 AM, moon soo Lee <m...@apache.org> wrote: >>> >>> Shabeel, thanks for the feedback about rest api and custom id. that >>> might help avoid multiple rest api calls. >>> >>> >>> >>> Thanks everyone for valuable feedback. Looks like all we're going to the >>> same direction. I have updated wiki. >>> >>> https://cwiki.apache.org/confluence/display/ZEPPELIN/Zeppelin+Roadmap >>> >>> Please take a look. >>> >>> >>> >>> I'm sure there're many missing details in this roadmap. I must say >>> something not on this roadmap doesn't mean community is not working on or >>> can't be included in the Zeppelin. Roadmap represents more like community >>> interest and overall direction. >>> >>> We're not changing roadmap everyday, but that doesn't mean roadmap is >>> set in stone and never be changed. We can improve it continuously. >>> >>> >>> >>> Please feel free to fork the this mail thread for any further discussion >>> on specific subject. (e.g. job scheduling) >>> >>> >>> >>> Thanks, >>> >>> moon >>> >>> >>> >>> On Wed, Mar 2, 2016 at 12:31 AM Shabeel Syed <shabeels...@gmail.com> >>> wrote: >>> >>> Also we need better rest api support for creating and fetching the >>> notebooks and paragraphs. >>> >>> for example if I can set custom defined notebookid and paragraphid , we >>> can avoid multiple rest api calls. >>> >>> >>> >>> http://localhost:8080/#/notebook/ >>> <notebookid>/paragraph/<paragraphid>?asIframe >>> >>> should return me error if notebook or paragraph deos not exists. >>> >>> >>> >>> and while creating notebook or paragraph I should be able to mention my >>> custom ids. >>> >>> >>> >>> Regards >>> >>> Shabeel >>> >>> >>> >>> On Wed, Mar 2, 2016 at 11:55 AM, Zhong Wang <wangzhong....@gmail.com> >>> wrote: >>> >>> +1 on @rick. quality is really important... I am still encountering bugs >>> consistently >>> >>> >>> >>> On Tue, Mar 1, 2016 at 10:16 AM, TEJA SRIVASTAV <tejasrivas...@gmail.com> >>> wrote: >>> >>> +1 on @rick >>> >>> >>> >>> On Tue, Mar 1, 2016 at 11:26 PM Benjamin Kim <bbuil...@gmail.com> wrote: >>> >>> I see in the Enterprise section that multi-tenancy will be included, >>> will this have user impersonation too? In this way, the user executing will >>> be the user owning the process. >>> >>> >>> >>> On Mar 1, 2016, at 12:51 AM, Shabeel Syed <shabeels...@gmail.com> wrote: >>> >>> >>> >>> +1 >>> >>> >>> >>> Hi Tamas, >>> >>> Pluggable external visualization is really a GREAT feature to have. >>> I'm looking forward to this :) >>> >>> >>> >>> Regards >>> >>> Shabeel >>> >>> >>> >>> On Tue, Mar 1, 2016 at 2:16 PM, Tamas Szuromi <tamas.szur...@odigeo.com> >>> wrote: >>> >>> Hey, >>> >>> >>> >>> Really promising roadmap. >>> >>> >>> >>> I'd only push more visualization options. I agree built in >>> visualization is needed with limited charting options but I think we also >>> need somehow 'inject' external js visualizations also. >>> >>> >>> >>> >>> >>> For scheduling Zeppelin notebooks we use >>> https://github.com/airbnb/airflow <https://github.com/airbnb/airflow> >>> through >>> the job rest api. It's an enterprise ready and very robust solution >>> right now. >>> >>> >>> >>> *Tamas* >>> >>> >>> >>> On 1 March 2016 at 09:12, Eran Witkon <eranwit...@gmail.com> wrote: >>> >>> One point to clarify, I don't want to suggest Oozie in specific, I want >>> to think about which features we develop and which ones we integrate >>> external, preferred Apache, technology? We don't think about building our >>> own storage services so why build our own scheduler? >>> Eran >>> >>> On Tue, 1 Mar 2016 at 09:49 moon soo Lee <m...@apache.org> wrote: >>> >>> @Vinayak, @Eran, @Benjamin, @Guilherme, @Sourav, @Rick >>> >>> Now I can see a lot of demands around enterprise level job scheduling. >>> Either external or built-in, I completely agree having enterprise level job >>> scheduling support on the roadmap. >>> >>> ZEPPELIN-137 <https://issues.apache.org/jira/browse/ZEPPELIN-137>, >>> ZEPPELIN-531 <https://issues.apache.org/jira/browse/ZEPPELIN-531> are >>> related issues i can find in our JIRA. >>> >>> >>> >>> @Vinayak >>> >>> Regarding importing notebook from github, Zeppelin has pluggable >>> notebook storage layer (see related package >>> <https://github.com/apache/incubator-zeppelin/tree/master/zeppelin-zengine/src/main/java/org/apache/zeppelin/notebook/repo>). >>> So, github notebook sync can be implemented easily. >>> >>> >>> >>> @Shabeel >>> >>> Right, we need better manage management to prevent such OOM. >>> >>> And i think table is one of the most frequently used way of displaying >>> data. So definitely, we'll need more features like filter, sort, etc. >>> >>> After this roadmap discussion, discussion for the next release will >>> follow. Then we'll get idea when those features will be available. >>> >>> >>> >>> @Prasad >>> >>> Thanks for mentioning HA and DR. They're really important subject for >>> enterprise use. Definitely Zeppelin will need to address them. >>> >>> And displaying meta information of notebook on top level page is good >>> idea. >>> >>> >>> >>> It's really great to hear many opinions and ideas. >>> >>> And thanks @Rick for sharing valuable view to Zeppelin project. >>> >>> >>> >>> Thanks, >>> >>> moon >>> >>> >>> >>> >>> >>> On Mon, Feb 29, 2016 at 11:14 PM Rick Moritz <rah...@gmail.com> wrote: >>> >>> Hi, >>> >>> For one, I know that there is rudimentary scheduling built into Zeppelin >>> already (at least I fixed a bug in the test for a scheduling feature a few >>> months ago). >>> >>> But another point is, that Zeppelin should also focus on quality, >>> reproduceability and portability. >>> >>> Although this doesn't offer exciting new features, it would make >>> development much easier. >>> >>> Cross-platform testability, Tests that pass when run sequentially, >>> compatibility with Firefox, and many more open issues that make it so much >>> harder to enhance Zeppelin and add features should be addressed soon, >>> preferably before more features are added. Already Zeppelin is suffering - >>> in my opinion - from quite a lot of feature creep, and we should avoid >>> putting in the kitchen sink, at the cost of quality and maintainability. >>> Instead modularity (ZEPPELIN-533 in particular) should be targeted. >>> >>> Oozie, in my opinion, is a dead end - it may de-facto still be in use on >>> many clusters, but it's not getting the love it needs, and I wouldn't bet >>> on it, when it comes to integrating scheduling. Instead, any external tool >>> should be able to use the REST-API to trigger executions, if you want >>> external scheduling. >>> >>> So, in conclusion, if we take Moon's list as a list of descending >>> priorities, I fully agree, under the condition that code quality is >>> included as a subset of enterprise-readyness. Auth* is paramount (Kerberos >>> SPNEGO SSO support is what we really want) with user and group rights >>> assignment on the notebook level. We probably also need Knox-integration >>> (ODP-Members looking at integrating Zeppelin should consider contributing >>> this), and integration of something like Spree ( >>> https://github.com/hammerlab/spree) to be able to profile jobs. >>> >>> I'm hopeful that soon I can resume contributing some quality-oriented >>> code, to drive this "necessary evil" forward ;) >>> >>> >>> >>> On Mon, Feb 29, 2016 at 8:27 PM, Sourav Mazumder < >>> sourav.mazumde...@gmail.com> wrote: >>> >>> I do agree with Vinayak. It need not be coupled with Oozie. >>> >>> Rather one should be able to call it from any scheduler typically used >>> in enterprise level. May be support for BPML. >>> >>> I believe the existing ability to call/execute a Zeppelin Notebook or a >>> specific paragraph within a notebook using REST API should take care of >>> this requirement to some extent. >>> >>> Regards, >>> >>> Sourav >>> >>> >>> >>> On Mon, Feb 29, 2016 at 11:23 AM, Vinayak Agrawal < >>> vinayakagrawa...@gmail.com> wrote: >>> >>> @Eran Witkon, >>> >>> Thanks for the suggestion Eran. I concur with your thought. >>> >>> If Zepplin can be integrated with oozie, that would be wonderful. Users >>> will also be able to leverage their Oozie skills. >>> >>> This would be promising for now. >>> >>> However, in the future Hadoop might not necessarily be installed in >>> Spark Cluster and Oozie (since its installs with Hadoop Distribution) might >>> not be available. >>> >>> So perhaps we should give a thought about this feature for the future. >>> Should it depend on oozie or should Zeppelin have its owns scheduling? >>> >>> As Benjamin has iterated, Databrick notebook has this as a core notebook >>> feature. >>> >>> >>> >>> Also, would anybody give any suggestions regarding "sync with github" >>> feature? >>> >>> -Exporting notebook to Github >>> >>> -Importing notebook from Github >>> >>> >>> >>> Thanks >>> >>> Vinayak >>> >>> >>> >>> >>> >>> On Mon, Feb 29, 2016 at 4:17 AM, Eran Witkon <eranwit...@gmail.com> >>> wrote: >>> >>> @*Vinayak Agrawal *I would suggest adding the ability to connect >>> zeppelin to existing scheduling tools\workflow tools such as >>> https://oozie.apache.org/. this requires betters hooks and status >>> reporting but doesn't make zeppeling and ETL\scheduler tool by itself/ >>> >>> >>> >>> >>> >>> On Mon, Feb 29, 2016 at 10:21 AM Vinayak Agrawal < >>> vinayakagrawa...@gmail.com> wrote: >>> >>> Moon, >>> >>> The new roadmap looks very promising. I am very happy to see security in >>> the list. >>> I have some suggestions regarding Enterprise Ready features: >>> >>> >>> 1. Job Scheduler - Can this be improved? >>> >>> Currently the scheduler can be used with Cron expression or a pre-set >>> time. But in an enterprise solution, a notebook might be one piece of the >>> workflow. Can we look towards the functionality of scheduling notebook's >>> based on other notebooks finishing their job successfully? >>> >>> This requirement would arise in any ETL workflow, where all the >>> downstream users wait for the ETL notebook to finish successfully. Only >>> after that, other business oriented notebooks can be executed. >>> >>> 2. Importing a notebook - Is there a current requirement or future plan >>> to implement a feature that allows import-notebook-from-github? This would >>> allow users to share notebooks seamlessly. >>> >>> Thanks >>> >>> Vinayak >>> >>> >>> >>> On Sun, Feb 28, 2016 at 11:22 PM, moon soo Lee <m...@apache.org> wrote: >>> >>> Zhong Wang, >>> >>> Right, Folder support would be quite useful. Thanks for the opinion. >>> >>> Hope i can finish the work pr-190 >>> <https://github.com/apache/incubator-zeppelin/pull/190>. >>> >>> >>> >>> Sourav, >>> >>> Regarding concurrent running, Zeppelin doesn't have limitation of run >>> paragraph/query concurrently. Interpreter can implement it's own scheduling >>> policy. For example, SparkSQL interpreter and ShellInterpreter can already >>> run paragraph/query concurrently. >>> >>> >>> >>> SparkInterpreter is implemented with FIFO scheduler considering nature >>> of scala compiler. That's why user can not run multiple paragraph >>> concurrently when they work with SparkInterpreter. >>> >>> But as Zhong Wang mentioned, pr-703 enables each notebook will have >>> separate scala compiler so paragraphs run concurrently, while they're in >>> different notebooks. >>> >>> Thanks for the feedback! >>> >>> >>> >>> Best, >>> >>> moon >>> >>> On Sat, Feb 27, 2016 at 8:59 PM Zhong Wang <wangzhong....@gmail.com> >>> wrote: >>> >>> Sourav: I think this newly merged PR can help you >>> https://github.com/apache/incubator-zeppelin/pull/703#issuecomment-185582537 >>> >>> >>> >>> On Sat, Feb 27, 2016 at 1:46 PM, Sourav Mazumder < >>> sourav.mazumde...@gmail.com> wrote: >>> >>> Hi Moon, >>> >>> This looks great. >>> >>> My only suggestion would be to include a PR/feature - Support for >>> Running Concurrent paragraphs/queries in Zeppelin. >>> >>> Right now if more than one user tries to run paragraphs in multiple >>> notebooks concurrently through a single Zeppelin instance (and single >>> interpreter instance) the performance is very slow. It is obvious that the >>> queue gets built up within the zeppelin process and interpreter process in >>> that scenario as the time taken to move the status from start to pending >>> and pending to running is very high compared to the actual running time of >>> a paragraph. >>> >>> Without this the multi tenancy support would be meaningless as no one >>> can practically use it in a situation where multiple users are trying to >>> connect to the same instance of Zeppelin (and the related interpreter). A >>> possible solution would be to spawn separate instance of the same >>> interpreter at every notebook/user level. >>> >>> Regards, >>> >>> Sourav >>> >>> On Sat, Feb 27, 2016 at 12:48 PM, moon soo Lee <m...@apache.org> wrote: >>> >>> Hi Zeppelin users and developers, >>> >>> >>> >>> The roadmap we have published at >>> >>> https://cwiki.apache.org/confluence/display/ZEPPELIN/Zeppelin+Roadmap >>> >>> is almost 9 month old, and it doesn't reflect where the community goes >>> anymore. It's time to update. >>> >>> >>> >>> Based on mailing list, jira issues, pullrequests, feedbacks from users, >>> conferences and meetings, I could summarize the major interest of users and >>> developers in 7 categories. Enterprise ready, Usability improvement, >>> Pluggability, Documentation, Backend integration, Notebook storage, and >>> Visualization. >>> >>> >>> >>> And i could list related subjects under each categories. >>> >>> >>> - Enterprise ready >>> >>> >>> - Authentication >>> >>> >>> - Shiro authentication ZEPPELIN-548 >>> <https://issues.apache.org/jira/browse/ZEPPELIN-548> >>> >>> >>> - Authorization >>> >>> >>> - Notebook authorization PR-681 >>> <https://github.com/apache/incubator-zeppelin/pull/681> >>> >>> >>> - Security >>> - Multi-tenancy >>> - Stability >>> >>> >>> - Usability Improvement >>> >>> >>> - UX improvement >>> - Better Table data support >>> >>> >>> - Download data as csv, etc PR-725 >>> <https://github.com/apache/incubator-zeppelin/pull/725>, PR-714 >>> <https://github.com/apache/incubator-zeppelin/pull/714>, PR-6 >>> <https://github.com/apache/incubator-zeppelin/pull/6>, PR-89 >>> <https://github.com/apache/incubator-zeppelin/pull/89> >>> >>> >>> - Featureful table data display (pagenation, etc) >>> >>> >>> - Pluggability ZEPPELIN-533 >>> <https://issues.apache.org/jira/browse/ZEPPELIN-533> >>> >>> >>> - Pluggable visualization >>> >>> >>> - Dynamic Interpreter, notebook, visualization loading >>> >>> >>> - Repository and registry for pluggable components >>> >>> >>> - Improve documentation >>> >>> >>> - Improve contents and readability >>> - more tutorials, examples >>> >>> >>> - Interpreter >>> >>> >>> - Generic JDBC Interpreter >>> - (spark)R Interpreter >>> - Cluster manager for interpreter (Proposal >>> >>> <https://cwiki.apache.org/confluence/display/ZEPPELIN/Cluster+Manager+Proposal> >>> ) >>> - more interpreters >>> >>> >>> - Notebook storage >>> >>> >>> - Versioning ZEPPELIN-540 >>> <http://issues.apache.org/jira/browse/ZEPPELIN-540> >>> - more notebook storages >>> >>> >>> - Visualization >>> >>> >>> - More visualizations PR-152 >>> <https://github.com/apache/incubator-zeppelin/pull/152>, PR-728 >>> <https://github.com/apache/incubator-zeppelin/pull/728>, PR-336 >>> <https://github.com/apache/incubator-zeppelin/pull/336>, PR-321 >>> <https://github.com/apache/incubator-zeppelin/pull/321> >>> >>> >>> - Customize graph (show/hide label, color, etc) >>> >>> It will help anyone quickly get overall interest of project and the >>> direction. And based on this roadmap, we can discuss and re-define the next >>> release 0.6.0 scope and it's schedule. >>> >>> >>> >>> What do you think? Any feedback would be appreciated. >>> >>> >>> >>> Thanks, >>> >>> moon >>> >>> >>> >>> >>> >>> >>> -- >>> >>> Vinayak Agrawal >>> >>> >>> >>> "To Strive, To Seek, To Find and Not to Yield!" >>> >>> ~Lord Alfred Tennyson >>> >>> >>> >>> >>> -- >>> >>> Vinayak Agrawal >>> >>> Big Data Analytics >>> >>> IBM >>> >>> "To Strive, To Seek, To Find and Not to Yield!" >>> >>> ~Lord Alfred Tennyson >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>
