Re: Wiki access
Hi again, I don't have write access to the wiki page: https://wiki.apache.org/incubator/ReefProposal. Could someone give me permission to update the page? Thanks. -Gon On Sat, Aug 2, 2014 at 10:43 AM, Byung-Gon Chun bgc...@gmail.com wrote: Hello, Can I have access to the incubator wiki? I'd like to post the REEF proposal on the wiki. My user name is Byung-Gon Chun. Thanks! - Gon -- Byung-Gon Chun -- Byung-Gon Chun
Re: [PROPOSAL] REEF for the Apache Incubator
Hi Roman, Thank you for the comment. We will add the following description that covers Helix to the proposal page. Apache Helix automates application-wide management operations which require global knowledge and coordination, such as repartitioning of resources and scheduling of maintenance tasks. Helix separates global coordination concerns from the functional tasks of the application with a state machine abstraction. REEF's generic layer makes it easy to program the functional and management tasks, which may span small or large groups within the application. Helix can work hand-in-hand with REEF, by providing the global management component for REEF applications. Thanks! - Gon --- Byung-Gon Chun On Tue, Aug 5, 2014 at 1:59 AM, Roman Shaposhnik r...@apache.org wrote: Hi! On Fri, Aug 1, 2014 at 12:14 AM, Byung-Gon Chun bgc...@gmail.com wrote: Since REEF is a library that makes it easy to write distributed applications on top of Apache YARN or Mesos, the Apache Software Foundation is the perfect home for hosting REEF. [ snip...snip...snip ] ## Relationships with Other Apache Products Really appreciated the detailed review of potential relationships, but was surprised not to see Apache Helix on the list of related projects. Given the exec summary of the project -- there must be some relationship. Or am I reading it incorrectly? Thanks, Roman. - To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org -- Byung-Gon Chun
Re: [PROPOSAL] REEF for the Apache Incubator
Hi Jake, Thank you for the comment. We had discussions on how to structure mailing lists with our mentors. We took our mentors' suggestions to start with a minimal set (two mailing lists) not to miss important discussions and to split them if there are demands. Thanks! -Gon --- Byung-Gon Chun On Tue, Aug 5, 2014 at 3:04 AM, Jake Farrell jfarr...@apache.org wrote: Would suggest you use the following format for the mailing lists (you have the older format listed) and also split the dev and commits. Also a lot of new projects have been also splitting out the jira issues from dev to cut down on noise on the dev list, would add issues@reef if you want to do this. private@reef for private PMC discussions dev@reef for technical discussions commits@reef notification about commits issues@reef jira notifications -Jake On Fri, Aug 1, 2014 at 3:14 AM, Byung-Gon Chun bgc...@gmail.com wrote: Hi everyone, I would like to propose REEF to be an Apache Incubator project. REEF is a scale-out computing fabric that eases the development of Big Data applications on top of resource managers such as Apache YARN and Mesos. The proposal is included in plain text below. I would also like to put this on wiki but I don't have privileges to create wiki pages. I look forward to hearing everyone's thoughts and feedback! -Gon -- Byung-Gon Chun === # REEFProposal - Incubator # Abstract REEF (Retainable Evaluator Execution Framework) is a scale-out computing fabric that eases the development of Big Data applications on top of resource managers such as Apache YARN and Mesos. # Proposal REEF is a Big Data system that makes it easy to implement scalable, fault-tolerant runtime environments for a range of data processing models (e.g., graph processing and machine learning) on top of resource managers such as Apache YARN and Mesos. REEF provides capabilities to run multiple heterogeneous frameworks and workflows of those efficiently. Additionally, REEF contains two libraries that are of independent value: Wake is an event-based-programming framework inspired by Rx and SEDA. Tang is a dependency injection framework inspired by Google Guice, but designed specifically for configuring distributed systems. # Background The resource management layer such as Apache YARN and Mesos has emerged as a critical layer in the new scale-out data processing stack; resource managers assume the responsibility of multiplexing a cluster of shared-nothing machines across heterogeneous applications. They operate behind an interface for leasing containers - a slice of a machine’s resources - to computations in an elastic fashion. However, building data processing frameworks directly on this layer comes at a high cost: each framework must tackle the same challenges (e.g., fault-tolerance, task scheduling and coordination) and reimplement common mechanisms (e.g., caching, bulk transfers). REEF provides a reusable control-plane for scheduling and coordinating task-level work on cluster resource managers. The REEF design enables sophisticated optimizations, such as container re-use and data caching, and facilitates workflows that span multiple frameworks. Examples include pipelining data between different operators in a relational system, retaining state across iterations in iterative or recursive data flow, and passing the result of a MapReduce job to a Machine Learning computation. # Rationale Since REEF is a library that makes it easy to write distributed applications on top of Apache YARN or Mesos, the Apache Software Foundation is the perfect home for hosting REEF. # Current Status REEF has been developed mostly by Microsoft, UCLA and the Seoul National University. The REEF codebase is open-sourced under Apache License 2.0 and is currently hosted in a public repository at github.com. # Meritocracy We plan to build a strong open community by following the Apache meritocracy principles. We will work with those who contribute significantly to the project and invite them to be its committers. # Community REEF is currently being used internally at Microsoft. Also, SK Telecom builds their data analytics infrastructure on top of REEF in collaboration with Seoul National University. We hope to extend our contributor base by becoming an Apache incubator project. REEF will attract developers who are interested in creating common building blocks for simplifying the development of large-scale big data applications. # Core Developers Core developers are engineers from Microsoft, Purestorage, UCB, UCLA, UW and Seoul National University. # Alignment REEF depends on many Apache projects and dependencies. REEF is built on resource managers such as Apache YARN and Apache Mesos. REEF also uses HDFS as a distributed storage layer.
Re: Start of the report contents
Hi, On Tue, Aug 5, 2014 at 2:11 AM, John D. Ament john.d.am...@gmail.com wrote: ...3. I believe I have all of the releases listed... There's also Apache DeviceMap data and java client 1.0.0 (the vote tally didn't include the [RESULT] tag, that's probably why you missed it) -Bertrand ... The following releases were made since the last Incubator report: Apache Samza 0.7.0-incubating RC0 Apache Tez-0.4.1-incubating RC0 Apache Slider 0.40-incubating RC0 Apache Blur 0.2.3-incubating RC2 Apache Metamodel incubating 4.2.0 Apache DeviceMap BrowserMap incubating 1.4.1 Apache HDT version 0.0.2.incubating (RC1) - To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org
[RESULT][VOTE] Apache DeviceMap data and java client 1.0.0
Resending this as the original tally is missing the [RESULT] subject header. -Bertrand On Sun, Jul 27, 2014 at 8:24 PM, Reza reza.nagh...@yahoo.com.invalid wrote: Thanks, the vote has passed. IPMC +1: Kevan Miller Bertrand Delacretaz Justin Mclean PPMC +1: Werner Keil Eberhard Speer Radu Cotescu Thanks again, Reza - To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org
Re: Wiki access
On Tue, 5 Aug 2014, Byung-Gon Chun wrote: I don't have write access to the wiki page: https://wiki.apache.org/incubator/ReefProposal. Could someone give me permission to update the page? My user name is Byung-Gon Chun. Karma granted Nick - To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org
Re: Wiki access
Thanks, Nick! On Tue, Aug 5, 2014 at 5:59 PM, Nick Burch apa...@gagravarr.org wrote: On Tue, 5 Aug 2014, Byung-Gon Chun wrote: I don't have write access to the wiki page: https://wiki.apache.org/incubator/ReefProposal. Could someone give me permission to update the page? My user name is Byung-Gon Chun. Karma granted Nick - To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org -- Byung-Gon Chun
Re: [VOTE] Apache Drill 0.4.0-incubating release
Hey Ted, Those SNAPSHOT dependencies are a bit misleading. Because Drill is so closely coupled to the Optiq and Parquet codebase, we need to generate separate artifacts. As part of doing monthly releases, it is unfeasible to bind Drill releases to incorporating of all patches into upstream projects. You can think of this as similar to how, for years, HBase released on top of a modified (non-released) Hadoop due to special requirements for append. As such, while slightly outside typical maven conventions, those SNAPSHOT are fixed and will not change in the future. The github hashes for each changeset are as follows: parquet-format: https://github.com/jacques-n/incubator-parquet-format/commit/7001502877e0cfbf81d429656989057ccc5fafb2 parquet-mr: https://github.com/jaltekruse/parquet-mr/commit/737500cbabd009eee065058fff2ccc8cc806c5b2 optiq: https://github.com/jacques-n/optiq/commit/4508b617bd3ffed2840055fe16e6684e1c0a35d7 Hopefully this satisfies the need to document the source references for third party dependencies within the Apache Drill 0.4.0 release. thanks, Jacques On Mon, Aug 4, 2014 at 12:07 PM, Ted Dunning ted.dunn...@gmail.com wrote: I checked the source artifact and found several internal SNAPSHOT dependencies (should be fixed) and one external SNAPSHOT dependency (must be fixed). The good news is that the external SNAPSHOT dependency is parquet version 1.5.0-SNAPSHOT. Since parquet 1.5.0 has been released, this should be a trivial fix. On Mon, Aug 4, 2014 at 8:07 AM, Jacques Nadeau jacq...@apache.org wrote: We've held a vote on drill-dev to release the 0.4.0-incubating release of Apache Drill. The vote thread can be found here: http://mail-archives.apache.org/mod_mbox/incubator-drill-dev/201407.mbox/%3CCAKa9qD%3DKQURAMcS3RQJbUABSU4%3DDEGSewK2s4MAAidu4c%3DOjBg%40mail.gmail.com%3E The vote passed with: +6 binding +7 non-binding A summary email can be found here: http://mail-archives.apache.org/mod_mbox/incubator-drill-dev/201408.mbox/%3CCAKa9qDnyKYS%3D3qbv3%3DfUrcuuNvBKRcvPt4AgWojLhy4tr2ZYLA%40mail.gmail.com%3E You can find the artifacts for the release at this location: http://people.apache.org/~jacques/apache-drill-0.4.0.rc1/ Please vote on this release. Thanks, Jacques
Re: Report Manager for August?
On Tue, Jul 29, 2014 at 3:43 AM, John D. Ament john.d.am...@gmail.com wrote: Looks good to me. I just copied general@ since I missed that they fell off. So yeah, whenever we can send it out. Thanks for consistently helping with the report -- this is extremely appreciated (especially by a guy who's just came back from a totally unplugged vacation!). @Roman I'm going to change your name to mine on the shepherding list, I forgot to put mine back on after last month's surgery. Perfect! That works beautifully. Thanks, Roman. - To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org
Re: [VOTE] Apache Drill 0.4.0-incubating release
Hmm That really is quite different. To add to the mix, the two projects in question are in the process of entering the Apache incubator. In Optiq's case, at least, that means that they have made what they have stated to be their last non-Apache release, but are not yet in a position to make an Apache release yet. That makes a temporary fork whose temporary nature is somewhat non-standardly indicated by the SNAPSHOT enclitic much more plausible. +1 from me, then. Thanks for the clarification. On Tue, Aug 5, 2014 at 1:44 PM, Jacques Nadeau jacq...@apache.org wrote: Hey Ted, Those SNAPSHOT dependencies are a bit misleading. Because Drill is so closely coupled to the Optiq and Parquet codebase, we need to generate separate artifacts. As part of doing monthly releases, it is unfeasible to bind Drill releases to incorporating of all patches into upstream projects. You can think of this as similar to how, for years, HBase released on top of a modified (non-released) Hadoop due to special requirements for append. As such, while slightly outside typical maven conventions, those SNAPSHOT are fixed and will not change in the future. The github hashes for each changeset are as follows: parquet-format: https://github.com/jacques-n/incubator-parquet-format/commit/7001502877e0cfbf81d429656989057ccc5fafb2 parquet-mr: https://github.com/jaltekruse/parquet-mr/commit/737500cbabd009eee065058fff2ccc8cc806c5b2 optiq: https://github.com/jacques-n/optiq/commit/4508b617bd3ffed2840055fe16e6684e1c0a35d7 Hopefully this satisfies the need to document the source references for third party dependencies within the Apache Drill 0.4.0 release. thanks, Jacques On Mon, Aug 4, 2014 at 12:07 PM, Ted Dunning ted.dunn...@gmail.com wrote: I checked the source artifact and found several internal SNAPSHOT dependencies (should be fixed) and one external SNAPSHOT dependency (must be fixed). The good news is that the external SNAPSHOT dependency is parquet version 1.5.0-SNAPSHOT. Since parquet 1.5.0 has been released, this should be a trivial fix. On Mon, Aug 4, 2014 at 8:07 AM, Jacques Nadeau jacq...@apache.org wrote: We've held a vote on drill-dev to release the 0.4.0-incubating release of Apache Drill. The vote thread can be found here: http://mail-archives.apache.org/mod_mbox/incubator-drill-dev/201407.mbox/%3CCAKa9qD%3DKQURAMcS3RQJbUABSU4%3DDEGSewK2s4MAAidu4c%3DOjBg%40mail.gmail.com%3E The vote passed with: +6 binding +7 non-binding A summary email can be found here: http://mail-archives.apache.org/mod_mbox/incubator-drill-dev/201408.mbox/%3CCAKa9qDnyKYS%3D3qbv3%3DfUrcuuNvBKRcvPt4AgWojLhy4tr2ZYLA%40mail.gmail.com%3E You can find the artifacts for the release at this location: http://people.apache.org/~jacques/apache-drill-0.4.0.rc1/ Please vote on this release. Thanks, Jacques
Re: [VOTE] Apache Drill 0.4.0-incubating release
Signatures files looks good LICENSE looks good NOTICE file needs to update year to 2014 = not a blocker but please open ticket to track it Version looks good No 3rd party exec files +1 - Henry On Mon, Aug 4, 2014 at 7:07 AM, Jacques Nadeau jacq...@apache.org wrote: We've held a vote on drill-dev to release the 0.4.0-incubating release of Apache Drill. The vote thread can be found here: http://mail-archives.apache.org/mod_mbox/incubator-drill-dev/201407.mbox/%3CCAKa9qD%3DKQURAMcS3RQJbUABSU4%3DDEGSewK2s4MAAidu4c%3DOjBg%40mail.gmail.com%3E The vote passed with: +6 binding +7 non-binding A summary email can be found here: http://mail-archives.apache.org/mod_mbox/incubator-drill-dev/201408.mbox/%3CCAKa9qDnyKYS%3D3qbv3%3DfUrcuuNvBKRcvPt4AgWojLhy4tr2ZYLA%40mail.gmail.com%3E You can find the artifacts for the release at this location: http://people.apache.org/~jacques/apache-drill-0.4.0.rc1/ Please vote on this release. Thanks, Jacques - To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org
