DRILL-2586
Project: http://git-wip-us.apache.org/repos/asf/drill/repo Commit: http://git-wip-us.apache.org/repos/asf/drill/commit/85e05623 Tree: http://git-wip-us.apache.org/repos/asf/drill/tree/85e05623 Diff: http://git-wip-us.apache.org/repos/asf/drill/diff/85e05623 Branch: refs/heads/gh-pages Commit: 85e0562371578852c7b4a91224ca2e03f517de5d Parents: f53226a Author: Kristine Hahn <kh...@maprtech.com> Authored: Mon Mar 30 12:52:25 2015 -0700 Committer: Bridget Bevens <bbev...@maprtech.com> Committed: Mon Mar 30 18:06:26 2015 -0700 ---------------------------------------------------------------------- _docs/001-arch.md | 2 - _docs/002-tutorial.md | 2 - _docs/003-yelp.md | 2 - _docs/004-install.md | 2 - _docs/005-connect.md | 2 - _docs/006-interfaces.md | 2 - _docs/007-query.md | 4 +- _docs/012-develop.md | 11 +- _docs/014-contribute.md | 9 - _docs/014-sample-ds.md | 10 + _docs/015-archived-pages.md | 8 + _docs/015-sample-ds.md | 10 - _docs/016-design.md | 13 - _docs/016-progress.md | 8 + _docs/017-archived-pages.md | 8 - _docs/017-bylaws.md | 170 +++++++++ _docs/018-progress.md | 8 - _docs/019-bylaws.md | 170 --------- _docs/arch/001-core-mod.md | 2 - _docs/arch/002-arch-hilite.md | 2 - _docs/arch/arch-hilite/001-flexibility.md | 2 - _docs/arch/arch-hilite/002-performance.md | 2 - _docs/connect/001-plugin-reg.md | 2 - _docs/connect/002-plugin-conf.md | 2 - _docs/connect/003-workspaces.md | 2 - _docs/connect/004-reg-fs.md | 2 - _docs/connect/005-reg-hbase.md | 2 - _docs/connect/006-reg-hive.md | 2 - _docs/connect/007-default-frmt.md | 2 - _docs/connect/008-mongo-plugin.md | 2 - _docs/connect/009-mapr-db-plugin.md | 2 - _docs/contribute/001-guidelines.md | 228 ------------ _docs/contribute/002-ideas.md | 158 --------- _docs/data-sources/001-hive-types.md | 37 +- _docs/data-sources/004-json-ref.md | 5 +- _docs/design/001-plan.md | 25 -- _docs/design/002-rpc.md | 19 - _docs/design/003-query-stages.md | 42 --- _docs/design/004-research.md | 48 --- _docs/design/005-value.md | 163 --------- _docs/develop/001-compile.md | 37 -- _docs/develop/001-develop-drill.md | 9 + _docs/develop/002-contribute.md | 10 + _docs/develop/002-setup.md | 5 - _docs/develop/003-design.md | 14 + _docs/develop/003-patch-tool.md | 160 --------- _docs/develop/contribute/001-guidelines.md | 228 ++++++++++++ _docs/develop/contribute/002-ideas.md | 158 +++++++++ _docs/develop/design/001-plan.md | 25 ++ _docs/develop/design/002-rpc.md | 19 + _docs/develop/design/003-query-stages.md | 42 +++ _docs/develop/design/004-research.md | 48 +++ _docs/develop/design/005-value.md | 163 +++++++++ _docs/develop/develop-drill/001-compile.md | 37 ++ _docs/develop/develop-drill/002-patch-tool.md | 160 +++++++++ _docs/img/image_1.png | Bin 0 -> 33528 bytes _docs/img/image_10.png | Bin 0 -> 53212 bytes _docs/img/image_11.png | Bin 0 -> 73527 bytes _docs/img/image_12.png | Bin 0 -> 114497 bytes _docs/img/image_13.png | Bin 0 -> 92812 bytes _docs/img/image_14.png | Bin 0 -> 18639 bytes _docs/img/image_15.png | Bin 0 -> 32379 bytes _docs/img/image_16.png | Bin 0 -> 45973 bytes _docs/img/image_17.png | Bin 0 -> 67424 bytes _docs/img/image_2.png | Bin 0 -> 30123 bytes _docs/img/image_3.png | Bin 0 -> 81685 bytes _docs/img/image_4.png | Bin 0 -> 169567 bytes _docs/img/image_5.png | Bin 0 -> 37764 bytes _docs/img/image_6.png | Bin 0 -> 57145 bytes _docs/img/image_7.png | Bin 0 -> 29885 bytes _docs/img/image_8.png | Bin 0 -> 64920 bytes _docs/img/image_9.png | Bin 0 -> 43775 bytes _docs/install/001-drill-in-10.md | 2 - _docs/install/002-deploy.md | 3 - _docs/install/003-install-embedded.md | 2 - _docs/install/004-install-distributed.md | 2 - .../install-embedded/001-install-linux.md | 2 - .../install/install-embedded/002-install-mac.md | 2 - .../install/install-embedded/003-install-win.md | 2 - _docs/interfaces/001-jdbc-squirrel.md | 2 - _docs/interfaces/002-odbc-linux.md | 2 - _docs/interfaces/003-odbc-win.md | 2 - _docs/interfaces/004-microstrategy.md | 158 +++++++++ .../odbc-linux/001-install-odbc-linux.md | 2 - .../odbc-linux/002-install-odbc-mac.md | 2 - .../odbc-linux/003-odbc-connections-linux.md | 2 - .../odbc-linux/004-odbc-driver-conf.md | 2 - .../odbc-linux/005-odbc-connect-str.md | 2 - .../interfaces/odbc-linux/006-odbc-adv-prop.md | 2 - .../odbc-linux/007-odbc-connections-test.md | 2 - .../interfaces/odbc-win/001-install-odbc-win.md | 2 - _docs/interfaces/odbc-win/002-conf-odbc-win.md | 2 - .../interfaces/odbc-win/003-connect-odbc-win.md | 2 - .../interfaces/odbc-win/004-tableau-examples.md | 2 - _docs/interfaces/odbc-win/005-browse-view.md | 2 - _docs/query/001-get-started.md | 2 - _docs/sql-ref/001-data-types.md | 199 +++++------ _docs/sql-ref/data-types/001-date.md | 28 +- _docs/sql-ref/functions/001-math.md | 50 ++- _docs/sql-ref/functions/002-data-type-fmt.md | 343 +++++++++++++++++++ _docs/sql-ref/nested/001-flatten.md | 2 +- _docs/sql-ref/nested/002-kvgen.md | 64 +++- _docs/sql-ref/nested/003-repeated-cnt.md | 2 +- _docs/sql-ref/nested/004-repeated-contains.md | 2 +- _docs/tutorial/001-install-sandbox.md | 2 - _docs/tutorial/002-get2kno-sb.md | 2 - _docs/tutorial/003-lesson1.md | 2 - _docs/tutorial/004-lesson2.md | 2 - _docs/tutorial/005-lesson3.md | 2 - _docs/tutorial/006-summary.md | 2 - .../install-sandbox/001-install-mapr-vm.md | 2 - .../install-sandbox/002-install-mapr-vb.md | 2 - 112 files changed, 1852 insertions(+), 1366 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/drill/blob/85e05623/_docs/001-arch.md ---------------------------------------------------------------------- diff --git a/_docs/001-arch.md b/_docs/001-arch.md index 458c2dd..8779a11 100644 --- a/_docs/001-arch.md +++ b/_docs/001-arch.md @@ -1,8 +1,6 @@ --- title: "Architectural Overview" --- -[Previous](/docs)<code> </code>[Back to Table of Contents](/docs)<code> </code>[Next](/docs/core-modules-within-a-drillbit) - Apache Drill is a low latency distributed query engine for large-scale datasets, including structured and semi-structured/nested data. Inspired by Googleâs Dremel, Drill is designed to scale to several thousands of nodes and http://git-wip-us.apache.org/repos/asf/drill/blob/85e05623/_docs/002-tutorial.md ---------------------------------------------------------------------- diff --git a/_docs/002-tutorial.md b/_docs/002-tutorial.md index e5c8bc1..9f69b71 100644 --- a/_docs/002-tutorial.md +++ b/_docs/002-tutorial.md @@ -1,8 +1,6 @@ --- title: "Apache Drill Tutorial" --- -[Previous](/docs/performance)<code> </code>[Back to Table of Contents](/docs)<code> </code>[Next](/docs/installing-the-apache-drill-sandbox) - This tutorial uses the MapR Sandbox, which is a Hadoop environment pre- configured with Apache Drill. http://git-wip-us.apache.org/repos/asf/drill/blob/85e05623/_docs/003-yelp.md ---------------------------------------------------------------------- diff --git a/_docs/003-yelp.md b/_docs/003-yelp.md index bf827d5..874f780 100644 --- a/_docs/003-yelp.md +++ b/_docs/003-yelp.md @@ -1,8 +1,6 @@ --- title: "Analyzing Yelp JSON Data with Apache Drill" --- -[Previous](/docs/summary)<code> </code>[Back to Table of Contents](/docs)<code> </code>[Next](/docs/install-drill) - [Apache Drill](https://www.mapr.com/products/apache-drill) is one of the fastest growing open source projects, with the community making rapid progress with monthly releases. The key difference is Drillâs agility and flexibility. http://git-wip-us.apache.org/repos/asf/drill/blob/85e05623/_docs/004-install.md ---------------------------------------------------------------------- diff --git a/_docs/004-install.md b/_docs/004-install.md index 0415cda..9dbfdc4 100644 --- a/_docs/004-install.md +++ b/_docs/004-install.md @@ -1,8 +1,6 @@ --- title: "Install Drill" --- -[Previous](/docs/analyzing-yelp-json-data-with-apache-drill)<code> </code>[Back to Table of Contents](/docs)<code> </code>[Next](/docs/apache-drill-in-10-minutes) - You can install Drill in embedded mode or in distributed mode. Installing Drill in embedded mode does not require any configuration, which means that you can quickly get started with Drill. If you want to use Drill in a http://git-wip-us.apache.org/repos/asf/drill/blob/85e05623/_docs/005-connect.md ---------------------------------------------------------------------- diff --git a/_docs/005-connect.md b/_docs/005-connect.md index 5b6b2a5..53de0a5 100644 --- a/_docs/005-connect.md +++ b/_docs/005-connect.md @@ -1,8 +1,6 @@ --- title: "Connect to a Data Source" --- -[Previous](/docs/installing-drill-in-distributed-mode)<code> </code>[Back to Table of Contents](/docs)<code> </code>[Next](/docs/storage-plugin-registration) - A storage plugin is an interface for connecting to a data source to read and write data. Apache Drill connects to a data source, such as a file on the file system or a Hive metastore, through a storage plugin. When you execute a query, Drill gets the plugin name you provide in FROM clause of your query or from the default you specify in the USE.<plugin name> command that precedes the query. . http://git-wip-us.apache.org/repos/asf/drill/blob/85e05623/_docs/006-interfaces.md ---------------------------------------------------------------------- diff --git a/_docs/006-interfaces.md b/_docs/006-interfaces.md index 688ab73..f971e63 100644 --- a/_docs/006-interfaces.md +++ b/_docs/006-interfaces.md @@ -1,8 +1,6 @@ --- title: "ODBC/JDBC Interfaces" --- -[Previous](/docs/mapr-db-format/)<code> </code>[Back to Table of Contents](/docs)<code> </code>[Next](/docs/using-the-mapr-odbc-driver-on-windows) - You can connect to Apache Drill through the following interfaces: * Drill shell (SQLLine) http://git-wip-us.apache.org/repos/asf/drill/blob/85e05623/_docs/007-query.md ---------------------------------------------------------------------- diff --git a/_docs/007-query.md b/_docs/007-query.md index 34902ab..c6bac8a 100644 --- a/_docs/007-query.md +++ b/_docs/007-query.md @@ -1,9 +1,6 @@ --- title: "Query Data" --- -[Previous]( -/docs/using-drill-explorer-to-browse-data-and-create-views)<code> </code>[Back to Table of Contents](/docs)<code> </code>[Next](/docs/getting-started-tutorial) - You can query local and distributed file systems, Hive, and HBase data sources registered with Drill. If you connected directly to a particular schema when you invoked SQLLine, you can issue SQL queries against that schema. If you did @@ -41,4 +38,5 @@ Remember the following tips when querying data with Drill: Example: ``SELECT * FROM dfs.default.`sample_data/my_sample.json`; `` * `CAST` data to `VARCHAR` if an expression in a query returns `VARBINARY` as the result type in order to view the `VARBINARY` types as readable data. If you do not use the `CAST` function, Drill returns the results as byte data. Example: `CAST (VARBINARY_expr as VARCHAR(50))` + * When querying schema-less data using SELECT *, the order of the returned columns might differ from the stored order and vary from query to query. http://git-wip-us.apache.org/repos/asf/drill/blob/85e05623/_docs/012-develop.md ---------------------------------------------------------------------- diff --git a/_docs/012-develop.md b/_docs/012-develop.md index 2b9ce67..c0442ee 100644 --- a/_docs/012-develop.md +++ b/_docs/012-develop.md @@ -1,9 +1,10 @@ --- -title: "Develop Drill" +title: "Developer Information" --- -To develop Drill, you compile Drill from source code and then set up a project -in Eclipse for use as your development environment. To review or contribute to -Drill code, you must complete the steps required to install and use the Drill -patch review tool. +The following quick links take you to key developer information: + +* TBD +* TBD +* TBD http://git-wip-us.apache.org/repos/asf/drill/blob/85e05623/_docs/014-contribute.md ---------------------------------------------------------------------- diff --git a/_docs/014-contribute.md b/_docs/014-contribute.md deleted file mode 100644 index 42108b9..0000000 --- a/_docs/014-contribute.md +++ /dev/null @@ -1,9 +0,0 @@ ---- -title: "Contribute to Drill" ---- -The Apache Drill community welcomes your support. Please read [Apache Drill -Contribution Guidelines](/docs/apache-drill-contribution-guidelines) for information about how to contribute to -the project. If you would like to contribute to the project and need some -ideas for what to do, please read [Apache Drill Contribution -Ideas](/docs/apache-drill-contribution-ideas). - http://git-wip-us.apache.org/repos/asf/drill/blob/85e05623/_docs/014-sample-ds.md ---------------------------------------------------------------------- diff --git a/_docs/014-sample-ds.md b/_docs/014-sample-ds.md new file mode 100644 index 0000000..c6f51e1 --- /dev/null +++ b/_docs/014-sample-ds.md @@ -0,0 +1,10 @@ +--- +title: "Sample Datasets" +--- +Use any of the following sample datasets provided to test Drill: + + * [AOL Search](/docs/aol-search) + * [Enron Emails](/docs/enron-emails) + * [Wikipedia Edit History](/docs/wikipedia-edit-history) + + http://git-wip-us.apache.org/repos/asf/drill/blob/85e05623/_docs/015-archived-pages.md ---------------------------------------------------------------------- diff --git a/_docs/015-archived-pages.md b/_docs/015-archived-pages.md new file mode 100644 index 0000000..d052579 --- /dev/null +++ b/_docs/015-archived-pages.md @@ -0,0 +1,8 @@ +--- +title: "Archived Pages" +--- +The following pages have been archived: + +* How to Run Drill with Sample Data +* Meet Apache Drill + http://git-wip-us.apache.org/repos/asf/drill/blob/85e05623/_docs/015-sample-ds.md ---------------------------------------------------------------------- diff --git a/_docs/015-sample-ds.md b/_docs/015-sample-ds.md deleted file mode 100644 index c6f51e1..0000000 --- a/_docs/015-sample-ds.md +++ /dev/null @@ -1,10 +0,0 @@ ---- -title: "Sample Datasets" ---- -Use any of the following sample datasets provided to test Drill: - - * [AOL Search](/docs/aol-search) - * [Enron Emails](/docs/enron-emails) - * [Wikipedia Edit History](/docs/wikipedia-edit-history) - - http://git-wip-us.apache.org/repos/asf/drill/blob/85e05623/_docs/016-design.md ---------------------------------------------------------------------- diff --git a/_docs/016-design.md b/_docs/016-design.md deleted file mode 100644 index 474052e..0000000 --- a/_docs/016-design.md +++ /dev/null @@ -1,13 +0,0 @@ ---- -title: "Design Docs" ---- -Review the Apache Drill design docs for early descriptions of Apache Drill -functionality, terms, and goals, and reference the research articles to learn -about Apache Drill's history: - - * [Drill Plan Syntax](/docs/drill-plan-syntax) - * [RPC Overview](/docs/rpc-overview) - * [Query Stages](/docs/query-stages) - * [Useful Research](/docs/useful-research) - * [Value Vectors](/docs/value-vectors) - http://git-wip-us.apache.org/repos/asf/drill/blob/85e05623/_docs/016-progress.md ---------------------------------------------------------------------- diff --git a/_docs/016-progress.md b/_docs/016-progress.md new file mode 100644 index 0000000..680290e --- /dev/null +++ b/_docs/016-progress.md @@ -0,0 +1,8 @@ +--- +title: "Progress Reports" +--- +Review the following Apache Drill progress reports for a summary of issues, +progression of the project, summary of mailing list discussions, and events: + + * [2014 Q1 Drill Report](/docs/2014-q1-drill-report) + http://git-wip-us.apache.org/repos/asf/drill/blob/85e05623/_docs/017-archived-pages.md ---------------------------------------------------------------------- diff --git a/_docs/017-archived-pages.md b/_docs/017-archived-pages.md deleted file mode 100644 index d052579..0000000 --- a/_docs/017-archived-pages.md +++ /dev/null @@ -1,8 +0,0 @@ ---- -title: "Archived Pages" ---- -The following pages have been archived: - -* How to Run Drill with Sample Data -* Meet Apache Drill - http://git-wip-us.apache.org/repos/asf/drill/blob/85e05623/_docs/017-bylaws.md ---------------------------------------------------------------------- diff --git a/_docs/017-bylaws.md b/_docs/017-bylaws.md new file mode 100644 index 0000000..2c35042 --- /dev/null +++ b/_docs/017-bylaws.md @@ -0,0 +1,170 @@ +--- +title: "Project Bylaws" +--- +## Introduction + +This document defines the bylaws under which the Apache Drill project +operates. It defines the roles and responsibilities of the project, who may +vote, how voting works, how conflicts are resolved, etc. + +Drill is a project of the [Apache Software +Foundation](http://www.apache.org/foundation/). The foundation holds the +copyright on Apache code including the code in the Drill codebase. The +[foundation FAQ](http://www.apache.org/foundation/faq.html) explains the +operation and background of the foundation. + +Drill is typical of Apache projects in that it operates under a set of +principles, known collectively as the _Apache Way_. If you are new to Apache +development, please refer to the [Incubator +project](http://incubator.apache.org/) for more information on how Apache +projects operate. + +## Roles and Responsibilities + +Apache projects define a set of roles with associated rights and +responsibilities. These roles govern what tasks an individual may perform +within the project. The roles are defined in the following sections. + +### Users + +The most important participants in the project are people who use our +software. The majority of our contributors start out as users and guide their +development efforts from the user's perspective. + +Users contribute to the Apache projects by providing feedback to contributors +in the form of bug reports and feature suggestions. As well, users participate +in the Apache community by helping other users on mailing lists and user +support forums. + +### Contributors + +All of the volunteers who are contributing time, code, documentation, or +resources to the Drill Project. A contributor that makes sustained, welcome +contributions to the project may be invited to become a committer, though the +exact timing of such invitations depends on many factors. + +### Committers + +The project's committers are responsible for the project's technical +management. Committers have access to a specified set of subproject's code +repositories. Committers on subprojects may cast binding votes on any +technical discussion regarding that subproject. + +Committer access is by invitation only and must be approved by lazy consensus +of the active PMC members. A Committer is considered _emeritus_ by his or her +own declaration or by not contributing in any form to the project for over six +months. An emeritus committer may request reinstatement of commit access from +the PMC which will be sufficient to restore him or her to active committer +status. + +Commit access can be revoked by a unanimous vote of all the active PMC members +(except the committer in question if he or she is also a PMC member). + +All Apache committers are required to have a signed [Contributor License +Agreement (CLA)](http://www.apache.org/licenses/icla.txt) on file with the +Apache Software Foundation. There is a [Committer +FAQ](http://www.apache.org/dev/committers.html) which provides more details on +the requirements for committers. + +A committer who makes a sustained contribution to the project may be invited +to become a member of the PMC. The form of contribution is not limited to +code. It can also include code review, helping out users on the mailing lists, +documentation, etc. + +### Project Management Committee + +The PMC is responsible to the board and the ASF for the management and +oversight of the Apache Drill codebase. The responsibilities of the PMC +include + + * Deciding what is distributed as products of the Apache Drill project. In particular all releases must be approved by the PMC. + * Maintaining the project's shared resources, including the codebase repository, mailing lists, websites. + * Speaking on behalf of the project. + * Resolving license disputes regarding products of the project. + * Nominating new PMC members and committers. + * Maintaining these bylaws and other guidelines of the project. + +Membership of the PMC is by invitation only and must be approved by a lazy +consensus of active PMC members. A PMC member is considered _emeritus_ by his +or her own declaration or by not contributing in any form to the project for +over six months. An emeritus member may request reinstatement to the PMC, +which will be sufficient to restore him or her to active PMC member. + +Membership of the PMC can be revoked by an unanimous vote of all the active +PMC members other than the member in question. + +The chair of the PMC is appointed by the ASF board. The chair is an office +holder of the Apache Software Foundation (Vice President, Apache Drill) and +has primary responsibility to the board for the management of the projects +within the scope of the Drill PMC. The chair reports to the board quarterly on +developments within the Drill project. + +The term of the chair is one year. When the current chair's term is up or if +the chair resigns before the end of his or her term, the PMC votes to +recommend a new chair using lazy consensus, but the decision must be ratified +by the Apache board. + +## Decision Making + +Within the Drill project, different types of decisions require different forms +of approval. For example, the previous section describes several decisions +which require 'lazy consensus' approval. This section defines how voting is +performed, the types of approvals, and which types of decision require which +type of approval. + +### Voting + +Decisions regarding the project are made by votes on the primary project +development mailing list +_[d...@drill.apache.org](mailto:d...@drill.apache.org)_. Where necessary, PMC +voting may take place on the private Drill PMC mailing list +[priv...@drill.apache.org](mailto:priv...@drill.apache.org). Votes are clearly +indicated by subject line starting with [VOTE]. Votes may contain multiple +items for approval and these should be clearly separated. Voting is carried +out by replying to the vote mail. Voting may take four flavors. + + <table ><tbody><tr><td valign="top" >Vote</td><td valign="top" > </td></tr><tr><td valign="top" >+1</td><td valign="top" >'Yes,' 'Agree,' or 'the action should be performed.' In general, this vote also indicates a willingness on the behalf of the voter in 'making it happen'.</td></tr><tr><td valign="top" >+0</td><td valign="top" >This vote indicates a willingness for the action under consideration to go ahead. The voter, however will not be able to help.</td></tr><tr><td valign="top" >-0</td><td valign="top" >This vote indicates that the voter does not, in general, agree with the proposed action but is not concerned enough to prevent the action going ahead.</td></tr><tr><td valign="top" >-1</td><td valign="top" >This is a negative vote. On issues where consensus is required, this vote counts as a <strong>veto</strong>. All vetoes must contain an explanation of why the veto is appropriate. Vetoes with no explanation are void. It may also be appropriate for a -1 vote to include an al ternative course of action.</td></tr></tbody></table> + +All participants in the Drill project are encouraged to show their agreement +with or against a particular action by voting. For technical decisions, only +the votes of active committers are binding. Non binding votes are still useful +for those with binding votes to understand the perception of an action in the +wider Drill community. For PMC decisions, only the votes of PMC members are +binding. + +Voting can also be applied to changes already made to the Drill codebase. +These typically take the form of a veto (-1) in reply to the commit message +sent when the commit is made. Note that this should be a rare occurrence. All +efforts should be made to discuss issues when they are still patches before +the code is committed. + +### Approvals + +These are the types of approvals that can be sought. Different actions require +different types of approvals. + +<table ><tbody><tr><td valign="top" >Approval Type</td><td valign="top" > </td></tr><tr><td valign="top" >Consensus</td><td valign="top" >For this to pass, all voters with binding votes must vote and there can be no binding vetoes (-1). Consensus votes are rarely required due to the impracticality of getting all eligible voters to cast a vote.</td></tr><tr><td valign="top" >Lazy Consensus</td><td valign="top" >Lazy consensus requires 3 binding +1 votes and no binding vetoes.</td></tr><tr><td valign="top" >Lazy Majority</td><td valign="top" >A lazy majority vote requires 3 binding +1 votes and more binding +1 votes that -1 votes.</td></tr><tr><td valign="top" >Lazy Approval</td><td valign="top" >An action with lazy approval is implicitly allowed unless a -1 vote is received, at which time, depending on the type of action, either lazy majority or lazy consensus approval must be obtained.</td></tr></tbody></table> + +### Vetoes + +A valid, binding veto cannot be overruled. If a veto is cast, it must be +accompanied by a valid reason explaining the reasons for the veto. The +validity of a veto, if challenged, can be confirmed by anyone who has a +binding vote. This does not necessarily signify agreement with the veto - +merely that the veto is valid. + +If you disagree with a valid veto, you must lobby the person casting the veto +to withdraw his or her veto. If a veto is not withdrawn, the action that has +been vetoed must be reversed in a timely manner. + +### Actions + +This section describes the various actions which are undertaken within the +project, the corresponding approval required for that action and those who +have binding votes over the action. It also specifies the minimum length of +time that a vote must remain open, measured in business days. In general votes +should not be called at times when it is known that interested members of the +project will be unavailable. + +<table ><tbody><tr><td valign="top" >Action</td><td valign="top" >Description</td><td valign="top" >Approval</td><td valign="top" >Binding Votes</td><td valign="top" >Minimum Length</td></tr><tr><td valign="top" >Code Change</td><td valign="top" >A change made to a codebase of the project and committed by a committer. This includes source code, documentation, website content, etc.</td><td valign="top" >Consensus approval of active committers, with a minimum of one +1. The code can be committed after the first +1</td><td valign="top" >Active committers</td><td valign="top" >1</td></tr><tr><td valign="top" >Release Plan</td><td valign="top" >Defines the timetable and actions for a release. The plan also nominates a Release Manager.</td><td valign="top" >Lazy majority</td><td valign="top" >Active committers</td><td valign="top" >3</td></tr><tr><td valign="top" >Product Release</td><td valign="top" >When a release of one of the project's products is ready, a vote is required to accept t he release as an official release of the project.</td><td valign="top" >Lazy Majority</td><td valign="top" >Active PMC members</td><td valign="top" >3</td></tr><tr><td valign="top" >Adoption of New Codebase</td><td valign="top" >When the codebase for an existing, released product is to be replaced with an alternative codebase. If such a vote fails to gain approval, the existing code base will continue. This also covers the creation of new sub-projects within the project.</td><td valign="top" >2/3 majority</td><td valign="top" >Active PMC members</td><td valign="top" >6</td></tr><tr><td valign="top" >New Committer</td><td valign="top" >When a new committer is proposed for the project.</td><td valign="top" >Lazy consensus</td><td valign="top" >Active PMC members</td><td valign="top" >3</td></tr><tr><td valign="top" >New PMC Member</td><td valign="top" >When a committer is proposed for the PMC.</td><td valign="top" >Lazy consensus</td><td valign="top" >Active PMC members</td><td valign ="top" >3</td></tr><tr><td valign="top" >Committer Removal</td><td valign="top" >When removal of commit privileges is sought. <em>Note: Such actions will also be referred to the ASF board by the PMC chair.</em></td><td valign="top" >Consensus</td><td valign="top" >Active PMC members (excluding the committer in question if a member of the PMC).</td><td valign="top" >6</td></tr><tr><td valign="top" >PMC Member Removal</td><td valign="top" >When removal of a PMC member is sought. <em>Note: Such actions will also be referred to the ASF board by the PMC chair.</em></td><td valign="top" >Consensus</td><td valign="top" >Active PMC members (excluding the member in question).</td><td valign="top" >6</td></tr><tr><td valign="top" >Modifying Bylaws</td><td valign="top" >Modifying this document.</td><td valign="top" >2/3 majority</td><td valign="top" >Active PMC members</td><td valign="top" >6</td></tr></tbody></table> + http://git-wip-us.apache.org/repos/asf/drill/blob/85e05623/_docs/018-progress.md ---------------------------------------------------------------------- diff --git a/_docs/018-progress.md b/_docs/018-progress.md deleted file mode 100644 index 680290e..0000000 --- a/_docs/018-progress.md +++ /dev/null @@ -1,8 +0,0 @@ ---- -title: "Progress Reports" ---- -Review the following Apache Drill progress reports for a summary of issues, -progression of the project, summary of mailing list discussions, and events: - - * [2014 Q1 Drill Report](/docs/2014-q1-drill-report) - http://git-wip-us.apache.org/repos/asf/drill/blob/85e05623/_docs/019-bylaws.md ---------------------------------------------------------------------- diff --git a/_docs/019-bylaws.md b/_docs/019-bylaws.md deleted file mode 100644 index 2c35042..0000000 --- a/_docs/019-bylaws.md +++ /dev/null @@ -1,170 +0,0 @@ ---- -title: "Project Bylaws" ---- -## Introduction - -This document defines the bylaws under which the Apache Drill project -operates. It defines the roles and responsibilities of the project, who may -vote, how voting works, how conflicts are resolved, etc. - -Drill is a project of the [Apache Software -Foundation](http://www.apache.org/foundation/). The foundation holds the -copyright on Apache code including the code in the Drill codebase. The -[foundation FAQ](http://www.apache.org/foundation/faq.html) explains the -operation and background of the foundation. - -Drill is typical of Apache projects in that it operates under a set of -principles, known collectively as the _Apache Way_. If you are new to Apache -development, please refer to the [Incubator -project](http://incubator.apache.org/) for more information on how Apache -projects operate. - -## Roles and Responsibilities - -Apache projects define a set of roles with associated rights and -responsibilities. These roles govern what tasks an individual may perform -within the project. The roles are defined in the following sections. - -### Users - -The most important participants in the project are people who use our -software. The majority of our contributors start out as users and guide their -development efforts from the user's perspective. - -Users contribute to the Apache projects by providing feedback to contributors -in the form of bug reports and feature suggestions. As well, users participate -in the Apache community by helping other users on mailing lists and user -support forums. - -### Contributors - -All of the volunteers who are contributing time, code, documentation, or -resources to the Drill Project. A contributor that makes sustained, welcome -contributions to the project may be invited to become a committer, though the -exact timing of such invitations depends on many factors. - -### Committers - -The project's committers are responsible for the project's technical -management. Committers have access to a specified set of subproject's code -repositories. Committers on subprojects may cast binding votes on any -technical discussion regarding that subproject. - -Committer access is by invitation only and must be approved by lazy consensus -of the active PMC members. A Committer is considered _emeritus_ by his or her -own declaration or by not contributing in any form to the project for over six -months. An emeritus committer may request reinstatement of commit access from -the PMC which will be sufficient to restore him or her to active committer -status. - -Commit access can be revoked by a unanimous vote of all the active PMC members -(except the committer in question if he or she is also a PMC member). - -All Apache committers are required to have a signed [Contributor License -Agreement (CLA)](http://www.apache.org/licenses/icla.txt) on file with the -Apache Software Foundation. There is a [Committer -FAQ](http://www.apache.org/dev/committers.html) which provides more details on -the requirements for committers. - -A committer who makes a sustained contribution to the project may be invited -to become a member of the PMC. The form of contribution is not limited to -code. It can also include code review, helping out users on the mailing lists, -documentation, etc. - -### Project Management Committee - -The PMC is responsible to the board and the ASF for the management and -oversight of the Apache Drill codebase. The responsibilities of the PMC -include - - * Deciding what is distributed as products of the Apache Drill project. In particular all releases must be approved by the PMC. - * Maintaining the project's shared resources, including the codebase repository, mailing lists, websites. - * Speaking on behalf of the project. - * Resolving license disputes regarding products of the project. - * Nominating new PMC members and committers. - * Maintaining these bylaws and other guidelines of the project. - -Membership of the PMC is by invitation only and must be approved by a lazy -consensus of active PMC members. A PMC member is considered _emeritus_ by his -or her own declaration or by not contributing in any form to the project for -over six months. An emeritus member may request reinstatement to the PMC, -which will be sufficient to restore him or her to active PMC member. - -Membership of the PMC can be revoked by an unanimous vote of all the active -PMC members other than the member in question. - -The chair of the PMC is appointed by the ASF board. The chair is an office -holder of the Apache Software Foundation (Vice President, Apache Drill) and -has primary responsibility to the board for the management of the projects -within the scope of the Drill PMC. The chair reports to the board quarterly on -developments within the Drill project. - -The term of the chair is one year. When the current chair's term is up or if -the chair resigns before the end of his or her term, the PMC votes to -recommend a new chair using lazy consensus, but the decision must be ratified -by the Apache board. - -## Decision Making - -Within the Drill project, different types of decisions require different forms -of approval. For example, the previous section describes several decisions -which require 'lazy consensus' approval. This section defines how voting is -performed, the types of approvals, and which types of decision require which -type of approval. - -### Voting - -Decisions regarding the project are made by votes on the primary project -development mailing list -_[d...@drill.apache.org](mailto:d...@drill.apache.org)_. Where necessary, PMC -voting may take place on the private Drill PMC mailing list -[priv...@drill.apache.org](mailto:priv...@drill.apache.org). Votes are clearly -indicated by subject line starting with [VOTE]. Votes may contain multiple -items for approval and these should be clearly separated. Voting is carried -out by replying to the vote mail. Voting may take four flavors. - - <table ><tbody><tr><td valign="top" >Vote</td><td valign="top" > </td></tr><tr><td valign="top" >+1</td><td valign="top" >'Yes,' 'Agree,' or 'the action should be performed.' In general, this vote also indicates a willingness on the behalf of the voter in 'making it happen'.</td></tr><tr><td valign="top" >+0</td><td valign="top" >This vote indicates a willingness for the action under consideration to go ahead. The voter, however will not be able to help.</td></tr><tr><td valign="top" >-0</td><td valign="top" >This vote indicates that the voter does not, in general, agree with the proposed action but is not concerned enough to prevent the action going ahead.</td></tr><tr><td valign="top" >-1</td><td valign="top" >This is a negative vote. On issues where consensus is required, this vote counts as a <strong>veto</strong>. All vetoes must contain an explanation of why the veto is appropriate. Vetoes with no explanation are void. It may also be appropriate for a -1 vote to include an al ternative course of action.</td></tr></tbody></table> - -All participants in the Drill project are encouraged to show their agreement -with or against a particular action by voting. For technical decisions, only -the votes of active committers are binding. Non binding votes are still useful -for those with binding votes to understand the perception of an action in the -wider Drill community. For PMC decisions, only the votes of PMC members are -binding. - -Voting can also be applied to changes already made to the Drill codebase. -These typically take the form of a veto (-1) in reply to the commit message -sent when the commit is made. Note that this should be a rare occurrence. All -efforts should be made to discuss issues when they are still patches before -the code is committed. - -### Approvals - -These are the types of approvals that can be sought. Different actions require -different types of approvals. - -<table ><tbody><tr><td valign="top" >Approval Type</td><td valign="top" > </td></tr><tr><td valign="top" >Consensus</td><td valign="top" >For this to pass, all voters with binding votes must vote and there can be no binding vetoes (-1). Consensus votes are rarely required due to the impracticality of getting all eligible voters to cast a vote.</td></tr><tr><td valign="top" >Lazy Consensus</td><td valign="top" >Lazy consensus requires 3 binding +1 votes and no binding vetoes.</td></tr><tr><td valign="top" >Lazy Majority</td><td valign="top" >A lazy majority vote requires 3 binding +1 votes and more binding +1 votes that -1 votes.</td></tr><tr><td valign="top" >Lazy Approval</td><td valign="top" >An action with lazy approval is implicitly allowed unless a -1 vote is received, at which time, depending on the type of action, either lazy majority or lazy consensus approval must be obtained.</td></tr></tbody></table> - -### Vetoes - -A valid, binding veto cannot be overruled. If a veto is cast, it must be -accompanied by a valid reason explaining the reasons for the veto. The -validity of a veto, if challenged, can be confirmed by anyone who has a -binding vote. This does not necessarily signify agreement with the veto - -merely that the veto is valid. - -If you disagree with a valid veto, you must lobby the person casting the veto -to withdraw his or her veto. If a veto is not withdrawn, the action that has -been vetoed must be reversed in a timely manner. - -### Actions - -This section describes the various actions which are undertaken within the -project, the corresponding approval required for that action and those who -have binding votes over the action. It also specifies the minimum length of -time that a vote must remain open, measured in business days. In general votes -should not be called at times when it is known that interested members of the -project will be unavailable. - -<table ><tbody><tr><td valign="top" >Action</td><td valign="top" >Description</td><td valign="top" >Approval</td><td valign="top" >Binding Votes</td><td valign="top" >Minimum Length</td></tr><tr><td valign="top" >Code Change</td><td valign="top" >A change made to a codebase of the project and committed by a committer. This includes source code, documentation, website content, etc.</td><td valign="top" >Consensus approval of active committers, with a minimum of one +1. The code can be committed after the first +1</td><td valign="top" >Active committers</td><td valign="top" >1</td></tr><tr><td valign="top" >Release Plan</td><td valign="top" >Defines the timetable and actions for a release. The plan also nominates a Release Manager.</td><td valign="top" >Lazy majority</td><td valign="top" >Active committers</td><td valign="top" >3</td></tr><tr><td valign="top" >Product Release</td><td valign="top" >When a release of one of the project's products is ready, a vote is required to accept t he release as an official release of the project.</td><td valign="top" >Lazy Majority</td><td valign="top" >Active PMC members</td><td valign="top" >3</td></tr><tr><td valign="top" >Adoption of New Codebase</td><td valign="top" >When the codebase for an existing, released product is to be replaced with an alternative codebase. If such a vote fails to gain approval, the existing code base will continue. This also covers the creation of new sub-projects within the project.</td><td valign="top" >2/3 majority</td><td valign="top" >Active PMC members</td><td valign="top" >6</td></tr><tr><td valign="top" >New Committer</td><td valign="top" >When a new committer is proposed for the project.</td><td valign="top" >Lazy consensus</td><td valign="top" >Active PMC members</td><td valign="top" >3</td></tr><tr><td valign="top" >New PMC Member</td><td valign="top" >When a committer is proposed for the PMC.</td><td valign="top" >Lazy consensus</td><td valign="top" >Active PMC members</td><td valign ="top" >3</td></tr><tr><td valign="top" >Committer Removal</td><td valign="top" >When removal of commit privileges is sought. <em>Note: Such actions will also be referred to the ASF board by the PMC chair.</em></td><td valign="top" >Consensus</td><td valign="top" >Active PMC members (excluding the committer in question if a member of the PMC).</td><td valign="top" >6</td></tr><tr><td valign="top" >PMC Member Removal</td><td valign="top" >When removal of a PMC member is sought. <em>Note: Such actions will also be referred to the ASF board by the PMC chair.</em></td><td valign="top" >Consensus</td><td valign="top" >Active PMC members (excluding the member in question).</td><td valign="top" >6</td></tr><tr><td valign="top" >Modifying Bylaws</td><td valign="top" >Modifying this document.</td><td valign="top" >2/3 majority</td><td valign="top" >Active PMC members</td><td valign="top" >6</td></tr></tbody></table> - http://git-wip-us.apache.org/repos/asf/drill/blob/85e05623/_docs/arch/001-core-mod.md ---------------------------------------------------------------------- diff --git a/_docs/arch/001-core-mod.md b/_docs/arch/001-core-mod.md index 7cd2fdf..17fa18d 100644 --- a/_docs/arch/001-core-mod.md +++ b/_docs/arch/001-core-mod.md @@ -2,8 +2,6 @@ title: "Core Modules within a Drillbit" parent: "Architectural Overview" --- -[Previous](/docs/architectural-overview)<code> </code>[Back to Table of Contents](/docs)<code> </code>[Next](/docs/architectural-highlights) - The following image represents components within each Drillbit:  http://git-wip-us.apache.org/repos/asf/drill/blob/85e05623/_docs/arch/002-arch-hilite.md ---------------------------------------------------------------------- diff --git a/_docs/arch/002-arch-hilite.md b/_docs/arch/002-arch-hilite.md index 3855745..5ac51bc 100644 --- a/_docs/arch/002-arch-hilite.md +++ b/_docs/arch/002-arch-hilite.md @@ -2,8 +2,6 @@ title: "Architectural Highlights" parent: "Architectural Overview" --- -[Previous](/docs/core-modules-within-a-drillbit)<code> </code>[Back to Table of Contents](/docs)<code> </code>[Next](/docs/flexibility) - The goal for Drill is to bring the **SQL Ecosystem** and **Performance** of the relational systems to **Hadoop scale** data **WITHOUT** compromising on the **Flexibility** of Hadoop/NoSQL systems. There are several core http://git-wip-us.apache.org/repos/asf/drill/blob/85e05623/_docs/arch/arch-hilite/001-flexibility.md ---------------------------------------------------------------------- diff --git a/_docs/arch/arch-hilite/001-flexibility.md b/_docs/arch/arch-hilite/001-flexibility.md index 3a10944..8c0ae3a 100644 --- a/_docs/arch/arch-hilite/001-flexibility.md +++ b/_docs/arch/arch-hilite/001-flexibility.md @@ -2,8 +2,6 @@ title: "Flexibility" parent: "Architectural Highlights" --- -[Previous](/docs/architectural-highlights)<code> </code>[Back to Table of Contents](/docs)<code> </code>[Next](/docs/performance) - The following features contribute to Drill's flexible architecture: **_Dynamic schema discovery_** http://git-wip-us.apache.org/repos/asf/drill/blob/85e05623/_docs/arch/arch-hilite/002-performance.md ---------------------------------------------------------------------- diff --git a/_docs/arch/arch-hilite/002-performance.md b/_docs/arch/arch-hilite/002-performance.md index 57ce3ab..c6271e0 100644 --- a/_docs/arch/arch-hilite/002-performance.md +++ b/_docs/arch/arch-hilite/002-performance.md @@ -2,8 +2,6 @@ title: "Performance" parent: "Architectural Highlights" --- -[Previous](/docs/flexibility)<code> </code>[Back to Table of Contents](/docs)<code> </code>[Next](/docs/apache-drill-tutorial) - Drill is designed from the ground up for high performance on large datasets. The following core elements of Drill processing are responsible for Drill's performance: http://git-wip-us.apache.org/repos/asf/drill/blob/85e05623/_docs/connect/001-plugin-reg.md ---------------------------------------------------------------------- diff --git a/_docs/connect/001-plugin-reg.md b/_docs/connect/001-plugin-reg.md index fd21773..b3c5cdc 100644 --- a/_docs/connect/001-plugin-reg.md +++ b/_docs/connect/001-plugin-reg.md @@ -2,8 +2,6 @@ title: "Storage Plugin Registration" parent: "Connect to a Data Source" --- -[Previous](/docs/connect-to-a-data-source)<code> </code>[Back to Table of Contents](/docs)<code> </code>[Next](/docs/storage-plugin-configuration) - You connect Drill to a file system, Hive, HBase, or other data source using storage plugins. Drill includes a number of storage plugins in the installation. On the Storage tab of the Web UI, you can view, create, reconfigure, and register a storage plugin. To open the Storage tab, go to `http://<IP address>:8047/storage`, where IP address is any one of the installed drill bits:  http://git-wip-us.apache.org/repos/asf/drill/blob/85e05623/_docs/connect/002-plugin-conf.md ---------------------------------------------------------------------- diff --git a/_docs/connect/002-plugin-conf.md b/_docs/connect/002-plugin-conf.md index 74c155a..caef2ad 100644 --- a/_docs/connect/002-plugin-conf.md +++ b/_docs/connect/002-plugin-conf.md @@ -2,8 +2,6 @@ title: "Storage Plugin Configuration" parent: "Connect to a Data Source" --- -[Previous](/docs/storage-plugin-registration)<code> </code>[Back to Table of Contents](/docs)<code> </code>[Next](/docs/workspaces) - When you add or update storage plugin instances on one Drill node in a Drill cluster, Drill broadcasts the information to all of the other Drill nodes to have identical storage plugin configurations. You do not need to http://git-wip-us.apache.org/repos/asf/drill/blob/85e05623/_docs/connect/003-workspaces.md ---------------------------------------------------------------------- diff --git a/_docs/connect/003-workspaces.md b/_docs/connect/003-workspaces.md index 3c6f7db..6e973e5 100644 --- a/_docs/connect/003-workspaces.md +++ b/_docs/connect/003-workspaces.md @@ -2,8 +2,6 @@ title: "Workspaces" parent: "Storage Plugin Configuration" --- -[Previous](/docs/storage-plugin-configuration)<code> </code>[Back to Table of Contents](/docs)<code> </code>[Next](/docs/file-system-storage-plugin) - When you register an instance of a file system data source, you can configure one or more workspaces for the instance. A workspace is a directory within the file system that you define. Drill searches the workspace to locate data when http://git-wip-us.apache.org/repos/asf/drill/blob/85e05623/_docs/connect/004-reg-fs.md ---------------------------------------------------------------------- diff --git a/_docs/connect/004-reg-fs.md b/_docs/connect/004-reg-fs.md index 02cd56a..2b3e287 100644 --- a/_docs/connect/004-reg-fs.md +++ b/_docs/connect/004-reg-fs.md @@ -2,8 +2,6 @@ title: "File System Storage Plugin" parent: "Storage Plugin Configuration" --- -[Previous](/docs/workspaces)<code> </code>[Back to Table of Contents](/docs)<code> </code>[Next](/docs/hbase-storage-plugin) - You can register a storage plugin instance that connects Drill to a local file system or a distributed file system registered in `core-site.xml`, such as S3 or HDFS. When you register a storage plugin instance for a file system, http://git-wip-us.apache.org/repos/asf/drill/blob/85e05623/_docs/connect/005-reg-hbase.md ---------------------------------------------------------------------- diff --git a/_docs/connect/005-reg-hbase.md b/_docs/connect/005-reg-hbase.md index 434ab88..627fb07 100644 --- a/_docs/connect/005-reg-hbase.md +++ b/_docs/connect/005-reg-hbase.md @@ -2,8 +2,6 @@ title: "HBase Storage Plugin" parent: "Storage Plugin Configuration" --- -[Previous](/docs/file-system-storage-plugin)<code> </code>[Back to Table of Contents](/docs)<code> </code>[Next](/docs/hive-storage-plugin) - Register a storage plugin instance and specify a zookeeper quorum to connect Drill to an HBase data source. When you register a storage plugin instance for an HBase data source, provide a unique name for the instance, and identify the http://git-wip-us.apache.org/repos/asf/drill/blob/85e05623/_docs/connect/006-reg-hive.md ---------------------------------------------------------------------- diff --git a/_docs/connect/006-reg-hive.md b/_docs/connect/006-reg-hive.md index 34a1229..dfb03dc 100644 --- a/_docs/connect/006-reg-hive.md +++ b/_docs/connect/006-reg-hive.md @@ -2,8 +2,6 @@ title: "Hive Storage Plugin" parent: "Storage Plugin Configuration" --- -[Previous](/docs/hbase-storage-plugin)<code> </code>[Back to Table of Contents](/docs)<code> </code>[Next](/docs/drill-default-input-format) - You can register a storage plugin instance that connects Drill to a Hive data source that has a remote or embedded metastore service. When you register a storage plugin instance for a Hive data source, provide a unique name for the http://git-wip-us.apache.org/repos/asf/drill/blob/85e05623/_docs/connect/007-default-frmt.md ---------------------------------------------------------------------- diff --git a/_docs/connect/007-default-frmt.md b/_docs/connect/007-default-frmt.md index 03b4f8e..fc10c16 100644 --- a/_docs/connect/007-default-frmt.md +++ b/_docs/connect/007-default-frmt.md @@ -2,8 +2,6 @@ title: "Drill Default Input Format" parent: "Storage Plugin Configuration" --- -[Previous](/docs/hive-storage-plugin)<code> </code>[Back to Table of Contents](/docs)<code> </code>[Next](/docs/mongodb-plugin-for-apache-drill) - You can define a default input format to tell Drill what file type exists in a workspace within a file system. Drill determines the file type based on file extensions and magic numbers when searching a workspace. http://git-wip-us.apache.org/repos/asf/drill/blob/85e05623/_docs/connect/008-mongo-plugin.md ---------------------------------------------------------------------- diff --git a/_docs/connect/008-mongo-plugin.md b/_docs/connect/008-mongo-plugin.md index b688428..bf2efdf 100644 --- a/_docs/connect/008-mongo-plugin.md +++ b/_docs/connect/008-mongo-plugin.md @@ -2,8 +2,6 @@ title: "MongoDB Plugin for Apache Drill" parent: "Connect to a Data Source" --- -[Previous](/docs/drill-default-input-format)<code> </code>[Back to Table of Contents](/docs)<code> </code>[Next](/docs/mapr-db-format/) - ## Overview You can leverage the power of Apache Drill to query data without any upfront http://git-wip-us.apache.org/repos/asf/drill/blob/85e05623/_docs/connect/009-mapr-db-plugin.md ---------------------------------------------------------------------- diff --git a/_docs/connect/009-mapr-db-plugin.md b/_docs/connect/009-mapr-db-plugin.md index 425f489..a0582f3 100644 --- a/_docs/connect/009-mapr-db-plugin.md +++ b/_docs/connect/009-mapr-db-plugin.md @@ -2,8 +2,6 @@ title: "MapR-DB Format" parent: "Connect to a Data Source" --- -[Previous](/docs/mongodb-plugin-for-apache-drill)<code> </code>[Back to Table of Contents](/docs)<code> </code>[Next](/docs/odbc-jdbc-interfaces) - Drill includes a `maprdb` format for reading MapR-DB data. The `dfs` storage plugin defines the format when you install Drill from the `mapr-drill` package on a MapR node. The `maprdb` format plugin improves the estimated number of rows that Drill uses to plan a query. It also enables you to query tables like you would query files in a file system because MapR-DB http://git-wip-us.apache.org/repos/asf/drill/blob/85e05623/_docs/contribute/001-guidelines.md ---------------------------------------------------------------------- diff --git a/_docs/contribute/001-guidelines.md b/_docs/contribute/001-guidelines.md deleted file mode 100644 index 7361e5c..0000000 --- a/_docs/contribute/001-guidelines.md +++ /dev/null @@ -1,228 +0,0 @@ ---- -title: "Apache Drill Contribution Guidelines" -parent: "Contribute to Drill" ---- -## How to Contribute to Apache Drill - -Disclaimer: These contribution guidelines are largely based on Apache Hive -contribution guidelines. - -This page describes the mechanics of _how_ to contribute software to Apache -Drill. For ideas about _what_ you might contribute, please see open tickets in -[Jira](https://issues.apache.org/jira/browse/DRILL). - -These guidelines include the following topics: - -* Getting the source code - * Making Changes - * Coding Convention - * Formatter configuration - * Understanding Maven - * Creating a patch - * Applying a patch - * Where is a good place to start contributing? - * Contributing your work - * JIRA Guidelines - * See Also - -### Getting the source code - -First, you need the Drill source code. - -Get the source code on your local drive using [Git](https://git-wip- -us.apache.org/repos/asf/incubator-drill.git). Most development is done on -"master": - - git clone https://git-wip-us.apache.org/repos/asf/drill.git - -### Making Changes - -Before you start, send a message to the [Drill developer mailing list](http -://mail-archives.apache.org/mod_mbox/incubator-drill-dev/), or file a bug -report in [JIRA](https://issues.apache.org/jira/browse/DRILL). Describe your -proposed changes and check that they fit in with what others are doing and -have planned for the project. Be patient, it may take folks a while to -understand your requirements. - -Modify the source code and add some features using your favorite IDE. - -#### Coding Convention - -Please take care about the following points - - * All public classes and methods should have informative [Javadoc comments](http://www.oracle.com/technetwork/java/javase/documentation/index-137868.html). - * Do not use @author tags. - * Code should be formatted according to [Sun's conventions](http://www.oracle.com/technetwork/java/codeconv-138413.html), with one exception: - * Indent two (2) spaces per level, not four (4). - * Line length limit is 120 chars, instead of 80 chars. - * Contributions should not introduce new Checkstyle violations. - * Contributions should pass existing unit tests. - * New unit tests should be provided to demonstrate bugs and fixes. [JUnit](http://www.junit.org) 4.1 is our test framework: - * You must implement a class that contain test methods annotated with JUnit's 4.x @Test annotation and whose class name ends with `Test`. - * Define methods within your class whose names begin with `test`, and call JUnit's many assert methods to verify conditions; these methods will be executed when you run `mvn clean test`. - -#### Formatter configuration - -Setting up IDE formatters is recommended and can be done by importing the -following settings into your browser: - -IntelliJ IDEA formatter: [settings -jar](https://cwiki.apache.org/confluence/download/attachments/30757399/idea-settings.jar?version=1&modificationDate=1363022308000&api=v2) - -Eclipse: [formatter xml from HBase](https://issues.apache.org/jira/secure/atta -chment/12474245/eclipse_formatter_apache.xml) - -#### Understanding Maven - -Drill is built by Maven, a Java build tool. - - * Good Maven tutorial: <http://maven.apache.org/guides/getting-started/maven-in-five-minutes.html> - -To build Drill, run - - mvn clean install - - -#### Creating a patch - -Check to see what files you have modified: - - git status - -Add any new files with: - - git add .../MyNewClass.java - git add .../TestMyNewClass.java - git add .../XXXXXX.q - git add .../XXXXXX.q.out - -In order to create a patch, type (from the base directory of drill): - - git format-patch origin/master --stdout > DRILL-1234.1.patch.txt - -This will report all modifications done on Drill sources on your local disk -and save them into the _DRILL-1234.1.patch.txt_ file. Read the patch file. -Make sure it includes ONLY the modifications required to fix a single issue. - -Please do not: - - * reformat code unrelated to the bug being fixed: formatting changes should be separate patches/commits. - * comment out code that is now obsolete: just remove it. - * insert comments around each change, marking the change: folks can use subversion to figure out what's changed and by whom. - * make things public which are not required by end users. - -Please do: - - * try to adhere to the coding style of files you edit; - * comment code whose function or rationale is not obvious; - * update documentation (e.g., _package.html_ files, this wiki, etc.) - -Updating a patch - -For patch updates, our convention is to number them like -DRILL-1856.1.patch.txt, DRILL-1856.2.patch.txt, etc. And then click the -"Submit Patch" button again when a new one is uploaded; this makes sure it -gets back into the review queue. Appending '.txt' to the patch file name makes -it easy to quickly view the contents of the patch in a web browser. - -#### Applying a patch - -To apply a patch either you generated or found from JIRA, you can issue - - git am < cool_patch.patch - -if you just want to check whether the patch applies you can run patch with ---dry-run option. - - - -####Review Process - - * Use Hadoop's [code review checklist](http://wiki.apache.org/hadoop/CodeReviewChecklist) as a rough guide when doing reviews. - * In JIRA, use attach file to notify that you've submitted a patch for that issue. - * Create a Review Request in [Review Board](https://reviews.apache.org/r/). The review request's name should start with the JIRA issue number (e.g. DRILL-XX) and should be assigned to the "drill-git" group. - * If a committer requests changes, set the issue status to 'Resume Progress', then once you're ready, submit an updated patch with necessary fixes and then request another round of review with 'Submit Patch' again. - * Once your patch is accepted, be sure to upload a final version which grants rights to the ASF. - -## Where is a good place to start contributing? - -After getting the source code, building and running a few simple queries, one -of the simplest places to start is to implement a DrillFunc. -DrillFuncs is way that Drill express all scalar functions (UDF or system). -First you can put together a JIRA for one of the DrillFunc's we don't yet have -but should (referencing the capabilities of something like Postgres -or SQL Server). Then try to implement one. - -One example DrillFunc: - -[https://github.com/apache/incubator- -drill/blob/103072a619741d5e228fdb181501ec2f82e111a3/sandbox/prototype/exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/impl/ComparisonFunction -s.java](https://github.com/apache/incubator- -drill/blob/103072a619741d5e228fdb181501ec2f82e111a3/sandbox/prototype/exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/impl/ComparisonFunction -s.java) - -Also one can visit the JIRA issues and implement one of those too. A list of -functions which need to be implemented can be found -[here](https://docs.google.com/spreadsheet/ccc?key=0AgAGbQ6asvQ- -dDRrUUxVSVlMVXRtanRoWk9JcHgteUE&usp=sharing#gid=0) (WIP). - -More contribution ideas are located on the [Contribution Ideas](/docs/apache-drill-contribution-ideas) page. - -### Contributing your work - -Finally, patches should be _attached_ to an issue report in -[JIRA](http://issues.apache.org/jira/browse/DRILL) via the **Attach File** -link on the issue's JIRA. Please add a comment that asks for a code review. -Please note that the attachment should be granted license to ASF for inclusion -in ASF works (as per the [Apache -License](http://www.apache.org/licenses/LICENSE-2.0)). - -Folks should run `mvn clean install` before submitting a patch. Tests should -all pass. If your patch involves performance optimizations, they should be -validated by benchmarks that demonstrate an improvement. - -If your patch creates an incompatibility with the latest major release, then -you must set the **Incompatible change** flag on the issue's JIRA 'and' fill -in the **Release Note** field with an explanation of the impact of the -incompatibility and the necessary steps users must take. - -If your patch implements a major feature or improvement, then you must fill in -the **Release Note** field on the issue's JIRA with an explanation of the -feature that will be comprehensible by the end user. - -A committer should evaluate the patch within a few days and either: commit it; -or reject it with an explanation. - -Please be patient. Committers are busy people too. If no one responds to your -patch after a few days, please make friendly reminders. Please incorporate -other's suggestions into your patch if you think they're reasonable. Finally, -remember that even a patch that is not committed is useful to the community. - -Should your patch receive a "-1" select the **Resume Progress** on the issue's -JIRA, upload a new patch with necessary fixes, and then select the **Submit -Patch** link again. - -Committers: for non-trivial changes, it is best to get another committer to -review your patches before commit. Use **Submit Patch** link like other -contributors, and then wait for a "+1" from another committer before -committing. Please also try to frequently review things in the patch queue. - -### JIRA Guidelines - -Please comment on issues in JIRA, making their concerns known. Please also -vote for issues that are a high priority for you. - -Please refrain from editing descriptions and comments if possible, as edits -spam the mailing list and clutter JIRA's "All" display, which is otherwise -very useful. Instead, preview descriptions and comments using the preview -button (on the right) before posting them. Keep descriptions brief and save -more elaborate proposals for comments, since descriptions are included in -JIRA's automatically sent messages. If you change your mind, note this in a -new comment, rather than editing an older comment. The issue should preserve -this history of the discussion. - -### See Also - - * [Apache contributor documentation](http://www.apache.org/dev/contributors.html) - * [Apache voting documentation](http://www.apache.org/foundation/voting.html) - http://git-wip-us.apache.org/repos/asf/drill/blob/85e05623/_docs/contribute/002-ideas.md ---------------------------------------------------------------------- diff --git a/_docs/contribute/002-ideas.md b/_docs/contribute/002-ideas.md deleted file mode 100644 index 2270112..0000000 --- a/_docs/contribute/002-ideas.md +++ /dev/null @@ -1,158 +0,0 @@ ---- -title: "Apache Drill Contribution Ideas" -parent: "Contribute to Drill" ---- - * Fixing JIRAs - * SQL functions - * Support for new file format readers/writers - * Support for new data sources - * New query language parsers - * Application interfaces - * BI Tool testing - * General CLI improvements - * Eco system integrations - * MapReduce - * Hive views - * YARN - * Spark - * Hue - * Phoenix - -## Fixing JIRAs - -This is a good place to begin if you are new to Drill. Feel free to pick -issues from the Drill JIRA list. When you pick an issue, assign it to -yourself, inform the team, and start fixing it. - -For any questions, seek help from the team by sending email to [drill- -d...@incubator.apache.org](mailto:drill-...@incubator.apache.org). - -[https://issues.apache.org/jira/browse/DRILL/?selectedTab=com.atlassian.jira -.jira-projects-plugin:summary-panel](https://issues.apache.org/jira/browse/DRILL/?selectedTab=com.atlassian.jira -.jira-projects-plugin:summary-panel) - -## SQL functions - -One of the next simple places to start is to implement a DrillFunc.â¨DrillFuncs -is way that Drill express all scalar functions (UDF or system).⨠First you can -put together a JIRA for one of the DrillFunc's we don't yet have but should -(referencing the capabilities of something like Postgresâ¨or SQL Server or your -own use case). Then try to implement one. - -One example DrillFunc: -[https://github.com/apache/incubator- -drill/blob/103072a619741d5e228fdb181501ec2f82e111a3/sandbox/prototype/exec -/java-exec/src/main/java/org/apache/drill/exec/expr/fn/impl/ComparisonFunction -s.java](https://github.com/apache/incubator- -drill/blob/103072a619741d5e228fdb181501ec2f82e111a3/sandbox/prototype/exec -/java-exec/src/main/java/org/apache/drill/exec/expr/fn/impl/ComparisonFunction -s.java)** ** - -**Additional ideas on functions that can be added to SQL support** - - * Madlib integration - * Machine learning functions - * Approximate aggregate functions (such as what is available in BlinkDB) - -## Support for new file format readers/writers - -Currently Drill supports text, JSON and Parquet file formats natively when -interacting with file system. More readers/writers can be introduced by -implementing custom storage plugins. Example formats include below. - - * AVRO - * Sequence - * RC - * ORC - * Protobuf - * XML - * Thrift - * .... - -## Support for new data sources - -Implement custom storage plugins for the following non-Hadoop data sources: - - * NoSQL databases (such as Mongo, Cassandra, Couch etc) - * Search engines (such as Solr, Lucidworks, Elastic Search etc) - * SQL databases (MySQL< PostGres etc) - * Generic JDBC/ODBC data sources - * HTTP URL - * \---- - -## New query language parsers - -Drill exposes strongly typed JSON APIs for logical and physical plans (plan -syntax at [https://docs.google.com/a/maprtech.com/document/d/1QTL8warUYS2KjldQ -rGUse7zp8eA72VKtLOHwfXy6c7I/edit#heading=h.n9gdb1ek71hf](https://docs.google.com/a/maprtech.com/document/d/1QTL8warUYS2KjldQ -rGUse7zp8eA72VKtLOHwfXy6c7I/edit#heading=h.n9gdb1ek71hf) ). Drill provides a -SQL language parser today, but any language parser that can generate -logical/physical plans can use Drill's power on the backend as the distributed -low latency query execution engine along with its support for self-describing -data and complex/multi-structured data. - - * Pig parser : Use Pig as the language to query data from Drill. Great for existing Pig users. - * Hive parser : Use HiveQL as the language to query data from Drill. Great for existing Hive users. - -## Application interfaces - -Drill currently provides JDBC/ODBC drivers for the applications to interact -along with a basic version of REST API and a C++ API. The following list -provides a few possible application interface opportunities: - - * Enhancements to REST APIs (<https://issues.apache.org/jira/browse/DRILL-77>) - * Expose Drill tables/views as REST APIs - * Language drivers for Drill (python etc) - * Thrift support - * .... - -### BI Tool testing - -Drill provides JDBC/ODBC drivers to connect to BI tools. We need to make sure -Drill works with all major BI tools. Doing a quick sanity testing with your -favorite BI tool is a good place to learn Drill and also uncover issues in -being able to do so. - -## General CLI improvements - -Currently Drill uses SQLLine as the CLI. The goal of this effort is to improve -the CLI experience by adding functionality such as execute statements from a -file, output results to a file, display version information, and so on. - -## Eco system integrations - -### MapReduce - -Allow using result set from Drill queries as input to the Hadoop/MapReduce -jobs. - -### Hive views - -Query data from existing Hive views using Drill queries. Drill needs to parse -the HiveQL and translate them appropriately (into Drill's SQL or -logical/physical plans) to execute the requests. - -### YARN - -[https://issues.apache.org/jira/browse/_DRILL_-1170](https://issues.apache.org -/jira/browse/DRILL-1170) - -## Spark - -Provide ability to invoke Drill queries as part of Apache Spark programs. This -gives ability for Spark developers/users to leverage Drill richness of the -query layer , for data source access and as low latency execution engine. - -### Hue - -Hue is a GUI for users to interact with various Hadoop eco system components -(such as Hive, Oozie, Pig, HBase, Impala ...). The goal of this project is to -expose Drill as an application inside Hue so users can explore Drill metadata -and do SQL queries. - -### Phoenix - -Phoenix provides a low latency query layer on HBase for operational -applications. The goal of this effort is to explore opportunities for -integrating Phoenix with Drill. - http://git-wip-us.apache.org/repos/asf/drill/blob/85e05623/_docs/data-sources/001-hive-types.md ---------------------------------------------------------------------- diff --git a/_docs/data-sources/001-hive-types.md b/_docs/data-sources/001-hive-types.md index 8482133..cbbbf77 100644 --- a/_docs/data-sources/001-hive-types.md +++ b/_docs/data-sources/001-hive-types.md @@ -2,7 +2,10 @@ title: "Hive-to-Drill Data Type Mapping" parent: "Data Sources and File Formats" --- -Using Drill you can read tables created in Hive that use data types compatible with Drill. Drill currently does not support writing Hive tables. The following table shows Drill support for Hive primitive types: +Using Drill you can read tables created in Hive that use data types compatible with Drill. Drill currently does not support writing Hive tables. Drill supports the following Hive types for querying: + +<!-- See DRILL-1570 --> + <table> <tr> <th>Supported SQL Type</th> @@ -10,67 +13,67 @@ Using Drill you can read tables created in Hive that use data types compatible w <th>Description</th> </tr> <tr> - <td>bigint</td> + <td>BIGINT</td> <td>BIGINT</td> <td>8-byte signed integer</td> </tr> <tr> - <td>boolean</td> + <td>BOOLEAN</td> <td>BOOLEAN</td> <td>TRUE (1) or FALSE (0)</td> </tr> <tr> - <td>N/A</td> <td>CHAR</td> - <td>Same as Varchar but having a fixed-length max 255</td> + <td>CHAR</td> + <td>Character string, fixed-length max 255</td> </tr> <tr> - <td>date</td> + <td>DATE</td> <td>DATE</td> <td>Years months and days in the form in the form YYYY-ÂMM-ÂDD</td> </tr> <tr> - <td>decimal</td> + <td>DECIMAL</td> <td>DECIMAL</td> <td>38-digit precision</td> </tr> <tr> - <td>float</td> + <td>FLOAT</td> <td>FLOAT</td> <td>4-byte single precision floating point number</td> </tr> <tr> - <td>double</td> + <td>DOUBLE</td> <td>DOUBLE</td> <td>8-byte double precision floating point number</td> </tr> <tr> - <td>integer</td> + <td>INT or INTEGER</td> <td>INT</td> <td>4-byte signed integer</td> </tr> <tr> - <td>interval</td> + <td>INTERVAL</td> <td>N/A</td> <td>Integer fields representing a period of time depending on the type of interval</td> </tr> <tr> - <td>intervalday</td> + <td>INTERVALDAY</td> <td>N/A</td> <td>Integer fields representing a day</td> </tr> <tr> - <td>intervalyear</td> + <td>INTERVALYEAR</td> <td>N/A</td> <td>Integer fields representing a year</td> </tr> <tr> - <td>smallint</td> + <td>SMALLINT</td> <td>SMALLINT</td> <td>2-byte signed integer</td> </tr> <tr> - <td>time</td> + <td>TIME</td> <td>N/A</td> <td>Hours minutes seconds 24-hour basis</td> </tr> @@ -80,7 +83,7 @@ Using Drill you can read tables created in Hive that use data types compatible w <td>Conventional UNIX Epoch timestamp.</td> </tr> <tr> - <td>timestamp</td> + <td>TIMESTAMP</td> <td>TIMESTAMP</td> <td>JDBC timestamp in yyyy-mm-dd hh:mm:ss format</td> </tr> @@ -91,7 +94,7 @@ Using Drill you can read tables created in Hive that use data types compatible w </tr> <tr> - <td>varchar</td> + <td>VARCHAR</td> <td>VARCHAR</td> <td>Character string variable length</td> </tr> http://git-wip-us.apache.org/repos/asf/drill/blob/85e05623/_docs/data-sources/004-json-ref.md ---------------------------------------------------------------------- diff --git a/_docs/data-sources/004-json-ref.md b/_docs/data-sources/004-json-ref.md index b80231e..358fc61 100644 --- a/_docs/data-sources/004-json-ref.md +++ b/_docs/data-sources/004-json-ref.md @@ -459,8 +459,9 @@ Complex arrays and maps can be difficult or impossible to query. Workaround: Separate lengthy objects into objects delimited by curly braces using the following functions: -[flatten](/docs/json-data-model#flatten-json-data) separates a set of nested JSON objects into individual rows in a DRILL table. -[kvgen](/docs/json-data-model#generate-key-value-pairs) separates objects having more elements than optimal for querying. +* [flatten](/docs/json-data-model#flatten-json-data) separates a set of nested JSON objects into individual rows in a DRILL table. + +* [kvgen](/docs/json-data-model#generate-key-value-pairs) separates objects having more elements than optimal for querying. ### Nested Column Names http://git-wip-us.apache.org/repos/asf/drill/blob/85e05623/_docs/design/001-plan.md ---------------------------------------------------------------------- diff --git a/_docs/design/001-plan.md b/_docs/design/001-plan.md deleted file mode 100644 index 67e2290..0000000 --- a/_docs/design/001-plan.md +++ /dev/null @@ -1,25 +0,0 @@ ---- -title: "Drill Plan Syntax" -parent: "Design Docs" ---- -### Whats the plan? - -This section is about the end-to-end plan flow for Drill. The incoming query -to Drill can be a SQL 2003 query/DrQL or MongoQL. The query is converted to a -_Logical Plan_ that is a Drill's internal representation of the query -(language-agnostic). Drill then uses its optimization rules over the Logical -Plan to optimize it for best performance and crafts out a _Physical Plan_. The -Physical Plan is the actual plan the Drill then executes for the final data -processing. Below is a diagram to illustrate the flow: - - - -**The Logical Plan** describes the abstract data flow of a language independent query i.e. it would be a representation of the input query which would not be dependent on the actual input query language. It generally tries to work with primitive operations without focus on optimization. This makes it more verbose than traditional query languages. This is to allow a substantial level of flexibility in defining higher-level query language features. It would be forwarded to the optimizer to get a physical plan. - -**The Physical Plan** is often called the execution plan, since it is the input to the execution engine. Its a description of the physical operations the execution engine will undertake to get the desired result. It is the output of the query planner and is a transformation of the logical plan after applying the optimization rules. - -Typically, the physical and execution plans will be represented using the same -JSON format as the logical plan. - -**Detailed document**: Here is a document that explains the Drill logical & physical plans in full detail. [Drill detailed plan syntax document](https://docs.google.com/document/d/1QTL8warUYS2KjldQrGUse7zp8eA72VKtLOHwfXy6c7I/edit). - http://git-wip-us.apache.org/repos/asf/drill/blob/85e05623/_docs/design/002-rpc.md ---------------------------------------------------------------------- diff --git a/_docs/design/002-rpc.md b/_docs/design/002-rpc.md deleted file mode 100644 index 05cb1d6..0000000 --- a/_docs/design/002-rpc.md +++ /dev/null @@ -1,19 +0,0 @@ ---- -title: "RPC Overview" -parent: "Design Docs" ---- -Drill leverages the Netty 4 project as an RPC underlayment. From there, we -built a simple protobuf based communication layer optimized to minimize the -requirement for on heap data transformations. Both client and server utilize -the CompleteRpcMessage protobuf envelope to communicate requests, responses -and errors. The communication model is that each endpoint sends a stream of -CompleteRpcMessages to its peer. The CompleteRpcMessage is prefixed by a -protobuf encoded length. - -CompleteRpcMessage is broken into three key components: RpcHeader, Protobuf -Body (bytes), RawBody (bytes). - -RpcHeader has the following fields: - -Drillbits communicate through the BitCom intermediary. BitCom manages... - http://git-wip-us.apache.org/repos/asf/drill/blob/85e05623/_docs/design/003-query-stages.md ---------------------------------------------------------------------- diff --git a/_docs/design/003-query-stages.md b/_docs/design/003-query-stages.md deleted file mode 100644 index 5c54249..0000000 --- a/_docs/design/003-query-stages.md +++ /dev/null @@ -1,42 +0,0 @@ ---- -title: "Query Stages" -parent: "Design Docs" ---- -## Overview - -Apache Drill is a system for interactive analysis of large-scale datasets. It -was designed to allow users to query across multiple large big data systems -using traditional query technologies such as SQL. It is built as a flexible -framework to support a wide variety of data operations, query languages and -storage engines. - -## Query Parsing - -A Drillbit is capable of parsing a provided query into a logical plan. In -theory, Drill is capable of parsing a large range of query languages. At -launch, this will likely be restricted to an enhanced SQL2003 language. - -## Physical Planning - -Once a query is parsed into a logical plan, a Drillbit will then translate the -plan into a physical plan. The physical plan will then be optimized for -performance. Since plan optimization can be computationally intensive, a -distributed in-memory cache will provide LRU retrieval of previously generated -optimized plans to speed query execution. - -## Execution Planning - -Once a physical plan is generated, the physical plan is then rendered into a -set of detailed executional plan fragments (EPFs). This rendering is based on -available resources, cluster load, query priority and detailed information -about data distribution. In the case of large clusters, a subset of nodes will -be responsible for rendering the EPFs. Shared state will be managed through -the use of a distributed in-memory cache. - -## Execution Operation - -Query execution starts with each Drillbit being provided with one or more EPFs -associated with query execution. A portion of these EPFs may be identified as -initial EPFs and thus they are executed immediately. Other EPFs are executed -as data flows into them. -
