This is an automated email from the ASF dual-hosted git repository.
npr pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/arrow-site.git
The following commit(s) were added to refs/heads/master by this push:
new e55cc8e ARROW-9335: [Website] Update website for 1.0 (#63)
e55cc8e is described below
commit e55cc8e8eb24c7f9b0e697617d2a4f4ecec920ac
Author: Neal Richardson <[email protected]>
AuthorDate: Fri Jul 24 11:19:22 2020 -0700
ARROW-9335: [Website] Update website for 1.0 (#63)
Co-authored-by: Antoine Pitrou <[email protected]>
Co-authored-by: François Saint-Jacques <[email protected]>
Co-authored-by: Greg <[email protected]>
Co-authored-by: David Li <[email protected]>
Co-authored-by: Ryan Murray <[email protected]>
Co-authored-by: Sutou Kouhei <[email protected]>
Co-authored-by: Wes McKinney <[email protected]>
Co-authored-by: Andy Grove <[email protected]>
---
.github/workflows/deploy.yml | 14 +-
_config.yml | 1 +
_data/committers.yml | 209 +++++++++++++++
_includes/blog_header.html | 25 +-
_includes/footer.html | 12 +-
_includes/header.html | 84 +++----
blog.html => _includes/mailing_list_links.html | 31 +--
_includes/top.html | 2 +-
_layouts/article.html | 18 ++
_layouts/default.html | 4 +-
_layouts/home.html | 21 ++
_layouts/overview.html | 18 ++
_layouts/post.html | 16 +-
_posts/2020-07-24-1.0.0-release.md | 299 ++++++++++++++++++++++
_sass/_font-awesome.scss | 25 --
blog.html | 7 +-
committers.html | 335 -------------------------
committers.md | 77 ++++++
community.md | 91 +++++++
css/main.scss | 91 ++++++-
faq.md | 207 +++++++++++++--
index.html | 84 ++-----
install.md | 35 ++-
overview.md | 95 +++++++
powered_by.md | 20 +-
release-announcement-template.md | 87 +++++++
security.md | 39 +++
use_cases.md | 98 ++++++++
28 files changed, 1476 insertions(+), 569 deletions(-)
diff --git a/.github/workflows/deploy.yml b/.github/workflows/deploy.yml
index 47966d2..374846a 100644
--- a/.github/workflows/deploy.yml
+++ b/.github/workflows/deploy.yml
@@ -36,7 +36,7 @@ jobs:
- name: Configure for production
run: |
echo "BASE_URL=" >> ../env.sh
- echo "ORIGIN=$(jq --raw-output .repository.full_name
${GITHUB_EVENT_PATH})" >> ../env.sh
+ echo "ORIGIN=${{ github.repository }}" >> ../env.sh
echo "TARGET_BRANCH=asf-site" >> ../env.sh
echo >> _extra_config.yml
if: |
@@ -45,7 +45,7 @@ jobs:
- name: Configure for GitHub Pages on master
run: |
owner=$(jq --raw-output .repository.owner.login ${GITHUB_EVENT_PATH})
- repository=$(jq .repository.name ${GITHUB_EVENT_PATH})
+ repository=$(jq --raw-output .repository.name ${GITHUB_EVENT_PATH})
echo "BASE_URL=/${repository}" >> ../env.sh
echo "ORIGIN=${owner}/${repository}" >> ../env.sh
echo "TARGET_BRANCH=gh-pages" >> ../env.sh
@@ -85,6 +85,13 @@ jobs:
if ! git checkout --track deploy/${TARGET_BRANCH}; then
git checkout -b ${TARGET_BRANCH} remotes/origin/asf-site
fi
+ if [ "$ORIGIN" != "apache/arrow-site" ]; then
+ # Pull latest asf-site (for docs etc.) if we're not already on it
+ git remote add apache https://github.com/apache/arrow-site.git
+ git fetch apache
+ git reset --hard apache/asf-site
+ PUSH_ARGS="-f"
+ fi
rsync \
-a \
--delete \
@@ -96,11 +103,12 @@ jobs:
-a \
../build/docs/ \
docs/
+ touch .nojekyll
if [ "$(git status --porcelain)" != "" ]; then
# There are changes to the built site
git add --all
git commit -m "Updating built site (build ${GITHUB_SHA})"
- git push deploy ${TARGET_BRANCH}
+ git push ${PUSH_ARGS} deploy ${TARGET_BRANCH}
else
echo "No changes to the built site"
fi
diff --git a/_config.yml b/_config.yml
index 5f647b7..ba87dcb 100644
--- a/_config.yml
+++ b/_config.yml
@@ -34,6 +34,7 @@ exclude:
- README.md
- vendor
- themes
+ - release-announcement-template.md
collections:
docs:
diff --git a/_data/committers.yml b/_data/committers.yml
new file mode 100644
index 0000000..73b3362
--- /dev/null
+++ b/_data/committers.yml
@@ -0,0 +1,209 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to you under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+# Database of Apache Arrow committers and PMC
+#
+- name: Jacques Nadeau
+ role: VP
+ alias: jacques
+ affiliation: Dremio
+- name: Ted Dunning
+ role: PMC
+ alias: tdunning
+ affiliation: MapR
+- name: P. Taylor Goetz
+ role: PMC
+ alias: ptgoetz
+ affiliation: Monetate
+- name: Julian Hyde
+ role: PMC
+ alias: jhyde
+ affiliation: Looker
+- name: Reynold Xin
+ role: PMC
+ alias: rxin
+ affiliation: Databricks
+- name: James Taylor
+ role: PMC
+ alias: jamestaylor
+ affiliation: Salesforce
+- name: Julien Le Dem
+ role: PMC
+ alias: julien
+ affiliation: Datakin
+- name: Jake Luciani
+ role: PMC
+ alias: jake
+ affiliation: DataStax
+- name: Jason Altekruse
+ role: PMC
+ alias: json
+ affiliation: Workday
+- name: Alex Levenson
+ role: PMC
+ alias: alexlevenson
+ affiliation: Twitter
+- name: Parth Chandra
+ role: PMC
+ alias: parthc
+ affiliation: MapR
+- name: Marcel Kornacker
+ role: PMC
+ alias: marcel
+ affiliation: Independent
+- name: Steven Phillips
+ role: PMC
+ alias: smp
+ affiliation: Dremio
+- name: Hanifi Gunes
+ role: PMC
+ alias: hg
+ affiliation: MZ
+- name: Abdelhakim Deneche
+ role: PMC
+ alias: adeneche
+ affiliation: Salesforce
+- name: Wes McKinney
+ role: PMC
+ alias: wesm
+ affiliation: Ursa Labs / RStudio
+- name: David Alves
+ role: Committer
+ alias: dralves
+ affiliation: CortexXus
+- name: Ippokratis Pandis
+ role: Committer
+ alias: ippokratis
+ affiliation: Amazon
+- name: Uwe L. Korn
+ role: PMC
+ alias: uwe
+ affiliation: Quantco
+- name: Kouhei Sutou
+ role: PMC
+ alias: kou
+ affiliation: ClearCode
+- name: Philipp Moritz
+ role: PMC
+ alias: pcmoritz
+ affiliation: Anyscale
+- name: Phillip Cloud
+ role: PMC
+ alias: cpcloud
+ affiliation: Standard Cognition
+- name: Bryan Cutler
+ role: Committer
+ alias: cutlerb
+ affiliation: IBM
+- name: Li Jin
+ role: Committer
+ alias: icexelloss
+ affiliation: Two Sigma
+- name: Siddharth Teotia
+ role: PMC
+ alias: siddteotia
+ affiliation: LinkedIn
+- name: Brian Hulette
+ role: Committer
+ alias: bhulette
+ affiliation: Google
+- name: Robert Nishihara
+ role: Committer
+ alias: robertnishihara
+ affiliation: Anyscale
+- name: Paul Taylor
+ role: Committer
+ alias: ptaylor
+ affiliation: NVIDIA
+- name: Antoine Pitrou
+ role: PMC
+ alias: apitrou
+ affiliation: Independent / Ursa Labs
+- name: Andy Grove
+ role: PMC
+ alias: agrove
+ affiliation: NVIDIA
+- name: Krisztián Szűcs
+ role: PMC
+ alias: kszucs
+ affiliation: Ursa Labs
+- name: Yosuke Shiro
+ role: Committer
+ alias: shiro
+ affiliation: Red Data Tools
+- name: Romain Francois
+ role: Committer
+ alias: romainfrancois
+ affiliation: RStudio
+- name: Sebastien Binet
+ role: PMC
+ alias: sbinet
+ affiliation: CERN
+- name: Paddy Horan
+ role: Committer
+ alias: paddyhoran
+ affiliation: MetLife
+- name: Chao Sun
+ role: Committer
+ alias: sunchao
+ affiliation: Uber
+- name: Micah Kornfield
+ role: PMC
+ alias: emkornfield
+ affiliation: Google
+- name: Neville Dipale
+ role: Committer
+ alias: nevime
+ affiliation: commutenav
+- name: François Saint-Jacques
+ role: PMC
+ alias: fsaintjacques
+ affiliation: Ursa Labs / RStudio
+- name: Praveen Kumar
+ role: Committer
+ alias: praveenbingo
+ affiliation: Dremio
+- name: David Li
+ role: Committer
+ alias: lidavidm
+ affiliation: Two Sigma
+- name: Ben Kietzman
+ role: Committer
+ alias: bkietz
+ affiliation: Ursa Labs / RStudio
+- name: Kenta Murata
+ role: Committer
+ alias: mrkn
+ affiliation: Speee, Inc.
+- name: Neal Richardson
+ role: PMC
+ alias: npr
+ affiliation: Ursa Labs / RStudio
+- name: Eric Erhardt
+ role: Committer
+ alias: eerhardt
+ affiliation: Microsoft
+- name: Joris van den Bossche
+ role: Committer
+ alias: jorisvandenbossche
+ affiliation: Ursa Labs
+- name: Liya Fan
+ role: Committer
+ alias: liyafan
+ affiliation: Alibaba
+- name: Ji Liu
+ role: Committer
+ alias: tianchen
+ affiliation: Alibaba
diff --git a/_includes/blog_header.html b/_includes/blog_header.html
index 233862d..0e13a6d 100644
--- a/_includes/blog_header.html
+++ b/_includes/blog_header.html
@@ -15,8 +15,9 @@ See the License for the specific language governing
permissions and
limitations under the License.
{% endcomment %}
<h1>
- {{ post.title }}
+ {{ page.title }}
</h1>
+<hr class="mt-4 mb-3">
{% capture discard %}
{% comment %}
@@ -25,41 +26,41 @@ limitations under the License.
{% endcomment %}
{% assign author_name = nil %}
{% for c in site.data.contributors %}
- {% if c.apacheId == post.author %}
+ {% if c.apacheId == page.author %}
{% assign author_name = c.name %}
{% if c.homepage %}
{% assign homepage = c.homepage %}
{% else if c.githubId %}
- {% capture homepage %}http://github.com/{{ c.githubId }}{% endcapture
%}
+ {% capture homepage %}https://github.com/{{ c.githubId }}{% endcapture
%}
{% else %}
- {% capture homepage %}http://people.apache.org/~{{ post.author }}{%
endcapture %}
+ {% capture homepage %}https://people.apache.org/~{{ page.author }}{%
endcapture %}
{% endif %}
{% if c.avatar %}
{% assign avatar = c.avatar %}
{% else %}
- {% capture avatar %}http://github.com/{{ c.githubId }}.png{%
endcapture %}
+ {% capture avatar %}https://github.com/{{ c.githubId }}.png{%
endcapture %}
{% endif %}
{% endif %}
{% endfor %}
{% endcapture %}{% assign discard = nil %}
-<p>
+<p class="mb-4 pb-1">
<span class="badge badge-secondary">Published</span>
- <span class="published">
- {{ post.date | date_to_string }}
+ <span class="published mr-3">
+ {{ page.date | date_to_string }}
</span>
<br />
<span class="badge badge-secondary">By</span>
{% if author_name %}
- <a href="{{ homepage }}">{{ author_name }} ({{ post.author }}) </a>
+ <a class="mr-3" href="{{ homepage }}">{{ author_name }} ({{ page.author
}}) </a>
{% else %}
- {{ post.author }}
+ {{ page.author }}
{% endif %}
- {% if post.translations %}
+ {% if page.translations %}
<br />
<span class="badge badge-secondary">Translations</span>
- {% for trans in post.translations %}
+ {% for trans in page.translations %}
<a href="{{ site.baseurl }}/blog/{{ trans.url }}">{{ trans.language
}}</a>
{% endfor %}
{% endif %}
diff --git a/_includes/footer.html b/_includes/footer.html
index f57f409..064c39b 100644
--- a/_includes/footer.html
+++ b/_includes/footer.html
@@ -1,6 +1,14 @@
<hr/>
<footer class="footer">
- <p>Apache Arrow, Arrow, Apache, the Apache feather logo, and the Apache
Arrow project logo are either registered trademarks or trademarks of The Apache
Software Foundation in the United States and other countries.</p>
- <p>© 2016-{{ site.time | date: "%Y" }} The Apache Software
Foundation</p>
+ <div class="row">
+ <div class="col-md-9">
+ <p>Apache Arrow, Arrow, Apache, the Apache feather logo, and the Apache
Arrow project logo are either registered trademarks or trademarks of The Apache
Software Foundation in the United States and other countries.</p>
+ <p>© 2016-{{ site.time | date: "%Y" }} The Apache Software
Foundation</p>
+ </div>
+ <div class="col-md-3">
+ <a class="d-sm-none d-md-inline pr-2"
href="https://www.apache.org/events/current-event.html";>
+ <img src="https://www.apache.org/events/current-event-234x60.png"/>
+ </a>
+ </div>
{% asset main.js %}
</footer>
diff --git a/_includes/header.html b/_includes/header.html
index fc4df47..ef70a86 100644
--- a/_includes/header.html
+++ b/_includes/header.html
@@ -1,47 +1,29 @@
<nav class="navbar navbar-expand-md navbar-dark bg-dark">
- <a class="navbar-brand" href="{{ site.baseurl }}/"><img src="{{ site.baseurl
}}/img/arrow-inverse-300px.png" height="60px"/></a>
- <button class="navbar-toggler" type="button" data-toggle="collapse"
data-target="#arrow-navbar" aria-controls="arrow-navbar" aria-expanded="false"
aria-label="Toggle navigation">
+ {% if page.url != "/" %}
+ <a class="navbar-brand no-padding" href="{{ site.baseurl }}/"><img src="{{
site.baseurl }}/img/arrow-inverse-300px.png" height="40px"/></a>
+ {% endif %}
+ <button class="navbar-toggler ml-auto" type="button" data-toggle="collapse"
data-target="#arrow-navbar" aria-controls="arrow-navbar" aria-expanded="false"
aria-label="Toggle navigation">
<span class="navbar-toggler-icon"></span>
</button>
<!-- Collect the nav links, forms, and other content for toggling -->
- <div class="collapse navbar-collapse" id="arrow-navbar">
+ <div class="collapse navbar-collapse justify-content-end"
id="arrow-navbar">
<ul class="nav navbar-nav">
+ <li class="nav-item"><a class="nav-link" href="{{ site.baseurl
}}/overview/" role="button" aria-haspopup="true"
aria-expanded="false">Overview</a></li>
+ <li class="nav-item"><a class="nav-link" href="{{ site.baseurl
}}/faq/" role="button" aria-haspopup="true" aria-expanded="false">FAQ</a></li>
+ <li class="nav-item"><a class="nav-link" href="{{ site.baseurl
}}/blog" role="button" aria-haspopup="true" aria-expanded="false">Blog</a></li>
<li class="nav-item dropdown">
<a class="nav-link dropdown-toggle" href="#"
- id="navbarDropdownProjectLinks" role="button"
data-toggle="dropdown"
+ id="navbarDropdownGetArrow" role="button" data-toggle="dropdown"
aria-haspopup="true" aria-expanded="false">
- Project Links
+ Get Arrow
</a>
- <div class="dropdown-menu"
aria-labelledby="navbarDropdownProjectLinks">
- <a class="dropdown-item" href="{{ site.baseurl
}}/install/">Installation</a>
+ <div class="dropdown-menu" aria-labelledby="navbarDropdownGetArrow">
+ <a class="dropdown-item" href="{{ site.baseurl
}}/install/">Install</a>
<a class="dropdown-item" href="{{ site.baseurl
}}/release/">Releases</a>
- <a class="dropdown-item" href="{{ site.baseurl }}/faq/">FAQ</a>
- <a class="dropdown-item" href="{{ site.baseurl }}/blog/">Blog</a>
<a class="dropdown-item"
href="https://github.com/apache/arrow";>Source Code</a>
- <a class="dropdown-item"
href="https://issues.apache.org/jira/browse/ARROW";>Issue Tracker</a>
- </div>
- </li>
- <li class="nav-item dropdown">
- <a class="nav-link dropdown-toggle" href="#"
- id="navbarDropdownCommunity" role="button" data-toggle="dropdown"
- aria-haspopup="true" aria-expanded="false">
- Community
- </a>
- <div class="dropdown-menu" aria-labelledby="navbarDropdownCommunity">
- <a class="dropdown-item"
href="http://mail-archives.apache.org/mod_mbox/arrow-user/";>User Mailing
List</a>
- <a class="dropdown-item"
href="http://mail-archives.apache.org/mod_mbox/arrow-dev/";>Dev Mailing List</a>
- <a class="dropdown-item"
href="https://cwiki.apache.org/confluence/display/ARROW";>Developer Wiki</a>
- <a class="dropdown-item" href="{{ site.baseurl
}}/committers/">Committers</a>
- <a class="dropdown-item" href="{{ site.baseurl
}}/powered_by/">Powered By</a>
</div>
</li>
- <li class="nav-item">
- <a class="nav-link" href="{{ site.baseurl
}}/docs/format/Columnar.html"
- role="button" aria-haspopup="true" aria-expanded="false">
- Specification
- </a>
- </li>
<li class="nav-item dropdown">
<a class="nav-link dropdown-toggle" href="#"
id="navbarDropdownDocumentation" role="button"
data-toggle="dropdown"
@@ -50,22 +32,45 @@
</a>
<div class="dropdown-menu"
aria-labelledby="navbarDropdownDocumentation">
<a class="dropdown-item" href="{{ site.baseurl }}/docs">Project
Docs</a>
- <a class="dropdown-item" href="{{ site.baseurl
}}/docs/python">Python</a>
+ <a class="dropdown-item" href="{{ site.baseurl
}}/docs/format/Columnar.html">Format</a>
+ <hr/>
+ <a class="dropdown-item" href="{{ site.baseurl }}/docs/c_glib">C
GLib</a>
<a class="dropdown-item" href="{{ site.baseurl }}/docs/cpp">C++</a>
+ <a class="dropdown-item"
href="https://github.com/apache/arrow/blob/master/csharp/README.md";>C#</a>
+ <a class="dropdown-item"
href="https://godoc.org/github.com/apache/arrow/go/arrow";>Go</a>
<a class="dropdown-item" href="{{ site.baseurl
}}/docs/java">Java</a>
- <a class="dropdown-item" href="{{ site.baseurl }}/docs/c_glib">C
GLib</a>
<a class="dropdown-item" href="{{ site.baseurl
}}/docs/js">JavaScript</a>
+ <a class="dropdown-item"
href="https://github.com/apache/arrow/blob/master/matlab/README.md";>MATLAB</a>
+ <a class="dropdown-item" href="{{ site.baseurl
}}/docs/python">Python</a>
<a class="dropdown-item" href="{{ site.baseurl }}/docs/r">R</a>
+ <a class="dropdown-item"
href="https://github.com/apache/arrow/blob/master/ruby/README.md";>Ruby</a>
+ <a class="dropdown-item"
href="https://docs.rs/crate/arrow/";>Rust</a>
+ </div>
+ </li>
+ <li class="nav-item dropdown">
+ <a class="nav-link dropdown-toggle" href="#"
+ id="navbarDropdownCommunity" role="button" data-toggle="dropdown"
+ aria-haspopup="true" aria-expanded="false">
+ Community
+ </a>
+ <div class="dropdown-menu" aria-labelledby="navbarDropdownCommunity">
+ <a class="dropdown-item" href="{{ site.baseurl
}}/community/">Communication</a>
+ <a class="dropdown-item" href="{{ site.baseurl
}}/docs/developers/contributing.html">Contributing</a>
+ <a class="dropdown-item"
href="https://issues.apache.org/jira/browse/ARROW";>Issue Tracker</a>
+ <a class="dropdown-item" href="{{ site.baseurl
}}/committers/">Governance</a>
+ <a class="dropdown-item" href="{{ site.baseurl }}/use_cases/">Use
Cases</a>
+ <a class="dropdown-item" href="{{ site.baseurl
}}/powered_by/">Powered By</a>
+ <a class="dropdown-item" href="{{ site.baseurl
}}/security/">Security</a>
+ <a class="dropdown-item"
href="https://www.apache.org/foundation/policies/conduct.html";>Code of
Conduct</a>
</div>
</li>
- <!-- <li><a href="{{ site.baseurl }}/blog">Blog</a></li> -->
<li class="nav-item dropdown">
<a class="nav-link dropdown-toggle" href="#"
id="navbarDropdownASF" role="button" data-toggle="dropdown"
aria-haspopup="true" aria-expanded="false">
ASF Links
</a>
- <div class="dropdown-menu" aria-labelledby="navbarDropdownASF">
+ <div class="dropdown-menu dropdown-menu-right"
aria-labelledby="navbarDropdownASF">
<a class="dropdown-item" href="http://www.apache.org/";>ASF
Website</a>
<a class="dropdown-item"
href="http://www.apache.org/licenses/";>License</a>
<a class="dropdown-item"
href="http://www.apache.org/foundation/sponsorship.html";>Donate</a>
@@ -74,14 +79,5 @@
</div>
</li>
</ul>
- <div class="flex-row justify-content-end ml-md-auto">
- <a class="d-sm-none d-md-inline pr-2"
href="https://www.apache.org/events/current-event.html";>
- <img src="https://www.apache.org/events/current-event-234x60.png"/>
- </a>
- <a href="http://www.apache.org/";>
- <img src="{{ site.baseurl }}/img/asf_logo.svg" width="120px"/>
- </a>
- </div>
- </div><!-- /.navbar-collapse -->
- </div>
+ </div><!-- /.navbar-collapse -->
</nav>
diff --git a/blog.html b/_includes/mailing_list_links.html
similarity index 50%
copy from blog.html
copy to _includes/mailing_list_links.html
index 8cb4500..7393053 100644
--- a/blog.html
+++ b/_includes/mailing_list_links.html
@@ -1,9 +1,3 @@
----
-layout: default
-title: Project News and Blog
-description: Updates and announcements about the Apache Arrow project
-permalink: /blog/
----
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
@@ -21,27 +15,4 @@ See the License for the specific language governing
permissions and
limitations under the License.
{% endcomment %}
-<h2>Project News and Blog</h2>
-<hr>
-
-{% for post in site.posts %}
- {% unless post.categories contains 'translation' %}
- <p>
- <h3>
- <a href="{{ site.baseurl }}{{ post.url }}">{{ post.title }}</a>
- </h3>
- {% if post.translations %}
- <em>Translations:</em>
- {% for trans in post.translations %}
- <a href="{{ site.baseurl }}/blog/{{ trans.url }}">{{ trans.language
}}</a>
- {% endfor %}
- {% endif %}
- <p>
- <span class="blog-list-date">
- {{ post.date | date: "%-d %B %Y" }}
- </span>
- </p>
- {{ post.content | strip_html | truncatewords: 50 }}
- </p>
- {% endunless %}
-{% endfor %}
+(<a href="mailto:{{ include.list
}}[email protected]?subject=Subscribe">subscribe</a>, <a
href="mailto:{{ include.list
}}[email protected]?subject=Unubscribe">unsubscribe</a>, <a
href="https://lists.apache.org/list.html?{{ include.list
}}@arrow.apache.org">archives</a>)
diff --git a/_includes/top.html b/_includes/top.html
index 6804f99..ffc75d5 100644
--- a/_includes/top.html
+++ b/_includes/top.html
@@ -10,7 +10,7 @@
{% else %}
<title>{{ site.title }}</title>
{% endif %}
-
+
{% seo title=false %}
<!-- favicons -->
diff --git a/_layouts/article.html b/_layouts/article.html
new file mode 100644
index 0000000..3f17a6c
--- /dev/null
+++ b/_layouts/article.html
@@ -0,0 +1,18 @@
+{% include top.html %}
+
+<body class="wrap">
+ <header>
+ {% include header.html %}
+ </header>
+
+ <div class="container p-4 pt-5">
+ <div class="col-lg-8 mx-auto">
+ <main role="main" class="pb-5">
+ {{ content }}
+ </main>
+ </div>
+
+ {% include footer.html %}
+ </div>
+</body>
+</html>
diff --git a/_layouts/default.html b/_layouts/default.html
index 21c000c..669884c 100644
--- a/_layouts/default.html
+++ b/_layouts/default.html
@@ -5,8 +5,8 @@
{% include header.html %}
</header>
- <div class="container p-lg-4">
- <main role="main">
+ <div class="container p-4 pt-5">
+ <main role="main" class="pb-5">
{{ content }}
</main>
diff --git a/_layouts/home.html b/_layouts/home.html
new file mode 100644
index 0000000..c58651f
--- /dev/null
+++ b/_layouts/home.html
@@ -0,0 +1,21 @@
+{% include top.html %}
+
+<body class="wrap">
+ <header>
+ {% include header.html %}
+ </header>
+ <div class="big-arrow-bg">
+ <div class="container px-4 py-5 centered">
+ <img src="{{ site.baseurl }}/img/arrow-inverse.png" style="max-width:
70%;"/>
+ <p class="lead">A cross-language development platform for in-memory
analytics</p>
+ </div>
+ </div>
+ <div class="container p-4 pt-5">
+ <main role="main" class="pb-5">
+ {{ content }}
+ </main>
+
+ {% include footer.html %}
+ </div>
+</body>
+</html>
diff --git a/_layouts/overview.html b/_layouts/overview.html
new file mode 100644
index 0000000..aa36422
--- /dev/null
+++ b/_layouts/overview.html
@@ -0,0 +1,18 @@
+{% include top.html %}
+
+<body class="wrap">
+ <header>
+ {% include header.html %}
+ </header>
+
+ <div class="container p-4 pt-5">
+ <div class="col-lg-10 mx-auto">
+ <main role="main" class="pb-5">
+ {{ content }}
+ </main>
+ </div>
+
+ {% include footer.html %}
+ </div>
+</body>
+</html>
diff --git a/_layouts/post.html b/_layouts/post.html
index 1e9db93..336cd66 100644
--- a/_layouts/post.html
+++ b/_layouts/post.html
@@ -5,16 +5,14 @@
{% include header.html %}
</header>
- <div class="container p-lg-4">
- <main role="main">
- {% comment %}
- This 'assign' is to match the variable name used in
_includes/blog_header.html
- {% endcomment %}
- {% assign post = page %}
- {% include blog_header.html %}
+ <div class="container p-4 pt-5">
+ <div class="col-md-8 mx-auto">
+ <main role="main" class="pb-5">
+ {% include blog_header.html %}
- {{ content }}
- </main>
+ {{ content }}
+ </main>
+ </div>
{% include footer.html %}
</div>
diff --git a/_posts/2020-07-24-1.0.0-release.md
b/_posts/2020-07-24-1.0.0-release.md
new file mode 100644
index 0000000..33e5fcd
--- /dev/null
+++ b/_posts/2020-07-24-1.0.0-release.md
@@ -0,0 +1,299 @@
+---
+layout: post
+title: "Apache Arrow 1.0.0 Release"
+date: "2020-07-24 00:00:00 -0600"
+author: pmc
+categories: [release]
+---
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+The Apache Arrow team is pleased to announce the 1.0.0 release. This covers
+over 3 months of development work and includes [**810 resolved issues**][1]
from
+[**100 distinct contributors**][2]. See the Install Page to learn how to get
the
+libraries for your platform.
+
+Despite a "1.0.0" version, this is the 18th major release of Apache Arrow and
+marks a transition to binary stability of the columnar format (which was
+already informally backward-compatible going back to December 2017) and a
+transition to Semantic Versioning for the Arrow software libraries.
+
+The release notes below are not exhaustive and only expose selected highlights
+of the release. Many other bugfixes and improvements have been made: we refer
+you to the [complete changelog][3].
+
+## 1.0.0 Columnar Format and Stability Guarantees
+
+The 1.0.0 release indicates that the Arrow columnar format is declared stable,
+with [forward and backward compatibility guarantees][5].
+
+The Arrow columnar format received several recent changes and additions,
+leading to the 1.0.0 format version:
+
+* The metadata version was bumped to a new version V5, indicating an
+ incompatible change in the buffer layout of Union types. All
+ other types keep the same layout as in V4. V5 also includes format additions
+ to assist with forward compatibility (detecting unsupported changes sent by
+ future library versions). Libraries remain backward compatible with data
+ generated by all libraries back to 0.8.0 (December 2017) and the Java and C++
+ libraries are capable of generating V4-compatible messages (for sending data
+ to applications using 0.8.0 to 0.17.1).
+
+* Dictionary indices are now allowed to be unsigned integers rather than only
+ signed integers. Using UInt64 is still discouraged because of
+ poor Java support.
+
+* A "Feature" enum has been added to announce the use of specific optional
+ features in an IPC stream, such as buffer compression. This
+ new field is not used by any implementation yet.
+
+* Optional buffer compression using LZ4 or ZStandard was added to the IPC
+ format.
+
+* Decimal types now have an optional "bitWidth" field, defaulting to 128.
+ This will allow for future support of other decimal widths
+ such as 32- and 64-bit.
+
+* The validity bitmap buffer has been removed from Union types. The nullity of
+ a slot in a Union array is determined exclusively by the constituent arrays
+ forming the union.
+
+Integration testing has been expanded to test for extension types and
+nested dictionaries. See the [implementation matrix][8] for details.
+
+## Community
+
+Since the last release, we have added two new committers:
+
+* Liya Fan
+* Ji Liu
+
+Thank you for all your contributions!
+
+## Arrow Flight RPC notes
+
+Flight now offers DoExchange, a fully bidirectional data endpoint, in addition
+to DoGet and DoPut, in C++, Java, and Python. Middlewares in all languages now
+expose binary-valued headers. Additionally, servers and clients can set Arrow
+IPC read/write options in all languages, making compatibility easier with
earlier
+versions of Arrow Flight.
+
+In C++ and Python, Flight now exposes more options from gRPC, including the
+address of the client (on the server) and the ability to set low-level gRPC
+client options. Flight also supports mutual TLS authentication and the ability
+for a client to control the size of a data message on the wire.
+
+## C++ notes
+
+* Support for static linking with Arrow has been vastly improved, including the
+ introduction of a `libarrow_bundled_dependencies.a` library bundling all
+ external dependencies that are built from source by Arrow's build system
+ rather than installed by an external package manager. This makes
+ it significantly easier to create dependency-free applications with all
+ libraries statically-linked.
+* Following the Arrow format changes, Union arrays cannot have a top-level
+ bitmap anymore.
+* A number of improvements were made to reduce the overall generated binary
+ size in the Arrow library.
+* A convenience API `GetBuildInfo` allows querying the characteristics of the
+ Arrow library. We encourage you to suggest any desired addition to the
+ returned information.
+* We added an optional dependency to the `utf8proc` library, used in several
+ compute functions (see below).
+* Instead of sharing the same concrete classes, sparse and dense unions now
+ have separated classes (`SparseUnionType` and `DenseUnionType`, as well as
+ `SparseUnionArray`, `DenseUnionArray`, `SparseUnionScalar`,
+ `DenseUnionScalar`).
+* Arrow can now be built for iOS using the right set of CMake options, though
+ we don't officially support it. See [this writeup][7] for details.
+
+### Compute functions
+
+The compute kernel layer was extensively reworked. It now offers
+a generic function lookup, dispatch and execution mechanism. Furthermore, new
+internal scaffoldings make it vastly easier to write new function kernels, with
+many common details like type checking and function dispatch based on type
+combinations handled by the framework rather than implemented manually by the
+function developer.
+
+Around 30 new array compute functions have been added. For example,
+Unicode-compliant predicates and transforms, such as lowercase and uppercase
+transforms, are now available.
+
+The available compute functions are listed exhaustively in the Sphinx-generated
+[documentation][10].
+
+### Datasets
+
+Datasets can now be read from CSV files.
+
+Datasets can be expanded to their component fragments, enabling fine grained
+interoperability with other consumers of data files. Where applicable, metadata
+is available as a property of the fragment, including partition information and
+(for the parquet format) per-column statistics.
+
+Datasets of parquet files can now be assembled from a single `_metadata` file,
+such as those created by systems like Dask and Spark. `_metadata`
+contains the metadata of all fragments, allowing construction of a statistics-
+aware dataset with a single IO call.
+
+### Feather
+
+The Feather format is now available in version 2, which is simply the Arrow
+IPC file format with another name.
+
+### IPC
+
+By default, we now write IPC streams with metadata V5. However, metadata V4
+can be requested by setting the appropriate member in `IpcWriteOptions`. V4 as
+well as V5 metadata IPC streams can be read properly, with one exception: a V4
+metadata stream containing Union arrays with top-level null values will refuse
+reading.
+
+As noted above, there are no changes between V4 and V5 that break
+backwards compatibility. For forward compatibility scenarios (where you need to
+generate data to be read by an older Arrow library), you can set the V4
+compatibility mode.
+
+Support for dictionary replacement and dictionary delta was implemented.
+
+### Parquet
+
+Writing files with the LZ4 codec is disabled because it produces files
+incompatible with the widely-used Hadoop Parquet implementation. Support will
+be reenabled once we align the LZ4 implementation with the special buffer
+encoding expected by Hadoop.
+
+## Java notes
+
+The Java package introduces a number of low level changes in this release.
+Most notable are the work in support of allocating large arrow buffers and
+removing Netty from the public API. Users will have to update their
+dependencies to use one of the two supported allocators Netty:
+`arrow-memory-netty` or Unsafe (internal java api for direct memory)
+`arrow-memory-unsafe`.
+
+The Java Vector implementation has improved its interoperability having
+verified `LargeVarChar`, `LargeBinary`, `LargeList`, `Union`, Extension types
+and duplicate field names in `Structs` are binary compatible with C++ and the
+specification.
+
+## Python notes
+
+The size of wheel packages is significantly reduced, up to 75%. One side
+effect is that these wheels do not enable Gandiva anymore (which requires the
+LLVM runtime to be statically-linked). We are interested in providing Gandiva
+as an add-on package as a separate Python wheel in the future.
+
+The Scalar class hierarchy was reworked to more closely follow its C++
+counterpart.
+
+TLS CA certificates are looked up more reliably when using the S3 filesystem,
+especially with manylinux wheels.
+
+The encoding of CSV files can now be specified explicitly, defaulting to UTF8.
+Custom timestamp parsers can now be used for CSV files.
+
+Filesystems can now be implemented in pure Python. As a result,
+[fsspec][9]-based filesystems can now
+be used in datasets.
+
+`parquet.read_table` is now backed by the dataset API by default, enabling
+filters on any column and more flexible partitioning.
+
+## R notes
+
+The R package added support for converting to and from many additional Arrow
+types. Tables showing how R types are mapped to Arrow types and vice versa have
+been added to the [introductory vignette][6], and nearly all types are handled.
+In addition, R `attributes` like custom classes and metadata are now preserved
+when converting a `data.frame` to an Arrow Table and are restored when loading
+them back into R.
+
+For more on what’s in the 1.0.0 R package, see the [R changelog][4].
+
+## Ruby and C GLib notes
+
+The Ruby and C GLib packages added support for the new compute function
+framework, in which users can find a
+compute function dynamically and call it. Users don't need to wait for a C
+GLib binding for new compute functions: if the C++ package provides a
+new compute function, users can use it
+without additional code in the Ruby and C GLib packages.
+
+The Ruby and C GLib packages added support for Apache Arrow
+Dataset. The Ruby package provides a new gem for Apache Arrow Dataset,
+`red-arrow-dataset`. The C GLib package provides a new module for
+Apache Arrow Dataset, `arrow-dataset-glib`. They just have a few
+features for now but we will add more in future releases.
+
+The Ruby and C GLib packages added support for reading only the
+specified row group in an Apache Parquet file.
+
+### Ruby
+
+The Ruby package added support for column level compression in writing
+Apache Parquet files.
+
+The Ruby package changed the `Arrow::DictionaryArray#[]` behavior. It now
+returns the dictionary value instead of the dictionary index. This is a
+backwards-incompatible change.
+
+## Rust notes
+
+* A new integration test crate has been added, allowing the Rust
+ implementation to participate in integration testing.
+* A new benchmark crate has been added for benchmarking performance
+ against popular data sets. The initial examples run SQL queries against
+ the NYC Taxi data set using DataFusion. This is useful for comparing
+ performance against other Arrow implementations.
+* Rust toolchain has been upgraded to 1.44 nightly.
+
+### Arrow Core
+
+* Support for binary, string, and list arrays with i64 offsets to support
+ large lists.
+* A new sort kernel has been added.
+* There have been various improvements to dictionary array support.
+* CSV reader enhancements include a new CsvReadOptions struct and support
+ for schema inference from multiple CSV files.
+* There are significant (10x - 40x) performance improvements to SIMD
+ comparison kernels.
+
+### DataFusion
+
+* There are numerous UX improvements to LogicalPlan and LogicalPlanBuilder,
+ including support for named columns.
+* General improvements to code base, such as removing many uses of `Arc`
+ and using slices instead of `&Vec` as function arguments.
+* ParquetScanExec performance improvement (almost 2x).
+* ExecutionContext can now be shared between threads.
+* Rust closures can now be used as Scalar UDFs.
+* Sort support has been added to SQL and LogicalPlan.
+
+[1]:
https://issues.apache.org/jira/issues/?jql=project%20%3D%20ARROW%20AND%20status%20%3D%20Resolved%20AND%20fixVersion%20%3D%201.0.0
+[2]: {{ site.baseurl }}/release/1.0.0.html#contributors
+[3]: {{ site.baseurl }}/release/1.0.0.html
+[4]: {{ site.baseurl }}/docs/r/news/
+[5]: {{ site.baseurl }}/docs/format/Versioning.html
+[6]: {{ site.baseurl }}/docs/r/articles/arrow.html
+[7]:
https://github.com/UnfoldedInc/deck.gl-native-dependencies/blob/master/docs/iOS-BUILD.md#arrow-v0170
+[8]: {{ site.baseurl }}/docs/status.html
+[9]: https://filesystem-spec.readthedocs.io
+[10]: {{ site.baseurl }}/docs/cpp/compute.html
diff --git a/_sass/_font-awesome.scss b/_sass/_font-awesome.scss
deleted file mode 100644
index d90676c..0000000
--- a/_sass/_font-awesome.scss
+++ /dev/null
@@ -1,25 +0,0 @@
-/*!
- * Font Awesome 4.2.0 by @davegandy - http://fontawesome.io - @fontawesome
- * License - http://fontawesome.io/license (Font: SIL OFL 1.1, CSS: MIT
License)
- */
-@font-face {
- font-family: 'FontAwesome';
- src: url('../fonts/fontawesome-webfont.eot?v=4.2.0');
- src: url('../fonts/fontawesome-webfont.eot?#iefix&v=4.2.0')
format('embedded-opentype'), url('../fonts/fontawesome-webfont.woff?v=4.2.0')
format('woff'), url('../fonts/fontawesome-webfont.ttf?v=4.2.0')
format('truetype'),
url('../fonts/fontawesome-webfont.svg?v=4.2.0#fontawesomeregular')
format('svg');
- font-weight: normal;
- font-style: normal;
-}
-.fa {
- display: inline-block;
- font: normal normal normal 14px/1 FontAwesome;
- font-size: inherit;
- text-rendering: auto;
- -webkit-font-smoothing: antialiased;
- -moz-osx-font-smoothing: grayscale;
-}
-.fa-link:before {
- content: "\f0c1";
-}
-.fa-pencil:before {
- content: "\f040";
-}
diff --git a/blog.html b/blog.html
index 8cb4500..7591c67 100644
--- a/blog.html
+++ b/blog.html
@@ -1,5 +1,5 @@
---
-layout: default
+layout: article
title: Project News and Blog
description: Updates and announcements about the Apache Arrow project
permalink: /blog/
@@ -21,8 +21,9 @@ See the License for the specific language governing
permissions and
limitations under the License.
{% endcomment %}
-<h2>Project News and Blog</h2>
-<hr>
+<h1>Project News and Blog</h1>
+
+<hr class="mb-5">
{% for post in site.posts %}
{% unless post.categories contains 'translation' %}
diff --git a/committers.html b/committers.html
deleted file mode 100644
index 76b79fe..0000000
--- a/committers.html
+++ /dev/null
@@ -1,335 +0,0 @@
----
-layout: default
-title: Committers
-description: "List of project-management committee (PMC) members and
committers on the Apache Arrow project."
----
- <div class="container">
-<h2>Committers</h2>
-<table class="table"><thead>
-<tr>
-<th>Name</th>
-<th>Role</th>
-<th>Alias (email is <alias>@apache.org)</th>
-<th>Affiliation</th>
-</tr>
-</thead><tbody>
-<tr>
-<td>Jacques Nadeau</td>
-<td>VP, PMC</td>
-<td>jacques</td>
-<td>Dremio</td>
-</tr>
-<tr>
-<td>Ted Dunning</td>
-<td>PMC</td>
-<td>tdunning</td>
-<td>MapR</td>
-</tr>
-<tr>
-<td>P. Taylor Goetz</td>
-<td>PMC</td>
-<td>ptgoetz</td>
-<td>Monetate</td>
-</tr>
-<tr>
-<td>Julian Hyde</td>
-<td>PMC</td>
-<td>jhyde</td>
-<td>Looker</td>
-</tr>
-<tr>
-<td>Reynold Xin</td>
-<td>PMC</td>
-<td>rxin</td>
-<td>Databricks</td>
-</tr>
-<tr>
-<td>James Taylor</td>
-<td>PMC</td>
-<td>jamestaylor</td>
-<td>Salesforce</td>
-</tr>
-<tr>
-<td>Julien Le Dem</td>
-<td>PMC</td>
-<td>julien</td>
-<td>WeWork</td>
-</tr>
-<tr>
-<td>Jake Luciani</td>
-<td>PMC</td>
-<td>jake</td>
-<td>DataStax</td>
-</tr>
-<tr>
-<td>Jason Altekruse</td>
-<td>PMC</td>
-<td>json</td>
-<td>Workday</td>
-</tr>
-<tr>
-<td>Alex Levenson</td>
-<td>PMC</td>
-<td>alexlevenson</td>
-<td>Twitter</td>
-</tr>
-<tr>
-<td>Parth Chandra</td>
-<td>PMC</td>
-<td>parthc</td>
-<td>MapR</td>
-</tr>
-<tr>
-<td>Marcel Kornacker</td>
-<td>PMC</td>
-<td>marcel</td>
-<td>Independent</td>
-</tr>
-<tr>
-<td>Steven Phillips</td>
-<td>PMC</td>
-<td>smp</td>
-<td>Dremio</td>
-</tr>
-<tr>
-<td>Hanifi Gunes</td>
-<td>PMC</td>
-<td>hg</td>
-<td>MZ</td>
-</tr>
-<tr>
-<td>Abdelhakim Deneche</td>
-<td>PMC</td>
-<td>adeneche</td>
-<td>Dremio</td>
-</tr>
-<tr>
-<td>Wes McKinney</td>
-<td>PMC</td>
-<td>wesm</td>
-<td>Ursa Labs / RStudio</td>
-</tr>
-<tr>
-<td>David Alves</td>
-<td>Committer</td>
-<td>dralves</td>
-<td>Cloudera</td>
-</tr>
-<tr>
-<td>Ippokratis Pandis</td>
-<td>Committer</td>
-<td>ippokratis</td>
-<td>Amazon</td>
-</tr>
-<tr>
-<td>Uwe L. Korn</td>
-<td>PMC</td>
-<td>uwe</td>
-<td>Blue Yonder GmbH</td>
-</tr>
-<tr>
-<td>Kouhei Sutou</td>
-<td>PMC</td>
-<td>kou</td>
-<td>ClearCode</td>
-</tr>
-<tr>
-<td>Philipp Moritz</td>
-<td>PMC</td>
-<td>pcmoritz</td>
-<td>UC Berkeley RISELab</td>
-</tr>
-<tr>
-<td>Phillip Cloud</td>
-<td>PMC</td>
-<td>cpcloud</td>
-<td>Two Sigma</td>
-</tr>
-<tr>
-<td>Bryan Cutler</td>
-<td>Committer</td>
-<td>cutlerb</td>
-<td>IBM</td>
-</tr>
-<tr>
-<td>Li Jin</td>
-<td>Committer</td>
-<td>icexelloss</td>
-<td>Two Sigma</td>
-</tr>
-<tr>
-<td>Siddharth Teotia</td>
-<td>PMC</td>
-<td>siddteotia</td>
-<td>Dremio</td>
-</tr>
-<tr>
-<td>Brian Hulette</td>
-<td>Committer</td>
-<td>bhulette</td>
-<td>Google</td>
-</tr>
-<tr>
-<td>Robert Nishihara</td>
-<td>Committer</td>
-<td>robertnishihara</td>
-<td>UC Berkeley RISELab</td>
-</tr>
-<tr>
-<td>Paul Taylor</td>
-<td>Committer</td>
-<td>ptaylor</td>
-<td>Graphistry</td>
-</tr>
-<tr>
-<td>Antoine Pitrou</td>
-<td>PMC</td>
-<td>apitrou</td>
-<td>Independent / Ursa Labs</td>
-</tr>
-<tr>
-<td>Andy Grove</td>
-<td>PMC</td>
-<td>agrove</td>
-<td>NVIDIA</td>
-</tr>
-<tr>
-<td>Krisztián Szűcs</td>
-<td>PMC</td>
-<td>kszucs</td>
-<td>Ursa Labs</td>
-</tr>
-<tr>
-<td>Yosuke Shiro</td>
-<td>Committer</td>
-<td>shiro</td>
-<td>Red Data Tools</td>
-</tr>
-<tr>
-<td>Romain Francois</td>
-<td>Committer</td>
-<td>romainfrancois</td>
-<td>RStudio</td>
-</tr>
-<tr>
-<td>Sebastien Binet</td>
-<td>PMC</td>
-<td>sbinet</td>
-<td>CERN</td>
-</tr>
-<tr>
-<td>Paddy Horan</td>
-<td>Committer</td>
-<td>paddyhoran</td>
-<td>MetLife</td>
-</tr>
-<tr>
-<td>Chao Sun</td>
-<td>Committer</td>
-<td>sunchao</td>
-<td>Uber</td>
-</tr>
-<tr>
-<td>Micah Kornfield</td>
-<td>PMC</td>
-<td>emkornfield</td>
-<td>Google</td>
-</tr>
-<tr>
-<td>Neville Dipale</td>
-<td>Committer</td>
-<td>nevime</td>
-<td>commutenav</td>
-</tr>
-<tr>
-<td>François Saint-Jacques</td>
-<td>PMC</td>
-<td>fsaintjacques</td>
-<td>Ursa Labs / RStudio</td>
-</tr>
-<tr>
-<td>Praveen Kumar</td>
-<td>Committer</td>
-<td>praveenbingo</td>
-<td>Dremio</td>
-</tr>
-<tr>
-<td>David Li</td>
-<td>Committer</td>
-<td>lidavidm</td>
-<td>Two Sigma</td>
-</tr>
-<tr>
-<td>Ben Kietzman</td>
-<td>Committer</td>
-<td>bkietz</td>
-<td>Ursa Labs / RStudio</td>
-</tr>
-<tr>
-<td>Kenta Murata</td>
-<td>Committer</td>
-<td>mrkn</td>
-<td>Speee, Inc.</td>
-</tr>
-<tr>
-<td>Neal Richardson</td>
-<td>PMC</td>
-<td>npr</td>
-<td>Ursa Labs / RStudio</td>
-</tr>
-<tr>
-<td>Eric Erhardt</td>
-<td>Committer</td>
-<td>eerhardt</td>
-<td>Microsoft</td>
-</tr>
-<tr>
-<td>Joris van den Bossche</td>
-<td>Committer</td>
-<td>jorisvandenbossche</td>
-<td>Ursa Labs</td>
-</tr>
-<tr>
- <td>Liya Fan</td>
- <td>Committer</td>
- <td>liyafan</td>
- <td>Alibaba</td>
-</tr>
-<tr>
- <td>Ji Liu</td>
- <td>Committer</td>
- <td>tianchen</td>
- <td>Alibaba</td>
-</tr>
-</tbody></table>
-
- </div> <!-- /container -->
-
- <div class="container">
-<h2>Emeritus PMC Members</h2>
-<table class="table"><thead>
-<tr>
-<th>Name</th>
-<th>Role</th>
-<th>Alias (email is <alias>@apache.org)</th>
-<th>Affiliation</th>
-</tr>
-</thead><tbody>
-<tr>
-<td>Todd Lipcon</td>
-<td>PMC</td>
-<td>todd</td>
-<td>Cloudera</td>
-</tr>
-<tr>
-<td>Michael Stack</td>
-<td>PMC</td>
-<td>stack</td>
-<td>Cloudera</td>
-</tr>
-</tbody></table>
-
- </div> <!-- /container -->
-
- </body>
-</html>
diff --git a/committers.md b/committers.md
new file mode 100644
index 0000000..0eb4212
--- /dev/null
+++ b/committers.md
@@ -0,0 +1,77 @@
+---
+layout: article
+title: Committers
+description: "List of project-management committee (PMC) members and
committers on the Apache Arrow project."
+---
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+## Apache Arrow Project Governance
+
+The Arrow project is a part of the Apache Software Foundation and follows its
[project management
guidelines](https://www.apache.org/foundation/how-it-works.html#management),
which promote community-led consensus decisionmaking,
[independent](https://community.apache.org/projectIndependence.html) of
commercial influence.
+
+### Project Management Committee (PMC)
+
+The [PMC](https://www.apache.org/foundation/how-it-works.html#pmc-members)
governs the project. Members
[vote](https://www.apache.org/foundation/voting.html) on important decisions,
including releases and inviting committers to join the PMC.
+
+<table class="table table-striped"><thead>
+<tr>
+<th>Name</th>
+<th>Affiliation</th>
+</tr>
+</thead><tbody>
+ {% assign sorted_committers = site.data.committers | sort: "name" %}
+ {% for person in sorted_committers %}
+ {% if person.role == "VP" %}
+ <tr>
+ <td><a href="https://people.apache.org/phonebook.html?uid={{ person.alias
}}">{{ person.name }}</a> (Chair)</td>
+ <td>{{ person.affiliation }}</td>
+ </tr>
+ {% endif %}
+ {% endfor %}
+ {% for person in sorted_committers %}
+ {% if person.role == "PMC" %}
+ <tr>
+ <td><a href="https://people.apache.org/phonebook.html?uid={{ person.alias
}}">{{ person.name }}</a></td>
+ <td>{{ person.affiliation }}</td>
+ </tr>
+ {% endif %}
+ {% endfor %}
+</tbody></table>
+
+### Committers
+
+Contributors who have demonstrated a sustained commitment to the project, not
only authoring code but also reviewing others' patches and exercising good
judgment, may be invited by the PMC to become
[committers](https://www.apache.org/foundation/how-it-works.html#committers).
Committers are authorized to merge code patches to the project and serve as
non-voting project maintainers.
+
+<table class="table table-striped"><thead>
+<tr>
+<th>Name</th>
+<th>Affiliation</th>
+</tr>
+</thead><tbody>
+ {% assign sorted_committers = site.data.committers | sort: "name" %}
+ {% for person in sorted_committers %}
+ {% if person.role == "Committer" %}
+ <tr>
+ <td><a href="https://people.apache.org/phonebook.html?uid={{ person.alias
}}">{{ person.name }}</a></td>
+ <td>{{ person.affiliation }}</td>
+ </tr>
+ {% endif %}
+ {% endfor %}
+</tbody></table>
diff --git a/community.md b/community.md
new file mode 100644
index 0000000..75a951a
--- /dev/null
+++ b/community.md
@@ -0,0 +1,91 @@
+---
+layout: article
+title: Apache Arrow Community
+description: Links and resources for participating in Apache Arrow
+---
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+# Apache Arrow Community
+
+<hr class="mt-4 mb-3">
+
+We welcome participation from everyone and encourage you to join us, ask
+questions, and get involved.
+
+All participation in the Apache Arrow project is governed by the Apache
+Software Foundation's [code of
conduct](https://www.apache.org/foundation/policies/conduct.html).
+
+## Questions?
+
+### Mailing lists
+
+These arrow.apache.org mailing lists are for project discussion:
+
+<ul>
+ <li> <code>user@</code> is for questions on using Apache Arrow libraries {%
include mailing_list_links.html list="user" %} </li>
+ <li> <code>dev@</code> is for discussions about contributing to the project
development {% include mailing_list_links.html list="dev" %} </li>
+</ul>
+
+When emailing one of the lists, you may want to prefix the subject line with
+one or more tags, like `[C++] why did this segfault?`, `[Python] trouble with
+wheels`, etc., so that the appropriate people in the community notice the
+message.
+
+You may also wish to subscribe to these lists, which capture some activity
streams:
+
+<ul>
+ <li> <code>issues@</code> for the creation of JIRA issues {% include
mailing_list_links.html list="issues" %} </li>
+ <li> <code>commits@</code> for commits to the <a
href="https://github.com/apache/arrow";>apache/arrow</a> and <a
href="https://github.com/apache/arrow-site";>apache/arrow-site</a> repositories
(typically to <code>master</code> only) {% include mailing_list_links.html
list="commits" %} </li>
+</ul>
+
+In addition, we have some "firehose" lists, which exist so that development
+activity is captured in email form for archival purposes.
+
+<ul>
+ <li> <code>jira@</code> for all activity on JIRA issues {% include
mailing_list_links.html list="jira" %} </li>
+ <li> <code>github@</code> for all activity on the GitHub repositories {%
include mailing_list_links.html list="github" %} </li>
+</ul>
+
+### Stack Overflow
+
+For questions on how to use Arrow libraries, you may want to use the Stack
+Overflow tag
+[apache-arrow](https://stackoverflow.com/questions/tagged/apache-arrow) in
+addition to the programming language. Some languages and subprojects may have
+their own tags (for example,
+[pyarrow](https://stackoverflow.com/questions/tagged/pyarrow)).
+
+### GitHub issues
+
+We support GitHub issues as a lightweight way to ask questions and engage with
+the Arrow developer community. That said, we use
+[JIRA](https://issues.apache.org/jira/browse/ARROW) for maintaining a queue of
+development work and as the public record for work on the project, and we use
+the mailing lists for development discussions, so to keep things in a single
+place, we prefer not to have lengthy discussions on GitHub issues.
+
+If you know your question is actually a bug report or feature request, we
+encourage you to go straight to JIRA. If you're not sure, feel free to make a
+GitHub issue to ask your question and we can triage/redirect your question
+there.
+
+## Contributing
+
+As mentioned above, we use [JIRA](https://issues.apache.org/jira/browse/ARROW)
for our issue tracker and [GitHub](https://github.com/apache/arrow) for source
control. See the [contribution guidelines]({{ site.baseurl
}}/docs/developers/contributing.html) for more.
diff --git a/css/main.scss b/css/main.scss
index 14f202f..e844dfb 100644
--- a/css/main.scss
+++ b/css/main.scss
@@ -2,9 +2,98 @@
---
@import "bootstrap";
-@import "font-awesome";
.blog-list-date {
color: #888888;
font-style: italic;
}
+
+a {
+ color: #d74633;
+}
+
+.dropdown-item.active, .dropdown-item:active {
+ background-color: #d74633;
+}
+
+body {
+ font-family: Lato,Roboto,"Helvetica Neue",Arial,sans-serif;
+ font-size: 1.04em;
+ line-height: 1.52;
+}
+
+.footer {
+ font-size: .8em;
+}
+
+.centered {
+ text-align: center;
+}
+
+.big-arrow-bg {
+ background-color: #343a40;
+}
+
+.lead {
+ font-size: 1.25em;
+ font-weight: 400;
+ color: #FFFFFF;
+}
+
+.lead-dark {
+ font-size: 1.25em;
+ font-weight: 400;
+ color: #212529;
+}
+
+.no-padding {
+ padding: 0;
+}
+
+h1, .h1 {
+ font-size: 2.5em;
+ margin-bottom: 30px;
+}
+
+h2, .h2 {
+ font-size: 2.2rem;
+}
+
+h3, .h3 {
+ font-size: 1.6em;
+}
+
+.h2, .h3, .h4, .h5, .h6, h2, h3, h4, h5, h6 {
+ margin-top: 40px;
+}
+.badge {
+ padding: 0.33em .8em;
+ font-size: 70%;
+ font-weight: 500;
+ border-radius: .15rem;
+ text-transform: uppercase;
+ margin-right: 5px;
+}
+
+@media only screen and (max-width: 767px) {
+ .dropdown-menu {
+ padding: 0 0 .75rem 0;
+ background-color: transparent;
+ border: 0px;
+ }
+ .dropdown-item {
+ color: rgba(255,255,255,1);
+ }
+}
+
+p code, li code {
+ font-size: 90% !important;
+ color: #000;
+ word-break: break-word;
+ background-color: #f8f8f8;
+ padding: 3px 4px;
+}
+
+p a code {
+ color: inherit;
+}
diff --git a/faq.md b/faq.md
index 3a6377b..556a831 100644
--- a/faq.md
+++ b/faq.md
@@ -1,5 +1,5 @@
---
-layout: default
+layout: article
title: FAQ
description: Frequently asked questions about the Apache Arrow project
---
@@ -24,32 +24,205 @@ limitations under the License.
# Frequently Asked Questions
-### How stable is the Arrow format? Is it safe to use in my application?
+<hr class="mt-4 mb-3">
-The Arrow in-memory format is considered stable, and we intend to make only
backwards-compatible changes, such as additional data types. We do not yet
recommend the Arrow file format for long-term disk persistence of data; that
said, it is perfectly acceptable to write Arrow memory to disk for purposes of
memory mapping and caching.
+## General
-We encourage people to start building Arrow-based in-memory computing
applications now, and choose a suitable file format for disk storage if
necessary. The Arrow libraries include adapters for several file formats,
including Parquet, ORC, CSV, and JSON.
+#### **What *is* Apache Arrow?**
-### What is the difference between Apache Arrow and Apache Parquet?
+Apache Arrow is a software development platform for building high performance
+applications that process and transport large data sets. It is designed to both
+improve the performance of analytical algorithms and the efficiency of moving
+data from one system (or programming language to another).
-In short, Parquet files are designed for disk storage, while Arrow is designed
for in-memory use, but you can put it on disk and then memory-map later. Arrow
and Parquet are intended to be compatible with each other and used together in
applications.
+A critical component of Apache Arrow is its **in-memory columnar format**, a
+standardized, language-agnostic specification for representing structured,
+table-like datasets in-memory. This data format has a rich data type system
+(included nested and user-defined data types) designed to support the needs of
+analytic database systems, data frame libraries, and more.
-Parquet is a columnar file format for data serialization. Reading a Parquet
file requires decompressing and decoding its contents into some kind of
in-memory data structure. It is designed to be space/IO-efficient at the
expensive CPU utilization for decoding. It does not provide any data structures
for in-memory computing. Parquet is a streaming format which must be decoded
from start-to-end; while some "index page" facilities have been added to the
storage format recently, random acces [...]
+The project also contains implementations of the Arrow columnar format in many
+languages, along with utilities for reading and writing it to many common
+storage formats. These official libraries enable third-party projects to work
+with Arrow data without having to implement the Arrow columnar format
+themselves. For those that want to implement a small subset of the format, the
+Arrow project contains some tools, such as a C data interface, to assist with
+interoperability with the official Arrow libraries.
-Arrow on the other hand is first and foremost a library providing columnar
data structures for *in-memory computing*. When you read a Parquet file, you
can decompress and decode the data *into* Arrow columnar data structures so
that you can then perform analytics in-memory on the decoded data. The Arrow
columnar format has some nice properties: random access is O(1) and each value
cell is next to the previous and following one in memory, so it's efficient to
iterate over.
+The Arrow libraries contain many software components that assist with systems
+problems related to getting data in and out of remote storage systems and
+moving Arrow-formatted data over network interfaces. Some of these components
+can be used even in scenarios where the columnar format is not used at all.
-What about "Arrow files" then? Apache Arrow defines a binary "serialization"
protocol for arranging a collection of Arrow columnar arrays (called a "record
batch") that can be used for messaging and interprocess communication. You can
put the protocol anywhere, including on disk, which can later be memory-mapped
or read into memory and sent elsewhere.
+Lastly, alongside software that helps with data access and IO-related issues,
+there are libraries of algorithms for performing analytical operations or
+queries against Arrow datasets.
-This Arrow protocol is designed so that you can "map" a blob of Arrow data
without doing any deserialization, so performing analytics on Arrow protocol
data on disk can use memory-mapping and pay effectively zero cost. The protocol
is used for many other things as well, such as streaming data between Spark SQL
and Python for running pandas functions against chunks of Spark SQL data (these
are called "pandas udfs").
+#### **Why define a standard for columnar in-memory?**
-In some applications, Parquet and Arrow can be used interchangeably for
on-disk data serialization. Some things to keep in mind:
+Traditionally, data processing engine developers have created custom data
+structures to represent datasets in-memory while they are being
+processed. Given the "custom" nature of these data structures, they must also
+develop serialization interfaces to convert between these data structures and
+different file formats, network wire protocols, database clients, and other
+data transport interface. The net result of this is an incredible amount of
+waste both in developer time and in CPU cycles spend serializing data from one
+format to another.
-* Parquet is intended for "archival" purposes, meaning if you write a file
today, we expect that any system that says they can "read Parquet" will be able
to read the file in 5 years or 7 years. We are not yet making this assertion
about long-term stability of the Arrow format.
-* Parquet is generally a lot more expensive to read because it must be decoded
into some other data structure. Arrow protocol data can simply be memory-mapped.
-* Parquet files are often much smaller than Arrow-protocol-on-disk because of
the data encoding schemes that Parquet uses. If your disk storage or network is
slow, Parquet may be a better choice.
+The rationale for Arrow's in-memory columnar data format is to provide an
+out-of-the-box solution to several interrelated problems:
-### How does Arrow relate to Flatbuffers?
+* A general purpose tabular data representation that is highly efficient to
+ process on modern hardware while also being suitable for a wide spectrum of
+ use cases. We believe that fewer and fewer systems will create their own data
+ structures and simply use Arrow.
+* Supports both random access and streaming / scan-based workloads.
+* A standardized memory format facilitates reuse of libraries of
+ algorithms. When custom in-memory data formats are used, common algorithms
+ must often be rewritten to target those custom data formats.
+* Systems that both use or support Arrow can transfer data between them at
+ little-to-no cost. This results in a radical reduction in the amount of
+ serialization overhead in analytical workloads that can often represent
+ 80-90% of computing costs.
+* The language-agnostic design of the Arrow format enables systems written in
+ different programming languages (even running on the JVM) to communicate
+ datasets without serialization overhead. For example, a Java application can
+ call a C or C++ algorithm on data that originated in the JVM.
-Flatbuffers is a domain-agnostic low-level building block for binary data
formats. It cannot be used directly for data analysis tasks without a lot of
manual scaffolding. Arrow is a data layer aimed directly at the needs of data
analysis, providing elaborate data types (including extensible logical types),
built-in support for "null" values (a.k.a "N/A"), and an expanding toolbox of
I/O and computing facilities.
+<hr class="my-5">
-The Arrow file format does use Flatbuffers under the hood to facilitate
low-level metadata serialization. However, Arrow data has much richer semantics
than Flatbuffers data.
+## Project status
+
+#### **How stable is the Arrow format? Is it safe to use in my application?**
+
+The Arrow columnar format and protocol is considered stable, and we intend to
+make only backwards-compatible changes, such as additional data types. It is
+used by many applications already, and you can trust that compatibility will
+not be broken. See [the documentation]({{ site.baseurl
+}}/docs/format/Versioning.html) for details on Arrow format versioning and
+stability.
+
+#### **How stable are the Arrow libraries?**
+
+We refer you to the [implementation matrix]({{ site.baseurl
}}/docs/status.html).
+
+<hr class="my-5">
+
+## Getting started
+
+#### **Where can I get Arrow libraries?**
+
+Arrow libraries for many languages are available through the usual package
+managers. See the [install]({{ site.baseurl }}/install/) page for specifics.
+
+<hr class="my-5">
+
+## Getting involved
+
+#### **I have some questions. How can I get help?**
+
+The [Arrow mailing lists]({{ site.baseurl }}/community/) are the best place
+to ask questions. Don't be shy--we're here to help.
+
+#### **I tried to use Arrow and it didn't work. Can you fix it?**
+
+Hopefully! Please make a detailed bug report--that's a valuable contribution to
+the project itself. See the [contribution guidelines]({{ site.baseurl
+}}/docs/developers/contributing.html) for how to make a report.
+
+#### **Arrow looks great and I'd totally use it if it only did X. When will it
be done?**
+
+We use [JIRA](https://issues.apache.org/jira/browse/ARROW) for our issue
+tracker. Search for an issue that matches your need. If you find one, feel
+free to comment on it and describe your use case--that will help whoever picks
+up the task. If you don't find one, make it.
+
+Ultimately, Arrow is software written by and for the community. If you don't
+see someone else in the community working on your issue, the best way to get it
+done is to pitch in yourself. We're more than willing to help you contribute
+successfully to the project.
+
+#### **How can I report a security vulnerability?**
+
+Please send an email to
[[email protected]](mailto:[email protected]).
+See the [security]({{ site.baseurl }}/security/) page for more.
+
+<hr class="my-5">
+
+## Relation to other projects
+
+#### **What is the difference between Apache Arrow and Apache Parquet?**
+
+Parquet is not a "runtime in-memory format"; in general, file formats almost
+always have to be deserialized into some in-memory data structure for
+processing. We intend for Arrow to be that in-memory data structure.
+
+Parquet is a storage format designed for maximum space efficiency, using
+advanced compression and encoding techniques. It is ideal when wanting to
+minimize disk usage while storing gigabytes of data, or perhaps more.
+This efficiency comes at the cost of relatively expensive reading into memory,
+as Parquet data cannot be directly operated on but must be decoded in
+large chunks.
+
+Conversely, Arrow is an in-memory format meant for direct and efficient use
+for computational purposes. Arrow data is not compressed (or only lightly so,
+when using dictionary encoding) but laid out in natural format for the CPU,
+so that data can be accessed at arbitrary places at full speed.
+
+Therefore, Arrow and Parquet complement each other
+and are commonly used together in applications. Storing your data on disk
+using Parquet and reading it into memory in the Arrow format will allow
+you to make the most of your computing hardware.
+
+#### **What about "Arrow files" then?**
+
+Apache Arrow defines an inter-process communication (IPC) mechanism to
+transfer a collection of Arrow columnar arrays (called a "record batch").
+It can be used synchronously between processes using the Arrow "stream format",
+or asynchronously by first persisting data on storage using the Arrow "file
format".
+
+The Arrow IPC mechanism is based on the Arrow in-memory format, such that
+there is no translation necessary between the on-disk representation and
+the in-memory representation. Therefore, performing analytics on an Arrow
+IPC file can use memory-mapping, avoiding any deserialization cost and extra
copies.
+
+Some things to keep in mind when comparing the Arrow IPC file format and the
+Parquet format:
+
+* Parquet is designed for long-term storage and archival purposes, meaning if
+ you write a file today, you can expect that any system that says they can
+ "read Parquet" will be able to read the file in 5 years or 10 years.
+ While the Arrow on-disk format is stable and will be readable by future
+ versions of the libraries, it does not prioritize the requirements of
+ long-term archival storage.
+
+* Reading Parquet files generally requires efficient yet relatively complex
+ decoding, while reading Arrow IPC files does not involve any decoding because
+ the on-disk representation is the same as the in-memory representation.
+
+* Parquet files are often much smaller than Arrow IPC files because of the
+ columnar data compression strategies that Parquet uses. If your disk storage
or network
+ is slow, Parquet may be a better choice even for short-term storage or
caching.
+
+#### **What about the "Feather" file format?**
+
+The Feather v1 format was a simplified custom container for writing a subset of
+the Arrow format to disk prior to the development of the Arrow IPC file format.
+"Feather version 2" is now exactly the Arrow IPC file format and we have
+retained the "Feather" name and APIs for backwards compatibility.
+
+#### **How does Arrow relate to Flatbuffers?**
+
+Flatbuffers is a low-level building block for binary data serialization.
+It is not adapted to the representation of large, structured, homogenous
+data, and does not sit at the right abstraction layer for data analysis tasks.
+
+Arrow is a data layer aimed directly at the needs of data analysis, providing a
+comprehensive collection of data types required to analytics, built-in support
+for "null" values (representing missing data), and an expanding toolbox of I/O
+and computing facilities.
+
+The Arrow file format does use Flatbuffers under the hood to serialize schemas
+and other metadata needed to implement the Arrow binary IPC protocol,
+but the Arrow data format uses its own representation
+for optimal access and computation.
diff --git a/index.html b/index.html
index 707c0ae..8d48459 100644
--- a/index.html
+++ b/index.html
@@ -1,72 +1,26 @@
---
-layout: default
+layout: home
+title: Apache Arrow
+description: A cross-language development platform for in-memory analytics
---
-<div class="jumbotron">
- <h1>Apache Arrow</h1>
- <p class="lead">A cross-language development platform for in-memory
data</p>
- <p>
- <a class="btn btn-lg btn-success" style="white-space: normal;"
href="mailto:[email protected]"; role="button">Join Mailing List</a>
- <a class="btn btn-lg btn-primary" style="white-space: normal;"
href="{{ site.baseurl }}/install/" role="button">Install
({{site.data.versions['current'].number}} Release -
{{site.data.versions['current'].date}})</a>
- </p>
-</div>
-<h5>
- Interested in contributing?
- <small class="text-muted">Join the <a
href="http://mail-archives.apache.org/mod_mbox/arrow-dev/";><strong>mailing
list</strong></a> or check out the <a
href="https://cwiki.apache.org/confluence/display/ARROW";><strong>developer
wiki</strong></a>.</small>
-</h5>
-<h5>
- <a href="{{ site.baseurl }}/blog/"><strong>See Latest News</strong></a>
-</h5>
-<p>
- {{ site.description }}
-</p>
-<hr />
+<h2>What is Arrow?</h2>
<div class="row">
- <div class="col-lg-4">
- <h2 class="mt-3">Fast</h2>
- <p>Apache Arrow™ enables execution engines to take advantage of
the latest SIMD (Single instruction, multiple data) operations included in
modern processors, for native vectorized optimization of analytical data
processing. Columnar layout is optimized for data locality for better
performance on modern hardware like CPUs and GPUs.</p>
- <p>The Arrow memory format supports <strong>zero-copy reads</strong> for
lightning-fast data access without serialization overhead.</p>
- </div>
- <div class="col-lg-4">
- <h2 class="mt-3">Flexible</h2>
- <p>Arrow acts as a new high-performance interface between various
systems. It is also focused on supporting a wide variety of industry-standard
programming languages. C, C++, C#, Go, Java, JavaScript, MATLAB, Python, R,
Ruby, and Rust implementations are in progress and more languages are welcome.
- </p>
- </div>
- <div class="col-lg-4">
- <h2 class="mt-3">Standard</h2>
- <p>Apache Arrow is backed by key developers of 13 major open source
projects, including Calcite, Cassandra, Drill, Hadoop, HBase, Ibis, Impala,
Kudu, Pandas, Parquet, Phoenix, Spark, and Storm making it the de-facto
standard for columnar in-memory analytics.</p>
- <p>Learn more about projects that are <a href="{{ site.baseurl
}}/powered_by/">Powered By Apache Arrow</a></p>
+ <div class="col-lg-4 mt-3">
+ <h3>Format</h3>
+ <p>Apache Arrow defines a language-independent columnar memory format
for flat and hierarchical data, organized for efficient analytic operations on
modern hardware like CPUs and GPUs. The Arrow memory format also supports
zero-copy reads for lightning-fast data access without serialization
overhead.</p>
+ <p><a href="{{ site.baseurl }}/overview/">Learn more</a> about the
design or
+ <a href="{{ site.baseurl }}/docs/format/Columnar.html">read the
specification</a>.</p>
</div>
-</div>
-<hr />
-<div class="row">
- <div class="col-md-6">
- <h2>Performance Advantage of Columnar In-Memory</h2>
- <p class="lead">
- Columnar memory layout allows applications to avoid unnecessary IO and
- accelerate analytical processing performance on modern CPUs and GPUs.
- </p>
- </div>
- <div class="col-md-6">
- <img src="{{ site.baseurl }}/img/simd.png" alt="SIMD" class="img-fluid
mx-auto" />
- </div>
-</div>
-<hr />
-<h2>Advantages of a Common Data Layer</h2>
-<div class="row">
- <div class="col-md-6">
- <img src="{{ site.baseurl }}/img/copy.png" alt="common data layer"
class="img-fluid mx-auto" />
- <ul>
- <li>Each system has its own internal memory format</li>
- <li>70-80% computation wasted on serialization and deserialization</li>
- <li>Similar functionality implemented in multiple projects</li>
- </ul>
+ <div class="col-lg-4 mt-3">
+ <h3>Libraries</h3>
+ <p>Arrow's libraries implement the format and provide building blocks
for a range of <a href="{{ site.baseurl }}/use_cases/">use cases</a>, including
high performance analytics. <a href="{{ site.baseurl }}/powered_by/">Many
popular projects</a> use Arrow to ship columnar data efficiently or as the
basis for analytic engines.</p>
+ <p>Libraries are available for <a href="{{ site.baseurl
}}/docs/c_glib/">C</a>, <a href="{{ site.baseurl }}/docs/cpp/">C++</a>, <a
href="https://github.com/apache/arrow/blob/master/csharp/README.md";>C#</a>, <a
href="https://godoc.org/github.com/apache/arrow/go/arrow";>Go</a>, <a href="{{
site.baseurl }}/docs/java/">Java</a>, <a href="{{ site.baseurl
}}/docs/js/">JavaScript</a>, <a
href="https://github.com/apache/arrow/blob/master/matlab/README.md";>MATLAB</a>,
<a href="{{ site.baseur [...]
+ See <a href="{{ site.baseurl }}/install/">how to install</a> and get
started.
</div>
- <div class="col-md-6">
- <img src="{{ site.baseurl }}/img/shared.png" alt="common data layer"
class="img-fluid mx-auto" />
- <ul>
- <li>All systems utilize the same memory format</li>
- <li>No overhead for cross-system communication</li>
- <li>Projects can share functionality (eg, Parquet-to-Arrow reader)</li>
- </ul>
+ <div class="col-lg-4 mt-3">
+ <h3>Ecosystem</h3>
+ <p>Apache Arrow is software created by and for the developer community.
We are <a href="https://www.apache.org/theapacheway/";>dedicated</a> to open,
kind communication and consensus decisionmaking. Our <a href="{{ site.baseurl
}}/committers/">committers</a> come from a range of organizations and
backgrounds, and <a
href="https://www.apache.org/foundation/policies/conduct.html";>we welcome
all</a> to participate with us. </p>
+
+ <p><a href="{{ site.baseurl }}/community/">Learn more</a> about how you
can ask questions and get involved in the Arrow project.</p>
</div>
</div>
diff --git a/install.md b/install.md
index a444e13..bdba426 100644
--- a/install.md
+++ b/install.md
@@ -22,11 +22,15 @@ limitations under the License.
{% endcomment %}
-->
-## Current Version: {{site.data.versions['current'].number}}
+# Install Apache Arrow
-### Released: {{site.data.versions['current'].date}}
+## Current Version: {{site.data.versions['current'].number}}
({{site.data.versions['current'].date}})
-See the [release notes][10] for more about what's new.
+See the [release notes][10] for more about what's new. For information on
previous releases, see [here][19].
+
+This page is a reference listing of release artifacts and package managers.
For language-specific user guides, see the pages listed in the "Documentation"
menu above.
+
+----
### Source Release
@@ -134,17 +138,18 @@ For convenience, we also provide packages through several
package managers. Many
### C++ and Python Conda Packages
-Binary conda packages are on [conda-forge][5] for the following
-platforms:
+Binary conda packages are on [conda-forge][5] for Linux, macOS, and Windows
+for the following versions:
-* Linux and macOS (Python 2.7, 3.6 and 3.7)
-* Windows (Python 3.6 and 3.7)
+* Python 3.6, 3.7, 3.8
+* R 3.6, 4.0
Install them with:
```shell
conda install arrow-cpp={{site.data.versions['current'].pinned_number}} -c
conda-forge
conda install pyarrow={{site.data.versions['current'].pinned_number}} -c
conda-forge
+conda install r-arrow={{site.data.versions['current'].pinned_number}} -c
conda-forge
```
### C++ and GLib (C) Packages on Homebrew
@@ -167,13 +172,13 @@ brew install apache-arrow-glib
The MSYS2 packages include [Apache Arrow C++ and GLib (C)
package][16]. You can install the package by `pacman`.
-64bit version:
+64-bit version:
```shell
pacman -S --noconfirm mingw-w64-x86_64-arrow
```
-32bit version:
+32-bit version:
```shell
pacman -S --noconfirm mingw-w64-i686-arrow
@@ -193,13 +198,21 @@ cd vcpkg
The Apache Arrow C++ port in vcpkg is kept up to date by Microsoft team
members and community contributors. If the version is out of date, please
[create an issue or pull request][18] on the vcpkg repository.
+### R Package on CRAN
+
+Install the R package from [CRAN][20] with
+
+```r
+install.packages("arrow")
+```
+
[1]: {{site.data.versions['current'].mirrors}}
[2]: {{site.data.versions['current'].github-tag-link}}
[4]: {{site.data.versions['current'].java-artifacts}}
[5]: https://conda-forge.github.io
[6]: {{site.data.versions['current'].mirrors-tar}}
-[10]: {{site.data.versions['current'].release-notes}}
+[10]: {{ site.baseurl }}/release/{{site.data.versions['current'].number}}.html
[11]: https://downloads.apache.org/arrow/KEYS
[12]: https://www.apache.org/dyn/closer.cgi#verify
[13]: {{site.data.versions['current'].asc}}
@@ -208,3 +221,5 @@ The Apache Arrow C++ port in vcpkg is kept up to date by
Microsoft team members
[16]: https://github.com/msys2/MINGW-packages/tree/master/mingw-w64-arrow
[17]: https://brew.sh/
[18]: https://github.com/Microsoft/vcpkg
+[19]: {{ site.baseurl }}/release/
+[20]: https://cran.r-project.org/
diff --git a/overview.md b/overview.md
new file mode 100644
index 0000000..7d0ab7c
--- /dev/null
+++ b/overview.md
@@ -0,0 +1,95 @@
+---
+layout: overview
+title: Format
+description: Arrow Format
+---
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+## Apache Arrow Overview
+
+Apache Arrow is a software development platform for building high performance
+applications that process and transport large data sets. It is designed to both
+improve the performance of analytical algorithms and the efficiency of moving
+data from one system or programming language to another.
+
+A critical component of Apache Arrow is its **in-memory columnar format**, a
+standardized, language-agnostic specification for representing structured,
+table-like datasets in-memory. This data format has a rich data type system
+(included nested and user-defined data types) designed to support the needs of
+analytic database systems, data frame libraries, and more.
+
+<div class="row mt-4">
+ <div class="col-md-6">
+ <h3>Columnar is Fast</h3>
+ <p>
+ The Apache Arrow format allows computational routines and execution
+ engines to maximize their efficiency when scanning and iterating large
+ chunks of data. In particular, the contiguous columnar layout enables
+ vectorization using the latest SIMD (Single Instruction, Multiple Data)
+ operations included in modern processors.
+ </p>
+ </div>
+ <div class="col-md-6">
+ <img src="{{ site.baseurl }}/img/simd.png" alt="SIMD" class="img-fluid
mx-auto" />
+ </div>
+</div>
+
+<div class="row mt-4">
+ <div class="col-md-6">
+ <img src="{{ site.baseurl }}/img/copy.png" alt="common data layer"
class="img-fluid mx-auto" />
+ <img src="{{ site.baseurl }}/img/shared.png" alt="common data layer"
class="img-fluid mx-auto" />
+ </div>
+ <div class="col-md-6">
+ <h3>Standardization Saves</h3>
+ <p>
+ Without a standard columnar data format, every database and language has
+ to implement its own internal data format. This generates a lot of
+ waste. Moving data from one system to another involves costly
+ serialization and deserialization. In addition, common algorithms must
+ often be rewritten for each data format.
+ </p>
+ <p>
+ Arrow's in-memory columnar data format is an out-of-the-box solution to
+ these problems. Systems that use or support Arrow can transfer data
+ between them at little-to-no cost. Moreover, they don't need to implement
+ custom connectors for every other system. On top of these savings, a
+ standardized memory format facilitates reuse of libraries of algorithms,
+ even across languages.
+ </p>
+ </div>
+</div>
+
+<div class="row mt-4">
+ <div class="col-md-12">
+ <h3>Arrow Libraries</h3>
+ <p>
+ The Arrow project contains libraries that enable you to work with data
in the Arrow columnar format in many languages. The <a href="{{ site.baseurl
}}/docs/cpp/">C++</a>, <a
href="https://github.com/apache/arrow/blob/master/csharp/README.md";>C#</a>, <a
href="https://godoc.org/github.com/apache/arrow/go/arrow";>Go</a>, <a href="{{
site.baseurl }}/docs/java/">Java</a>, <a href="{{ site.baseurl
}}/docs/js/">JavaScript</a>, and <a
href="https://docs.rs/crate/arrow/";>Rust</a> libraries
+ contain distinct implementations of the Arrow format. These libraries
are <a href="{{ site.baseurl }}/docs/status.html">integration-tested</a>
against each other to ensure their fidelity to the format. In addition, Arrow
libraries for <a href="{{ site.baseurl }}/docs/c_glib/">C (Glib)</a>, <a
href="https://github.com/apache/arrow/blob/master/matlab/README.md";>MATLAB</a>,
<a href="{{ site.baseurl }}/docs/python/">Python</a>, <a href="{{ site.baseurl
}}/docs/r/">R</a>, and <a href="h [...]
+ </p>
+ <p>
+ These official libraries enable third-party projects to work with Arrow
+ data without having to implement the Arrow columnar format
+ themselves. They also contain many software components that assist with
+ systems problems related to getting data in and out of remote storage
+ systems and moving Arrow-formatted data over network interfaces, among
+ other <a href="{{ site.baseurl }}/use_cases/">use cases</a>.
+ </p>
+ </div>
+</div>
diff --git a/powered_by.md b/powered_by.md
index ca02c6c..48731c3 100644
--- a/powered_by.md
+++ b/powered_by.md
@@ -1,5 +1,5 @@
---
-layout: default
+layout: article
title: Powered by
description: List of projects powered by Apache Arrow
---
@@ -75,6 +75,15 @@ short description of your use case.
Dremio reads data from any source (RDBMS, HDFS, S3, NoSQL) into Arrow
buffers, and provides fast SQL access via ODBC, JDBC, and REST for BI,
Python, R, and more (all backed by Apache Arrow).
+* **[Falcon][25]:** An interactive data exploration tool with coordinated
views.
+ Falcon loads Arrow files using the Arrow JavaScript module. Since Arrow does
+ not need to be parsed (like text-based formats like CSV and JSON), startup
cost
+ is significantly minimized.
+* **[FASTDATA.io][26]**: Plasma Engine (unrelated to Arrow's Plasma In-Memory
+ Object Store) exploits the massive parallel processing power of GPUs for
+ stream and batch processing. It supports Arrow as input and output, uses
+ Arrow internally to maximize performance, and can be used with existing
+ Apache Spark™ APIs.
* **[Fletcher][20]:** Fletcher is a framework that can integrate FPGA
accelerators with tools and frameworks that use the Apache Arrow in-memory
format. From a set of Arrow Schemas, Fletcher generates highly optimized
@@ -145,15 +154,6 @@ short description of your use case.
Database Connectivity (ODBC) interface. It provides the ability to return
Arrow Tables and RecordBatches in addition to the Python Database API
Specification 2.0.
-* **[Falcon][25]:** An interactive data exploration tool with coordinated
views.
- Falcon loads Arrow files using the Arrow JavaScript module. Since Arrow does
- not need to be parsed (like text-based formats like CSV and JSON), startup
cost
- is significantly minimized.
-* **[FASTDATA.io][26]**: Plasma Engine (unrelated to Arrow's Plasma In-Memory
- Object Store) exploits the massive parallel processing power of GPUs for
- stream and batch processing. It supports Arrow as input and output, uses
- Arrow internally to maximize performance, and can be used with existing
- Apache Spark™ APIs.
[1]: https://www.apache.org/foundation/marks/
[2]: https://www.apache.org/foundation/marks/faq/
diff --git a/release-announcement-template.md b/release-announcement-template.md
new file mode 100644
index 0000000..cb351a5
--- /dev/null
+++ b/release-announcement-template.md
@@ -0,0 +1,87 @@
+---
+layout: post
+title: "Apache Arrow x.y.z Release"
+date: "2020-04-21 00:00:00 -0600"
+author: pmc
+categories: [release]
+---
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+<!--
+
+To use this template:
+
+* Copy this template file to the _posts directory, naming it
YYYY-MM-DD-x.y.z-release.md
+* Replace all instances of "x.y.z" with the current release number
+* Update the date in the front matter
+* Update all "XX" values with the appropriate numbers (you can get the
resolved issues and contributors count from `_release/x.y.z.md`)
+* Fill in the various sections below. Note that the audience is the broader
user community, not Arrow developers, so please write clearly using terms they
will understand and care about. Delete any sections that don't have any content
(as in, there are no changes to announce)
+* Delete this introductory comment
+
+ -->
+
+
+The Apache Arrow team is pleased to announce the x.y.z release. This covers
+over XX months of development work and includes [**XX resolved issues**][1]
+from [**XX distinct contributors**][2]. See the Install Page to learn how to
+get the libraries for your platform.
+
+The release notes below are not exhaustive and only expose selected highlights
+of the release. Many other bugfixes and improvements have been made: we refer
+you to the [complete changelog][3].
+
+## Community
+
+<!-- Acknowledge and link to any new committers and PMC members since the last
release. See previous release announcements for examples. -->
+
+## Columnar Format Notes
+
+## Arrow Flight RPC notes
+
+## C++ notes
+
+## C# notes
+
+## Go notes
+
+## Java notes
+
+## JavaScript notes
+
+## Python notes
+
+## R notes
+
+
+For more on what’s in the x.y.z R package, see the [R changelog][4].
+
+## Ruby and C GLib notes
+
+### Ruby
+
+### C GLib
+
+## Rust notes
+
+
+[1]:
https://issues.apache.org/jira/issues/?jql=project%20%3D%20ARROW%20AND%20status%20%3D%20Resolved%20AND%20fixVersion%20%3D%20x.y.z
+[2]: {{ site.baseurl }}/release/x.y.z.html#contributors
+[3]: {{ site.baseurl }}/release/x.y.z.html
+[4]: {{ site.baseurl }}/docs/r/news/
diff --git a/security.md b/security.md
new file mode 100644
index 0000000..00dd184
--- /dev/null
+++ b/security.md
@@ -0,0 +1,39 @@
+---
+layout: default
+title: Security
+description: Security
+---
+
+# Reporting Security Issues
+
+Apache Arrow uses the standard process outlined by the [Apache Security
Team](https://www.apache.org/security/) for reporting vulnerabilities. Note
that vulnerabilities should not be publicly disclosed until the project has
responded.
+
+To report a possible security vulnerability, please email
[[email protected]](mailto:[email protected]).
+
+<hr class="my-5">
+
+###
[CVE-2019-12408](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2019-12408):
Uninitialized Memory in C++ ArrayBuilder
+
+**Severity**: High
+
+**Vendor**: The Apache Software Foundation
+
+**Versions affected**: 0.14.x
+
+**Description**: It was discovered that the C++ implementation (which
underlies the R, Python and Ruby implementations) of Apache Arrow 0.14.0 to
0.14.1 had a uninitialized memory bug when building arrays with null values in
some cases. This can lead to uninitialized memory being unintentionally shared
if Arrow Arrays are transmitted over the wire (for instance with Flight) or
persisted in the streaming IPC and file formats.
+
+**Mitigation**: Upgrade to version 0.15.1 or greater.
+
+<hr class="my-5">
+
+###
[CVE-2019-12410](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2019-12410):
Uninitialized Memory in C++ Reading from Parquet
+
+**Severity**: High
+
+**Vendor**: The Apache Software Foundation
+
+**Versions affected**: 0.12.0 - 0.14.1
+
+**Description**: While investigating UBSAN errors in
[ARROW-6549](https://github.com/apache/arrow/pull/5365) it was discovered
Apache Arrow versions 0.12.0 to 0.14.1 left memory Array data uninitialized
when reading RLE null data from parquet. This affected the C++, Python, Ruby,
and R implementations. The uninitialized memory could potentially be shared if
are transmitted over the wire (for instance with Flight) or persisted in the
streaming IPC and file formats.
+
+**Mitigation**: Upgrade to version 0.15.1 or greater.
diff --git a/use_cases.md b/use_cases.md
new file mode 100644
index 0000000..5dffaf8
--- /dev/null
+++ b/use_cases.md
@@ -0,0 +1,98 @@
+---
+layout: default
+title: Use cases
+description: Example use cases for the Apache Arrow project
+---
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+# Use Cases
+
+Here are some example applications of the Apache Arrow format and libraries.
+For more, see our [blog]({{ site.baseurl }}/blog/) and the list of projects
+[powered by Arrow]({{ site.baseurl }}/powered_by/).
+
+## Reading/writing columnar storage formats
+
+Many Arrow libraries provide convenient methods for reading and writing
+columnar file formats, including the Arrow IPC file format ("Feather")
+and the [Apache Parquet](https://parquet.apache.org/) format.
+
+<!-- Link to implementation matrix? -->
+
+* Feather: C++, [Python]({{ site.baseurl }}/docs/python/feather.html }}),
+ [R]({{ site.baseurl }}/docs/r/reference/read_feather.html)
+* Parquet: [C++]({{ site.baseurl }}/docs/cpp/parquet.html),
+ [Python]({{ site.baseurl }}/docs/python/parquet.html),
+ [R]({{ site.baseurl }}/docs/r/reference/read_parquet.html)
+
+In addition to single-file readers, some libraries (C++,
+[Python]({{ site.baseurl }}/docs/python/dataset.html),
+[R]({{ site.baseurl }}/docs/r/articles/dataset.html)) support reading
+entire directories of files and treating them as a single dataset. These
+datasets may be on the local file system or on a remote storage system, such
+as HDFS, S3, etc.
+
+## Sharing memory locally
+
+Arrow IPC files can be memory-mapped locally, which allow you to work with
+data bigger than memory and to share data across languages and processes.
+<!-- example? -->
+
+The Arrow project includes [Plasma]({% post_url
2017-08-08-plasma-in-memory-object-store %}),
+a shared-memory object store written in C++ and exposed in Python. Plasma
+holds immutable objects in shared memory so that they can be accessed
+efficiently by many clients across process boundaries.
+
+The Arrow format also defines a [C data interface]({% post_url
2020-05-04-introducing-arrow-c-data-interface %}),
+which allows zero-copy data sharing inside a single process without any
+build-time or link-time dependency requirements. This allows, for example,
+[R users to access `pyarrow`-based projects]({{ site.baseurl
}}/docs/r/articles/python.html)
+using the `reticulate` package.
+
+## Moving data over the network
+
+The Arrow format allows serializing and shipping columnar data
+over the network - or any kind of streaming transport.
+[Apache Spark](https://spark.apache.org/) uses Arrow as a
+data interchange format, and both [PySpark]({% post_url 2017-07-26-spark-arrow
%})
+and [sparklyr]({% post_url 2019-01-25-r-spark-improvements %}) can take
+advantage of Arrow for significant performance gains when transferring data.
+[Google BigQuery](https://cloud.google.com/bigquery/docs/reference/storage),
+[TensorFlow](https://www.tensorflow.org/tfx),
+[AWS
Athena](https://docs.aws.amazon.com/athena/latest/ug/connect-to-a-data-source.html),
+and [others]({{ site.baseurl }}/powered_by/) also use Arrow similarly.
+
+The Arrow project also defines [Flight]({% post_url
2019-09-30-introducing-arrow-flight %}),
+a client-server RPC framework to build rich services exchanging data according
+to application-defined semantics.
+
+<!-- turbodbc -->
+
+## In-memory data structure for analytics
+
+The Arrow format is designed to enable fast computation. Some projects have
+begun to take advantage of that design. Within the Apache Arrow project,
+[DataFusion]({% post_url 2019-02-04-datafusion-donation %}) is a query engine
+using Arrow data built in Rust.
+
+<!--
+* Rapids?
+* Dremio?
+-->
![https://www.apache.org/events/current-event-234x60.png"/](https://www.apache.org/events/current-event-234x60.png"/){kind=link}