This is an automated email from the ASF dual-hosted git repository.
alamb pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/arrow-site.git
The following commit(s) were added to refs/heads/master by this push:
new 03fd97f521 ARROW-16846: [Website] June 2022 Rust Apache Arrow and
Parquet 16.0.0 Highlights (#220)
03fd97f521 is described below
commit 03fd97f521b658a47dad93c187291fe56afb8eb2
Author: Andrew Lamb <[email protected]>
AuthorDate: Thu Jun 16 17:00:13 2022 -0400
ARROW-16846: [Website] June 2022 Rust Apache Arrow and Parquet 16.0.0
Highlights (#220)
* June 2022 Rust Apache Arrow and Parquet 16.0.0 Highlights
* Update _posts/2022-06-11-rust-16.0.0.md
Co-authored-by: Raphael Taylor-Davies
<[email protected]>
* Minor edits
* Remove todos, clean up links
* Add more links, small edits
* style and typos
* Date for publishing
Co-authored-by: Raphael Taylor-Davies
<[email protected]>
Co-authored-by: Wakahisa <[email protected]>
---
_posts/2022-06-16-rust-16.0.0.md | 141 +++++++++++++++++++++++++++++++++++++++
1 file changed, 141 insertions(+)
diff --git a/_posts/2022-06-16-rust-16.0.0.md b/_posts/2022-06-16-rust-16.0.0.md
new file mode 100644
index 0000000000..37399caf01
--- /dev/null
+++ b/_posts/2022-06-16-rust-16.0.0.md
@@ -0,0 +1,141 @@
+---
+layout: post
+title: "June 2022 Rust Apache Arrow and Parquet 16.0.0 Highlights"
+date: "2022-06-16 00:00:00 -0600"
+author: pmc
+categories: [release]
+---
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+
+# Introduction
+
+We recently celebrated releasing version 16.0.0 of the Rust implementation of
[Apache Arrow](https://arrow.apache.org/). While we still get a few comments on
“most rust libraries use versions 0.x.0, why are you at 16.0.0?”, our
versioning scheme appears to be working well, and permits quick releases of new
features and API evolution in a semver compatible way without breaking
downstream projects.
+
+This post contains highlights from the last four months (versions
+10.0.0 to 16.0.0) of
+[arrow-rs](https://github.com/apache/arrow-rs/arrow) and
+[parquet-rs](https://github.com/apache/arrow-rs/parquet) development
+as well as a roadmap of future work. The full list of awesomeness can
+be found in the
+[CHANGELOG](https://github.com/apache/arrow-rs/blob/master/CHANGELOG.md).
+
+As you [may remember](https://github.com/apache/arrow-rs/issues/1715), the
arrow and parquet implementations are in the same crate, on the same release
schedule, and in this same blog. This is not for technical reasons, but helps
to keep the maintenance burden for delivering great Apache software reasonable,
and allows easier development of optimized conversion between Arrow <-->
Parquet formats.
+
+# Parquet
+The [parquet crate](https://crates.io/crates/parquet) has seen a return to
substantial improvements after being relatively dormant for several years. The
current major areas of focus are
+
+1. **Performance**: Improving the raw performance for reading and writing
mirroring the efforts that went into the C++ version a few years ago.
+2. **API Ease of Use**: Improving the API so it is easy to use efficiently
with modern Rust for two preeminent use cases: 1) reading from local disk and
2) reading `async`hronously from remote object stores.
+
+Some Major Highlights:
+
+* **Advanced Metadata Access**: API access to advanced parquet metadata, such
as [PageEncoding](https://github.com/apache/arrow-rs/pull/1322),
[BloomFilters](https://github.com/apache/arrow-rs/pull/1309) and
[PageIndex](https://github.com/apache/arrow-rs/pull/1762).
+* **Improved API Usability**: For example, the [parquet writer now uses
`std:io::Write`](https://github.com/apache/arrow-rs/pull/1719) rather than a
custom `ParquetWriter` trait, making it more interoperable with the rest of the
Rust ecosystem and the [projection API is easier to use with nested
types](https://github.com/apache/arrow-rs/pull/1716).
+* **Rewritten support for nested types (e.g. struct, lists)** : @tustvold has
revamped / rewritten support for
[reading](https://github.com/apache/arrow-rs/pull/1682) and
[writing](https://github.com/apache/arrow-rs/pull/1746) structured types, which
both improved the support for arbitrary nested schemas, and is 30% faster.
+
+Looking Forward:
+
+* **Even Faster**: We are actively working to make [writing even
faster](https://github.com/apache/arrow-rs/issues/1764) and expect to see some
major improvements over the next few releases.
+* **Object Store Integration**: Support for easily and efficiently
reading/writing to/from object storage is improving, and we expect it will soon
work well out of the box, fetching the minimal bytes, etc... More on this to
follow in a separate blog post.
+* **Parallel Decode**: We intend to transparently support high performance
parallel decoding of parquet to arrow arrays, when invoked from a
[rayon](https://crates.io/crates/rayon) threadpool.
+
+# Arrow
+
+The Rust arrow implementation has also had substantial improvements, in
addition to bug fixes and performance improvements.
+
+Some Major Highlights:
+
+* **Ecosystem Compatibility**: [@viriya](https://github.com/viirya) has put in
a massive effort to improve (and prove) compatibility with other Arrow
implementations via the Rust IPC integration tests. There have been major
improvements for corner cases involving nested structures, nullability, nested
dictionaries, etc.
+* **Safety**: We continue to improve the safety of arrow, and it is not
possible to trigger undefined behavior using `safe` apis -- checkout the
[README](https://github.com/apache/arrow-rs/tree/master/arrow#safety) and the
[module level
rustdocs](https://docs.rs/arrow/16.0.0/arrow/#safety-and-security) for more
details. Among other things, we have added additional validation checking to
[string kernels](https://github.com/apache/arrow-rs/issues/1575) and
[`DecimalArrays`](https://github. [...]
+* **Performance**: There have been several major performance improvements such
as [much faster filter
kernels](https://github.com/apache/arrow-rs/issues/1288), thanks to
[@tustvold](https://github.com/tustvold).
+* **Easier to Use APIs**: Several of the APIs are now easier to use (e.g.
[#1645](https://github.com/apache/arrow-rs/pull/1645) and
[#1739](https://github.com/apache/arrow-rs/pull/1739) which lowers the barrier
to entry of using `arrow-rs`, thanks to
[@HaoYang670](https://github.com/HaoYang670).
+* **DataType::Null support**: is much improved, such as [in the cast
kernels](https://github.com/apache/arrow-rs/pull/1572), thanks to
[@WinkerDu](https://github.com/WinkerDu).
+* **Improved JSON reader**: The JSON reader is [now easier to
use](https://github.com/apache/arrow-rs/pull/1451) thanks to
[@sum12](https://github.com/sum12).
+
+Looking Forward:
+* **Make ArrayData Easier to use Safely**: Some amount of `unsafe` will likely
always be required in arrow (for fast IPC, for example), but we are also
working to improve the underlying `ArrayData` structure to make it more
compatible with the ecosystem (e.g. use `Bytes`), support faster to decode from
parquet, and to avoid bugs related to offsets (slicing) which are a [frequent
pain point](https://github.com/apache/arrow-rs/issues/1799).
+* **FlightSQL** – we have some [initial support for Flight
SQL](https://github.com/apache/arrow-rs/pulls?q=flightsql) thank to
[@wangfenjin](https://github.com/wangfenjin) and
[@timvw](https://github.com/timvw), though we would love to see some additional
contributors. Such help can include a basic FlightSQL server, and starting work
on clients.
+
+Some areas looking for help include:
+
+* **Decimal 256 support**: <https://github.com/apache/arrow-rs/issues/131>
+* **Support for negative Decimal scale**:
<https://github.com/apache/arrow-rs/issues/1785>
+* **Support IPC file compression**:
<https://github.com/apache/arrow-rs/issues/1709>
+* **Zero-copy bitmap slicing**:
<https://github.com/apache/arrow-rs/issues/1802>
+
+
+# Contributors:
+
+While some open source software can be created mostly by a single contributor,
we believe the greatest software with the largest impact and reach is built
around its community. Thus, Arrow is part of the [Apache Software
Foundation](https://www.apache.org/) and our releases both past and present are
a result of our amazing community's effort.
+
+We would like to thank everyone who has contributed to the arrow-rs repository
since the `9.0.2` release. Keep up the great work and we look forward to
continued improvements:
+
+```console
+git shortlog -sn 9.0.0..16.0.0
+ 47 Liang-Chi Hsieh
+ 45 Raphael Taylor-Davies
+ 43 Andrew Lamb
+ 40 Remzi Yang
+ 8 Sergey Glushchenko
+ 7 Jörn Horstmann
+ 6 Shani Solomon
+ 6 dependabot[bot]
+ 5 Yang Jiang
+ 4 jakevin
+ 4 Chao Sun
+ 4 Yijie Shen
+ 3 kazuhiko kikuchi
+ 2 Sumit
+ 2 Ismail-Maj
+ 2 Kamil Konior
+ 2 tfeda
+ 2 Matthew Turner
+ 1 iyupeng
+ 1 ryan-jacobs1
+ 1 Alex Qyoun-ae
+ 1 tjwilson90
+ 1 Andy Grove
+ 1 Atef Sawaed
+ 1 Daniël Heres
+ 1 DuRipeng
+ 1 Helgi Kristvin Sigurbjarnarson
+ 1 Kun Liu
+ 1 Kyle Barron
+ 1 Marc Garcia
+ 1 Peter C. Jentsch
+ 1 Remco Verhoef
+ 1 Sven Cattell
+ 1 Thomas Peiselt
+ 1 Tiphaine Ruy
+ 1 Trent Feda
+ 1 Wang Fenjin
+ 1 Ze'ev Maor
+ 1 diana
+```
+
+# Join the community
+
+If you are interested in contributing to the Rust subproject in Apache Arrow,
you can find a list of open issues
+suitable for beginners
[here](https://github.com/apache/arrow-rs/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22)
+and the full list [here](https://github.com/apache/arrow-rs/issues).
+
+Other ways to get involved include trying out Arrow on some of your data and
filing bug reports, and helping to
+improve the documentation.
