tustvold commented on code in PR #220:
URL: https://github.com/apache/arrow-site/pull/220#discussion_r895012905
##########
_posts/2022-06-11-rust-16.0.0.md:
##########
@@ -0,0 +1,142 @@
+---
+layout: post
+title: "June 2022 Rust Apache Arrow and Parquet 16.0.0 Highlights"
+date: "2022-06-11 00:00:00 -0600"
+author: pmc
+categories: [release]
+---
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+
+# Introduction
+
+
+We recently celebrated releasing version 16.0.0 of the Rust implementation of
[Apache Arrow](https://arrow.apache.org/). While we still get a few comments on
“most rust libraries use versions 0.x.0, why are you at 16.0.0?”, our
versioning scheme appears to be working well, and permits quick releases of new
features and API evolution in a semver compatible way without breaking
downstream projects.
+
+This post contains highlights from the last four months (versions
+10.0.0 to 16.0.0) of
+[arrow-rs](https://github.com/apache/arrow-rs/arrow) and
+[parquet-rs](https://github.com/apache/arrow-rs/parquet) development
+as well as a roadmap of future work. The full list of awesomeness can
+be found in the
+[CHANGELOG](https://github.com/apache/arrow-rs/blob/master/CHANGELOG.md).
+
+As you [may remember](https://github.com/apache/arrow-rs/issues/1715), the
arrow and parquet implementations are in the same crate, on the same release
schedule, and in this same blog. This is not for technical reasons, but helps
to keep the maintenance burden for delivering great Apache software reasonable,
and focus on optimized conversion between arrow <--> parquet.
+
+# Parquet
+The [parquet crate](https://crates.io/crates/parquet) has seen a return to
substantial improvements after being relatively dormant for several years. The
current major areas of focus are
+
+1. **Performance**: Improving the raw performance for reading and writing
mirroring the efforts that went into the C++ version a few years ago
+2. **API Ease of Use**: Improving the API so it is easy to use efficiently
with modern Rust for two preeminent use cases: 1) reading from local disk and
2) reading `async`hronously from remote object stores.
+
+Some Major Highlights:
+
+* **Advanced Metadata Access**: APIs have been added to access more advanced
metadata, such as [PageEncoding](https://github.com/apache/arrow-rs/pull/1322),
[BloomFilters](https://github.com/apache/arrow-rs/pull/1309) and
[PageIndex](https://github.com/apache/arrow-rs/pull/1762).
+* **Improved usability API**: For example, the [parquet writer now uses
`std:io::Write`](https://github.com/apache/arrow-rs/pull/1719)) rather custom
`ParquetWriter` trait , making it more interoperable with the rest of the Rust
ecosystem and the [projection API is easier to use with nested
types](https://github.com/apache/arrow-rs/pull/1716).
+* **Rewritten support for nested types (e.g. struct, lists)** : @tustvold has
revamped / rewritten support for
[reading](https://github.com/apache/arrow-rs/pull/1682_ and
[writing](https://github.com/apache/arrow-rs/pull/1746) structured types,
improving the support for arbitrary nested schemas, and is 30% faster.
+
+(TODO maybe a picture / example code to make this blog post less textful)
+
+Looking Forward:
+
+* **Even Faster**: We are actively working to make [writing even
faster](https://github.com/apache/arrow-rs/issues/1764) and expect to see some
major improvements over the next few releases.
+* **Object Store Integration**: Support for easily and efficiently
reading/writing to/from object storage is improving, and we expect it will soon
work well out of the box, fetching the minimal bytes, etc... More on this to
follow in a separate blog post
+* **Parallel Decode**: We intend to transparently support high performance
parallel decoding of parquet to arrow arrays, when invoked from a
[rayon](https://crates.io/crates/rayon) threadpool.
+
+# Arrow
+
+The Rust arrow implementation has also had substantial improvements, in
addition to bug fixes and performance improvements.
+
+Some Major Highlights:
+
+* **Ecosystem Compatibility**: @viriya has put in a massive effort to improve
(and prove) compatibility with other Arrow implementations via the rust IPC
integration tests. There have been major improvements for corner cases
involving nested structures, nullabulity, nested dictionaries, etc.
+* **Safety**: We continue to improve the safety of arrow (checkout README
https://github.com/apache/arrow-rs/tree/master/arrow#safety and the module
level rustdocs TODO linkaftr 16.0.0 is released ). Among other things, we have
added additional validation checking to string kernels
https://github.com/apache/arrow-rs/issues/1575 and DecimalArrays
(https://github.com/apache/arrow-rs/pull/1767).
+* **Performance**: There have been several major performance improvements such
as much caster filter kernels https://github.com/apache/arrow-rs/issues/1288-
thanks @tustvold)
Review Comment:
```suggestion
* **Performance**: There have been several major performance improvements
such as much faster filter kernels
https://github.com/apache/arrow-rs/issues/1288- thanks @tustvold)
```
##########
_posts/2022-06-11-rust-16.0.0.md:
##########
@@ -0,0 +1,142 @@
+---
+layout: post
+title: "June 2022 Rust Apache Arrow and Parquet 16.0.0 Highlights"
+date: "2022-06-11 00:00:00 -0600"
+author: pmc
+categories: [release]
+---
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+
+# Introduction
+
+
+We recently celebrated releasing version 16.0.0 of the Rust implementation of
[Apache Arrow](https://arrow.apache.org/). While we still get a few comments on
“most rust libraries use versions 0.x.0, why are you at 16.0.0?”, our
versioning scheme appears to be working well, and permits quick releases of new
features and API evolution in a semver compatible way without breaking
downstream projects.
+
+This post contains highlights from the last four months (versions
+10.0.0 to 16.0.0) of
+[arrow-rs](https://github.com/apache/arrow-rs/arrow) and
+[parquet-rs](https://github.com/apache/arrow-rs/parquet) development
+as well as a roadmap of future work. The full list of awesomeness can
+be found in the
+[CHANGELOG](https://github.com/apache/arrow-rs/blob/master/CHANGELOG.md).
+
+As you [may remember](https://github.com/apache/arrow-rs/issues/1715), the
arrow and parquet implementations are in the same crate, on the same release
schedule, and in this same blog. This is not for technical reasons, but helps
to keep the maintenance burden for delivering great Apache software reasonable,
and focus on optimized conversion between arrow <--> parquet.
+
+# Parquet
+The [parquet crate](https://crates.io/crates/parquet) has seen a return to
substantial improvements after being relatively dormant for several years. The
current major areas of focus are
+
+1. **Performance**: Improving the raw performance for reading and writing
mirroring the efforts that went into the C++ version a few years ago
+2. **API Ease of Use**: Improving the API so it is easy to use efficiently
with modern Rust for two preeminent use cases: 1) reading from local disk and
2) reading `async`hronously from remote object stores.
+
+Some Major Highlights:
+
+* **Advanced Metadata Access**: APIs have been added to access more advanced
metadata, such as [PageEncoding](https://github.com/apache/arrow-rs/pull/1322),
[BloomFilters](https://github.com/apache/arrow-rs/pull/1309) and
[PageIndex](https://github.com/apache/arrow-rs/pull/1762).
+* **Improved usability API**: For example, the [parquet writer now uses
`std:io::Write`](https://github.com/apache/arrow-rs/pull/1719)) rather custom
`ParquetWriter` trait , making it more interoperable with the rest of the Rust
ecosystem and the [projection API is easier to use with nested
types](https://github.com/apache/arrow-rs/pull/1716).
+* **Rewritten support for nested types (e.g. struct, lists)** : @tustvold has
revamped / rewritten support for
[reading](https://github.com/apache/arrow-rs/pull/1682_ and
[writing](https://github.com/apache/arrow-rs/pull/1746) structured types,
improving the support for arbitrary nested schemas, and is 30% faster.
+
+(TODO maybe a picture / example code to make this blog post less textful)
+
+Looking Forward:
+
+* **Even Faster**: We are actively working to make [writing even
faster](https://github.com/apache/arrow-rs/issues/1764) and expect to see some
major improvements over the next few releases.
+* **Object Store Integration**: Support for easily and efficiently
reading/writing to/from object storage is improving, and we expect it will soon
work well out of the box, fetching the minimal bytes, etc... More on this to
follow in a separate blog post
+* **Parallel Decode**: We intend to transparently support high performance
parallel decoding of parquet to arrow arrays, when invoked from a
[rayon](https://crates.io/crates/rayon) threadpool.
+
+# Arrow
+
+The Rust arrow implementation has also had substantial improvements, in
addition to bug fixes and performance improvements.
+
+Some Major Highlights:
+
+* **Ecosystem Compatibility**: @viriya has put in a massive effort to improve
(and prove) compatibility with other Arrow implementations via the rust IPC
integration tests. There have been major improvements for corner cases
involving nested structures, nullabulity, nested dictionaries, etc.
+* **Safety**: We continue to improve the safety of arrow (checkout README
https://github.com/apache/arrow-rs/tree/master/arrow#safety and the module
level rustdocs TODO linkaftr 16.0.0 is released ). Among other things, we have
added additional validation checking to string kernels
https://github.com/apache/arrow-rs/issues/1575 and DecimalArrays
(https://github.com/apache/arrow-rs/pull/1767).
Review Comment:
Another todo
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
