andygrove commented on code in PR #254:
URL: https://github.com/apache/arrow-site/pull/254#discussion_r1002479373
##########
_posts/2022-10-20-datafusion-13.0.0.md:
##########
@@ -0,0 +1,284 @@
+---
+layout: post
+title: "Apache Arrow DataFusion 13.0.0 Project Update"
+date: "2022-10-20 00:00:00"
+author: pmc
+categories: [release]
+---
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+# Introduction
+
+[Apache Arrow DataFusion](https://arrow.apache.org/datafusion/)
[`13.0.0`](https://crates.io/crates/datafusion) is released, and this blog
contains an update on the project for the 5 months since our [last update in
May 2022](https://arrow.apache.org/blog/2022/05/16/datafusion-8.0.0/).
+
+DataFusion is an extensible and embeddable query engine, written in Rust used
to create modern, fast and efficient data pipelines, ETL processes, and
database systems. You may want to check out DataFusion to extend your Rust
project with:
+
+- [SQL
support](https://arrow.apache.org/datafusion/user-guide/sql/sql_status.html),
+- [DataFrame
API](https://docs.rs/datafusion/13.0.0/datafusion/dataframe/struct.DataFrame.html),
+- A custom Domain Specific Query Language
+- The ability to easily and quickly read and process Parquet, JSON, Avro or
CSV data.
+- To read from remote object stores such as AWS S3, Azure Blob Storage, GCP.
+
+Even though DataFusion is 4 years "young," it has seen significant community
growth in the last few months and the momentum continues to accelerate.
+
+# Background
+
+
+DataFusion is used as the engine in [many open source and commercial
projects](https://github.com/apache/arrow-datafusion#known-uses) and was one of
the early open source projects to provide this capability. 2022 has validated
our belief in the need for such a ["LLVM for database and AI
systems"](https://www.slideshare.net/AndrewLamb32/20220623-apache-arrow-and-datafusion-changing-the-game-for-implementing-database-systemspdf)
with announcements such as the [release of FaceBook's
Velox](https://engineering.fb.com/2022/08/31/open-source/velox/) engine, the
major investments in
[Acero](https://arrow.apache.org/docs/cpp/streaming_execution.html) as well as
the continued popularity of [Apache Calcite](https://calcite.apache.org/) and
other similar technologies.
+
+While Velox and Acero focus on execution engines, DataFusion provides the
entire suite of components needed to build most analytic systems, including a
SQL frontend, a dataframe API, and extension points for just about everything.
Some [DataFusion users](https://github.com/apache/arrow-datafusion#known-uses)
use a subset of the features such as the frontend (e.g.
(dask-sql)[https://dask-sql.readthedocs.io/en/latest/] or the execution engine,
such as [Blaze](https://github.com/blaze-init/blaze), and some users use many
different components to build both SQL based and customized DSL based systems
such as [InfluxDB IOx](https://github.com/influxdata/influxdb_iox/pulls) and
[VegaFusion](https://github.com/vegafusion/vegafusion).
+
+One of DataFusion’s advantages is its implementation in
[Rust](https://www.rust-lang.org/) and thus its easy integration with the
broader Rust ecosystem. Rust continues to be a major source of benefit, from
the [ease of parallelization with the high quality and standardized `async`
ecosystem](https://www.influxdata.com/blog/using-rustlangs-async-tokio-runtime-for-cpu-bound-tasks/)
, as well as its modern dependency management system and wonderful
performance. <!-- I wonder if we should link to clickbench?? -->
+<!--While we haven’t invested in the benchmarking ratings game datafusion
continues to be quite speedy (todo quantity this, with some evidence) – maybe
clickbench?-->
+
+<!-- I am not sure this section needed -->
+DataFusion belongs to and builds on the Apache Arrow ecosystem and its family
of related technologies and projects in rust. This include
+- [Arrow](https://arrow.apache.org/): low level array representation
+- [Parquet](https://parquet.apache.org/): high performance, full featured
support for the parquet file format
+
+
+
+# DataFusion in Action
+
+While DataFusion really shines as an embeddable query engine, if you want to
try it out and get a feel for its power, you can use the
basic[`datafusion-cli`](https://docs.rs/datafusion-cli/13.0.0/datafusion_cli/)
tool to get a sense for what is possible to add in your application
+
+(TODO example here of using datafusion-cli to query from local parquet files
on disk)
+
+TODO: also mention you can use the same thing to query data from S3
+
+
+
+# Summary
+
+We have increased the frequency of DataFusion releases to monthly instead of
quarterly. This
+makes it easier for the increasing number of projects that now depend on
DataFusion.
+
+We have also completed the "graduation" of [Ballista to its own top-level
arrow-ballista
repository](https://docs.google.com/document/d/1jNRbadyStSrV5kifwn0khufAwq6OnzGczG4z8oTQJP4/edit?usp=sharing)
+which decouples the two projects and allows each project to move even faster.
+
+Along with numerous other bug fixes and smaller improvements, here are some of
the major advances:
+
+# Improved Support for Cloud Object Stores
+
+DataFusion now has improved support for the major cloud object stores (Amazon
S3, Azure Blog Storage, and Google Cloud Storage) via the
[object_store](https://crates.io/crates/object_store) crate. Using this
integration, DataFusion optimizes reading parquet files by reading only the
parts of the files that are needed.
Review Comment:
```suggestion
DataFusion now has improved support for the major cloud object stores
(Amazon S3, Azure Blob Storage, and Google Cloud Storage) via the
[object_store](https://crates.io/crates/object_store) crate. Using this
integration, DataFusion optimizes reading parquet files by reading only the
parts of the files that are needed.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@arrow.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
