jayzhan211 commented on code in PR #457:
URL: https://github.com/apache/arrow-site/pull/457#discussion_r1451824243
##########
_posts/2024-01-25-datafusion-34.0.0.md:
##########
@@ -0,0 +1,345 @@
+---
+layout: post
+title: "Apache Arrow DataFusion 34.0.0 Released, Looking Forward to 2024"
+date: "2024-01-01 00:00:00"
+author: pmc
+categories: [release]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+## Introduction
+
+[Apache Arrow DataFusion] is an extensible query engine, written in [Rust],
that
+uses [Apache Arrow] as its in-memory format. DataFusion is used by developers
to
+create new, fast data centric systems such as databases, dataframe libraries,
+machine learning and streaming applications. While [DataFusion’s primary design
+goal] is to accelerate creating other data centric systems, it has a
+reasonably good experience out of the box as a [dataframe library] and
+[command line SQL tool].
+
+[DataFusion’s primary design goal]:
https://arrow.apache.org/datafusion/user-guide/introduction.html#project-goals
+[dataframe library]: https://arrow.apache.org/datafusion-python/
+[command line SQL tool]:
https://arrow.apache.org/datafusion/user-guide/cli.html
+
+
+[apache arrow datafusion]: https://arrow.apache.org/datafusion/
+[apache arrow]: https://arrow.apache.org
+[rust]: https://www.rust-lang.org/
+
+We recently [released DataFusion 34.0.0]. This blog highlights some of the
major
+improvements since we [released DataFusion 26.0.0] (spoiler alert it is a lot)
+and a preview of where the community will likely spend time in the next 6
months.
+
+[released DataFusion 26.0.0]:
https://arrow.apache.org/blog/2023/06/24/datafusion-25.0.0/.
+[released DataFusion 34.0.0]: https://crates.io/crates/datafusion/34.0.0
+
+This may also be our last update blog post on the Apache Arrow Site. Future
+updates will likely be on the DataFusion website as we are working to [graduate
+to a top level project] (Apache Arrow DataFusion → Apache DataFusion!) which
+will help focus governance and project growth. Also exciting, our [first
+DataFusion in person meetup] is planned for March 2024.
+
+[graduate to a top level project]:
https://github.com/apache/arrow-datafusion/discussions/6475
+[first DataFusion in person meetup]:
https://github.com/apache/arrow-datafusion/discussions/8522
+
+DataFusion is very much a community endeavor. The core thesis is that as a
+community we can build much better and advanced technology than any of us a
+individuals or companies could alone. In the last 6 months between `26.0.0` and
+`34.0.0`, community growth has been strong. We accepted and reviewed over a
+thousand PRs from 124 different committers, created over 650 and closed 517
+issues.
+
+<!--
+$ git log --pretty=oneline 26.0.0..34.0.0 . | wc -l
+ 1009
+
+$ git shortlog -sn 26.0.0..34.0.0 . | wc -l
+ 124
+
+https://crates.io/crates/datafusion/26.0.0
+DataFusion 26 released June 7, 2023
+
+https://crates.io/crates/datafusion/34.0.0
+DataFusion 34 released Dec 17, 2023
+
+Issues created in this time: 214 open, 437 closed
+https://github.com/apache/arrow-datafusion/issues?q=is%3Aissue+created%3A2023-06-23..2023-12-17
+
+Issues closes: 517
+https://github.com/apache/arrow-datafusion/issues?q=is%3Aissue+closed%3A2023-06-23..2023-12-17+
+
+PRs merged in this time 908
+https://github.com/apache/arrow-datafusion/pulls?q=is%3Apr+merged%3A2023-06-23..2023-12-17
+-->
+
+The rest of this post highlights a small portion of how we have improved
+DataFusion over the last 6 months previews where we are heading. You can
+see a list of all changes in the detailed [CHANGELOG].
+
+[CHANGELOG]:
https://github.com/apache/arrow-datafusion/blob/main/datafusion/CHANGELOG.md
+
+# Improved Performance 🚀
+
+Performance is a key feature of DataFusion. We have made major improvements
+since `25.0.0`, resulting in a 2x overall runtime improvement in the
+[ClickBench] queries.
+
+<!--
+ Source:
https://docs.google.com/spreadsheets/d/1FtI3652WIJMC5LmJbLfT3G06w0JQIxEPG4yfMafexh8/edit#gid=1879366976
+ Average runtime on 25.0.0: 7.2s (for the queries that actually ran)
+ Average runtime on 34.0.0: 3.6s (for the same queries that ran in 25.0.0)
+-->
+
+[ClickBench]: https://benchmark.clickhouse.com/
+
+<figure style="text-align: center;">
+ <img src="{{ site.baseurl }}/img/datafusion-34.0.0/compare.png" width="100%"
class="img-responsive" alt="Fig 1: Adaptive Arrow schema architecture
overview.">
+ <figcaption>
+ Fig 1: Performance improvements between DataFusion 25.0.0 and DataFusion
34.0.0.
+ Note that several queries don't run on <code>25.0.0</code>, for various
reasons such as requiring too much memory (Q33)
+ or unsupported SQL features.
+ </figcaption>
+</figure>
+
+Here are some of the specific enhancements we made:
+* [2-3x Better aggregation performance with many distinct groups]
+* Partially ordered grouping / streaming grouping
+* [Specialized operator for "TopK" `ORDER BY LIMIT XXX`]
+* [Specialized operator for `min(col) GROUP BY .. ORDER by min(col) LIMIT XXX`]
+* [Improved Join Performance]
+* Eliminate redundant sorting with sort order aware optimizers
+
+[2-3x Better aggregation performance with many distinct groups]:
https://arrow.apache.org/blog/2023/08/05/datafusion_fast_grouping/
+[Specialized operator for `min(col) GROUP BY .. ORDER by min(col) LIMIT XXX`]:
https://github.com/apache/arrow-datafusion/pull/7192
+[Specialized operator for "TopK" `ORDER BY LIMIT XXX`]:
https://github.com/apache/arrow-datafusion/pull/7721
+[Improved Join Performance]:
https://github.com/apache/arrow-datafusion/pull/8126
+
+# New Features ✨
+
+## DML / Insert / Creating Files
+
+DataFusion now supports writing data in parallel, to individual or multiple
+files, using `Parquet`, `CSV`, `JSON`, `ARROW` and user defined formats.
+
+You can do this using [`CREATE EXTERNAL TABLE` statement] for example:
+
+```sql
+❯ CREATE EXTERNAL TABLE awesome_table(x INT) STORED AS PARQUET LOCATION
'/tmp/my_awesome_table';
+0 rows in set. Query took 0.003 seconds.
+
+❯ INSERT INTO awesome_table SELECT x * 10 FROM my_source_table;
++-------+
+| count |
++-------+
+| 3 |
++-------+
+1 row in set. Query took 0.024 seconds.
+```
+
+[`CREATE EXTERNAL TABLE` statement]:
https://arrow.apache.org/datafusion/user-guide/sql/ddl.html#create-external-table
+
+You can also create files using the [`COPY` command], similarly to [DuckDB’s
`COPY`] command:
+
+[`COPY` command]:
https://arrow.apache.org/datafusion/user-guide/sql/dml.html#copy
+[DuckDB’s `COPY`]: https://duckdb.org/docs/sql/statements/copy.html
+
+```sql
+❯ COPY (SELECT x + 1 FROM my_source_table) TO '/tmp/output.json';
++-------+
+| count |
++-------+
+| 3 |
++-------+
+1 row in set. Query took 0.014 seconds.
+```
+
+```shell
+$ cat /tmp/output.json
+{"x":1}
+{"x":2}
+{"x":3}
+
+$ python3
+Python 3.11.7 (main, Dec 4 2023, 18:10:11) [Clang 15.0.0
(clang-1500.1.0.2.5)] on darwin
+Type "help", "copyright", "credits" or "license" for more information.
+>>> import pyarrow.feather as ft
+>>> table = ft.read_table("/tmp/output.arrow")
+>>> print(table)
+pyarrow.Table
+x: int32
+----
+x: [[1,2,3]]
+```
+
+## Improved `STRUCT` and `ARRAY` support
Review Comment:
nope. I think all of them are new features compare to 6 months before.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
