alamb commented on code in PR #217:
URL: https://github.com/apache/datafusion-site/pull/217#discussion_r4647985579


##########
content/blog/2026-11-20-datafusion-56.0.0.md:
##########
@@ -0,0 +1,668 @@
+---
+layout: post
+title: Apache DataFusion 56.0.0 Released
+date: 2026-11-20
+author: pmc
+categories: [release]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements.  See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License.  You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+[TOC]
+
+<!-- TODO: update the post date (front matter and file name) to the actual 
release date -->
+
+We are proud to announce the release of [DataFusion 56.0.0], which includes 
many new features, including
+`ASOF` joins, SQL registration of external catalogs, Avro and ALP write 
support,
+better support for extension types, and many performance and stability 
improvements. The complete list of
+changes is available in the [changelog]. This release represents roughly 12
+weeks of development and 1200 commits. Thanks to the [247 contributors].
+
+[DataFusion 56.0.0]: https://crates.io/crates/datafusion/56.0.0
+[DataFusion 55.0.0]: 
https://datafusion.apache.org/blog/2026/08/25/datafusion-55.0.0/
+[changelog]: 
https://github.com/apache/datafusion/blob/branch-56/dev/changelog/56.0.0.md
+[247 contributors]: 
https://github.com/apache/datafusion/blob/branch-56/dev/changelog/56.0.0.md#credits
+
+## New Features ✨
+
+### `ASOF` Joins
+
+DataFusion 56 adds `ASOF JOIN`, a long-standing feature request ([#318]) 

Review Comment:
   FYI @Xuanwo 🫶 



##########
content/blog/2026-11-20-datafusion-56.0.0.md:
##########
@@ -0,0 +1,668 @@
+---
+layout: post
+title: Apache DataFusion 56.0.0 Released
+date: 2026-11-20
+author: pmc
+categories: [release]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements.  See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License.  You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+[TOC]
+
+<!-- TODO: update the post date (front matter and file name) to the actual 
release date -->
+
+We are proud to announce the release of [DataFusion 56.0.0], which includes 
many new features, including
+`ASOF` joins, SQL registration of external catalogs, Avro and ALP write 
support,
+better support for extension types, and many performance and stability 
improvements. The complete list of
+changes is available in the [changelog]. This release represents roughly 12
+weeks of development and 1200 commits. Thanks to the [247 contributors].
+
+[DataFusion 56.0.0]: https://crates.io/crates/datafusion/56.0.0
+[DataFusion 55.0.0]: 
https://datafusion.apache.org/blog/2026/08/25/datafusion-55.0.0/
+[changelog]: 
https://github.com/apache/datafusion/blob/branch-56/dev/changelog/56.0.0.md
+[247 contributors]: 
https://github.com/apache/datafusion/blob/branch-56/dev/changelog/56.0.0.md#credits
+
+## New Features ✨
+
+### `ASOF` Joins
+
+DataFusion 56 adds `ASOF JOIN`, a long-standing feature request ([#318]) 
+that matches each row on the left with the
+*closest* row on the right rather than an exact match. Such joins are
+common in time-series workloads such as finding the most recent quote for each
+trade, or aligning two sensors that sample at different rates.
+
+For example, given tables of `temperature` and `humidity` readings from two
+sensors that report at different times:
+
+| id | location | ts                  | temp |
+|----|----------|---------------------|------|
+| T1 | roof     | 2026-11-20T09:30:05 | 11.5 |
+| T2 | roof     | 2026-11-20T09:30:12 | 11.7 |
+| T3 | lobby    | 2026-11-20T09:30:07 | 20.1 |
+
+| id | location | ts                  | humidity |
+|----|----------|---------------------|----------|
+| H1 | roof     | 2026-11-20T09:30:00 | 71.0     |
+| H2 | roof     | 2026-11-20T09:30:10 | 70.5     |
+| H3 | lobby    | 2026-11-20T09:30:10 | 45.2     |
+
+Since no temperature reading has the exact same timestamp as any humidity
+reading, a regular equality join on `location` and `ts` would match nothing.
+Instead, an `ASOF` join efficiently pairs each temperature reading with the 
most
+recent humidity reading at or before it:
+
+```sql
+SELECT t.id AS temp_id, h.id AS humidity_id, t.location, t.ts, t.temp, 
h.humidity
+FROM temperature AS t ASOF JOIN humidity AS h
+MATCH_CONDITION (t.ts >= h.ts)
+ON t.location = h.location
+ORDER BY t.id;
+```
+
+```text
++---------+-------------+----------+---------------------+------+----------+
+| temp_id | humidity_id | location | ts                  | temp | humidity |
++---------+-------------+----------+---------------------+------+----------+
+| T1      | H1          | roof     | 2026-11-20T09:30:05 | 11.5 | 71.0     |
+| T2      | H2          | roof     | 2026-11-20T09:30:12 | 11.7 | 70.5     |
+| T3      | NULL        | lobby    | 2026-11-20T09:30:07 | 20.1 | NULL     |
++---------+-------------+----------+---------------------+------+----------+
+```
+
+<details>
+<summary>Full Example</summary>
+
+```sql
+CREATE TABLE temperature (id VARCHAR, location VARCHAR, ts TIMESTAMP, temp 
DOUBLE) AS VALUES
+  ('T1', 'roof',  '2026-11-20T09:30:05', 11.5),
+  ('T2', 'roof',  '2026-11-20T09:30:12', 11.7),
+  ('T3', 'lobby', '2026-11-20T09:30:07', 20.1);
+
+CREATE TABLE humidity (id VARCHAR, location VARCHAR, ts TIMESTAMP, humidity 
DOUBLE) AS VALUES
+  ('H1', 'roof',  '2026-11-20T09:30:00', 71.0),
+  ('H2', 'roof',  '2026-11-20T09:30:10', 70.5),
+  ('H3', 'lobby', '2026-11-20T09:30:10', 45.2);
+
+SELECT t.id AS temp_id, h.id AS humidity_id, t.location, t.ts, t.temp, 
h.humidity
+FROM temperature AS t ASOF JOIN humidity AS h
+MATCH_CONDITION (t.ts >= h.ts)
+ON t.location = h.location
+ORDER BY t.id;
+```
+
+</details>
+
+DataFusion supports the [Snowflake `ASOF JOIN` syntax][Snowflake ASOF JOIN
+syntax], please see the [`ASOF JOIN` documentation][ASOF JOIN documentation] 
for
+more details. Thanks to [@Xuanwo] for contributing this feature  and to
+[@jayzhan211], [@Kristofferjoha]  and [@2010YOUY01],  [@xudong963], [@alamb],
+and [@jonathanc-n] for reviews and improvements.
+
+[Snowflake ASOF JOIN syntax]: 
https://docs.snowflake.com/en/sql-reference/constructs/asof-join
+[ASOF JOIN documentation]: 
https://datafusion.apache.org/user-guide/sql/select.html#asof-join
+
+### External Catalogs in SQL
+
+Connecting DataFusion to an external catalog, such as an Iceberg REST catalog,
+previously required writing Rust code. DataFusion 56 adds support for
+`CREATE EXTERNAL CATALOG` and `DROP CATALOG` statements ([#25112]), to
+attach catalogs with SQL in the same way as external tables:
+
+```sql
+CREATE EXTERNAL CATALOG my_catalog
+STORED AS ICEBERG
+LOCATION 's3://bucket/warehouse'
+OPTIONS ('catalog.uri' 'http://rest-catalog:8181', 'warehouse' 'my_catalog');
+
+SELECT * FROM my_catalog.sales.orders LIMIT 10;
+```
+
+Note that DataFusion does not include any catalog implementations itself.
+Instead users register a [`CatalogProviderFactory`][CatalogProviderFactory] for
+each `STORED AS` type, an then attach catalogs with SQL; see the [`CREATE
+EXTERNAL CATALOG` documentation][CREATE EXTERNAL CATALOG documentation] for 
more
+details.
+
+Thanks to [@pepijnve] and [@TTyChud] for this work.
+
+[CatalogProviderFactory]: 
https://docs.rs/datafusion/56.0.0/datafusion/catalog/trait.CatalogProviderFactory.html
+[CREATE EXTERNAL CATALOG documentation]: 
https://datafusion.apache.org/user-guide/sql/ddl.html#create-external-catalog
+
+### File Format Improvements
+
+**Writing Avro**: DataFusion has been able to read [Apache Avro] files for a
+long time, and DataFusion 56 adds support for writing them:
+
+```sql
+COPY orders TO 'orders.avro' STORED AS AVRO;
+```
+
+<details>
+<summary>Full Example</summary>
+
+```sql
+CREATE TABLE orders (id INT, customer VARCHAR, total DOUBLE) AS VALUES
+  (1, 'alice', 42.50),
+  (2, 'bob', 17.25),
+  (3, 'carol', 99.99);
+
+-- Write the table to an Avro file
+COPY orders TO 'orders.avro' STORED AS AVRO;
+
+-- Read it back
+SELECT * FROM 'orders.avro';
++----+----------+-------+
+| id | customer | total |
++----+----------+-------+
+| 1  | alice    | 42.5  |
+| 2  | bob      | 17.25 |
+| 3  | carol    | 99.99 |
++----+----------+-------+
+```
+
+</details>
+
+**Parquet ALP Encoding**: [ALP] (Adaptive Lossless floating-Point) is a new
+Parquet encoding that often stores `FLOAT` and `DOUBLE` columns much more
+compactly. DataFusion 56 can read and write ALP-encoded files:
+
+```sql
+COPY measurements TO 'measurements.parquet' STORED AS PARQUET
+OPTIONS (
+  'format.encoding::price' 'alp',
+  'format.dictionary_enabled::price' 'false'
+);
+```
+
+<details>
+<summary>Full Example</summary>
+
+```sql
+-- One million prices with two decimal places
+CREATE TABLE measurements AS
+SELECT i AS id, round(100000 * random(), 2) AS price
+FROM generate_series(1, 1000000) AS t(i);
+
+-- Write with the default settings, and with ALP for the price column
+COPY measurements TO 'default.parquet' STORED AS PARQUET;
+COPY measurements TO 'alp.parquet' STORED AS PARQUET
+OPTIONS (
+  'format.encoding::price' 'alp',
+  'format.dictionary_enabled::price' 'false'
+);
+
+-- Compare the size of the price column in the two files
+-- note that the ALP-encoded column is about a third smaller than with
+-- the default setting. 
+SELECT 'default' AS file, encodings, total_compressed_size
+FROM parquet_metadata('default.parquet') WHERE path_in_schema = '"price"'
+UNION ALL
+SELECT 'alp', encodings, total_compressed_size
+FROM parquet_metadata('alp.parquet') WHERE path_in_schema = '"price"';
++---------+------------------------------+-----------------------+
+| file    | encodings                    | total_compressed_size |
++---------+------------------------------+-----------------------+
+| default | [PLAIN, RLE, RLE_DICTIONARY] | 4481181               |
+| alp     | [RLE, ALP]                   | 3018657               |
++---------+------------------------------+-----------------------+
+```
+
+</details>
+
+Thanks to [@pepijnve] and [@Surajsuthar] for this work.
+
+[Apache Avro]: https://avro.apache.org/
+[ALP]: https://github.com/apache/parquet-format/blob/master/Encodings.md
+
+### More SQL Support
+
+**`MATERIALIZED` CTEs**: By default, DataFusion inlines Common Table 
Expressions
+*(CTEs), so a CTE that is referenced multiple times is evaluated multiple 
times.
+*DataFusion 56 supports the Postgres `WITH ... AS [NOT] MATERIALIZED` syntax to
+*evaluate a CTE once and reuse the result ([#25580]). The syntax is currently
+*only recognized by the `postgres` SQL dialect:
+
+```sql
+SET datafusion.sql_parser.dialect = 'postgres';
+
+-- The CTE runs once, so both references see the same random value
+WITH r AS MATERIALIZED (SELECT random() AS v)
+SELECT a.v = b.v AS same FROM r AS a, r AS b;
++------+
+| same |
++------+
+| true |
++------+
+
+-- Without MATERIALIZED, each reference evaluates random() separately
+WITH r AS (SELECT random() AS v)
+SELECT a.v = b.v AS same FROM r AS a, r AS b;
++-------+
+| same  |
++-------+
+| false |
++-------+
+```
+
+Thanks to [@adriangb] for this work.
+
+### `datafusion-cli` Improvements
+
+`datafusion-cli` adds [`parquet_file_metadata`][parquet_file_metadata] and
+[`parquet_page_index`][parquet_page_index] table functions for inspecting
+Parquet files.
+
+[parquet_file_metadata]: 
https://datafusion.apache.org/user-guide/cli/functions.html#parquet-file-metadata
+[parquet_page_index]: 
https://datafusion.apache.org/user-guide/cli/functions.html#parquet-page-index
+
+### Versioned Documentation
+
+The [DataFusion documentation] site previously always reflected the latest
+`main` branch, making it hard to tell which features were available in what
+version. The site now has a version picker and publishes a snapshot of the
+documentation for each release. Thanks to [@yiaany] for this work and
+[@Omega359] for the original request.
+
+<img
+src="/blog/images/datafusion-56.0.0/versioned_docs.png"
+width="100%"
+class="img-fluid"
+alt="Screenshot of the DataFusion documentation site showing the new version 
picker, a drop-down menu listing Development (main) and releases 55.2.0, 
54.1.0, 53.0.0, 52.0.0, 51.0.0, and 50.0.0."
+/>
+
+**Figure 2**: The new version picker on the [DataFusion documentation] site,
+which selects between the documentation for the development version (`main`)
+and for previous releases.
+
+[DataFusion documentation]: https://datafusion.apache.org/
+
+### Improved support for Extension Types
+
+As described in [the User Defined Types blog], DataFusion's type system can be 
extended
+using [Arrow extension types] to attach a logical type to an
+Arrow physical type. This technique allows the DataFusion ecosystem to add
+entirely new data types *without* changes to the DataFusion core (see
+[Ecosystem Highlights] below). DataFusion 56 improves this support in several 
ways:
+
+- The SQL types `UUID` and `JSON` now map to
+  the Arrow [canonical extension types] `arrow.uuid` and `arrow.json`.
+- `DESCRIBE`, `SHOW COLUMNS`, and
+  `information_schema.columns` show the extension type name rather than the
+  storage type.
+- Extension types are preserved when reading Parquet
+  files with the default settings.
+- Extension types are preserved through more operators such as
+  casts, `min` / `max`, scalar subqueries, and `INSERT`.
+
+For example, a column declared with the SQL `JSON` type is stored as a standard
+Arrow string and tagged with the `arrow.json` extension type:
+
+```sql
+CREATE TABLE events (doc JSON);
+INSERT INTO events VALUES ('{"a":1}');
+SELECT doc,
+       arrow_typeof(doc)                           AS doc_type,
+       arrow_metadata(doc)['ARROW:extension:name'] AS doc_extension
+FROM events;
++---------+----------+---------------+
+| doc     | doc_type | doc_extension |
++---------+----------+---------------+
+| {"a":1} | Utf8     | arrow.json    |
++---------+----------+---------------+
+```
+
+Thanks to [@cblp], [@cyberbeam524], [@paleolimbot], [@M-Tesla], and
+[@osipovartem] for this work.
+
+[Arrow extension types]: 
https://arrow.apache.org/docs/format/Columnar.html#extension-types
+[canonical extension types]: 
https://arrow.apache.org/docs/format/CanonicalExtensions.html
+[Ecosystem Highlights]: #ecosystem-highlights
+[the User Defined Types blog]: 
https://datafusion.apache.org/blog/2025/09/21/custom-types-using-metadata
+
+## Ecosystem Highlights 🌱
+
+DataFusion is designed to be extended, and this section highlights two 
+projects in the DataFusion ecosystem that are made possible by the
+extension points in DataFusion 56.
+
+### Semi-Structured Data (e.g. JSON)

Review Comment:
   FYI @sdf-jkl I am hoping to higlight Variant here



##########
content/blog/2026-11-20-datafusion-56.0.0.md:
##########
@@ -0,0 +1,668 @@
+---
+layout: post
+title: Apache DataFusion 56.0.0 Released
+date: 2026-11-20
+author: pmc
+categories: [release]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements.  See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License.  You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+[TOC]
+
+<!-- TODO: update the post date (front matter and file name) to the actual 
release date -->
+
+We are proud to announce the release of [DataFusion 56.0.0], which includes 
many new features, including
+`ASOF` joins, SQL registration of external catalogs, Avro and ALP write 
support,
+better support for extension types, and many performance and stability 
improvements. The complete list of
+changes is available in the [changelog]. This release represents roughly 12
+weeks of development and 1200 commits. Thanks to the [247 contributors].
+
+[DataFusion 56.0.0]: https://crates.io/crates/datafusion/56.0.0
+[DataFusion 55.0.0]: 
https://datafusion.apache.org/blog/2026/08/25/datafusion-55.0.0/
+[changelog]: 
https://github.com/apache/datafusion/blob/branch-56/dev/changelog/56.0.0.md
+[247 contributors]: 
https://github.com/apache/datafusion/blob/branch-56/dev/changelog/56.0.0.md#credits
+
+## New Features ✨
+
+### `ASOF` Joins
+
+DataFusion 56 adds `ASOF JOIN`, a long-standing feature request ([#318]) 
+that matches each row on the left with the
+*closest* row on the right rather than an exact match. Such joins are
+common in time-series workloads such as finding the most recent quote for each
+trade, or aligning two sensors that sample at different rates.
+
+For example, given tables of `temperature` and `humidity` readings from two
+sensors that report at different times:
+
+| id | location | ts                  | temp |
+|----|----------|---------------------|------|
+| T1 | roof     | 2026-11-20T09:30:05 | 11.5 |
+| T2 | roof     | 2026-11-20T09:30:12 | 11.7 |
+| T3 | lobby    | 2026-11-20T09:30:07 | 20.1 |
+
+| id | location | ts                  | humidity |
+|----|----------|---------------------|----------|
+| H1 | roof     | 2026-11-20T09:30:00 | 71.0     |
+| H2 | roof     | 2026-11-20T09:30:10 | 70.5     |
+| H3 | lobby    | 2026-11-20T09:30:10 | 45.2     |
+
+Since no temperature reading has the exact same timestamp as any humidity
+reading, a regular equality join on `location` and `ts` would match nothing.
+Instead, an `ASOF` join efficiently pairs each temperature reading with the 
most
+recent humidity reading at or before it:
+
+```sql
+SELECT t.id AS temp_id, h.id AS humidity_id, t.location, t.ts, t.temp, 
h.humidity
+FROM temperature AS t ASOF JOIN humidity AS h
+MATCH_CONDITION (t.ts >= h.ts)
+ON t.location = h.location
+ORDER BY t.id;
+```
+
+```text
++---------+-------------+----------+---------------------+------+----------+
+| temp_id | humidity_id | location | ts                  | temp | humidity |
++---------+-------------+----------+---------------------+------+----------+
+| T1      | H1          | roof     | 2026-11-20T09:30:05 | 11.5 | 71.0     |
+| T2      | H2          | roof     | 2026-11-20T09:30:12 | 11.7 | 70.5     |
+| T3      | NULL        | lobby    | 2026-11-20T09:30:07 | 20.1 | NULL     |
++---------+-------------+----------+---------------------+------+----------+
+```
+
+<details>
+<summary>Full Example</summary>
+
+```sql
+CREATE TABLE temperature (id VARCHAR, location VARCHAR, ts TIMESTAMP, temp 
DOUBLE) AS VALUES
+  ('T1', 'roof',  '2026-11-20T09:30:05', 11.5),
+  ('T2', 'roof',  '2026-11-20T09:30:12', 11.7),
+  ('T3', 'lobby', '2026-11-20T09:30:07', 20.1);
+
+CREATE TABLE humidity (id VARCHAR, location VARCHAR, ts TIMESTAMP, humidity 
DOUBLE) AS VALUES
+  ('H1', 'roof',  '2026-11-20T09:30:00', 71.0),
+  ('H2', 'roof',  '2026-11-20T09:30:10', 70.5),
+  ('H3', 'lobby', '2026-11-20T09:30:10', 45.2);
+
+SELECT t.id AS temp_id, h.id AS humidity_id, t.location, t.ts, t.temp, 
h.humidity
+FROM temperature AS t ASOF JOIN humidity AS h
+MATCH_CONDITION (t.ts >= h.ts)
+ON t.location = h.location
+ORDER BY t.id;
+```
+
+</details>
+
+DataFusion supports the [Snowflake `ASOF JOIN` syntax][Snowflake ASOF JOIN
+syntax], please see the [`ASOF JOIN` documentation][ASOF JOIN documentation] 
for
+more details. Thanks to [@Xuanwo] for contributing this feature  and to
+[@jayzhan211], [@Kristofferjoha]  and [@2010YOUY01],  [@xudong963], [@alamb],
+and [@jonathanc-n] for reviews and improvements.
+
+[Snowflake ASOF JOIN syntax]: 
https://docs.snowflake.com/en/sql-reference/constructs/asof-join
+[ASOF JOIN documentation]: 
https://datafusion.apache.org/user-guide/sql/select.html#asof-join
+
+### External Catalogs in SQL
+
+Connecting DataFusion to an external catalog, such as an Iceberg REST catalog,
+previously required writing Rust code. DataFusion 56 adds support for
+`CREATE EXTERNAL CATALOG` and `DROP CATALOG` statements ([#25112]), to
+attach catalogs with SQL in the same way as external tables:
+
+```sql
+CREATE EXTERNAL CATALOG my_catalog
+STORED AS ICEBERG
+LOCATION 's3://bucket/warehouse'
+OPTIONS ('catalog.uri' 'http://rest-catalog:8181', 'warehouse' 'my_catalog');
+
+SELECT * FROM my_catalog.sales.orders LIMIT 10;
+```
+
+Note that DataFusion does not include any catalog implementations itself.
+Instead users register a [`CatalogProviderFactory`][CatalogProviderFactory] for
+each `STORED AS` type, an then attach catalogs with SQL; see the [`CREATE
+EXTERNAL CATALOG` documentation][CREATE EXTERNAL CATALOG documentation] for 
more
+details.
+
+Thanks to [@pepijnve] and [@TTyChud] for this work.
+
+[CatalogProviderFactory]: 
https://docs.rs/datafusion/56.0.0/datafusion/catalog/trait.CatalogProviderFactory.html
+[CREATE EXTERNAL CATALOG documentation]: 
https://datafusion.apache.org/user-guide/sql/ddl.html#create-external-catalog
+
+### File Format Improvements
+
+**Writing Avro**: DataFusion has been able to read [Apache Avro] files for a
+long time, and DataFusion 56 adds support for writing them:
+
+```sql
+COPY orders TO 'orders.avro' STORED AS AVRO;
+```
+
+<details>
+<summary>Full Example</summary>
+
+```sql
+CREATE TABLE orders (id INT, customer VARCHAR, total DOUBLE) AS VALUES
+  (1, 'alice', 42.50),
+  (2, 'bob', 17.25),
+  (3, 'carol', 99.99);
+
+-- Write the table to an Avro file
+COPY orders TO 'orders.avro' STORED AS AVRO;
+
+-- Read it back
+SELECT * FROM 'orders.avro';
++----+----------+-------+
+| id | customer | total |
++----+----------+-------+
+| 1  | alice    | 42.5  |
+| 2  | bob      | 17.25 |
+| 3  | carol    | 99.99 |
++----+----------+-------+
+```
+
+</details>
+
+**Parquet ALP Encoding**: [ALP] (Adaptive Lossless floating-Point) is a new
+Parquet encoding that often stores `FLOAT` and `DOUBLE` columns much more
+compactly. DataFusion 56 can read and write ALP-encoded files:
+
+```sql
+COPY measurements TO 'measurements.parquet' STORED AS PARQUET
+OPTIONS (
+  'format.encoding::price' 'alp',
+  'format.dictionary_enabled::price' 'false'
+);
+```
+
+<details>
+<summary>Full Example</summary>
+
+```sql
+-- One million prices with two decimal places
+CREATE TABLE measurements AS
+SELECT i AS id, round(100000 * random(), 2) AS price
+FROM generate_series(1, 1000000) AS t(i);
+
+-- Write with the default settings, and with ALP for the price column
+COPY measurements TO 'default.parquet' STORED AS PARQUET;
+COPY measurements TO 'alp.parquet' STORED AS PARQUET
+OPTIONS (
+  'format.encoding::price' 'alp',
+  'format.dictionary_enabled::price' 'false'
+);
+
+-- Compare the size of the price column in the two files
+-- note that the ALP-encoded column is about a third smaller than with
+-- the default setting. 
+SELECT 'default' AS file, encodings, total_compressed_size
+FROM parquet_metadata('default.parquet') WHERE path_in_schema = '"price"'
+UNION ALL
+SELECT 'alp', encodings, total_compressed_size
+FROM parquet_metadata('alp.parquet') WHERE path_in_schema = '"price"';
++---------+------------------------------+-----------------------+
+| file    | encodings                    | total_compressed_size |
++---------+------------------------------+-----------------------+
+| default | [PLAIN, RLE, RLE_DICTIONARY] | 4481181               |
+| alp     | [RLE, ALP]                   | 3018657               |
++---------+------------------------------+-----------------------+
+```
+
+</details>
+
+Thanks to [@pepijnve] and [@Surajsuthar] for this work.
+
+[Apache Avro]: https://avro.apache.org/
+[ALP]: https://github.com/apache/parquet-format/blob/master/Encodings.md
+
+### More SQL Support
+
+**`MATERIALIZED` CTEs**: By default, DataFusion inlines Common Table 
Expressions
+*(CTEs), so a CTE that is referenced multiple times is evaluated multiple 
times.
+*DataFusion 56 supports the Postgres `WITH ... AS [NOT] MATERIALIZED` syntax to
+*evaluate a CTE once and reuse the result ([#25580]). The syntax is currently
+*only recognized by the `postgres` SQL dialect:
+
+```sql
+SET datafusion.sql_parser.dialect = 'postgres';
+
+-- The CTE runs once, so both references see the same random value
+WITH r AS MATERIALIZED (SELECT random() AS v)
+SELECT a.v = b.v AS same FROM r AS a, r AS b;
++------+
+| same |
++------+
+| true |
++------+
+
+-- Without MATERIALIZED, each reference evaluates random() separately
+WITH r AS (SELECT random() AS v)
+SELECT a.v = b.v AS same FROM r AS a, r AS b;
++-------+
+| same  |
++-------+
+| false |
++-------+
+```
+
+Thanks to [@adriangb] for this work.
+
+### `datafusion-cli` Improvements
+
+`datafusion-cli` adds [`parquet_file_metadata`][parquet_file_metadata] and
+[`parquet_page_index`][parquet_page_index] table functions for inspecting
+Parquet files.
+
+[parquet_file_metadata]: 
https://datafusion.apache.org/user-guide/cli/functions.html#parquet-file-metadata
+[parquet_page_index]: 
https://datafusion.apache.org/user-guide/cli/functions.html#parquet-page-index
+
+### Versioned Documentation
+
+The [DataFusion documentation] site previously always reflected the latest
+`main` branch, making it hard to tell which features were available in what
+version. The site now has a version picker and publishes a snapshot of the
+documentation for each release. Thanks to [@yiaany] for this work and
+[@Omega359] for the original request.
+
+<img
+src="/blog/images/datafusion-56.0.0/versioned_docs.png"
+width="100%"
+class="img-fluid"
+alt="Screenshot of the DataFusion documentation site showing the new version 
picker, a drop-down menu listing Development (main) and releases 55.2.0, 
54.1.0, 53.0.0, 52.0.0, 51.0.0, and 50.0.0."
+/>
+
+**Figure 2**: The new version picker on the [DataFusion documentation] site,
+which selects between the documentation for the development version (`main`)
+and for previous releases.
+
+[DataFusion documentation]: https://datafusion.apache.org/
+
+### Improved support for Extension Types
+
+As described in [the User Defined Types blog], DataFusion's type system can be 
extended
+using [Arrow extension types] to attach a logical type to an
+Arrow physical type. This technique allows the DataFusion ecosystem to add
+entirely new data types *without* changes to the DataFusion core (see
+[Ecosystem Highlights] below). DataFusion 56 improves this support in several 
ways:
+
+- The SQL types `UUID` and `JSON` now map to
+  the Arrow [canonical extension types] `arrow.uuid` and `arrow.json`.
+- `DESCRIBE`, `SHOW COLUMNS`, and
+  `information_schema.columns` show the extension type name rather than the
+  storage type.
+- Extension types are preserved when reading Parquet
+  files with the default settings.
+- Extension types are preserved through more operators such as
+  casts, `min` / `max`, scalar subqueries, and `INSERT`.
+
+For example, a column declared with the SQL `JSON` type is stored as a standard
+Arrow string and tagged with the `arrow.json` extension type:
+
+```sql
+CREATE TABLE events (doc JSON);
+INSERT INTO events VALUES ('{"a":1}');
+SELECT doc,
+       arrow_typeof(doc)                           AS doc_type,
+       arrow_metadata(doc)['ARROW:extension:name'] AS doc_extension
+FROM events;
++---------+----------+---------------+
+| doc     | doc_type | doc_extension |
++---------+----------+---------------+
+| {"a":1} | Utf8     | arrow.json    |
++---------+----------+---------------+
+```
+
+Thanks to [@cblp], [@cyberbeam524], [@paleolimbot], [@M-Tesla], and
+[@osipovartem] for this work.
+
+[Arrow extension types]: 
https://arrow.apache.org/docs/format/Columnar.html#extension-types
+[canonical extension types]: 
https://arrow.apache.org/docs/format/CanonicalExtensions.html
+[Ecosystem Highlights]: #ecosystem-highlights
+[the User Defined Types blog]: 
https://datafusion.apache.org/blog/2025/09/21/custom-types-using-metadata
+
+## Ecosystem Highlights 🌱
+
+DataFusion is designed to be extended, and this section highlights two 
+projects in the DataFusion ecosystem that are made possible by the
+extension points in DataFusion 56.
+
+### Semi-Structured Data (e.g. JSON)
+
+Parquet [Variant] is a type for storing semi-structured data, similar to JSON
+with an efficient binary encoding recently added to Apache Parquet and Apache
+Iceberg (see the [blog on the Parquet site] for more details).  The
+[datafusion-variant] crate extends DataFusion to add the `VARIANT` SQL type and
+Spark-compatible functions. Register it with a single call to `register_all` on
+a `SessionContext`:
+
+```rust
+let mut ctx = SessionContext::new();
+datafusion_variant::register_all(&mut ctx)?;
+```
+
+Then you can load, query, and store semi-structured data using SQL:
+
+```sql
+-- Load JSON data into a VARIANT column
+CREATE TABLE events AS
+SELECT parse_json(json_data) AS event
+FROM (VALUES
+    ('{"service":"api","status":200,"latency_ms":12,"tags":["cached"]}'),
+    ('{"service":"api","status":500,"latency_ms":85,"error":null}'),
+    ('{"service":"api","status":504,"error":"Somethings not working"}')
+) AS input(json_data);
+
+-- Extract fields with the Spark-style `:` operator
+SELECT event:service::VARCHAR, event:latency_ms::BIGINT
+FROM events
+WHERE event:status::BIGINT >= 400;
+```
+
+```text
++----------------------+-------------------------+
+| events.event:service | events.event:latency_ms |
++----------------------+-------------------------+
+| api                  | 85                      |
+| api                  | NULL                    |
++----------------------+-------------------------+
+```
+
+Variant columns can be written to Parquet files with `COPY` statements,
+including optionally [*shredding* commonly used fields][Variant shredding] 
into typed Parquet columns
+for faster queries:
+
+```sql
+COPY events TO 'shredded_events.parquet' STORED AS PARQUET
+OPTIONS ('variant.shredding' '{"event": "STRUCT<service VARCHAR, status 
BIGINT>"}');
+```
+
+The `datafusion-variant` crate is under active development, refer to the 
project
+documentation for more details. 
+
+<!-- TODO: thank the datafusion-variant contributors -->
+
+[Variant]: https://parquet.apache.org/docs/file-format/types/variantencoding/
+[blog on the Parquet site]: 
https://parquet.apache.org/blog/2026/02/27/variant-type-in-apache-parquet-for-semi-structured-data/
+[datafusion-variant]: https://datafusion-contrib.github.io/datafusion-variant/
+[Variant shredding]: 
https://parquet.apache.org/docs/file-format/types/variantshredding/
+[datafusion-variant roadmap]: 
https://github.com/datafusion-contrib/datafusion-variant/issues/69
+
+### Geospatial Data 

Review Comment:
   FYI @Xuanwo I was hoping to feature geodatafusion for this blog post -- do 
you think that is a good idea / would the datafusion 56 integration be ready 
relatively shortly after datafusion 56 is released?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to