alamb commented on code in PR #66:
URL: https://github.com/apache/datafusion-site/pull/66#discussion_r2035305854


##########
content/blog/2025-04-04-datafusion-userdefined-window-functions.md:
##########
@@ -0,0 +1,339 @@
+---
+layout: post
+title: User defined Window Functions in DataFusion 
+date: 2025-04-04
+author: Aditya Singh Rathore
+categories: [tutorial]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements.  See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License.  You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+## Introduction
+Window functions are a powerful feature in SQL, allowing for complex 
analytical computations over a subset of data. However, efficiently 
implementing them, especially sliding windows, can be quite challenging. With 
DataFusion's recent support for user-defined window functions , developers now 
have more flexibility and performance improvements at their disposal.
+
+In this post, we'll explore:
+
+- What window functions are and why they matter

Review Comment:
   I think it would be good if this list lined up with the section headings -- 
for example, the "## User Defined Window Functions" section probably should be 
in the outline 



##########
content/blog/2025-04-04-datafusion-userdefined-window-functions.md:
##########
@@ -0,0 +1,339 @@
+---
+layout: post
+title: User defined Window Functions in DataFusion 
+date: 2025-04-04
+author: Aditya Singh Rathore
+categories: [tutorial]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements.  See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License.  You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+## Introduction
+Window functions are a powerful feature in SQL, allowing for complex 
analytical computations over a subset of data. However, efficiently 
implementing them, especially sliding windows, can be quite challenging. With 
DataFusion's recent support for user-defined window functions , developers now 
have more flexibility and performance improvements at their disposal.
+
+In this post, we'll explore:
+
+- What window functions are and why they matter
+
+- Understanding sliding windows
+
+- The challenges of computing window aggregates efficiently
+
+- How DataFusion optimizes these computations
+
+- Alternative approaches and their trade-offs
+
+## Understanding Window Functions in SQL 
+
+Imagine you're analyzing sales data and want insights without losing the finer 
details. This is where **window functions** come into play. Unlike **GROUP 
BY**, which condenses data, window functions let you retain each row while 
performing calculations over a defined **range** —like having a moving lens 
over your dataset.

Review Comment:
   This is a great introduction -- concise enough that someone can understand 
what window functions are without going to some other article if they aren't 
already familar with window functions
   
   Maybe you could also add a link for further background, such as
   
   ```suggestion
   
   Imagine you're analyzing sales data and want insights without losing the 
finer details. This is where **[window functions]** come into play. Unlike 
**GROUP BY**, which condenses data, window functions let you retain each row 
while performing calculations over a defined **range** —like having a moving 
lens over your dataset.
   
   [window functions]: https://en.wikipedia.org/wiki/Window_function_(SQL)
   
   ```
   
   



##########
content/blog/2025-04-04-datafusion-userdefined-window-functions.md:
##########
@@ -0,0 +1,339 @@
+---
+layout: post
+title: User defined Window Functions in DataFusion 
+date: 2025-04-04
+author: Aditya Singh Rathore
+categories: [tutorial]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements.  See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License.  You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+## Introduction
+Window functions are a powerful feature in SQL, allowing for complex 
analytical computations over a subset of data. However, efficiently 
implementing them, especially sliding windows, can be quite challenging. With 
DataFusion's recent support for user-defined window functions , developers now 
have more flexibility and performance improvements at their disposal.
+
+In this post, we'll explore:
+
+- What window functions are and why they matter
+
+- Understanding sliding windows
+
+- The challenges of computing window aggregates efficiently
+
+- How DataFusion optimizes these computations
+
+- Alternative approaches and their trade-offs
+
+## Understanding Window Functions in SQL 
+
+Imagine you're analyzing sales data and want insights without losing the finer 
details. This is where **window functions** come into play. Unlike **GROUP 
BY**, which condenses data, window functions let you retain each row while 
performing calculations over a defined **range** —like having a moving lens 
over your dataset.
+
+Picture a business tracking daily sales. They need a running total to 
understand cumulative revenue trends without collapsing individual 
transactions. SQL makes this easy:
+```sql
+SELECT id, value, SUM(value) OVER (ORDER BY id) AS running_total
+FROM sales;
+```
+```text
+example:
++------------+--------+-------------------------------+
+|   Date     | Sales  | Rows Considered               |
++------------+--------+-------------------------------+
+| Jan 01     | 100    | [100]                         |
+| Jan 02     | 120    | [100, 120]                    |
+| Jan 03     | 130    | [100, 120, 130]               |
+| Jan 04     | 150    | [100, 120, 130, 150]          |
+| Jan 05     | 160    | [100, 120, 130, 150, 160]     |
+| Jan 06     | 180    | [100, 120, 130, 150, 160, 180]|
+| Jan 07     | 170    | [100, ..., 170] (7 days)      |
+| Jan 08     | 175    | [120, ..., 175]               |
++------------+--------+-------------------------------+
+A row-by-row representation of how a 7-day moving average includes the 
previous 6 days and the current one.

Review Comment:
   I suggest putting the caption below the text like: 
   
   ```suggestion
   **Figure 1**: A row-by-row representation of how a 7-day moving average 
includes the previous 6 days and the current one.
   ```



##########
content/blog/2025-04-04-datafusion-userdefined-window-functions.md:
##########
@@ -0,0 +1,339 @@
+---
+layout: post
+title: User defined Window Functions in DataFusion 
+date: 2025-04-04
+author: Aditya Singh Rathore
+categories: [tutorial]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements.  See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License.  You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+## Introduction
+Window functions are a powerful feature in SQL, allowing for complex 
analytical computations over a subset of data. However, efficiently 
implementing them, especially sliding windows, can be quite challenging. With 
DataFusion's recent support for user-defined window functions , developers now 
have more flexibility and performance improvements at their disposal.
+
+In this post, we'll explore:
+
+- What window functions are and why they matter
+
+- Understanding sliding windows
+
+- The challenges of computing window aggregates efficiently
+
+- How DataFusion optimizes these computations
+
+- Alternative approaches and their trade-offs
+
+## Understanding Window Functions in SQL 
+
+Imagine you're analyzing sales data and want insights without losing the finer 
details. This is where **window functions** come into play. Unlike **GROUP 
BY**, which condenses data, window functions let you retain each row while 
performing calculations over a defined **range** —like having a moving lens 
over your dataset.
+
+Picture a business tracking daily sales. They need a running total to 
understand cumulative revenue trends without collapsing individual 
transactions. SQL makes this easy:
+```sql
+SELECT id, value, SUM(value) OVER (ORDER BY id) AS running_total
+FROM sales;
+```
+```text
+example:
++------------+--------+-------------------------------+
+|   Date     | Sales  | Rows Considered               |
++------------+--------+-------------------------------+
+| Jan 01     | 100    | [100]                         |
+| Jan 02     | 120    | [100, 120]                    |
+| Jan 03     | 130    | [100, 120, 130]               |
+| Jan 04     | 150    | [100, 120, 130, 150]          |
+| Jan 05     | 160    | [100, 120, 130, 150, 160]     |
+| Jan 06     | 180    | [100, 120, 130, 150, 160, 180]|
+| Jan 07     | 170    | [100, ..., 170] (7 days)      |
+| Jan 08     | 175    | [120, ..., 175]               |
++------------+--------+-------------------------------+
+A row-by-row representation of how a 7-day moving average includes the 
previous 6 days and the current one.
+```
+
+
+This helps in analytical queries where we need cumulative sums, moving 
averages, or ranking without losing individual records.
+
+
+## User Defined Window Functions
+Built-in window functions serve many use cases, but sometimes custom logic is 
needed—for example:
+
+- Calculating moving averages with complex conditions
+
+- Implementing a custom ranking strategy
+
+- Tracking non-standard cumulative logic
+
+Thus, **User-Defined Window Functions (UDWFs)** allow developers to define 
their own behavior using a combination of SQL and *bit of logic*.
+
+Writing a user defined window function is slightly more complex than an 
aggregate function due
+to the variety of ways that window functions are called. I recommend reviewing 
the
+[online 
documentation](https://datafusion.apache.org/library-user-guide/adding-udfs.html#registering-a-window-udf)
+for a description of which functions need to be implemented. 
+
+## Understaing Sliding Window 
+
+Sliding windows define a **moving range** of data over which aggregations are 
computed. Unlike simple cumulative functions, these windows are dynamically 
updated as new data arrives.
+
+For instance, if we want a 7-day moving average of sales:
+
+```sql
+SELECT date, sales, 
+       AVG(sales) OVER (ORDER BY date ROWS BETWEEN 6 PRECEDING AND CURRENT 
ROW) AS moving_avg
+FROM sales;
+```
+Here, each row’s result is computed based on the last 7 days, making it 
computationally intensive as data grows.
+
+## Why Computing Sliding Windows Is Hard
+
+Imagine you’re at a café, and the barista is preparing coffee orders. If they 
made each cup from scratch without using pre-prepared ingredients, the process 
would be painfully slow. This is exactly the problem with naïve sliding window 
computations.
+
+Computing sliding windows efficiently is tricky because:
+
+- **High Computation Costs:** Just like making coffee from scratch for each 
customer, recalculating aggregates for every row is expensive.
+
+- **Data Shuffling:** In large distributed systems, data must often be 
shuffled between nodes, causing delays—like passing orders between multiple 
baristas who don’t communicate efficiently.
+
+- **State Management:** Keeping track of past computations is like remembering 
previous orders without writing them down—error-prone and inefficient.
+
+Many traditional query engines struggle to optimize these computations 
effectively, leading to sluggish performance.
+
+## How DataFusion Evaluates Window Functions Quickly
+In the world of big data, every millisecond counts. Imagine you’re analyzing 
stock market data, tracking sensor readings from millions of IoT devices, or 
crunching through massive customer logs—speed matters. This is where 
[DataFusion](https://datafusion.apache.org/) shines, making window function 
computations blazing fast. Let’s break down how it achieves this remarkable 
performance.

Review Comment:
   This feels a little over dramatic -- perhaps we can tone down the wording 
some
   
   In general I don't think DataFusion's window function performance is 
significantly better (or worse) than other engines . I think the main 
difference is how extensible it is. I'll try and comment more inline



##########
content/blog/2025-04-04-datafusion-userdefined-window-functions.md:
##########
@@ -0,0 +1,339 @@
+---
+layout: post
+title: User defined Window Functions in DataFusion 
+date: 2025-04-04
+author: Aditya Singh Rathore
+categories: [tutorial]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements.  See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License.  You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+## Introduction
+Window functions are a powerful feature in SQL, allowing for complex 
analytical computations over a subset of data. However, efficiently 
implementing them, especially sliding windows, can be quite challenging. With 
DataFusion's recent support for user-defined window functions , developers now 
have more flexibility and performance improvements at their disposal.
+
+In this post, we'll explore:
+
+- What window functions are and why they matter
+
+- Understanding sliding windows
+
+- The challenges of computing window aggregates efficiently
+
+- How DataFusion optimizes these computations
+
+- Alternative approaches and their trade-offs
+
+## Understanding Window Functions in SQL 
+
+Imagine you're analyzing sales data and want insights without losing the finer 
details. This is where **window functions** come into play. Unlike **GROUP 
BY**, which condenses data, window functions let you retain each row while 
performing calculations over a defined **range** —like having a moving lens 
over your dataset.
+
+Picture a business tracking daily sales. They need a running total to 
understand cumulative revenue trends without collapsing individual 
transactions. SQL makes this easy:
+```sql
+SELECT id, value, SUM(value) OVER (ORDER BY id) AS running_total
+FROM sales;
+```
+```text
+example:
++------------+--------+-------------------------------+
+|   Date     | Sales  | Rows Considered               |
++------------+--------+-------------------------------+
+| Jan 01     | 100    | [100]                         |
+| Jan 02     | 120    | [100, 120]                    |
+| Jan 03     | 130    | [100, 120, 130]               |
+| Jan 04     | 150    | [100, 120, 130, 150]          |
+| Jan 05     | 160    | [100, 120, 130, 150, 160]     |
+| Jan 06     | 180    | [100, 120, 130, 150, 160, 180]|
+| Jan 07     | 170    | [100, ..., 170] (7 days)      |
+| Jan 08     | 175    | [120, ..., 175]               |
++------------+--------+-------------------------------+
+A row-by-row representation of how a 7-day moving average includes the 
previous 6 days and the current one.
+```
+
+
+This helps in analytical queries where we need cumulative sums, moving 
averages, or ranking without losing individual records.
+
+
+## User Defined Window Functions
+Built-in window functions serve many use cases, but sometimes custom logic is 
needed—for example:
+
+- Calculating moving averages with complex conditions
+
+- Implementing a custom ranking strategy
+
+- Tracking non-standard cumulative logic
+
+Thus, **User-Defined Window Functions (UDWFs)** allow developers to define 
their own behavior using a combination of SQL and *bit of logic*.
+
+Writing a user defined window function is slightly more complex than an 
aggregate function due
+to the variety of ways that window functions are called. I recommend reviewing 
the
+[online 
documentation](https://datafusion.apache.org/library-user-guide/adding-udfs.html#registering-a-window-udf)
+for a description of which functions need to be implemented. 
+
+## Understaing Sliding Window 
+
+Sliding windows define a **moving range** of data over which aggregations are 
computed. Unlike simple cumulative functions, these windows are dynamically 
updated as new data arrives.
+
+For instance, if we want a 7-day moving average of sales:
+
+```sql
+SELECT date, sales, 
+       AVG(sales) OVER (ORDER BY date ROWS BETWEEN 6 PRECEDING AND CURRENT 
ROW) AS moving_avg
+FROM sales;
+```
+Here, each row’s result is computed based on the last 7 days, making it 
computationally intensive as data grows.
+
+## Why Computing Sliding Windows Is Hard
+
+Imagine you’re at a café, and the barista is preparing coffee orders. If they 
made each cup from scratch without using pre-prepared ingredients, the process 
would be painfully slow. This is exactly the problem with naïve sliding window 
computations.
+
+Computing sliding windows efficiently is tricky because:
+
+- **High Computation Costs:** Just like making coffee from scratch for each 
customer, recalculating aggregates for every row is expensive.
+
+- **Data Shuffling:** In large distributed systems, data must often be 
shuffled between nodes, causing delays—like passing orders between multiple 
baristas who don’t communicate efficiently.
+
+- **State Management:** Keeping track of past computations is like remembering 
previous orders without writing them down—error-prone and inefficient.
+
+Many traditional query engines struggle to optimize these computations 
effectively, leading to sluggish performance.
+
+## How DataFusion Evaluates Window Functions Quickly
+In the world of big data, every millisecond counts. Imagine you’re analyzing 
stock market data, tracking sensor readings from millions of IoT devices, or 
crunching through massive customer logs—speed matters. This is where 
[DataFusion](https://datafusion.apache.org/) shines, making window function 
computations blazing fast. Let’s break down how it achieves this remarkable 
performance.
+
+DataFusion now supports [user-defined window aggregates 
(UDWAs)](https://datafusion.apache.org/library-user-guide/adding-udfs.html), 
meaning you can bring your own aggregation logic and use it within a window 
function.
+
+For example, we will declare a user defined window function that computes a 
moving average.
+```sql

Review Comment:
   ```suggestion
   ```rust
   ```



##########
content/blog/2025-04-04-datafusion-userdefined-window-functions.md:
##########
@@ -0,0 +1,339 @@
+---
+layout: post
+title: User defined Window Functions in DataFusion 
+date: 2025-04-04
+author: Aditya Singh Rathore
+categories: [tutorial]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements.  See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License.  You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+## Introduction
+Window functions are a powerful feature in SQL, allowing for complex 
analytical computations over a subset of data. However, efficiently 
implementing them, especially sliding windows, can be quite challenging. With 
DataFusion's recent support for user-defined window functions , developers now 
have more flexibility and performance improvements at their disposal.
+
+In this post, we'll explore:
+
+- What window functions are and why they matter
+
+- Understanding sliding windows
+
+- The challenges of computing window aggregates efficiently
+
+- How DataFusion optimizes these computations
+
+- Alternative approaches and their trade-offs
+
+## Understanding Window Functions in SQL 
+
+Imagine you're analyzing sales data and want insights without losing the finer 
details. This is where **window functions** come into play. Unlike **GROUP 
BY**, which condenses data, window functions let you retain each row while 
performing calculations over a defined **range** —like having a moving lens 
over your dataset.
+
+Picture a business tracking daily sales. They need a running total to 
understand cumulative revenue trends without collapsing individual 
transactions. SQL makes this easy:
+```sql
+SELECT id, value, SUM(value) OVER (ORDER BY id) AS running_total
+FROM sales;
+```
+```text
+example:
++------------+--------+-------------------------------+
+|   Date     | Sales  | Rows Considered               |
++------------+--------+-------------------------------+
+| Jan 01     | 100    | [100]                         |
+| Jan 02     | 120    | [100, 120]                    |
+| Jan 03     | 130    | [100, 120, 130]               |
+| Jan 04     | 150    | [100, 120, 130, 150]          |
+| Jan 05     | 160    | [100, 120, 130, 150, 160]     |
+| Jan 06     | 180    | [100, 120, 130, 150, 160, 180]|
+| Jan 07     | 170    | [100, ..., 170] (7 days)      |
+| Jan 08     | 175    | [120, ..., 175]               |
++------------+--------+-------------------------------+
+A row-by-row representation of how a 7-day moving average includes the 
previous 6 days and the current one.
+```
+
+
+This helps in analytical queries where we need cumulative sums, moving 
averages, or ranking without losing individual records.
+
+
+## User Defined Window Functions
+Built-in window functions serve many use cases, but sometimes custom logic is 
needed—for example:
+
+- Calculating moving averages with complex conditions
+
+- Implementing a custom ranking strategy
+
+- Tracking non-standard cumulative logic
+
+Thus, **User-Defined Window Functions (UDWFs)** allow developers to define 
their own behavior using a combination of SQL and *bit of logic*.
+
+Writing a user defined window function is slightly more complex than an 
aggregate function due
+to the variety of ways that window functions are called. I recommend reviewing 
the
+[online 
documentation](https://datafusion.apache.org/library-user-guide/adding-udfs.html#registering-a-window-udf)
+for a description of which functions need to be implemented. 
+
+## Understaing Sliding Window 
+
+Sliding windows define a **moving range** of data over which aggregations are 
computed. Unlike simple cumulative functions, these windows are dynamically 
updated as new data arrives.
+
+For instance, if we want a 7-day moving average of sales:
+
+```sql
+SELECT date, sales, 
+       AVG(sales) OVER (ORDER BY date ROWS BETWEEN 6 PRECEDING AND CURRENT 
ROW) AS moving_avg
+FROM sales;
+```
+Here, each row’s result is computed based on the last 7 days, making it 
computationally intensive as data grows.
+
+## Why Computing Sliding Windows Is Hard
+
+Imagine you’re at a café, and the barista is preparing coffee orders. If they 
made each cup from scratch without using pre-prepared ingredients, the process 
would be painfully slow. This is exactly the problem with naïve sliding window 
computations.
+
+Computing sliding windows efficiently is tricky because:
+
+- **High Computation Costs:** Just like making coffee from scratch for each 
customer, recalculating aggregates for every row is expensive.
+
+- **Data Shuffling:** In large distributed systems, data must often be 
shuffled between nodes, causing delays—like passing orders between multiple 
baristas who don’t communicate efficiently.
+
+- **State Management:** Keeping track of past computations is like remembering 
previous orders without writing them down—error-prone and inefficient.
+
+Many traditional query engines struggle to optimize these computations 
effectively, leading to sluggish performance.

Review Comment:
   Unless we have examples of traditional query engines struggling, I would 
instead suggest we highlight how hard it is to optimize, and how many 
traditional engines don't permit user defined functions. 



##########
content/blog/2025-04-04-datafusion-userdefined-window-functions.md:
##########
@@ -0,0 +1,339 @@
+---
+layout: post
+title: User defined Window Functions in DataFusion 
+date: 2025-04-04
+author: Aditya Singh Rathore
+categories: [tutorial]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements.  See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License.  You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+## Introduction
+Window functions are a powerful feature in SQL, allowing for complex 
analytical computations over a subset of data. However, efficiently 
implementing them, especially sliding windows, can be quite challenging. With 
DataFusion's recent support for user-defined window functions , developers now 
have more flexibility and performance improvements at their disposal.
+
+In this post, we'll explore:
+
+- What window functions are and why they matter
+
+- Understanding sliding windows
+
+- The challenges of computing window aggregates efficiently
+
+- How DataFusion optimizes these computations
+
+- Alternative approaches and their trade-offs
+
+## Understanding Window Functions in SQL 
+
+Imagine you're analyzing sales data and want insights without losing the finer 
details. This is where **window functions** come into play. Unlike **GROUP 
BY**, which condenses data, window functions let you retain each row while 
performing calculations over a defined **range** —like having a moving lens 
over your dataset.
+
+Picture a business tracking daily sales. They need a running total to 
understand cumulative revenue trends without collapsing individual 
transactions. SQL makes this easy:
+```sql
+SELECT id, value, SUM(value) OVER (ORDER BY id) AS running_total
+FROM sales;
+```
+```text
+example:
++------------+--------+-------------------------------+
+|   Date     | Sales  | Rows Considered               |
++------------+--------+-------------------------------+
+| Jan 01     | 100    | [100]                         |
+| Jan 02     | 120    | [100, 120]                    |
+| Jan 03     | 130    | [100, 120, 130]               |
+| Jan 04     | 150    | [100, 120, 130, 150]          |
+| Jan 05     | 160    | [100, 120, 130, 150, 160]     |
+| Jan 06     | 180    | [100, 120, 130, 150, 160, 180]|
+| Jan 07     | 170    | [100, ..., 170] (7 days)      |
+| Jan 08     | 175    | [120, ..., 175]               |
++------------+--------+-------------------------------+
+A row-by-row representation of how a 7-day moving average includes the 
previous 6 days and the current one.
+```
+
+
+This helps in analytical queries where we need cumulative sums, moving 
averages, or ranking without losing individual records.
+
+
+## User Defined Window Functions
+Built-in window functions serve many use cases, but sometimes custom logic is 
needed—for example:
+
+- Calculating moving averages with complex conditions
+
+- Implementing a custom ranking strategy
+
+- Tracking non-standard cumulative logic
+
+Thus, **User-Defined Window Functions (UDWFs)** allow developers to define 
their own behavior using a combination of SQL and *bit of logic*.
+
+Writing a user defined window function is slightly more complex than an 
aggregate function due
+to the variety of ways that window functions are called. I recommend reviewing 
the
+[online 
documentation](https://datafusion.apache.org/library-user-guide/adding-udfs.html#registering-a-window-udf)
+for a description of which functions need to be implemented. 
+
+## Understaing Sliding Window 
+
+Sliding windows define a **moving range** of data over which aggregations are 
computed. Unlike simple cumulative functions, these windows are dynamically 
updated as new data arrives.
+
+For instance, if we want a 7-day moving average of sales:
+
+```sql
+SELECT date, sales, 
+       AVG(sales) OVER (ORDER BY date ROWS BETWEEN 6 PRECEDING AND CURRENT 
ROW) AS moving_avg
+FROM sales;
+```
+Here, each row’s result is computed based on the last 7 days, making it 
computationally intensive as data grows.
+
+## Why Computing Sliding Windows Is Hard
+
+Imagine you’re at a café, and the barista is preparing coffee orders. If they 
made each cup from scratch without using pre-prepared ingredients, the process 
would be painfully slow. This is exactly the problem with naïve sliding window 
computations.
+
+Computing sliding windows efficiently is tricky because:
+
+- **High Computation Costs:** Just like making coffee from scratch for each 
customer, recalculating aggregates for every row is expensive.
+
+- **Data Shuffling:** In large distributed systems, data must often be 
shuffled between nodes, causing delays—like passing orders between multiple 
baristas who don’t communicate efficiently.
+
+- **State Management:** Keeping track of past computations is like remembering 
previous orders without writing them down—error-prone and inefficient.
+
+Many traditional query engines struggle to optimize these computations 
effectively, leading to sluggish performance.
+
+## How DataFusion Evaluates Window Functions Quickly
+In the world of big data, every millisecond counts. Imagine you’re analyzing 
stock market data, tracking sensor readings from millions of IoT devices, or 
crunching through massive customer logs—speed matters. This is where 
[DataFusion](https://datafusion.apache.org/) shines, making window function 
computations blazing fast. Let’s break down how it achieves this remarkable 
performance.
+
+DataFusion now supports [user-defined window aggregates 
(UDWAs)](https://datafusion.apache.org/library-user-guide/adding-udfs.html), 
meaning you can bring your own aggregation logic and use it within a window 
function.
+
+For example, we will declare a user defined window function that computes a 
moving average.
+```sql
+use datafusion::arrow::{array::{ArrayRef, Float64Array, AsArray}, 
datatypes::Float64Type};
+use datafusion::logical_expr::{PartitionEvaluator};
+use datafusion::common::ScalarValue;
+use datafusion::error::Result;
+/// This implements the lowest level evaluation for a window function
+///
+/// It handles calculating the value of the window function for each
+/// distinct values of `PARTITION BY`
+#[derive(Clone, Debug)]
+struct MyPartitionEvaluator {}
+
+impl MyPartitionEvaluator {
+    fn new() -> Self {
+        Self {}
+    }
+}
+
+/// Different evaluation methods are called depending on the various
+/// settings of WindowUDF. This example uses the simplest and most
+/// general, `evaluate`. See `PartitionEvaluator` for the other more
+/// advanced uses.
+impl PartitionEvaluator for MyPartitionEvaluator {

Review Comment:
   In this blog form, it might be easier to read if you broke the comments out 
of the text so something lik e
   
   ```suggestion
   ```(end text)
   Different evaluation methods are called depending on the various
   settings of WindowUDF. In the first example, we  use the simplest and most
   general, `evaluate`. we will seee how to use `PartitionEvaluator` for the 
other more
    advanced uses later in the article.
    
    ```rust
   impl PartitionEvaluator for MyPartitionEvaluator {
   ```



##########
content/blog/2025-04-04-datafusion-userdefined-window-functions.md:
##########
@@ -0,0 +1,339 @@
+---
+layout: post
+title: User defined Window Functions in DataFusion 
+date: 2025-04-04
+author: Aditya Singh Rathore
+categories: [tutorial]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements.  See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License.  You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+## Introduction
+Window functions are a powerful feature in SQL, allowing for complex 
analytical computations over a subset of data. However, efficiently 
implementing them, especially sliding windows, can be quite challenging. With 
DataFusion's recent support for user-defined window functions , developers now 
have more flexibility and performance improvements at their disposal.
+
+In this post, we'll explore:
+
+- What window functions are and why they matter
+
+- Understanding sliding windows
+
+- The challenges of computing window aggregates efficiently
+
+- How DataFusion optimizes these computations
+
+- Alternative approaches and their trade-offs
+
+## Understanding Window Functions in SQL 
+
+Imagine you're analyzing sales data and want insights without losing the finer 
details. This is where **window functions** come into play. Unlike **GROUP 
BY**, which condenses data, window functions let you retain each row while 
performing calculations over a defined **range** —like having a moving lens 
over your dataset.
+
+Picture a business tracking daily sales. They need a running total to 
understand cumulative revenue trends without collapsing individual 
transactions. SQL makes this easy:
+```sql
+SELECT id, value, SUM(value) OVER (ORDER BY id) AS running_total
+FROM sales;
+```
+```text
+example:
++------------+--------+-------------------------------+
+|   Date     | Sales  | Rows Considered               |
++------------+--------+-------------------------------+
+| Jan 01     | 100    | [100]                         |
+| Jan 02     | 120    | [100, 120]                    |
+| Jan 03     | 130    | [100, 120, 130]               |
+| Jan 04     | 150    | [100, 120, 130, 150]          |
+| Jan 05     | 160    | [100, 120, 130, 150, 160]     |
+| Jan 06     | 180    | [100, 120, 130, 150, 160, 180]|
+| Jan 07     | 170    | [100, ..., 170] (7 days)      |
+| Jan 08     | 175    | [120, ..., 175]               |
++------------+--------+-------------------------------+
+A row-by-row representation of how a 7-day moving average includes the 
previous 6 days and the current one.
+```
+
+
+This helps in analytical queries where we need cumulative sums, moving 
averages, or ranking without losing individual records.
+
+
+## User Defined Window Functions
+Built-in window functions serve many use cases, but sometimes custom logic is 
needed—for example:
+
+- Calculating moving averages with complex conditions
+
+- Implementing a custom ranking strategy
+
+- Tracking non-standard cumulative logic
+
+Thus, **User-Defined Window Functions (UDWFs)** allow developers to define 
their own behavior using a combination of SQL and *bit of logic*.

Review Comment:
   ```suggestion
   Thus, **User-Defined Window Functions (UDWFs)** allow developers to define 
their own behavior while allowing DataFusion to handle the calculations of the  
windows and grouping specified in the `OVER` clause
   ```



##########
content/blog/2025-04-04-datafusion-userdefined-window-functions.md:
##########
@@ -0,0 +1,339 @@
+---
+layout: post
+title: User defined Window Functions in DataFusion 
+date: 2025-04-04
+author: Aditya Singh Rathore
+categories: [tutorial]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements.  See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License.  You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+## Introduction
+Window functions are a powerful feature in SQL, allowing for complex 
analytical computations over a subset of data. However, efficiently 
implementing them, especially sliding windows, can be quite challenging. With 
DataFusion's recent support for user-defined window functions , developers now 
have more flexibility and performance improvements at their disposal.

Review Comment:
   ```suggestion
   Window functions are a powerful feature in SQL, allowing for complex 
analytical computations over a subset of data. However, efficiently 
implementing them, especially sliding windows, can be quite challenging. With 
[Apache DataFusion]'s user-defined window functions, developers can easily take 
advantage of all the effort put into DataFusion's implementation.
   ```



##########
content/blog/2025-04-04-datafusion-userdefined-window-functions.md:
##########
@@ -0,0 +1,339 @@
+---
+layout: post
+title: User defined Window Functions in DataFusion 
+date: 2025-04-04
+author: Aditya Singh Rathore
+categories: [tutorial]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements.  See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License.  You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+## Introduction
+Window functions are a powerful feature in SQL, allowing for complex 
analytical computations over a subset of data. However, efficiently 
implementing them, especially sliding windows, can be quite challenging. With 
DataFusion's recent support for user-defined window functions , developers now 
have more flexibility and performance improvements at their disposal.
+
+In this post, we'll explore:
+
+- What window functions are and why they matter
+
+- Understanding sliding windows
+
+- The challenges of computing window aggregates efficiently
+
+- How DataFusion optimizes these computations
+
+- Alternative approaches and their trade-offs
+
+## Understanding Window Functions in SQL 
+
+Imagine you're analyzing sales data and want insights without losing the finer 
details. This is where **window functions** come into play. Unlike **GROUP 
BY**, which condenses data, window functions let you retain each row while 
performing calculations over a defined **range** —like having a moving lens 
over your dataset.
+
+Picture a business tracking daily sales. They need a running total to 
understand cumulative revenue trends without collapsing individual 
transactions. SQL makes this easy:
+```sql
+SELECT id, value, SUM(value) OVER (ORDER BY id) AS running_total
+FROM sales;
+```
+```text
+example:
++------------+--------+-------------------------------+
+|   Date     | Sales  | Rows Considered               |
++------------+--------+-------------------------------+
+| Jan 01     | 100    | [100]                         |
+| Jan 02     | 120    | [100, 120]                    |
+| Jan 03     | 130    | [100, 120, 130]               |
+| Jan 04     | 150    | [100, 120, 130, 150]          |
+| Jan 05     | 160    | [100, 120, 130, 150, 160]     |
+| Jan 06     | 180    | [100, 120, 130, 150, 160, 180]|
+| Jan 07     | 170    | [100, ..., 170] (7 days)      |
+| Jan 08     | 175    | [120, ..., 175]               |
++------------+--------+-------------------------------+
+A row-by-row representation of how a 7-day moving average includes the 
previous 6 days and the current one.
+```
+
+
+This helps in analytical queries where we need cumulative sums, moving 
averages, or ranking without losing individual records.
+
+
+## User Defined Window Functions
+Built-in window functions serve many use cases, but sometimes custom logic is 
needed—for example:
+
+- Calculating moving averages with complex conditions
+
+- Implementing a custom ranking strategy
+
+- Tracking non-standard cumulative logic
+
+Thus, **User-Defined Window Functions (UDWFs)** allow developers to define 
their own behavior using a combination of SQL and *bit of logic*.
+
+Writing a user defined window function is slightly more complex than an 
aggregate function due
+to the variety of ways that window functions are called. I recommend reviewing 
the
+[online 
documentation](https://datafusion.apache.org/library-user-guide/adding-udfs.html#registering-a-window-udf)
+for a description of which functions need to be implemented. 
+
+## Understaing Sliding Window 
+
+Sliding windows define a **moving range** of data over which aggregations are 
computed. Unlike simple cumulative functions, these windows are dynamically 
updated as new data arrives.
+
+For instance, if we want a 7-day moving average of sales:
+
+```sql
+SELECT date, sales, 
+       AVG(sales) OVER (ORDER BY date ROWS BETWEEN 6 PRECEDING AND CURRENT 
ROW) AS moving_avg
+FROM sales;
+```
+Here, each row’s result is computed based on the last 7 days, making it 
computationally intensive as data grows.

Review Comment:
   I am not sure I would say a 7 day window is computationally expensive -- if 
you changed the example to be `UNBOUNDED PRECEDING` you could make the point 
that the window size kept growing. Or 365 days maybe would make the point 
better?



##########
content/blog/2025-04-04-datafusion-userdefined-window-functions.md:
##########
@@ -0,0 +1,339 @@
+---
+layout: post
+title: User defined Window Functions in DataFusion 
+date: 2025-04-04
+author: Aditya Singh Rathore
+categories: [tutorial]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements.  See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License.  You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+## Introduction
+Window functions are a powerful feature in SQL, allowing for complex 
analytical computations over a subset of data. However, efficiently 
implementing them, especially sliding windows, can be quite challenging. With 
DataFusion's recent support for user-defined window functions , developers now 
have more flexibility and performance improvements at their disposal.
+
+In this post, we'll explore:
+
+- What window functions are and why they matter
+
+- Understanding sliding windows
+
+- The challenges of computing window aggregates efficiently
+
+- How DataFusion optimizes these computations
+
+- Alternative approaches and their trade-offs
+
+## Understanding Window Functions in SQL 
+
+Imagine you're analyzing sales data and want insights without losing the finer 
details. This is where **window functions** come into play. Unlike **GROUP 
BY**, which condenses data, window functions let you retain each row while 
performing calculations over a defined **range** —like having a moving lens 
over your dataset.
+
+Picture a business tracking daily sales. They need a running total to 
understand cumulative revenue trends without collapsing individual 
transactions. SQL makes this easy:
+```sql
+SELECT id, value, SUM(value) OVER (ORDER BY id) AS running_total
+FROM sales;
+```
+```text
+example:
++------------+--------+-------------------------------+
+|   Date     | Sales  | Rows Considered               |
++------------+--------+-------------------------------+
+| Jan 01     | 100    | [100]                         |
+| Jan 02     | 120    | [100, 120]                    |
+| Jan 03     | 130    | [100, 120, 130]               |
+| Jan 04     | 150    | [100, 120, 130, 150]          |
+| Jan 05     | 160    | [100, 120, 130, 150, 160]     |
+| Jan 06     | 180    | [100, 120, 130, 150, 160, 180]|
+| Jan 07     | 170    | [100, ..., 170] (7 days)      |
+| Jan 08     | 175    | [120, ..., 175]               |
++------------+--------+-------------------------------+
+A row-by-row representation of how a 7-day moving average includes the 
previous 6 days and the current one.
+```
+
+
+This helps in analytical queries where we need cumulative sums, moving 
averages, or ranking without losing individual records.
+
+
+## User Defined Window Functions
+Built-in window functions serve many use cases, but sometimes custom logic is 
needed—for example:

Review Comment:
   I suggest adding a link here to what functions are built in and giving some 
examples. Something like
   
   ```suggestion
   DataFusion's Built-in window functions such as `first_value`, `rank` and 
`row_number` serve many common use cases, but sometimes custom logic is 
needed—for example:
   ```
   
   
   And then link `Built-in window functions to 
https://datafusion.apache.org/user-guide/sql/window_functions.html



##########
content/blog/2025-04-04-datafusion-userdefined-window-functions.md:
##########
@@ -0,0 +1,154 @@
+---
+layout: post
+title: User defined Window Functions in DataFusion 
+date: 2025-04-04
+author: Aditya Singh Rathore
+categories: [tutorial]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements.  See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License.  You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+## Introduction
+Window functions are a powerful feature in SQL, allowing for complex 
analytical computations over a subset of data. However, efficiently 
implementing them, especially sliding windows, can be quite challenging. With 
DataFusion's recent support for user-defined window functions , developers now 
have more flexibility and performance improvements at their disposal.
+
+In this post, we'll explore:
+
+- What window functions are and why they matter
+
+- Understanding sliding windows
+
+- The challenges of computing window aggregates efficiently
+
+- How DataFusion optimizes these computations
+
+- Alternative approaches and their trade-offs
+
+## Understanding Window Functions in SQL 
+
+Imagine you're analyzing sales data and want insights without losing the finer 
details. This is where **window functions** come into play. Unlike **GROUP 
BY**, which condenses data, window functions let you retain each row while 
performing calculations over a defined **range** —like having a moving lens 
over your dataset.
+
+Picture a business tracking daily sales. They need a running total to 
understand cumulative revenue trends without collapsing individual 
transactions. SQL makes this easy:
+```sql
+SELECT id, value, SUM(value) OVER (ORDER BY id) AS running_total
+FROM sales;
+```
+This helps in analytical queries where we need cumulative sums, moving 
averages, or ranking without losing individual records.
+
+
+## User Defined Window Functions
+
+Writing a user defined window function is slightly more complex than an 
aggregate function due
+to the variety of ways that window functions are called. I recommend reviewing 
the
+[online 
documentation](https://datafusion.apache.org/library-user-guide/adding-udfs.html)
+for a description of which functions need to be implemented. The details of 
how to implement
+these generally follow the same patterns as described above for aggregate 
functions.
+
+## Understaing Sliding Window 
+
+Sliding windows define a **moving range** of data over which aggregations are 
computed. Unlike simple cumulative functions, these windows are dynamically 
updated as new data arrives.
+
+For instance, if we want a 7-day moving average of sales:
+
+```sql
+SELECT date, sales, 
+       AVG(sales) OVER (ORDER BY date ROWS BETWEEN 6 PRECEDING AND CURRENT 
ROW) AS moving_avg
+FROM sales;
+```
+Here, each row’s result is computed based on the last 7 days, making it 
computationally intensive as data grows.
+
+## Why Computing Sliding Windows Is Hard
+
+Imagine you’re at a café, and the barista is preparing coffee orders. If they 
made each cup from scratch without using pre-prepared ingredients, the process 
would be painfully slow. This is exactly the problem with naïve sliding window 
computations.
+
+Computing sliding windows efficiently is tricky because:
+
+- **High Computation Costs:** Just like making coffee from scratch for each 
customer, recalculating aggregates for every row is expensive.
+
+- **Data Shuffling:** In large distributed systems, data must often be 
shuffled between nodes, causing delays—like passing orders between multiple 
baristas who don’t communicate efficiently.
+
+- **State Management:** Keeping track of past computations is like remembering 
previous orders without writing them down—error-prone and inefficient.
+
+Many traditional query engines struggle to optimize these computations 
effectively, leading to sluggish performance.
+
+## How DataFusion Making it fast
+In the world of big data, every millisecond counts. Imagine you’re analyzing 
stock market data, tracking sensor readings from millions of IoT devices, or 
crunching through massive customer logs—speed matters. This is where 
[DataFusion](https://datafusion.apache.org/) shines, making window function 
computations blazing fast. Let’s break down how it achieves this remarkable 
performance.
+
+DataFusion now supports [user-defined window aggregates 
(UDWAs)](https://datafusion.apache.org/library-user-guide/adding-udfs.html), 
meaning you can bring your own aggregation logic and use it within a window 
function.
+
+```sql
+let my_udwa = create_my_custom_udwa();
+ctx.register_udaf("my_moving_avg", my_udwa);
+
+// Then use in SQL:
+SELECT
+  user_id,
+  my_moving_avg(score) OVER (
+    PARTITION BY user_id
+    ORDER BY game_time
+    ROWS BETWEEN 2 PRECEDING AND CURRENT ROW
+  ) AS moving_score
+FROM leaderboard;
+```
+This gives you full flexibility to build **domain-specific logic** that plugs 
seamlessly into DataFusion’s engine — all without sacrificing performance.
+
+
+## Performance Gains
+
+To demonstrate efficiency, we benchmarked a 1-million row dataset with a 
sliding window aggregate.

Review Comment:
   I suggest we avoid comparisons with other systems and simply focus on 
DataFusion's own performance. 
   
   



##########
content/blog/2025-04-04-datafusion-userdefined-window-functions.md:
##########
@@ -0,0 +1,339 @@
+---
+layout: post
+title: User defined Window Functions in DataFusion 
+date: 2025-04-04
+author: Aditya Singh Rathore
+categories: [tutorial]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements.  See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License.  You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+## Introduction
+Window functions are a powerful feature in SQL, allowing for complex 
analytical computations over a subset of data. However, efficiently 
implementing them, especially sliding windows, can be quite challenging. With 
DataFusion's recent support for user-defined window functions , developers now 
have more flexibility and performance improvements at their disposal.
+
+In this post, we'll explore:
+
+- What window functions are and why they matter
+
+- Understanding sliding windows
+
+- The challenges of computing window aggregates efficiently
+
+- How DataFusion optimizes these computations
+
+- Alternative approaches and their trade-offs
+
+## Understanding Window Functions in SQL 
+
+Imagine you're analyzing sales data and want insights without losing the finer 
details. This is where **window functions** come into play. Unlike **GROUP 
BY**, which condenses data, window functions let you retain each row while 
performing calculations over a defined **range** —like having a moving lens 
over your dataset.
+
+Picture a business tracking daily sales. They need a running total to 
understand cumulative revenue trends without collapsing individual 
transactions. SQL makes this easy:
+```sql
+SELECT id, value, SUM(value) OVER (ORDER BY id) AS running_total
+FROM sales;
+```
+```text
+example:
++------------+--------+-------------------------------+
+|   Date     | Sales  | Rows Considered               |
++------------+--------+-------------------------------+
+| Jan 01     | 100    | [100]                         |
+| Jan 02     | 120    | [100, 120]                    |
+| Jan 03     | 130    | [100, 120, 130]               |
+| Jan 04     | 150    | [100, 120, 130, 150]          |
+| Jan 05     | 160    | [100, 120, 130, 150, 160]     |
+| Jan 06     | 180    | [100, 120, 130, 150, 160, 180]|
+| Jan 07     | 170    | [100, ..., 170] (7 days)      |
+| Jan 08     | 175    | [120, ..., 175]               |
++------------+--------+-------------------------------+
+A row-by-row representation of how a 7-day moving average includes the 
previous 6 days and the current one.
+```
+
+
+This helps in analytical queries where we need cumulative sums, moving 
averages, or ranking without losing individual records.
+
+
+## User Defined Window Functions
+Built-in window functions serve many use cases, but sometimes custom logic is 
needed—for example:
+
+- Calculating moving averages with complex conditions

Review Comment:
   ```suggestion
   - Calculating moving averages with complex conditions (e.g. exponential 
averages, integrals, etc)
   ```



##########
content/blog/2025-04-04-datafusion-userdefined-window-functions.md:
##########
@@ -0,0 +1,339 @@
+---
+layout: post
+title: User defined Window Functions in DataFusion 
+date: 2025-04-04
+author: Aditya Singh Rathore
+categories: [tutorial]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements.  See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License.  You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+## Introduction
+Window functions are a powerful feature in SQL, allowing for complex 
analytical computations over a subset of data. However, efficiently 
implementing them, especially sliding windows, can be quite challenging. With 
DataFusion's recent support for user-defined window functions , developers now 
have more flexibility and performance improvements at their disposal.
+
+In this post, we'll explore:
+
+- What window functions are and why they matter
+
+- Understanding sliding windows
+
+- The challenges of computing window aggregates efficiently
+
+- How DataFusion optimizes these computations
+
+- Alternative approaches and their trade-offs
+
+## Understanding Window Functions in SQL 
+
+Imagine you're analyzing sales data and want insights without losing the finer 
details. This is where **window functions** come into play. Unlike **GROUP 
BY**, which condenses data, window functions let you retain each row while 
performing calculations over a defined **range** —like having a moving lens 
over your dataset.
+
+Picture a business tracking daily sales. They need a running total to 
understand cumulative revenue trends without collapsing individual 
transactions. SQL makes this easy:
+```sql
+SELECT id, value, SUM(value) OVER (ORDER BY id) AS running_total
+FROM sales;
+```
+```text
+example:
++------------+--------+-------------------------------+
+|   Date     | Sales  | Rows Considered               |
++------------+--------+-------------------------------+
+| Jan 01     | 100    | [100]                         |
+| Jan 02     | 120    | [100, 120]                    |
+| Jan 03     | 130    | [100, 120, 130]               |
+| Jan 04     | 150    | [100, 120, 130, 150]          |
+| Jan 05     | 160    | [100, 120, 130, 150, 160]     |
+| Jan 06     | 180    | [100, 120, 130, 150, 160, 180]|
+| Jan 07     | 170    | [100, ..., 170] (7 days)      |
+| Jan 08     | 175    | [120, ..., 175]               |
++------------+--------+-------------------------------+
+A row-by-row representation of how a 7-day moving average includes the 
previous 6 days and the current one.
+```
+
+
+This helps in analytical queries where we need cumulative sums, moving 
averages, or ranking without losing individual records.
+
+
+## User Defined Window Functions
+Built-in window functions serve many use cases, but sometimes custom logic is 
needed—for example:
+
+- Calculating moving averages with complex conditions
+
+- Implementing a custom ranking strategy
+
+- Tracking non-standard cumulative logic
+
+Thus, **User-Defined Window Functions (UDWFs)** allow developers to define 
their own behavior using a combination of SQL and *bit of logic*.
+
+Writing a user defined window function is slightly more complex than an 
aggregate function due
+to the variety of ways that window functions are called. I recommend reviewing 
the
+[online 
documentation](https://datafusion.apache.org/library-user-guide/adding-udfs.html#registering-a-window-udf)
+for a description of which functions need to be implemented. 
+
+## Understaing Sliding Window 
+
+Sliding windows define a **moving range** of data over which aggregations are 
computed. Unlike simple cumulative functions, these windows are dynamically 
updated as new data arrives.
+
+For instance, if we want a 7-day moving average of sales:
+
+```sql
+SELECT date, sales, 
+       AVG(sales) OVER (ORDER BY date ROWS BETWEEN 6 PRECEDING AND CURRENT 
ROW) AS moving_avg
+FROM sales;
+```
+Here, each row’s result is computed based on the last 7 days, making it 
computationally intensive as data grows.
+
+## Why Computing Sliding Windows Is Hard
+
+Imagine you’re at a café, and the barista is preparing coffee orders. If they 
made each cup from scratch without using pre-prepared ingredients, the process 
would be painfully slow. This is exactly the problem with naïve sliding window 
computations.
+
+Computing sliding windows efficiently is tricky because:
+
+- **High Computation Costs:** Just like making coffee from scratch for each 
customer, recalculating aggregates for every row is expensive.
+
+- **Data Shuffling:** In large distributed systems, data must often be 
shuffled between nodes, causing delays—like passing orders between multiple 
baristas who don’t communicate efficiently.
+
+- **State Management:** Keeping track of past computations is like remembering 
previous orders without writing them down—error-prone and inefficient.
+
+Many traditional query engines struggle to optimize these computations 
effectively, leading to sluggish performance.
+
+## How DataFusion Evaluates Window Functions Quickly
+In the world of big data, every millisecond counts. Imagine you’re analyzing 
stock market data, tracking sensor readings from millions of IoT devices, or 
crunching through massive customer logs—speed matters. This is where 
[DataFusion](https://datafusion.apache.org/) shines, making window function 
computations blazing fast. Let’s break down how it achieves this remarkable 
performance.
+
+DataFusion now supports [user-defined window aggregates 
(UDWAs)](https://datafusion.apache.org/library-user-guide/adding-udfs.html), 
meaning you can bring your own aggregation logic and use it within a window 
function.
+
+For example, we will declare a user defined window function that computes a 
moving average.
+```sql
+use datafusion::arrow::{array::{ArrayRef, Float64Array, AsArray}, 
datatypes::Float64Type};
+use datafusion::logical_expr::{PartitionEvaluator};
+use datafusion::common::ScalarValue;
+use datafusion::error::Result;
+/// This implements the lowest level evaluation for a window function
+///
+/// It handles calculating the value of the window function for each
+/// distinct values of `PARTITION BY`
+#[derive(Clone, Debug)]
+struct MyPartitionEvaluator {}
+
+impl MyPartitionEvaluator {
+    fn new() -> Self {
+        Self {}
+    }
+}
+
+/// Different evaluation methods are called depending on the various
+/// settings of WindowUDF. This example uses the simplest and most
+/// general, `evaluate`. See `PartitionEvaluator` for the other more
+/// advanced uses.
+impl PartitionEvaluator for MyPartitionEvaluator {
+    /// Tell DataFusion the window function varies based on the value
+    /// of the window frame.
+    fn uses_window_frame(&self) -> bool {
+        true
+    }
+
+    /// This function is called once per input row.
+    ///
+    /// `range`specifies which indexes of `values` should be
+    /// considered for the calculation.
+    ///
+    /// Note this is the SLOWEST, but simplest, way to evaluate a
+    /// window function. It is much faster to implement
+    /// evaluate_all or evaluate_all_with_rank, if possible
+    fn evaluate(
+        &mut self,
+        values: &[ArrayRef],
+        range: &std::ops::Range<usize>,
+    ) -> Result<ScalarValue> {
+        // Again, the input argument is an array of floating
+        // point numbers to calculate a moving average
+        let arr: &Float64Array = 
values[0].as_ref().as_primitive::<Float64Type>();
+
+        let range_len = range.end - range.start;
+
+        // our smoothing function will average all the values in the
+        let output = if range_len > 0 {
+            let sum: f64 = 
arr.values().iter().skip(range.start).take(range_len).sum();
+            Some(sum / range_len as f64)
+        } else {
+            None
+        };
+
+        Ok(ScalarValue::Float64(output))
+    }
+}
+
+/// Create a `PartitionEvaluator` to evaluate this function on a new
+/// partition.
+fn make_partition_evaluator() -> Result<Box<dyn PartitionEvaluator>> {
+    Ok(Box::new(MyPartitionEvaluator::new()))
+}
+```
+### Registering a Window UDF
+To register a Window UDF, you need to wrap the function implementation in a 
`WindowUDF` struct and then register it with the `SessionContext`. DataFusion 
provides the `create_udwf` helper functions to make this easier. There is a 
lower level API with more functionality but is more complex, that is documented 
in 
[advanced_udwf.rs](https://github.com/apache/datafusion/blob/main/datafusion-examples/examples/advanced_udwf.rs).

Review Comment:
   It would be nice if you could add links to the doc for `WindowUDF`, 
`SessionContext`, etc here



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org
For additional commands, e-mail: github-h...@datafusion.apache.org

Reply via email to