Hi,

*1. incremental loads*

There is mention of incremental processing / incremental loads in the doc.

E.g.

This is particularly useful for large datasets that need to be processed
> incrementally or updated periodically.


And

> Facilitating Incremental Processing: Many modern data processing
> strategies rely on incremental updates


But there are no examples re how this solves for that use case.

I think it's actually not good to think of or talk about incremental loads
as "partitioned".

Let me explain.

An incremental load might track an `updated_at` column.  The data it
processes is the data with an updated `updated_at` column.  But you would
not be correct in calling this a partition of data.  Because when the data
is updated again, it would now be in another partition.  That's not what
partitioning is.

If this is supposed to solve for incremental loads, I think an example is
needed.  If it's not, let's call it out explicitly and say this is not
solving for incremental loads.

*2. support for tasks*

I see this is specific to tasks defined with the asset syntax.  What's the
story with "normal" dags and tasks e.g. with task flow or classic
operators.  Is this AIP adding support only for assets?  Is there some plan
for that?

Reply via email to