ashb opened a new pull request, #47061:
URL: https://github.com/apache/airflow/pull/47061
There was a number of issues here that prevented us from correctly resuming a
task after it's trigger fired.
First off, only ever ran the execute method and we didn't respect
next_method.
So we added that info the the TI context we send in the server in response
to the `.../run` endpoint.
Once we'd fixed that, we then ran into the net problem that we were
incorrectly setting the same value for trigger_kwargs (which are the kwargs
we
pass to the trigger constructor) and to the next_kwargs (which are the kwargs
to use when resuming the task) -- they needed to be different. This involved
adding the new field (`kwargs`) onto the TIDeferredStatePayload.
The next complication after that was the "ExtendedJSON" type on the
next_kwargs column of TI: this is a type decorator that automatically applies
the BaseSerialization encode/decode step (__var and __type etc). The problem
with that is that we need to do the serialization on the client in order to
send a JSON HTTP request, so we don't want to encode _again_ on the server
ideally. I was able to do that easily on the write/update side but not so
easily on the read side -- there I left a comment and for now we will hae
SQLA
decode it for us, and then we have to encode it again. Not the best, but not
a
disaster either.
The other change I did here was to have the DeferTask automatically apply the
serde encoding when serializing, just so that there are fewer places in the
code that need to be aware of that detail (So the Task subprocess will encode
it before making the request toe the Supervisor, and in the Supervisor it
will be
kept encoded and passed along as is to it's HTTP request). This means you can
once again pass datetime objects to a trigger "natively", not only strings.
For consistency with the user facing code I renamed `next_method` on
`DeferTask` message to `method_name`. I'm not sure this really makes sense on
in the API request to the API server though.
Closes #47013
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->
<!--
Thank you for contributing! Please make sure that your code changes
are covered with tests. And in case of new features or big changes
remember to adjust the documentation.
Feel free to ping committers for the review!
In case of an existing issue, reference it using one of the following:
closes: #ISSUE
related: #ISSUE
How to write a good git commit message:
http://chris.beams.io/posts/git-commit/
-->
<!-- Please keep an empty line above the dashes. -->
---
**^ Add meaningful description above**
Read the **[Pull Request
Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines)**
for more information.
In case of fundamental code changes, an Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals))
is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party
License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in a
newsfragment file, named `{pr_number}.significant.rst` or
`{issue_number}.significant.rst`, in
[newsfragments](https://github.com/apache/airflow/tree/main/newsfragments).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
