henry3260 opened a new pull request, #77092:
URL: https://github.com/apache/airflow/pull/77092
## Why
A Go task can return a value that a downstream Go task cannot decode,
although both use the same Go type.
The Go SDK writes a task's return value with msgpack. Since #70209 the
encoder takes field names from `json` struct tags, but nothing else from
`encoding/json`: a `,string` tag option and a `MarshalJSON` method have no
effect. Argument binding (`pkg/binding`) and
`TaskStateStore.UnmarshalJSONValue` read the value back with `encoding/json`,
which does apply them.
```go
type Order struct {
ID int64 `json:"id,string"`
}
```
- `encoding/json` writes `{"id": "123"}`; the SDK stored `{"id": 123}`.
- The upstream task succeeds, because the stored value is valid JSON.
- The downstream task fails with `json: cannot unmarshal number into Go
struct field Order.id of type int64`.
- Every retry fails the same way, because each retry reads the same stored
value.
A type with `MarshalJSON`/`UnmarshalJSON` fails in the same way.
related: #70209, #75601
## What
- **`go-sdk/pkg/execution/client.go`**: `PushXCom` and `TaskStateStore.Set`
now encode a value the way `encoding/json` writes it and then pack that as
msgpack (`encodeJSONValue`, replacing `validateJSONRepresentable`).
`airflow.Literal` and a `TriggerDagRun` conf already copy their values by way
of JSON like this.
- **`go-sdk/internal/jsonvalue`**: `resolveNumbers` moved here unchanged
from `go-sdk/airflow`, so `pkg/execution` can use it.
- **Docs**: the `PushXCom` and `Set` godoc and the task state section of
`go.rst`.
### Behavior changes
| Value | Before | After |
| --- | --- | --- |
| a `,string` field, a `MarshalJSON` or `MarshalText` type | stored in a
shape its own type could not decode | decodes back into its type |
| `time.Time`, `uuid.UUID`, `[]byte` | refused before sending (#75601) |
stored as the string `encoding/json` gives them; decode back into the same type
|
| a `float64` with no fractional part, such as `5.0` | stored as a float |
stored as the integer `5`, since JSON text does not record that it was a float |
| a struct with only `msgpack:` tags | keys from the `msgpack` tags | keys
from the Go field names; use `json` tags |
| struct keys | declaration order | sorted, so the same value is always
stored the same way |
| a value with a pointer cycle | crashed the task process with a stack
overflow | returns an error |
| a non-finite float, an integer past 64 bits | refused | still refused |
Two of these are worth a reviewer's attention:
- **Parity with the other SDKs.** The Go task state store now accepts a
`time.Time` and bytes, which the Python task state store still refuses
(`SetTaskStateStore.value` is a pydantic `JsonValue`). The Java SDK still
refuses a byte array. Only a Go task reads the state a Go task stores, and it
reads a time or bytes back into the same type.
- **Whole-number floats.** Typed reads are unaffected: binding into a
`float64` parameter and `UnmarshalJSONValue` into a `float64` still give a
`float64`. `Get` and `GetXCom`, which return `any`, now give an `int64` for
`5.0`, and a Python consumer sees an `int`.
### Cost
The value is now encoded twice. Client-side encoding only (no supervisor,
HTTP or database), median of 5 runs, old and new encoder in the same benchmark
binary:
| Value | Before | After |
| --- | --- | --- |
| small map (3 keys) | 0.38 µs | 1.59 µs |
| one struct (4 fields) | 0.49 µs | 1.91 µs |
| 1,000 structs | 0.44 ms, 0.7 MB | 1.73 ms, 1.7 MB |
| 100,000 ints | 2.85 ms, 2.9 MB | 18.2 ms, 24.4 MB |
| control (`decodeFrame`, untouched) | 125 ns | 125 ns |
A typical push costs about 1 µs more. A very large return value costs
several times more time and memory to encode.
### Tests
- `TestCoordinatorClientPushXComRoundTripsThroughBinding` pushes a value and
decodes it with `binding.DecodeLiteral`. The `,string` and `MarshalJSON` rows
fail on `main` with the errors above.
- `TestTaskStateStoreSetRoundTripsThroughUnmarshalJSONValue` does the same
through `Set` and `UnmarshalJSONValue`.
- The tests that expected a `time.Time`, a `uuid.UUID` and a `[]byte` to be
refused now expect them to round-trip.
Checks run locally, all passing: `go test ./...` and `go test -race ./...`
in `go-sdk`, `go vet`, and the `prek` pre-commit and manual stages against
`upstream/main`. `breeze build-docs` was not run (Docker was not available).
---
##### Was generative AI tooling used to co-author this PR?
- [X] Yes — Claude Code (Fable 5.2)
Generated-by: Claude Code (Fable 5.2) following [the
guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]