This is an automated email from the ASF dual-hosted git repository.

jason810496 pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git


The following commit(s) were added to refs/heads/main by this push:
     new 2935fcb9fbb Go SDK: read the Dag run conf into a struct with typed 
fields (#75602)
2935fcb9fbb is described below

commit 2935fcb9fbb313f7e91d182c0224a75f33c30111
Author: Henry Chen <[email protected]>
AuthorDate: Sat Nov 14 21:59:49 2026 +0800

    Go SDK: read the Dag run conf into a struct with typed fields (#75602)
    
    * Go SDK: read the Dag run conf into a struct with typed fields
    
    A conf read from the map gives the value of a key as an any, so a task
    that reads several keys carries a type assertion per key, and a wrong
    one fails as a panic or as a silent zero value rather than as an error
    the task can report.
    
    Reading the conf into a struct leaves the types to the task author, but
    the names do not line up on their own: a conf key is written as
    target_table, where the Go field has to be TargetTable, and
    encoding/json matches a key to a field only when the two differ at most
    in case. Folding case and underscores is how a TaskFlow argument
    already reaches an untagged field, so a conf key reaches its field by
    the same folding.
    
    * Go SDK: decode a conf number for an any field as a json.Number
    
    A number that lands in a field of type any, or in the values of a
    map[string]any or []any field, was decoded as a float64, which cannot
    hold every integer a conf carries: an id above 2^53 could reach the
    task with its last digits changed and no error to show for it.
    
    Decoding with UseNumber keeps the literal, so the task reads an integer
    exactly through Int64. A field with a Go number type is unaffected,
    because the decoder converts into it as before.
    
    * Go SDK: fix two cases of reading the Dag run conf
    
    encoding/json cannot allocate an embedded pointer whose field name is
    unexported, so a conf key that reached one of its fields through a nil
    pointer failed the whole read. Keys are matched per type, and a type
    cannot tell a nil pointer from one the task set before the call, so the
    SDK matches no conf key to the fields of such a pointer either way.
    
    Reading a DagRun whose Conf is nil, such as one a task's unit test
    builds, changed the target it was given:
    
    - a struct or map target that does not decode itself got an empty
      object, which allocated a nil map or a nil pointer to either;
    - any other target got null, which reset an any, a slice or a pointer
      to nil and passed null to UnmarshalJSON.
    
    Airflow stores an empty conf for a run triggered without one, so such a
    run still reads as an empty object.
    
    * Go SDK: point the conf docs at the UnmarshalConf godoc
    
    The Go SDK docs listed the key-to-field matching rules and the error
    cases that DagRun.UnmarshalConf's godoc already states, and the two had
    already drifted apart, so a reader could follow either and get a
    different answer. The example stays, so the page still shows how a task
    makes the call.
    
    The godoc also did not say that the rest of a `json:` tag keeps its
    encoding/json meaning, which left a `,string` field looking as though a
    numeric conf value would reach it.
    
    * Go SDK: share one name-folding helper with the argument binder
    
    The Dag run conf reader and the TaskFlow argument binder each carried a
    byte-identical fold helper, so the two could start folding names
    differently without anything failing. pkg/binding imports sdk, so the
    helper cannot live in pkg/binding, and in sdk it would have to be
    exported; an internal package shares it without adding to the API.
---
 .../authoring-and-scheduling/language-sdks/go.rst  |  33 +-
 go-sdk/internal/fieldname/fieldname.go             |  30 ++
 go-sdk/pkg/binding/binding.go                      |  11 +-
 go-sdk/sdk/conf.go                                 | 385 +++++++++++++++++++++
 go-sdk/sdk/conf_test.go                            | 353 +++++++++++++++++++
 go-sdk/sdk/context.go                              |   2 +
 6 files changed, 806 insertions(+), 8 deletions(-)

diff --git a/airflow-core/docs/authoring-and-scheduling/language-sdks/go.rst 
b/airflow-core/docs/authoring-and-scheduling/language-sdks/go.rst
index 66738f562d7..8ec64c5aece 100644
--- a/airflow-core/docs/authoring-and-scheduling/language-sdks/go.rst
+++ b/airflow-core/docs/authoring-and-scheduling/language-sdks/go.rst
@@ -478,7 +478,8 @@ that runs under ``context.WithoutCancel(actx)``. A helper 
that takes a plain ``c
 ``airflow.Context`` back with ``airflow.FromContext``.
 
 ``Conf`` is a ``map[string]any`` whose values follow the :ref:`XCom type 
mapping <go-sdk/types>`. Use a map lookup
-or ``len`` to check for configuration entries.
+or ``len`` to check for configuration entries, or :ref:`read the conf into a 
struct <go-sdk/dag-run-conf>` with
+typed fields.
 
 A value the task returns becomes its ``return_value`` XCom. A non-nil 
``error``, or a panic, fails the task, and
 retries it if the task has retries.
@@ -494,6 +495,36 @@ retries it if the task has retries.
     )
     require.NoError(t, transform(actx, "uk", map[string]any{"rows": 42}))
 
+.. _go-sdk/dag-run-conf:
+
+Reading the Dag run conf
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+Reading ``Conf`` directly gives the value of a key as an ``any``, so a task 
that reads several keys asserts a Go
+type per key. ``UnmarshalConf`` reads the conf into a struct with typed fields 
instead:
+
+.. code-block:: go
+
+    type etlConf struct {
+        TargetTable string
+        Limit       int
+        Since       time.Time
+    }
+
+    func load(actx airflow.Context) error {
+        etl := etlConf{Limit: 100}
+        if err := actx.DagRun().UnmarshalConf(&etl); err != nil {
+            return err
+        }
+        ...
+    }
+
+For a run triggered with the conf ``{"target_table": "sales", "since": 
"2024-01-31T00:00:00Z"}``,
+``etl.TargetTable`` is ``"sales"``, ``etl.Since`` is that date, and 
``etl.Limit`` stays ``100``.
+
+For how keys match fields and when it returns an error,
+see `DagRun.UnmarshalConf 
<https://pkg.go.dev/github.com/apache/airflow/go-sdk/sdk#DagRun.UnmarshalConf>`__.
+
 .. _go-sdk/client:
 
 The ``sdk.Client``
diff --git a/go-sdk/internal/fieldname/fieldname.go 
b/go-sdk/internal/fieldname/fieldname.go
new file mode 100644
index 00000000000..ec3f6edb6be
--- /dev/null
+++ b/go-sdk/internal/fieldname/fieldname.go
@@ -0,0 +1,30 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+// Package fieldname folds the name of a Go field and the name it is matched
+// against onto one token. The Dag run conf reader in sdk and the TaskFlow
+// argument binder in pkg/binding both match names this way. pkg/binding
+// imports sdk, and an exported helper in sdk would become public API, so the
+// shared helper lives here.
+package fieldname
+
+import "strings"
+
+// Fold lowercases name and drops underscores, so target_table matches 
TargetTable.
+func Fold(name string) string {
+       return strings.ToLower(strings.ReplaceAll(name, "_", ""))
+}
diff --git a/go-sdk/pkg/binding/binding.go b/go-sdk/pkg/binding/binding.go
index 7f1882e0eaa..59b6e0dd3f3 100644
--- a/go-sdk/pkg/binding/binding.go
+++ b/go-sdk/pkg/binding/binding.go
@@ -40,6 +40,7 @@ import (
        "strings"
        "sync"
 
+       "github.com/apache/airflow/go-sdk/internal/fieldname"
        "github.com/apache/airflow/go-sdk/pkg/execution/genmodels"
        "github.com/apache/airflow/go-sdk/pkg/sdkcontext"
        "github.com/apache/airflow/go-sdk/sdk"
@@ -260,7 +261,7 @@ func (p *Plan) resolveLoneStructParam(
                        continue
                }
                byName[a.ArgName()] = i
-               key := foldArgName(a.ArgName())
+               key := fieldname.Fold(a.ArgName())
                if _, dup := byFolded[key]; dup {
                        ambiguous[key] = true
                        continue
@@ -278,7 +279,7 @@ func (p *Plan) resolveLoneStructParam(
        for _, sf := range plan.fields {
                idx, ok := byName[sf.argName]
                if !ok && !sf.tagged {
-                       key := foldArgName(sf.argName)
+                       key := fieldname.Fold(sf.argName)
                        if !ambiguous[key] {
                                idx, ok = byFolded[key]
                        }
@@ -685,7 +686,7 @@ func collectStructFields(
                        )
                }
                claimed[sf.argName] = f.Name
-               key := foldArgName(sf.argName)
+               key := fieldname.Fold(sf.argName)
                if existing, ok := folded[key]; ok {
                        return fmt.Errorf(
                                "task function %s: parameter %d: struct fields 
%s and %s bind arg names that "+
@@ -700,10 +701,6 @@ func collectStructFields(
        return nil
 }
 
-func foldArgName(name string) string {
-       return strings.ToLower(strings.ReplaceAll(name, "_", ""))
-}
-
 type schemaShape struct {
        jsonType string
        format   string
diff --git a/go-sdk/sdk/conf.go b/go-sdk/sdk/conf.go
new file mode 100644
index 00000000000..ff29dc87b90
--- /dev/null
+++ b/go-sdk/sdk/conf.go
@@ -0,0 +1,385 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package sdk
+
+import (
+       "bytes"
+       "encoding"
+       "encoding/json"
+       "fmt"
+       "maps"
+       "reflect"
+       "slices"
+       "strings"
+       "sync"
+
+       "github.com/apache/airflow/go-sdk/internal/fieldname"
+)
+
+// UnmarshalConf reads [DagRun.Conf] into target, which must be a non-nil
+// pointer, usually to a struct with one field per conf key:
+//
+//     type etlConf struct {
+//             TargetTable string
+//             Limit       int
+//             Since       time.Time
+//     }
+//
+//     etl := etlConf{Limit: 100}
+//     if err := actx.DagRun().UnmarshalConf(&etl); err != nil {
+//             return err
+//     }
+//
+// For a run triggered with {"target_table": "sales", "since": 
"2024-01-31T00:00:00Z"},
+// etl.TargetTable is "sales", etl.Since is that date, and etl.Limit stays 100.
+//
+// Each key goes to the field whose name matches it when case and underscores
+// are ignored, so target_table reaches TargetTable. A `json:` tag renames the
+// field, and the tag name matches the same way. If one key matches a field
+// name exactly, that key wins over keys that match it only once case and
+// underscores are ignored. Objects nested in the conf follow the same rules,
+// including objects in slices and objects that are map values. The keys of a
+// map field stay as written, because they are data rather than field names.
+// Other `json:` tag options keep their encoding/json meaning, so a `,string`
+// field needs its conf value as a quoted string.
+//
+// Keys that match no field are ignored, because other tasks in the run may
+// read them. A field with no matching key keeps the value target already
+// holds, so set the defaults on target before the call, as Limit does above.
+// The conf carries only the values the run was triggered with: it does not
+// carry the defaults of the Dag's params, so in a scheduled run every field
+// keeps the value it came in with.
+//
+// The fields of an embedded pointer whose field name is unexported take no
+// key, even when target already points it at a struct, because the decoder
+// cannot allocate such a pointer and the keys are matched per type. Embed the
+// struct by value, or give its type an exported name, to fill those fields.
+//
+// The `arg:` tag that binds a TaskFlow argument has no effect here. Read a key
+// that no field name can match, such as target-table, from [DagRun.Conf], or
+// name it in a `json:` tag.
+//
+// A number decoded into an any, such as a field of type any or a value of a
+// map[string]any or []any field, arrives as a [json.Number] rather than as the
+// number type [DagRun.Conf] carries, because a float64 cannot hold every
+// integer a conf can. Read it with its Int64 or Float64 method, or give the
+// field a Go integer or float type to have the decoder convert it. Of those,
+// only an integer type keeps every digit of a large integer.
+//
+// It returns an error when a value does not fit the type of its field, when
+// two keys match one field only once case and underscores are ignored, or when
+// two fields of a struct differ only in case or underscores.
+func (d DagRun) UnmarshalConf(target any) error {
+       rv := reflect.ValueOf(target)
+       if rv.Kind() != reflect.Pointer || rv.IsNil() {
+               return fmt.Errorf(
+                       "sdk.DagRun.UnmarshalConf: target must be a non-nil 
pointer, got %T",
+                       target,
+               )
+       }
+       matched, err := matchConfValue(d.Conf, rv.Type().Elem())
+       if err != nil {
+               return err
+       }
+       // A nil conf has nothing to decode. Encoding what matchConfValue 
returns
+       // for it gives {} or null, and decoding either would still allocate 
parts
+       // of target, reset them, or call UnmarshalJSON.
+       if d.Conf == nil {
+               return nil
+       }
+       blob, err := json.Marshal(matched)
+       if err != nil {
+               return fmt.Errorf("sdk.DagRun.UnmarshalConf: cannot encode the 
Dag run conf: %w", err)
+       }
+       dec := json.NewDecoder(bytes.NewReader(blob))
+       // A number that reaches an any-typed destination keeps its literal, so 
the
+       // task reads an exact integer instead of the float64 encoding/json 
would
+       // otherwise decode it into.
+       dec.UseNumber()
+       if err := dec.Decode(target); err != nil {
+               return fmt.Errorf(
+                       "sdk.DagRun.UnmarshalConf: cannot read the Dag run conf 
into %s: %w",
+                       rv.Type().Elem(),
+                       err,
+               )
+       }
+       return nil
+}
+
+// matchConfValue renames the keys of every conf object that becomes a struct
+// when the conf is decoded into target. encoding/json matches a key to a field
+// only when the two differ at most in case, so renaming the keys to the exact
+// field names is what lets target_table reach TargetTable.
+//
+// The walk follows target rather than the conf, so it reaches objects nested
+// in slices, arrays and map values, and stops at a type that decodes itself.
+func matchConfValue(value any, target reflect.Type) (any, error) {
+       for target.Kind() == reflect.Pointer {
+               target = target.Elem()
+       }
+       // A type such as time.Time decodes from a value of its own shape, and 
its
+       // Go fields are not conf keys.
+       if implementsConfUnmarshaler(target) {
+               return value, nil
+       }
+
+       switch target.Kind() {
+       case reflect.Struct:
+               if obj, ok := value.(map[string]any); ok {
+                       return matchConfFields(obj, target)
+               }
+       case reflect.Map:
+               if obj, ok := value.(map[string]any); ok {
+                       matched := make(map[string]any, len(obj))
+                       for key, elem := range obj {
+                               elem, err := matchConfValue(elem, target.Elem())
+                               if err != nil {
+                                       return nil, err
+                               }
+                               matched[key] = elem
+                       }
+                       return matched, nil
+               }
+       case reflect.Slice, reflect.Array:
+               if list, ok := value.([]any); ok {
+                       matched := make([]any, len(list))
+                       for i, elem := range list {
+                               elem, err := matchConfValue(elem, target.Elem())
+                               if err != nil {
+                                       return nil, err
+                               }
+                               matched[i] = elem
+                       }
+                       return matched, nil
+               }
+       }
+       // A value whose shape does not fit the field, such as a string for a
+       // struct, passes through so that the decoder reports the mismatch.
+       return value, nil
+}
+
+// matchConfFields renames each key of obj to the name of the target field the
+// key matches, and drops a key that matches no field. Keys are visited in
+// sorted order so that an error names the same pair of keys on every run.
+func matchConfFields(obj map[string]any, target reflect.Type) (map[string]any, 
error) {
+       fields, err := confFieldsOf(target)
+       if err != nil {
+               return nil, err
+       }
+
+       matched := make(map[string]any, len(obj))
+       keyOfField := make(map[string]string, len(obj))
+       for _, key := range slices.Sorted(maps.Keys(obj)) {
+               field, ok := fields.byName[key]
+               if !ok {
+                       // A key that folds to a field claims that field only 
when no other
+                       // key names it exactly.
+                       folded, found := fields.byFold[fieldname.Fold(key)]
+                       if !found {
+                               continue
+                       }
+                       if _, exact := obj[folded.name]; exact {
+                               continue
+                       }
+                       field = folded
+               }
+               if other, dup := keyOfField[field.name]; dup {
+                       return nil, fmt.Errorf(
+                               "sdk.DagRun.UnmarshalConf: Dag run conf keys %q 
and %q both match field %s when case and "+
+                                       "underscores are ignored, so the SDK 
cannot tell which one to use; name the field or "+
+                                       "its json tag exactly as the key it 
should take",
+                               other,
+                               key,
+                               field.goName,
+                       )
+               }
+               keyOfField[field.name] = key
+
+               value, err := matchConfValue(obj[key], field.typ)
+               if err != nil {
+                       return nil, err
+               }
+               matched[field.name] = value
+       }
+       return matched, nil
+}
+
+// confField is one field a conf key can reach.
+type confField struct {
+       // name is the key the decoder matches the field by: the `json:` tag 
name
+       // when the field has one, and the Go field name otherwise.
+       name string
+       // goName is the field as an error names it, qualified with the struct 
that
+       // declares it, which is an embedded struct for a promoted field.
+       goName string
+       typ    reflect.Type
+}
+
+type confFields struct {
+       byName map[string]confField
+       byFold map[string]confField
+       err    error
+}
+
+// Indexing a struct costs far more than renaming the keys of one object of
+// that struct, so a conf holding a list of many objects would otherwise spend
+// most of its time indexing the same struct again.
+var confFieldCache sync.Map // reflect.Type -> *confFields
+
+func confFieldsOf(target reflect.Type) (*confFields, error) {
+       if cached, ok := confFieldCache.Load(target); ok {
+               fields := cached.(*confFields)
+               return fields, fields.err
+       }
+       fields := indexConfFields(target)
+       confFieldCache.Store(target, fields)
+       return fields, fields.err
+}
+
+func indexConfFields(target reflect.Type) *confFields {
+       fields := &confFields{
+               byName: map[string]confField{},
+               byFold: map[string]confField{},
+       }
+       // Breadth-first over embedded structs, because a field of the outer 
struct
+       // shadows the field of the same name it promotes, as encoding/json 
does.
+       // A struct can embed a pointer to itself, so each one is visited once.
+       visited := map[reflect.Type]bool{target: true}
+       for level := []reflect.Type{target}; len(level) > 0; {
+               var embedded []reflect.Type
+               for _, structType := range level {
+                       for _, inner := range collectConfFields(structType, 
fields) {
+                               if !visited[inner] {
+                                       visited[inner] = true
+                                       embedded = append(embedded, inner)
+                               }
+                       }
+               }
+               if fields.err != nil {
+                       return fields
+               }
+               level = embedded
+       }
+       return fields
+}
+
+// collectConfFields adds the fields one struct declares and returns the
+// structs it embeds, whose fields belong to the next level.
+func collectConfFields(structType reflect.Type, fields *confFields) 
[]reflect.Type {
+       var embedded []reflect.Type
+       for i := range structType.NumField() {
+               f := structType.Field(i)
+               name, tagged := confFieldName(f)
+               // `json:"-"` on an embedded struct drops the fields it would 
promote,
+               // as encoding/json does, so the check comes before the walk 
into it.
+               // Promoting them instead can fail a struct the decoder fills, 
with an
+               // error naming a field no conf key can reach.
+               if name == "" {
+                       continue
+               }
+               if f.Anonymous && !tagged {
+                       // The decoder cannot allocate a pointer to an 
unexported struct.
+                       if f.Type.Kind() == reflect.Pointer && !f.IsExported() {
+                               continue
+                       }
+                       if inner := confStructType(f.Type); inner != nil {
+                               embedded = append(embedded, inner)
+                               continue
+                       }
+               }
+               if !f.IsExported() {
+                       continue
+               }
+               if _, shadowed := fields.byName[name]; shadowed {
+                       continue
+               }
+               field := confField{
+                       name:   name,
+                       goName: confTypeName(structType) + "." + f.Name,
+                       typ:    f.Type,
+               }
+               fields.byName[name] = field
+               folded := fieldname.Fold(name)
+               if clash, dup := fields.byFold[folded]; dup {
+                       fields.err = fmt.Errorf(
+                               "sdk.DagRun.UnmarshalConf: fields %s and %s 
differ only in case or underscores, so the SDK "+
+                                       "cannot tell which one a Dag run conf 
key should go to; rename one of them",
+                               clash.goName,
+                               field.goName,
+                       )
+                       return nil
+               }
+               fields.byFold[folded] = field
+       }
+       return embedded
+}
+
+// confFieldName returns the key the decoder matches a field by, and whether a
+// `json:` tag names it. An empty name means the field takes no key at all,
+// which is what `json:"-"` asks for.
+func confFieldName(f reflect.StructField) (name string, tagged bool) {
+       tag, ok := f.Tag.Lookup("json")
+       if !ok {
+               return f.Name, false
+       }
+       tagName, _, _ := strings.Cut(tag, ",")
+       switch tagName {
+       case "-":
+               // `json:"-,"` names the field "-"; only a bare `json:"-"` 
drops it.
+               if tag == "-" {
+                       return "", false
+               }
+               return "-", true
+       case "":
+               return f.Name, false
+       }
+       return tagName, true
+}
+
+// confStructType returns the struct an embedded field promotes fields from, or
+// nil when the field promotes no conf key.
+func confStructType(t reflect.Type) reflect.Type {
+       if t.Kind() == reflect.Pointer {
+               t = t.Elem()
+       }
+       if t.Kind() != reflect.Struct || implementsConfUnmarshaler(t) {
+               return nil
+       }
+       return t
+}
+
+func confTypeName(t reflect.Type) string {
+       if t.Name() == "" {
+               return t.String()
+       }
+       return t.Name()
+}
+
+func implementsConfUnmarshaler(t reflect.Type) bool {
+       for _, candidate := range []reflect.Type{t, reflect.PointerTo(t)} {
+               if candidate.Implements(jsonUnmarshalerType) || 
candidate.Implements(textUnmarshalerType) {
+                       return true
+               }
+       }
+       return false
+}
+
+var (
+       jsonUnmarshalerType = reflect.TypeFor[json.Unmarshaler]()
+       textUnmarshalerType = reflect.TypeFor[encoding.TextUnmarshaler]()
+)
diff --git a/go-sdk/sdk/conf_test.go b/go-sdk/sdk/conf_test.go
new file mode 100644
index 00000000000..0bee85ef5ca
--- /dev/null
+++ b/go-sdk/sdk/conf_test.go
@@ -0,0 +1,353 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package sdk
+
+import (
+       "encoding/json"
+       "testing"
+       "time"
+
+       "github.com/stretchr/testify/assert"
+       "github.com/stretchr/testify/require"
+)
+
+type etlTable struct {
+       TableName string
+}
+
+type etlSource struct {
+       BucketName string
+}
+
+type etlConf struct {
+       TargetTable  string
+       Limit        int
+       Since        time.Time
+       Source       *etlSource
+       Tables       []etlTable
+       TableArray   [1]etlTable
+       TablesByName map[string]etlTable
+       Options      map[string]any
+}
+
+type taggedConf struct {
+       Table  string `json:"target_table"`
+       Rows   int    `json:"row_count,omitempty"`
+       Secret string `json:"-"`
+}
+
+type commonConf struct {
+       Env   string
+       Limit int
+}
+
+type jobConf struct {
+       commonConf
+
+       Limit int
+       Name  string
+}
+
+type foldClashConf struct {
+       TargetTable string
+       Targettable string
+}
+
+// Tablename folds to the same token as the embedded TableName, which only the
+// `json:"-"` keeps from being a clash.
+type skippedEmbedConf struct {
+       etlTable `json:"-"`
+
+       Tablename string
+}
+
+type SourceTable struct {
+       SourceName string
+}
+
+// etlTable is unexported and SourceTable is exported, so only the fields of
+// *SourceTable take conf keys.
+type embeddedPointerConf struct {
+       *etlTable
+       *SourceTable
+
+       Zone string
+}
+
+type untypedConf struct {
+       RowCount int64
+       Options  map[string]any
+       Items    []any
+       Anything any
+}
+
+func dagRunWithConf(conf map[string]any) DagRun {
+       return DagRun{DagID: "dag1", RunID: "run1", Conf: conf}
+}
+
+func TestUnmarshalConfMatchesKeysIgnoringCaseAndUnderscores(t *testing.T) {
+       var conf etlConf
+
+       require.NoError(t, dagRunWithConf(map[string]any{
+               "target_table": "sales",
+               "LIMIT":        int64(5),
+       }).UnmarshalConf(&conf))
+
+       assert.Equal(t, "sales", conf.TargetTable)
+       assert.Equal(t, 5, conf.Limit)
+}
+
+// Another task of the run may read a key this task does not declare, so an
+// unmatched key is not an error, and the field keeps the default the task set.
+func TestUnmarshalConfIgnoresUnmatchedKeysAndKeepsDefaults(t *testing.T) {
+       conf := etlConf{Limit: 100}
+
+       require.NoError(t, dagRunWithConf(map[string]any{
+               "notify_email": "[email protected]",
+       }).UnmarshalConf(&conf))
+
+       assert.Empty(t, conf.TargetTable)
+       assert.Equal(t, 100, conf.Limit)
+}
+
+func TestUnmarshalConfPrefersAnExactKeyOverFoldedOnes(t *testing.T) {
+       var conf etlConf
+
+       require.NoError(t, dagRunWithConf(map[string]any{
+               "target_table": "folded",
+               "TargetTable":  "exact",
+               "TARGET_TABLE": "folded",
+       }).UnmarshalConf(&conf))
+
+       assert.Equal(t, "exact", conf.TargetTable)
+}
+
+func TestUnmarshalConfMatchesKeysOfNestedObjects(t *testing.T) {
+       var conf etlConf
+
+       require.NoError(t, dagRunWithConf(map[string]any{
+               "source":         map[string]any{"bucket_name": "raw"},
+               "tables":         []any{map[string]any{"table_name": "orders"}},
+               "table_array":    []any{map[string]any{"table_name": "items"}},
+               "tables_by_name": map[string]any{"Sales_Key": 
map[string]any{"table_name": "sales"}},
+       }).UnmarshalConf(&conf))
+
+       require.NotNil(t, conf.Source)
+       assert.Equal(t, "raw", conf.Source.BucketName)
+       assert.Equal(t, []etlTable{{TableName: "orders"}}, conf.Tables)
+       assert.Equal(t, [1]etlTable{{TableName: "items"}}, conf.TableArray)
+       assert.Equal(t, map[string]etlTable{"Sales_Key": {TableName: "sales"}}, 
conf.TablesByName)
+}
+
+// The keys of a map are data, so they reach the task as the run was triggered
+// with them, at every depth.
+func TestUnmarshalConfKeepsTheKeysOfAMapWithUntypedValues(t *testing.T) {
+       options := map[string]any{
+               "Rows":      []any{map[string]any{"Key_A": "a"}},
+               "Inner_Map": map[string]any{"Key_B": "b"},
+       }
+       var conf etlConf
+
+       require.NoError(t, dagRunWithConf(map[string]any{"options": 
options}).UnmarshalConf(&conf))
+
+       assert.Equal(t, options, conf.Options)
+}
+
+// The value is the first integer a float64 rounds, so the test also fails if
+// the number passes through a float64 before it becomes a json.Number.
+func TestUnmarshalConfReadsANumberIntoAnUntypedFieldAsAJSONNumber(t 
*testing.T) {
+       var conf untypedConf
+
+       require.NoError(t, dagRunWithConf(map[string]any{
+               "row_count": int64(9007199254740993),
+               "options":   map[string]any{"rows": int64(9007199254740993)},
+               "items":     []any{int64(9007199254740993)},
+               "anything":  int64(9007199254740993),
+       }).UnmarshalConf(&conf))
+
+       assert.Equal(t, int64(9007199254740993), conf.RowCount)
+       assert.Equal(t, json.Number("9007199254740993"), conf.Options["rows"])
+       assert.Equal(t, json.Number("9007199254740993"), conf.Items[0])
+       assert.Equal(t, json.Number("9007199254740993"), conf.Anything)
+}
+
+func TestUnmarshalConfReadsATimestampIntoATimeField(t *testing.T) {
+       var conf etlConf
+
+       require.NoError(t, dagRunWithConf(map[string]any{
+               "since": "2024-01-31T02:00:00+08:00",
+       }).UnmarshalConf(&conf))
+
+       assert.True(
+               t,
+               conf.Since.Equal(time.Date(2024, 1, 31, 2, 0, 0, 0, 
time.FixedZone("", 8*60*60))),
+       )
+}
+
+func TestUnmarshalConfMatchesAJSONTagAndSkipsASkippedField(t *testing.T) {
+       var conf taggedConf
+
+       require.NoError(t, dagRunWithConf(map[string]any{
+               "TargetTable": "sales",
+               "ROW_COUNT":   int64(5),
+               "secret":      "token",
+       }).UnmarshalConf(&conf))
+
+       assert.Equal(t, "sales", conf.Table)
+       assert.Equal(t, 5, conf.Rows)
+       assert.Empty(t, conf.Secret, `a json:"-" field takes no conf key`)
+}
+
+func TestUnmarshalConfFillsPromotedFieldsAndHonoursShadowing(t *testing.T) {
+       var conf jobConf
+
+       require.NoError(t, dagRunWithConf(map[string]any{
+               "env":   "prod",
+               "limit": int64(5),
+       }).UnmarshalConf(&conf))
+
+       assert.Equal(t, "prod", conf.Env)
+       assert.Equal(t, 5, conf.Limit)
+       assert.Zero(t, conf.commonConf.Limit, "the outer field shadows the one 
it promotes")
+}
+
+// encoding/json skips an embedded struct tagged `json:"-"` rather than
+// promoting its fields, so a conf key reaches none of them either.
+func TestUnmarshalConfPromotesNoFieldOfASkippedEmbeddedStruct(t *testing.T) {
+       var conf skippedEmbedConf
+
+       require.NoError(
+               t,
+               dagRunWithConf(map[string]any{"table_name": 
"sales"}).UnmarshalConf(&conf),
+       )
+
+       assert.Equal(t, "sales", conf.Tablename)
+       assert.Empty(t, conf.TableName, `a json:"-" embedded struct takes no 
conf key`)
+}
+
+// A conf key reaches no field of an embedded pointer whose field name is
+// unexported, even one the task set before the call, while the fields beside
+// it still take their keys.
+func TestUnmarshalConfPromotesNoFieldOfAnUnexportedEmbeddedPointer(t 
*testing.T) {
+       run := dagRunWithConf(map[string]any{
+               "table_name":  "sales",
+               "source_name": "raw",
+               "zone":        "eu",
+       })
+       var unset embeddedPointerConf
+       preset := embeddedPointerConf{etlTable: &etlTable{TableName: "default"}}
+
+       require.NoError(t, run.UnmarshalConf(&unset))
+       require.NoError(t, run.UnmarshalConf(&preset))
+
+       assert.Equal(
+               t,
+               embeddedPointerConf{SourceTable: &SourceTable{SourceName: 
"raw"}, Zone: "eu"},
+               unset,
+       )
+       assert.Equal(t, "default", preset.TableName)
+}
+
+func TestUnmarshalConfRejectsTwoKeysThatMatchOneField(t *testing.T) {
+       var conf etlConf
+
+       err := dagRunWithConf(map[string]any{
+               "target_table": "a",
+               "TARGET_TABLE": "b",
+       }).UnmarshalConf(&conf)
+
+       require.EqualError(
+               t,
+               err,
+               `sdk.DagRun.UnmarshalConf: Dag run conf keys "TARGET_TABLE" and 
"target_table" both match field `+
+                       `etlConf.TargetTable when case and underscores are 
ignored, so the SDK cannot tell which one to `+
+                       `use; name the field or its json tag exactly as the key 
it should take`,
+       )
+}
+
+// The struct, not the conf, is what has to be fixed, so the error does not
+// wait for a run whose conf happens to hold one of the two names.
+func TestUnmarshalConfRejectsFieldsThatDifferOnlyInCaseOrUnderscores(t 
*testing.T) {
+       var conf foldClashConf
+
+       err := DagRun{DagID: "dag1", RunID: "run1"}.UnmarshalConf(&conf)
+
+       require.EqualError(
+               t,
+               err,
+               "sdk.DagRun.UnmarshalConf: fields foldClashConf.TargetTable and 
foldClashConf.Targettable differ "+
+                       "only in case or underscores, so the SDK cannot tell 
which one a Dag run conf key should go "+
+                       "to; rename one of them",
+       )
+}
+
+func TestUnmarshalConfNamesTheTargetWhenAValueDoesNotFit(t *testing.T) {
+       var conf etlConf
+
+       err := dagRunWithConf(map[string]any{"limit": 
"many"}).UnmarshalConf(&conf)
+
+       require.ErrorContains(
+               t,
+               err,
+               "sdk.DagRun.UnmarshalConf: cannot read the Dag run conf into 
sdk.etlConf: ",
+       )
+       require.ErrorContains(
+               t,
+               err,
+               "cannot unmarshal string into Go struct field etlConf.Limit of 
type int",
+       )
+}
+
+func TestUnmarshalConfRejectsATargetItCannotFill(t *testing.T) {
+       var nilTarget *etlConf
+
+       for _, target := range []any{etlConf{}, nilTarget, nil} {
+               err := dagRunWithConf(map[string]any{"limit": 
int64(5)}).UnmarshalConf(target)
+
+               require.ErrorContains(
+                       t,
+                       err,
+                       "sdk.DagRun.UnmarshalConf: target must be a non-nil 
pointer, got ",
+               )
+       }
+}
+
+// A run triggered without a conf leaves every field as the task set it.
+func TestUnmarshalConfOfARunWithoutAConf(t *testing.T) {
+       conf := etlConf{Limit: 100}
+
+       require.NoError(t, DagRun{DagID: "dag1", RunID: 
"run1"}.UnmarshalConf(&conf))
+
+       assert.Equal(t, etlConf{Limit: 100}, conf)
+}
+
+// Nothing to decode means nothing to allocate or reset either.
+func TestUnmarshalConfOfARunWithoutAConfLeavesTheTargetAsItIs(t *testing.T) {
+       run := DagRun{DagID: "dag1", RunID: "run1"}
+       var nilMap map[string]any
+       var nilPointer *etlConf
+       var untyped any = "default"
+
+       require.NoError(t, run.UnmarshalConf(&nilMap))
+       require.NoError(t, run.UnmarshalConf(&nilPointer))
+       require.NoError(t, run.UnmarshalConf(&untyped))
+
+       assert.Nil(t, nilMap)
+       assert.Nil(t, nilPointer)
+       assert.Equal(t, "default", untyped)
+}
diff --git a/go-sdk/sdk/context.go b/go-sdk/sdk/context.go
index b9e7b98c71a..901c3aaa6da 100644
--- a/go-sdk/sdk/context.go
+++ b/go-sdk/sdk/context.go
@@ -106,5 +106,7 @@ type DagRun struct {
        RunType DagRunType
        // Conf is the run configuration, empty when none was supplied. 
Integers are
        // int64, or uint64 above math.MaxInt64; floating-point values are 
float64.
+       // [DagRun.UnmarshalConf] reads the conf into a struct with typed fields
+       // instead.
        Conf map[string]any
 }

Reply via email to