This is an automated email from the ASF dual-hosted git repository.
ryerraguntla pushed a change to branch feat/connectors-3503-opensearch-source
in repository https://gitbox.apache.org/repos/asf/iggy.git
from cb230c6dc Fixing review comments: key OpenSearch hits by tie-breaker,
drop origin_timestamp
add d0fc19a26 chore(deps): Bump nlohmann_json from 3.12.0.bcr.1 to
3.12.0.bcr.2 in /foreign/cpp in the cpp group across 1 directory (#4689)
add c4d86e3ff chore(deps): Bump the python group across 3 directories with
3 updates (#4752)
add 2487036c4 feat(python): expose consumer shutdown and offset drain
timeout (#4660)
add 09b5a02c0 fix(mcp): document allowed values for poll strategy and
partitioning (#4588)
add fff4f983c test(go): cover access token command serialization (#4640)
add f451fb91e feat(example): add a multi-tenant Go producer and consumer
(#4716)
add 7bcf398ea feat(python): add sink data producer example (#4710)
add df68779fc feat(connectors): deliver sink messages at least once (#4734)
add e49908dc1 test(partitions): stop the removal test panicking when run
as root (#4754)
add ff74972d4 feat(php): add PHP message envelope example (#4718)
add fbe348433 test(integration): pin the crash loss of replicated
partitions (#4738)
add 8d6d0fefe fix(connectors): skip Iceberg rows that do not match the
table schema (#4739)
add e4764be3c fix(connectors): retry failed Iceberg catalog commits (#4740)
add c59961ce4 feat(python): add multi-tenant producer and consumer
examples (#4751)
add 06cf04e5e fix(metadata): send metadata prepares in op order (#4755)
add ad7aaebdd fix(ci): prevent false Python example passes (#4762)
add ac27a023e chore(deps): Bump the cpp group across 1 directory with 2
updates (#4770)
add fdec51376 chore(deps): Bump the node group across 1 directory with 2
updates (#4769)
add a4a9a79a7 chore(deps): Bump the web group across 1 directory with 7
updates (#4768)
add 3c493f2f6 feat(connectors): deliver Iceberg sink messages at least
once (#4759)
add ec5d4b189 fix(connectors): tighten Iceberg sink redelivery skip and
error mapping (#4775)
add 89cb6d13f docs: let maintainers close agent-driven PRs (#4773)
add d82fc06ac Merge branch 'master' into
feat/connectors-3503-opensearch-source
No new revisions were added by this update.
Summary of changes:
.claude/skills/connector-sink/SKILL.md | 22 +-
.claude/skills/connectors-overview/SKILL.md | 2 +-
.github/config/components.yml | 2 +
.github/workflows/_test_examples.yml | 7 -
.github/workflows/issue-labeler-assigner.yml | 127 +-
.github/workflows/pr-triage-apply.yml | 2 +-
AGENTS.md | 6 +-
CONTRIBUTING.md | 51 +-
Cargo.lock | 3 +
Cargo.toml | 1 +
PULL_REQUEST_TEMPLATE.md | 3 +
bdd/python/uv.lock | 56 +-
core/ai/mcp/src/service/requests.rs | 28 +-
core/connectors/runtime/README.md | 82 +-
.../example_config/connectors/iceberg_sink.toml | 8 +
core/connectors/runtime/src/configs/connectors.rs | 145 ++
core/connectors/runtime/src/error.rs | 15 +
core/connectors/runtime/src/main.rs | 13 +
core/connectors/runtime/src/manager/sink.rs | 11 +
core/connectors/runtime/src/sink.rs | 895 +++++++---
core/connectors/runtime/src/sink_delivery.rs | 1811 ++++++++++++++++++++
core/connectors/sdk/README.md | 27 +
core/connectors/sdk/src/lib.rs | 88 +
core/connectors/sdk/src/sink.rs | 533 +++++-
core/connectors/sinks/http_sink/README.md | 18 +-
core/connectors/sinks/http_sink/src/lib.rs | 188 +-
core/connectors/sinks/iceberg_sink/Cargo.toml | 6 +-
core/connectors/sinks/iceberg_sink/README.md | 124 +-
core/connectors/sinks/iceberg_sink/config.toml | 4 +
core/connectors/sinks/iceberg_sink/src/lib.rs | 119 +-
core/connectors/sinks/iceberg_sink/src/props.rs | 4 +
.../sinks/iceberg_sink/src/router/commit.rs | 1033 +++++++++++
.../iceberg_sink/src/router/dynamic_router.rs | 436 +++--
.../sinks/iceberg_sink/src/router/mod.rs | 578 +++++--
.../sinks/iceberg_sink/src/router/redelivery.rs | 513 ++++++
.../sinks/iceberg_sink/src/router/static_router.rs | 258 ++-
core/connectors/sinks/iceberg_sink/src/sink.rs | 48 +-
core/integration/src/harness/disk.rs | 24 +
.../src/harness/handle/connectors_runtime.rs | 20 +-
core/integration/src/harness/mod.rs | 1 +
core/integration/tests/cluster/crash_durability.rs | 182 +-
.../tests/connectors/fixtures/http/container.rs | 7 +
.../tests/connectors/fixtures/http/mod.rs | 6 +-
.../tests/connectors/fixtures/http/sink.rs | 71 +
.../tests/connectors/fixtures/iceberg/container.rs | 267 ++-
.../tests/connectors/fixtures/iceberg/mod.rs | 6 +-
core/integration/tests/connectors/fixtures/mod.rs | 13 +-
.../tests/connectors/http/http_sink_delivery.rs | 237 +++
core/integration/tests/connectors/http/mod.rs | 1 +
.../tests/connectors/iceberg/iceberg_sink.rs | 181 +-
.../connectors/iceberg/iceberg_sink_delivery.rs | 229 +++
core/integration/tests/connectors/iceberg/mod.rs | 1 +
.../tests/connectors/stdout/stdout_sink.rs | 61 +-
core/journal/src/local_gate.rs | 193 ++-
core/journal/src/partition_journal.rs | 46 +-
core/metadata/src/impls/metadata.rs | 135 +-
core/partitions/src/partition_storage.rs | 7 +-
examples/go/README.md | 19 +-
examples/go/common/client.go | 94 +
examples/go/common/permissions.go | 92 +
examples/go/common/permissions_test.go | 63 +
examples/go/multi-tenant/consumer/main.go | 190 ++
examples/go/multi-tenant/consumer/main_test.go | 47 +
examples/go/multi-tenant/producer/main.go | 200 +++
.../go/multi-tenant/producer/main_test.go | 38 +-
examples/php/README.md | 13 +
examples/php/message-envelope/consumer.php | 108 ++
examples/php/message-envelope/producer.php | 78 +
examples/python/README.md | 56 +-
examples/python/basic/consumer.py | 2 +-
examples/python/getting-started/consumer.py | 5 +-
examples/python/multi-tenant/consumer.py | 138 ++
examples/python/multi-tenant/producer.py | 121 ++
examples/python/multi-tenant/tenants.py | 123 ++
examples/python/pyproject.toml | 3 +-
examples/python/sink-data-producer/producer.py | 171 ++
examples/python/uv.lock | 40 +-
foreign/cpp/MODULE.bazel | 6 +-
foreign/cpp/MODULE.bazel.lock | 12 +-
foreign/go/internal/command/access_token_test.go | 87 +-
foreign/node/package-lock.json | 149 +-
foreign/node/package.json | 2 +-
foreign/php/tests/IggySdkTest.php | 4 +
foreign/python/README.md | 15 +-
foreign/python/apache_iggy.pyi | 6 +-
foreign/python/src/client.rs | 12 +-
foreign/python/tests/test_consumer_group.py | 36 +-
foreign/python/uv.lock | 50 +-
scripts/ci/test-runtime-artifacts.py | 122 ++
scripts/run-examples-from-readme.sh | 12 +-
scripts/utils.sh | 6 +-
web/package-lock.json | 178 +-
web/package.json | 14 +-
93 files changed, 10209 insertions(+), 1058 deletions(-)
create mode 100644 core/connectors/runtime/src/sink_delivery.rs
create mode 100644 core/connectors/sinks/iceberg_sink/src/router/commit.rs
create mode 100644 core/connectors/sinks/iceberg_sink/src/router/redelivery.rs
create mode 100644 core/integration/tests/connectors/http/http_sink_delivery.rs
create mode 100644
core/integration/tests/connectors/iceberg/iceberg_sink_delivery.rs
create mode 100644 examples/go/common/client.go
create mode 100644 examples/go/common/permissions.go
create mode 100644 examples/go/common/permissions_test.go
create mode 100644 examples/go/multi-tenant/consumer/main.go
create mode 100644 examples/go/multi-tenant/consumer/main_test.go
create mode 100644 examples/go/multi-tenant/producer/main.go
copy foreign/go/internal/command/segment.go =>
examples/go/multi-tenant/producer/main_test.go (51%)
create mode 100644 examples/php/message-envelope/consumer.php
create mode 100644 examples/php/message-envelope/producer.php
create mode 100644 examples/python/multi-tenant/consumer.py
create mode 100644 examples/python/multi-tenant/producer.py
create mode 100644 examples/python/multi-tenant/tenants.py
create mode 100644 examples/python/sink-data-producer/producer.py
