This is an automated email from the ASF dual-hosted git repository.
hubcio pushed a change to branch feat/server-ng-perf
in repository https://gitbox.apache.org/repos/asf/iggy.git
omit 3a944be28 perf(server-ng): reach single-node latency parity with
legacy server
add 40d1e356e fix(server-ng): legacy-parity consumer group errors, roster
HTTP bind (#3754)
add ab944b558 chore(deps): Bump the csharp group with 8 updates (#3759)
add cfa104c37 feat(server-ng): add cluster.nodes.advertised_address for
clients (#3757)
add 89045771e feat(consensus): rework client table for session resume
across restarts (#3753)
add f88e8a862 feat(server-ng): harden configuration surface (#3756)
add b6a2f81c0 fix(sdk): forward unknown command codes instead of rejecting
them (#3766)
add ba541bb9c fix(server-ng): stop restart panic when WAL replay recovers
sessions (#3768)
add 91fcb9300 fix(ci): propagate failures in pre-commit hooks (#3743)
add 265bd265a chore(deps): Bump com.gradleup.shadow from 9.5.1 to 9.6.0 in
/foreign/java in the java group across 1 directory (#3771)
add 3153e456f chore(deps): Bump the github-actions group with 5 updates
(#3772)
add 3711d060d fix(ci): example-runner silently skips Python/Go examples
when exclude starts with a dash (#3770)
add f0eb0950d fix(server-ng): make partition reconvergence safe for
in-flight sends (#3761)
add f67f03873 fix(server-ng): fold io_uring remediation into
runtime-creation errors (#3775)
add 0fa657a94 fix(ci): stop trailing checks from skipping text files as
binary (#3778)
add 3b190c14e feat(js): add VSR framing to the Node SDK (#3763)
add 5ca0bbb40 chore(deps): Bump postcss from 8.5.16 to 8.5.23 in /web
(#3762)
add 74fb5e2a9 fix(server-ng): bind cluster TCP to [tcp].address, not the
roster ip (#3783)
add 53a7ff99e feat(server-ng): harden listener, auth, shutdown, and
disk-poll paths (#3774)
add 2afd4b12b feat(consensus): persist metadata VSR state in a durable
superblock (#3767)
add 102484af2 feat(python): expose Permissions and remaining user auth
methods (#3727)
add 1453114da feat(metadata): state transfer for snapshot and client table
(#3765)
add 71c67e432 feat(server-ng): enforce PAT limits and expose Prometheus
metrics (#3773)
add a59a1db01 chore(deps): Bump the node group across 1 directory with 3
updates (#3779)
add 01e721028 chore(deps): Bump github.com/klauspost/compress from 1.19.0
to 1.19.1 in /bdd/go in the go group across 1 directory (#3785)
add a8832ceed perf(server-ng): compute batch checksum in a single produce
pass
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (3a944be28)
\
N -- N -- N refs/heads/feat/server-ng-perf (a8832ceed)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
No new revisions were added by this update.
Summary of changes:
.github/actions/node-npm/pre-merge/action.yml | 76 +-
.github/actions/utils/server-start/action.yml | 15 +-
.github/config/components.yml | 2 +-
.github/workflows/_build_python_wheels.yml | 14 +-
.github/workflows/_build_rust_artifacts.yml | 6 +-
.github/workflows/_common.yml | 32 +-
.github/workflows/_detect.yml | 2 +-
.github/workflows/_publish_rust_crates.yml | 2 +-
.github/workflows/_test.yml | 17 +-
.github/workflows/_test_bdd.yml | 2 +-
.github/workflows/_test_examples.yml | 6 +-
.github/workflows/coverage-baseline.yml | 26 +-
.github/workflows/edge-release.yml | 2 +-
.github/workflows/post-merge.yml | 4 +-
.github/workflows/publish.yml | 16 +-
.pre-commit-config.yaml | 8 +-
Cargo.lock | 12 +-
Cargo.toml | 11 +-
bdd/go/go.mod | 2 +-
bdd/go/go.sum | 4 +-
codecov.yml | 6 +
core/binary_protocol/Cargo.toml | 2 +-
core/binary_protocol/src/consensus/command.rs | 19 +-
core/binary_protocol/src/consensus/header.rs | 392 +++-
core/binary_protocol/src/consensus/mod.rs | 3 +-
core/binary_protocol/src/consensus/operation.rs | 3 +-
core/binary_protocol/src/lib.rs | 5 +-
core/common/Cargo.toml | 2 +-
core/common/src/error/iggy_error.rs | 4 +-
core/common/src/types/identifier/mod.rs | 16 +-
core/configs/src/lib.rs | 5 +-
core/configs/src/server_ng_config/cluster.rs | 1047 +++++++++-
core/configs/src/server_ng_config/defaults.rs | 63 +-
core/configs/src/server_ng_config/displays.rs | 12 +-
core/configs/src/server_ng_config/message_bus.rs | 137 +-
core/configs/src/server_ng_config/metadata.rs | 89 +-
core/configs/src/server_ng_config/mod.rs | 1 +
core/configs/src/server_ng_config/partition.rs | 212 ++
core/configs/src/server_ng_config/server_ng.rs | 2 +
core/configs/src/server_ng_config/sharding.rs | 74 +
core/configs/src/server_ng_config/validators.rs | 383 +++-
core/configs/src/server_ng_config/websocket.rs | 105 +-
core/connectors/sinks/s3_sink/Cargo.toml | 6 +
core/consensus/Cargo.toml | 2 +
core/consensus/src/client_table.rs | 2156 +++++++++++++++-----
core/consensus/src/impls.rs | 1003 ++++++++-
core/consensus/src/le_cursor.rs | 115 ++
core/consensus/src/lib.rs | 20 +-
core/consensus/src/metadata_helpers.rs | 519 +++--
core/consensus/src/observability.rs | 4 +-
core/consensus/src/plane_helpers.rs | 131 +-
core/consensus/src/state_manifest.rs | 309 +++
core/consensus/src/vsr_state.rs | 224 ++
core/consensus/src/vsr_timeout.rs | 83 +-
core/integration/Cargo.toml | 13 +-
core/integration/src/harness/config/resolve.rs | 7 +-
core/integration/src/harness/handle/server.rs | 83 +-
.../src/harness/orchestrator/harness.rs | 26 +
.../cli/message/test_message_flush_command.rs | 5 +
.../tests/cluster/client_table_restart.rs | 388 +++-
.../tests/cluster/metadata_checkpoint_restart.rs | 276 +++
.../tests/cluster/metadata_state_transfer.rs | 406 ++++
core/integration/tests/cluster/mod.rs | 2 +
.../integration/tests/connectors/http/http_sink.rs | 72 +-
core/integration/tests/data_integrity/mod.rs | 11 +-
.../data_integrity/verify_after_server_restart.rs | 31 +
.../tests/sdk/consumer_group_membership.rs | 102 +
core/integration/tests/sdk/raw.rs | 13 +-
.../tests/server/cluster_view_durability_vsr.rs | 266 +++
core/integration/tests/server/flush_vsr.rs | 4 +
core/integration/tests/server/http_vsr.rs | 11 +
core/integration/tests/server/message_retrieval.rs | 51 +-
.../server/metadata_checkpoint_recovery_vsr.rs | 138 ++
core/integration/tests/server/mod.rs | 8 +
.../server/scenarios/message_cleanup_scenario.rs | 8 +-
.../server/scenarios/purge_delete_scenario.rs | 78 +-
.../scenarios/reconnect_after_restart_scenario.rs | 9 +
.../scenarios/segment_rotation_race_scenario.rs | 9 -
core/journal/Cargo.toml | 2 +
core/journal/src/lib.rs | 31 +
core/journal/src/prepare_journal.rs | 511 ++++-
core/journal/src/superblock.rs | 824 ++++++++
core/message_bus/src/cache/connection.rs | 2 -
core/message_bus/src/client_listener/mod.rs | 7 +-
core/message_bus/src/client_listener/quic.rs | 24 +-
core/message_bus/src/config.rs | 56 +-
core/message_bus/src/framing.rs | 8 +-
core/message_bus/src/installer/replica.rs | 2 +-
core/message_bus/src/replica/auth.rs | 106 +-
core/message_bus/src/replica/handshake.rs | 21 +-
core/message_bus/src/socket_opts.rs | 59 +-
core/message_bus/tests/common/mod.rs | 7 +-
core/message_bus/tests/replica_auth_handshake.rs | 28 +
core/metadata/src/impls/metadata.rs | 2144 ++++++++++++++++---
core/metadata/src/impls/recovery.rs | 1190 ++++++++++-
core/metadata/src/lib.rs | 5 +-
core/metadata/src/stm/consumer_group.rs | 281 ++-
core/metadata/src/stm/mod.rs | 10 +
core/metadata/src/stm/mux.rs | 40 +-
core/metadata/src/stm/result.rs | 71 +-
core/metadata/src/stm/snapshot.rs | 137 +-
core/metadata/src/stm/stream.rs | 499 ++++-
core/metadata/src/stm/user.rs | 71 +-
core/partitions/src/iggy_index_reader.rs | 35 +-
core/partitions/src/iggy_partition.rs | 309 +--
core/partitions/src/iggy_partitions.rs | 17 +-
core/partitions/src/journal.rs | 35 +-
core/partitions/src/lib.rs | 1 +
core/partitions/src/log.rs | 319 +--
core/partitions/src/poll_plan.rs | 228 +--
core/sdk/Cargo.toml | 2 +-
core/sdk/src/clients/client.rs | 5 +-
core/sdk/src/clients/consumer.rs | 7 +-
core/sdk/src/vsr.rs | 88 +-
core/server-ng/Cargo.toml | 6 +-
core/server-ng/config.toml | 218 +-
core/server-ng/src/auth.rs | 105 +-
core/server-ng/src/bootstrap.rs | 1283 ++++++++++--
core/server-ng/src/cluster_meta.rs | 81 +-
core/server-ng/src/dispatch.rs | 545 ++++-
core/server-ng/src/dispatch/authz.rs | 10 +-
core/server-ng/src/http.rs | 200 +-
core/server-ng/src/http/error.rs | 161 +-
core/server-ng/src/http/handlers.rs | 19 +-
core/server-ng/src/http/metrics.rs | 297 +++
core/server-ng/src/http/reads.rs | 114 +-
core/server-ng/src/http/session.rs | 73 +-
core/server-ng/src/http/state.rs | 117 +-
core/server-ng/src/http/submit.rs | 117 +-
core/server-ng/src/http/tls.rs | 29 +-
core/server-ng/src/login_register.rs | 9 +-
core/server-ng/src/main.rs | 13 +-
core/server-ng/src/partition_helpers.rs | 20 +-
core/server-ng/src/partition_reconciler.rs | 218 +-
core/server-ng/src/pat.rs | 182 +-
core/server-ng/src/responses.rs | 17 +-
core/server-ng/src/server_error.rs | 32 +-
core/server-ng/src/session_manager.rs | 26 +
core/server-ng/src/users.rs | 10 +-
core/server/src/main.rs | 40 +-
core/server_common/src/consensus_message.rs | 102 +-
core/server_common/src/diagnostics.rs | 108 +-
core/shard/src/coordinator.rs | 232 ++-
core/shard/src/lib.rs | 1946 ++++++++++++++++--
core/shard/src/router.rs | 37 +-
core/simulator/src/deps.rs | 131 ++
core/simulator/src/lib.rs | 555 ++++-
core/simulator/src/replica.rs | 126 +-
core/simulator/src/workload/mod.rs | 4 +-
.../src/workload/ops/delete_consumer_group.rs | 34 +-
.../Iggy_SDK.Examples.Basic.Consumer.csproj | 8 +-
.../Iggy_SDK.Examples.Basic.Producer.csproj | 10 +-
...ggy_SDK.Examples.GettingStarted.Consumer.csproj | 4 +-
...ggy_SDK.Examples.GettingStarted.Producer.csproj | 4 +-
.../Iggy_SDK.Examples.Shared.csproj | 2 +-
...gy_SDK.Examples.MessageEnvelope.Consumer.csproj | 10 +-
...gy_SDK.Examples.MessageEnvelope.Producer.csproj | 10 +-
...ggy_SDK.Examples.MessageHeaders.Consumer.csproj | 4 +-
...ggy_SDK.Examples.MessageHeaders.Producer.csproj | 4 +-
.../Iggy_SDK.Examples.NewSdk.Consumer.csproj | 10 +-
.../Iggy_SDK.Examples.NewSdk.Producer.csproj | 10 +-
.../Iggy_SDK.Examples.TcpTls.Consumer.csproj | 4 +-
.../Iggy_SDK.Examples.TcpTls.Producer.csproj | 4 +-
examples/go/go.mod | 2 +-
examples/go/go.sum | 4 +-
foreign/csharp/Directory.Packages.props | 6 +-
foreign/go/go.mod | 2 +-
foreign/go/go.sum | 4 +-
.../data-population-explanation.md | 3 +-
foreign/java/gradle/libs.versions.toml | 2 +-
foreign/node/README.md | 63 +-
foreign/node/package-lock.json | 489 +----
foreign/node/package.json | 12 +-
foreign/node/scripts/check-vsr-protocol.mjs | 340 +++
foreign/node/src/bdd/raw.ts | 5 +-
foreign/node/src/client/client.config.test.ts | 95 +
foreign/node/src/client/client.config.ts | 56 +
foreign/node/src/client/client.connection.test.ts | 455 +++++
foreign/node/src/client/client.connection.ts | 362 +++-
foreign/node/src/client/client.frame.test.ts | 160 ++
foreign/node/src/client/client.frame.ts | 227 +++
foreign/node/src/client/client.socket.test.ts | 677 ++++++
foreign/node/src/client/client.socket.ts | 410 +++-
foreign/node/src/client/client.ts | 73 +-
foreign/node/src/client/client.type.ts | 24 +-
foreign/node/src/client/index.ts | 1 +
foreign/node/src/e2e/tcp.consumer-group.e2e.ts | 8 +-
foreign/node/src/e2e/tcp.consumer-stream.e2e.ts | 8 +-
foreign/node/src/e2e/tcp.raw.e2e.ts | 38 +-
foreign/node/src/e2e/test-client.utils.ts | 2 +
foreign/node/src/index.ts | 6 +
foreign/node/src/stream/consumer-stream.ts | 9 +-
.../wire/cluster/get-cluster-metadata.command.ts | 2 +-
foreign/node/src/wire/command-set.test.ts | 10 +-
foreign/node/src/wire/command-set.ts | 10 +-
foreign/node/src/wire/command.code.ts | 5 +-
.../src/wire/command.utils.test.ts} | 39 +-
foreign/node/src/wire/command.utils.ts | 8 +-
.../wire/consumer-group/delete-group.command.ts | 19 +-
.../src/wire/consumer-group/get-group.command.ts | 16 +-
.../node/src/wire/consumer-group/group.utils.ts | 7 +
foreign/node/src/wire/consumer-group/index.ts | 1 +
.../src/wire/consumer-group/join-group.command.ts | 19 +-
.../src/wire/consumer-group/leave-group.command.ts | 19 +-
.../wire/consumer-group/sync-group.command.test.ts | 99 +
.../src/wire/consumer-group/sync-group.command.ts | 67 +
foreign/node/src/wire/error.code.test.ts | 40 +
foreign/node/src/wire/error.code.ts | 32 +-
foreign/node/src/wire/error.utils.ts | 22 +-
.../src/wire/message/poll-messages.command.test.ts | 423 ++++
.../node/src/wire/message/poll-messages.command.ts | 211 +-
foreign/node/src/wire/offset/get-offset.command.ts | 2 +-
foreign/node/src/wire/offset/offset.utils.ts | 4 +-
.../node/src/wire/offset/store-offset.command.ts | 2 +-
.../topic.utils.test.ts} | 60 +-
foreign/node/src/wire/topic/topic.utils.ts | 4 +-
foreign/node/src/wire/vsr/header.test.ts | 99 +
foreign/node/src/wire/vsr/header.ts | 165 ++
foreign/node/src/wire/vsr/index.ts | 157 ++
foreign/node/src/wire/vsr/namespace.test.ts | 342 ++++
foreign/node/src/wire/vsr/namespace.ts | 216 ++
foreign/node/src/wire/vsr/operation.test.ts | 107 +
foreign/node/src/wire/vsr/operation.ts | 158 ++
foreign/node/src/wire/vsr/register.test.ts | 93 +
foreign/node/src/wire/vsr/register.ts | 116 ++
foreign/node/src/wire/vsr/reply.test.ts | 196 ++
foreign/node/src/wire/vsr/reply.ts | 172 ++
foreign/node/src/wire/vsr/session.test.ts | 73 +
foreign/node/src/wire/vsr/session.ts | 117 ++
foreign/node/src/wire/vsr/vsr.test.ts | 131 ++
foreign/node/tsconfig.json | 2 +-
foreign/python/Cargo.toml | 2 +-
foreign/python/apache_iggy.pyi | 334 ++-
foreign/python/src/client.rs | 103 +-
foreign/python/src/lib.rs | 6 +
foreign/python/src/permissions.rs | 458 +++++
foreign/python/src/user.rs | 9 +
foreign/python/tests/test_permissions.py | 695 +++++++
foreign/python/tests/test_user.py | 113 +-
foreign/python/tests/utils.py | 23 +
justfile | 4 +-
scripts/ci/trailing-newline.sh | 2 +-
scripts/ci/trailing-whitespace.sh | 2 +-
scripts/utils.sh | 2 +-
web/package-lock.json | 16 +-
web/package.json | 2 +-
246 files changed, 28225 insertions(+), 4267 deletions(-)
create mode 100644 core/configs/src/server_ng_config/partition.rs
create mode 100644 core/consensus/src/le_cursor.rs
create mode 100644 core/consensus/src/state_manifest.rs
create mode 100644 core/consensus/src/vsr_state.rs
create mode 100644
core/integration/tests/cluster/metadata_checkpoint_restart.rs
create mode 100644 core/integration/tests/cluster/metadata_state_transfer.rs
create mode 100644 core/integration/tests/server/cluster_view_durability_vsr.rs
create mode 100644
core/integration/tests/server/metadata_checkpoint_recovery_vsr.rs
create mode 100644 core/journal/src/superblock.rs
create mode 100644 core/server-ng/src/http/metrics.rs
create mode 100644 foreign/node/scripts/check-vsr-protocol.mjs
create mode 100644 foreign/node/src/client/client.config.test.ts
create mode 100644 foreign/node/src/client/client.config.ts
create mode 100644 foreign/node/src/client/client.connection.test.ts
create mode 100644 foreign/node/src/client/client.frame.test.ts
create mode 100644 foreign/node/src/client/client.frame.ts
create mode 100644 foreign/node/src/client/client.socket.test.ts
copy foreign/{go/contracts/duration.go => node/src/wire/command.utils.test.ts}
(60%)
create mode 100644
foreign/node/src/wire/consumer-group/sync-group.command.test.ts
create mode 100644 foreign/node/src/wire/consumer-group/sync-group.command.ts
create mode 100644 foreign/node/src/wire/error.code.test.ts
create mode 100644 foreign/node/src/wire/message/poll-messages.command.test.ts
copy foreign/node/src/wire/{token/delete-token.command.test.ts =>
topic/topic.utils.test.ts} (51%)
create mode 100644 foreign/node/src/wire/vsr/header.test.ts
create mode 100644 foreign/node/src/wire/vsr/header.ts
create mode 100644 foreign/node/src/wire/vsr/index.ts
create mode 100644 foreign/node/src/wire/vsr/namespace.test.ts
create mode 100644 foreign/node/src/wire/vsr/namespace.ts
create mode 100644 foreign/node/src/wire/vsr/operation.test.ts
create mode 100644 foreign/node/src/wire/vsr/operation.ts
create mode 100644 foreign/node/src/wire/vsr/register.test.ts
create mode 100644 foreign/node/src/wire/vsr/register.ts
create mode 100644 foreign/node/src/wire/vsr/reply.test.ts
create mode 100644 foreign/node/src/wire/vsr/reply.ts
create mode 100644 foreign/node/src/wire/vsr/session.test.ts
create mode 100644 foreign/node/src/wire/vsr/session.ts
create mode 100644 foreign/node/src/wire/vsr/vsr.test.ts
create mode 100644 foreign/python/src/permissions.rs
create mode 100644 foreign/python/tests/test_permissions.py
