This is an automated email from the ASF dual-hosted git repository.
vvcephei pushed a change to branch kafka-10867-improved-task-idling
in repository https://gitbox.apache.org/repos/asf/kafka.git.
discard fa1af20 decrease log verbosity at DEBUG for PartitionGroup
discard 0634433 CR feedback
discard 57cce0e rebase fix
discard 1eeae1f Rollback streams-side metadata caching
discard 56fda48 Streams changes to cope with cached metadata response
discard 28b541d ditto
discard e205d27 disable idling for selected ITs
discard f59abc8 fix diff
discard 4e15d21 switch default
discard 93252f6 KAFKA-10867: Improved task idling
discard ffbaeab guard against early long-poll return due to metadata-only
response
discard c2ce4c8 CR feedback
discard 59cb4a5 drop beginningOffset from the metadata
discard ce1230a Rollback consumer-side metadata caching
discard 88cf06a Consumer changes to cope with cached metadata response
discard 79f8f1f spotbugs
discard 1ffa71c fix test
discard 8d64a7c fix core integration test failures
discard 746f235 tests
discard 5ce8141 KAFKA-10866: Add fetched metadata to ConsumerRecords
add 9689a31 MINOR: Drop enable.metadata.quorum config (#9934)
add 45550e9 MINOR: log 2min processing summary of StreamThread loop
(#9941)
add 7a1d1d9 KAFKA-12212; Bump Metadata API version to remove
`ClusterAuthorizedOperations` fields (KIP-700) (#9945)
add 019cd4a KAFKA-10357: Extract setup of repartition topics from Streams
partition assignor (#9848)
add 7b06a24 MINOR: Restore interrupt status when closing (#9863)
add 667a6b2 MINOR: fix record time in test
shouldWipeOutStandbyStateDirectoryIfCheckpointIsMissing (#9948)
add 91d6c55 MINOR: Upgrade ducktape to version 0.8.1 (#9933)
add 6d41162 MINOR: A few small group coordinator cleanups (#9952)
add b8a9201 MINOR: Update to Gradle 6.8.1 (#9953)
add 6f8ca66 MINOR: Tag `RaftEventSimulationTest` as `integration` and
tweak it (#9925)
add f4c2030 MINOR: Optimize assertions in unit tests (#9955)
add cb0e781 MINOR: Revert assertion in MockProducerTest (#9956)
add 24a2ed2 MINOR: Update zstd-jni to 1.4.8-2 (#9957)
add cd0b928 MINOR: MessageUtil: remove some deadcode (#9931)
add 35f03ec MINOR: Fix typo in Utils#toPositive (#9943)
add 411ac7d MINOR: Remove `toStruct` and `fromStruct` methods from
generated protocol classes (#9960)
add e1a4dcc KAFKA-12190: Fix setting of file permissions on non-POSIX
filesystems (#9947)
add 45b7a0a KAFKA-12229: Restore original class loader in integration
tests using EmbeddedConnectCluster during shutdown (#9942)
add 7230d14 MINOR: Fix visibility of Log.{unflushedMessages, addSegment}
methods (#9966)
add 0c6bb3d MINOR: remove unused code from MessageTest (#9961)
add 3d67761 MINOR: set initial capacity of ArrayList for all json
converters (#9962)
add f572545 KAFKA-10763: Fix incomplete cooperative rebalances preventing
connector/task revocations (#9765)
add a26db2a KAFKA-10694; Implement zero copy for FetchSnapshot (#9819)
add 6c8033a MINOR: Add docs for StreamJoined changes (#9951)
add 647c609 KAFKA-10555: Improve client state machine (#9720)
add 35b1e76 KAFKA-10793: move handling of FindCoordinatorFuture to fix
race condition (#9671)
add c830bce MINOR: Fix meaningless message in assertNull validation
(#9965)
add 4075a5c MINOR: Call logSegments.toBuffer only when required (#9971)
add 91c504c KAFKA-12233: Align the length passed to FileChannel by
`FileRecords.writeTo` (#9970)
add fc68c0f MINOR: Remove redundant apostrophe in doc (#9976)
add 8bdab2e MINOR: Remove outdated comment in Connect's WorkerCoordinator
(#9805)
add fb6c7be MINOR: Remove redundant casting and if condition from
ConnectSchema (#9959)
add 5cf9cfc MINOR: Update zookeeper to 3.5.9 (#9977)
add f4983f4 KAFKA-8930: MirrorMaker v2 documentation (#324) (#9983)
add bd990ab KAFKA-10866: Add fetched metadata to ConsumerRecords
add 2b0f202 tests
add c75b94e fix core integration test failures
add b0fde7c fix test
add b4d2b0d spotbugs
add 9502a9c Consumer changes to cope with cached metadata response
add cb61e58 Rollback consumer-side metadata caching
add 3fd6831 drop beginningOffset from the metadata
add fd952b8 CR feedback
add fc3ec40 guard against early long-poll return due to metadata-only
response
add 9ff83e5 KAFKA-10867: Improved task idling
add 5e5a706 switch default
add 026480b fix diff
add 67dbbdd disable idling for selected ITs
add 9cc574c ditto
add 902ee61 Streams changes to cope with cached metadata response
add 7decbfb Rollback streams-side metadata caching
add 969382c rebase fix
add d7d53cf CR feedback
add 12fb3c4 decrease log verbosity at DEBUG for PartitionGroup
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (fa1af20)
\
N -- N -- N refs/heads/kafka-10867-improved-task-idling (12fb3c4)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
No new revisions were added by this update.
Summary of changes:
checkstyle/import-control.xml | 1 +
.../org/apache/kafka/clients/NodeApiVersions.java | 4 +-
.../kafka/clients/consumer/ConsumerConfig.java | 2 +-
.../consumer/internals/AbstractCoordinator.java | 59 ++-
.../org/apache/kafka/common/PartitionInfo.java | 2 -
.../apache/kafka/common/config/SaslConfigs.java | 2 +-
.../org/apache/kafka/common/protocol/ApiKeys.java | 30 +-
.../org/apache/kafka/common/protocol/Message.java | 25 -
.../apache/kafka/common/protocol/MessageUtil.java | 4 +-
.../org/apache/kafka/common/protocol/Protocol.java | 2 +-
.../apache/kafka/common/protocol/SendBuilder.java | 4 +
.../kafka/common/record/AbstractRecords.java | 4 +-
.../kafka/common/record/CompressionType.java | 19 +-
.../kafka/common/record/DefaultRecordsSend.java | 6 +-
.../apache/kafka/common/record/FileRecords.java | 30 +-
.../record/LazyDownConversionRecordsSend.java | 2 +-
.../apache/kafka/common/record/MemoryRecords.java | 7 +-
.../org/apache/kafka/common/record/Records.java | 14 +-
...ltRecordsSend.java => TransferableRecords.java} | 27 +-
...tRecordsSend.java => UnalignedFileRecords.java} | 28 +-
.../common/record/UnalignedMemoryRecords.java | 56 ++
...tractRecordBatch.java => UnalignedRecords.java} | 21 +-
.../kafka/common/requests/ApiVersionsResponse.java | 4 +-
.../DefaultKafkaPrincipalBuilder.java | 15 +-
.../java/org/apache/kafka/common/utils/Utils.java | 28 +-
.../common/message/FetchSnapshotResponse.json | 4 +-
.../resources/common/message/MetadataRequest.json | 10 +-
.../resources/common/message/MetadataResponse.json | 10 +-
.../src/main/resources/common/message/README.md | 6 -
.../apache/kafka/clients/NodeApiVersionsTest.java | 8 +-
.../kafka/clients/admin/KafkaAdminClientTest.java | 2 +-
.../apache/kafka/common/message/MessageTest.java | 50 +-
.../kafka/common/message/RecordsSerdeTest.java | 26 +-
.../common/message/SimpleExampleMessageTest.java | 74 +--
.../kafka/common/protocol/SendBuilderTest.java | 33 ++
.../common/record/ControlRecordUtilsTest.java | 4 +-
.../kafka/common/record/FileRecordsTest.java | 29 +
.../common/record/UnalignedFileRecordsTest.java | 79 +++
.../common/requests/ApiVersionsResponseTest.java | 6 +-
.../common/requests/JoinGroupRequestTest.java | 14 +-
.../common/requests/OffsetFetchResponseTest.java | 16 +-
.../test/java/org/apache/kafka/test/TestUtils.java | 7 +-
.../apache/kafka/connect/data/ConnectSchema.java | 46 +-
.../MirrorConnectorsIntegrationBaseTest.java | 2 +-
.../runtime/distributed/DistributedHerder.java | 3 +-
.../runtime/distributed/WorkerCoordinator.java | 3 -
.../runtime/distributed/DistributedHerderTest.java | 108 +++-
.../runtime/distributed/WorkerCoordinatorTest.java | 16 +-
.../util/clusters/EmbeddedConnectCluster.java | 5 +
core/src/main/scala/kafka/Kafka.scala | 5 +-
.../kafka/coordinator/group/DelayedHeartbeat.scala | 4 +-
.../kafka/coordinator/group/DelayedJoin.scala | 1 -
.../kafka/coordinator/group/GroupCoordinator.scala | 23 +-
.../kafka/coordinator/group/GroupMetadata.scala | 16 +-
.../coordinator/group/GroupMetadataManager.scala | 1 -
.../kafka/coordinator/group/MemberMetadata.scala | 4 +-
core/src/main/scala/kafka/log/Log.scala | 10 +-
.../main/scala/kafka/network/RequestChannel.scala | 8 +-
.../main/scala/kafka/network/SocketServer.scala | 14 +-
.../scala/kafka/server/ForwardingManager.scala | 4 +-
core/src/main/scala/kafka/server/KafkaApis.scala | 46 +-
core/src/main/scala/kafka/server/KafkaConfig.scala | 21 +-
core/src/main/scala/kafka/server/KafkaServer.scala | 19 +-
.../main/scala/kafka/tools/TestRaftServer.scala | 7 +-
.../kafka/admin/BrokerApiVersionsCommandTest.scala | 2 +-
.../kafka/api/AuthorizerIntegrationTest.scala | 77 ++-
.../server/DynamicBrokerReconfigurationTest.scala | 4 +-
.../test/scala/unit/kafka/api/ApiVersionTest.scala | 2 +-
.../controller/ControllerChannelManagerTest.scala | 6 +-
.../coordinator/group/GroupCoordinatorTest.scala | 6 +-
.../group/GroupMetadataManagerTest.scala | 17 +-
.../coordinator/group/GroupMetadataTest.scala | 21 +-
.../coordinator/group/MemberMetadataTest.scala | 12 +-
.../kafka/integration/KafkaServerTestHarness.scala | 11 +-
.../test/scala/unit/kafka/log/LogManagerTest.scala | 2 +-
core/src/test/scala/unit/kafka/log/LogTest.scala | 2 +-
.../security/auth/SimpleAclAuthorizerTest.scala | 4 +-
.../security/authorizer/AclAuthorizerTest.scala | 4 +-
.../delegation/DelegationTokenManagerTest.scala | 2 +-
.../server/AbstractApiVersionsRequestTest.scala | 2 +-
.../CreateTopicsRequestWithForwardingTest.scala | 6 +-
.../unit/kafka/server/ForwardingManagerTest.scala | 2 +-
.../scala/unit/kafka/server/KafkaApisTest.scala | 13 +-
.../scala/unit/kafka/server/KafkaConfigTest.scala | 17 +
.../scala/unit/kafka/server/RequestQuotaTest.scala | 6 +-
.../scala/unit/kafka/utils/SchedulerTest.scala | 2 +-
.../test/scala/unit/kafka/utils/TestUtils.scala | 6 +-
docs/ops.html | 583 ++++++++++++++++++++-
docs/streams/upgrade-guide.html | 6 +
docs/toc.html | 20 +-
.../kafka/message/JsonConverterGenerator.java | 2 +-
.../apache/kafka/message/MessageDataGenerator.java | 370 +------------
gradle/dependencies.gradle | 6 +-
gradle/wrapper/gradle-wrapper.properties | 2 +-
gradlew | 2 +-
.../jmh/metadata/MetadataRequestBenchmark.java | 6 +-
raft/README.md | 9 +-
.../org/apache/kafka/raft/KafkaRaftClient.java | 42 +-
.../kafka/snapshot/FileRawSnapshotReader.java | 14 +-
.../kafka/snapshot/FileRawSnapshotWriter.java | 16 +-
.../apache/kafka/snapshot/RawSnapshotReader.java | 17 +-
.../apache/kafka/snapshot/RawSnapshotWriter.java | 36 +-
.../org/apache/kafka/snapshot/SnapshotWriter.java | 13 +-
.../kafka/raft/KafkaRaftClientSnapshotTest.java | 52 +-
.../test/java/org/apache/kafka/raft/MockLog.java | 27 +-
.../apache/kafka/raft/RaftEventSimulationTest.java | 86 ++-
.../apache/kafka/snapshot/FileRawSnapshotTest.java | 100 +++-
.../org/apache/kafka/streams/KafkaStreams.java | 111 ++--
.../internals/InternalTopologyBuilder.java | 2 +-
.../processor/internals/RepartitionTopics.java | 212 ++++++++
.../processor/internals/StateDirectory.java | 24 +-
.../streams/processor/internals/StreamThread.java | 73 ++-
.../internals/StreamsPartitionAssignor.java | 151 +-----
.../streams/processor/internals/TaskManager.java | 3 +-
.../org/apache/kafka/streams/KafkaStreamsTest.java | 93 +---
.../EOSUncleanShutdownIntegrationTest.java | 2 -
.../integration/EosBetaUpgradeIntegrationTest.java | 40 +-
.../integration/StandbyTaskEOSIntegrationTest.java | 16 +-
...amsUncaughtExceptionHandlerIntegrationTest.java | 8 +-
.../integration/SuppressionIntegrationTest.java | 2 +-
...HandlingSourceTopicDeletionIntegrationTest.java | 12 +-
.../processor/internals/RepartitionTopicsTest.java | 429 +++++++++++++++
.../processor/internals/StateDirectoryTest.java | 40 +-
.../state/internals/AbstractKeyValueStoreTest.java | 2 +-
tests/docker/Dockerfile | 2 +-
tests/setup.py | 2 +-
126 files changed, 2525 insertions(+), 1474 deletions(-)
copy
clients/src/main/java/org/apache/kafka/common/record/{DefaultRecordsSend.java
=> TransferableRecords.java} (60%)
copy
clients/src/main/java/org/apache/kafka/common/record/{DefaultRecordsSend.java
=> UnalignedFileRecords.java} (53%)
create mode 100644
clients/src/main/java/org/apache/kafka/common/record/UnalignedMemoryRecords.java
copy
clients/src/main/java/org/apache/kafka/common/record/{AbstractRecordBatch.java
=> UnalignedRecords.java} (70%)
create mode 100644
clients/src/test/java/org/apache/kafka/common/record/UnalignedFileRecordsTest.java
create mode 100644
streams/src/main/java/org/apache/kafka/streams/processor/internals/RepartitionTopics.java
create mode 100644
streams/src/test/java/org/apache/kafka/streams/processor/internals/RepartitionTopicsTest.java
