GitHub user sharad-develop opened a pull request:
https://github.com/apache/kafka/pull/2463
4706
4706 - Unify StreamsKafkaClient instances
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/apache/kafka 0.10.2
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/kafka/pull/2463.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2463
----
commit 7946b2f35c0e3811ba7d706fc9c761d73ece47bb
Author: Damian Guy <[email protected]>
Date: 2017-01-16T19:40:47Z
MINOR: Remove unused constructor param from ProcessorStateManager
Remove applicationId parameter as it is no longer used.
Author: Damian Guy <[email protected]>
Reviewers: Guozhang Wang <[email protected]>
Closes #2385 from dguy/minor-remove-unused-param
commit 621dff22e79dc64b9a8748186dd985774044f91a
Author: Rajini Sivaram <[email protected]>
Date: 2017-01-17T11:16:29Z
KAFKA-4363; Documentation for sasl.jaas.config property
Author: Rajini Sivaram <[email protected]>
Reviewers: Ismael Juma <[email protected]>
Closes #2316 from rajinisivaram/KAFKA-4363
commit e3f4cdd0e249f78a7f4e8f064533bcd15eb11cbf
Author: Rajini Sivaram <[email protected]>
Date: 2017-01-17T12:55:07Z
KAFKA-4590; SASL/SCRAM system tests
Runs sanity test and one replication test using SASL/SCRAM.
Author: Rajini Sivaram <[email protected]>
Reviewers: Ewen Cheslack-Postava <[email protected]>, Ismael Juma
<[email protected]>
Closes #2355 from rajinisivaram/KAFKA-4590
commit 2b19ad9d8c47fb0f78a6e90d2f5711df6110bf1f
Author: Rajini Sivaram <[email protected]>
Date: 2017-01-17T18:42:55Z
KAFKA-4580; Use sasl.jaas.config for some system tests
Switched console_consumer, verifiable_consumer and verifiable_producer to
use new sasl.jaas_config property instead of static JAAS configuration file
when used with SASL_PLAINTEXT.
Author: Rajini Sivaram <[email protected]>
Reviewers: Ewen Cheslack-Postava <[email protected]>, Ismael Juma
<[email protected]>
Closes #2323 from rajinisivaram/KAFKA-4580
(cherry picked from commit 3f6c4f63c9c17424cf717ca76c74554bcf3b2e9a)
Signed-off-by: Ismael Juma <[email protected]>
commit 60d759a227087079e6fd270c68fd9e38441cb34a
Author: Jason Gustafson <[email protected]>
Date: 2017-01-17T18:42:05Z
MINOR: Some cleanups and additional testing for KIP-88
Author: Jason Gustafson <[email protected]>
Reviewers: Vahid Hashemian <[email protected]>, Ismael Juma
<[email protected]>
Closes #2383 from hachikuji/minor-cleanup-kip-88
commit c9b9acf6a8b5433332c2d0d825c17a4a20cf3fa5
Author: Damian Guy <[email protected]>
Date: 2017-01-17T20:33:11Z
KAFKA-4588: Wait for topics to be created in
QueryableStateIntegrationTest.shouldNotMakeStoreAvailableUntilAllStoresAvailable
After debugging this i can see the times that it fails there is a race
between when the topic is actually created/ready on the broker and when the
assignment happens. When it fails `StreamPartitionAssignor.assign(..)` gets
called with a `Cluster` with no topics. Hence the test hangs as no tasks get
assigned. To fix this I added a `waitForTopics` method to
`EmbeddedKafkaCluster`. This will wait until the topics have been created.
Author: Damian Guy <[email protected]>
Reviewers: Matthias J. Sax, Guozhang Wang
Closes #2371 from dguy/integration-test-fix
(cherry picked from commit 825f225bc5706b16af8ec44ca47ee1452c11e6f3)
Signed-off-by: Guozhang Wang <[email protected]>
commit 6f72a5a53c444278187fa6be58031168bcaffb26
Author: Damian Guy <[email protected]>
Date: 2017-01-17T22:13:46Z
KAFKA-3452 Follow-up: Refactoring StateStore hierarchies
This is a follow up of https://github.com/apache/kafka/pull/2166 -
refactoring the store hierarchies as requested
Author: Damian Guy <[email protected]>
Reviewers: Guozhang Wang <[email protected]>
Closes #2360 from dguy/state-store-refactor
(cherry picked from commit 73b7ae0019d387407375f3865e263225c986a6ce)
Signed-off-by: Guozhang Wang <[email protected]>
commit eb62e5695506ae13bd37102c3c08e8a067eca0c8
Author: Ismael Juma <[email protected]>
Date: 2017-01-18T02:43:10Z
KAFKA-4591; Create Topic Policy follow-up
1. Added javadoc to public classes
2. Removed `s` from config name for consistency with interface name
3. The policy interface now implements Configurable and AutoCloseable as
per the KIP
4. Use `null` instead of `-1` in `RequestMetadata`
5. Perform all broker validation before invoking the policy
6. Add tests
Author: Ismael Juma <[email protected]>
Reviewers: Jason Gustafson <[email protected]>
Closes #2388 from ijuma/create-topic-policy-docs-and-config-name-change
(cherry picked from commit fd6d7bcf335166a524dc9a29a50c96af8f1c1c02)
Signed-off-by: Ismael Juma <[email protected]>
commit e38794e020951adec5a5d0bbfe42c57294bf67bd
Author: Guozhang Wang <[email protected]>
Date: 2017-01-18T04:29:55Z
KAFKA-3502; move RocksDB options construction to init()
In RocksDBStore, options / wOptions / fOptions are constructed in the
constructor, which needs to be dismissed in the close() call; however in some
tests, the generated topology is not initialized at all, and hence the
corresponding state stores are supposed to not be able to be closed as well
since their `init` function is not called. This could cause the above option
objects to be not released.
This is fixed in this patch to move the logic out of constructor and inside
`init` functions, so that no RocksDB objects will be created in the constructor
only. Also some minor cleanups:
1. In KStreamTestDriver.close(), we lost the logic to close the state
stores but only call `flush`; it is now changed back to call both.
2. Moved the forwarding logic from KStreamTestDriver to
MockProcessorContext to remove the mutual dependency: these functions should
really be in ProcessorContext, not the test driver.
Author: Guozhang Wang <[email protected]>
Reviewers: Damian Guy <[email protected]>, Matthias J. Sax
<[email protected]>, Jason Gustafson <[email protected]>
Closes #2381 from guozhangwang/K3502-pure-virtual-function-unit-tests
(cherry picked from commit 1974e1b0e54abe5fdebd8ff3338df864b7ab60f3)
Signed-off-by: Jason Gustafson <[email protected]>
commit 919548ba0e1d0e231008c381f824f91a8924dc6a
Author: Rajini Sivaram <[email protected]>
Date: 2017-01-18T11:05:58Z
KAFKA-4568; Simplify test code for multiple SASL mechanisms
Remove workaround for testing multiple SASL mechanisms using
sasl.jaas.config and the new support for multiple client
modules within a JVM.
Author: Rajini Sivaram <[email protected]>
Reviewers: Sriharsha Chintalapani <[email protected]>, Ismael Juma
<[email protected]>
Closes #2373 from rajinisivaram/KAFKA-4568
(cherry picked from commit 4c492975a05e4a646f7140c560ed403c48925331)
Signed-off-by: Ismael Juma <[email protected]>
commit 9b751f0c54e0409380632c4ee4618d78daca87d5
Author: Damian Guy <[email protected]>
Date: 2017-01-18T15:17:05Z
MINOR: Fix javadoc typos in KStream#process
interface for `Processor` in comments incorrectly had `transform` rather
than `process`.
Author: Damian Guy <[email protected]>
Reviewers: Michael G. Noll, Ismael Juma <[email protected]>
Closes #2396 from dguy/minor-javadoc
(cherry picked from commit ccb183f9fca20a37b9b39761dedcd197e3cf2033)
Signed-off-by: Ismael Juma <[email protected]>
commit 638a82b7bbf5b375f951e5827983753e9771dda0
Author: Damian Guy <[email protected]>
Date: 2017-01-18T17:19:06Z
HOTFIX: ChangeLoggingKeyValueStore.name() returns null
This class doesn't need to override this method as it is handled
appropriately by the super class
Author: Damian Guy <[email protected]>
Reviewers: Guozhang Wang <[email protected]>
Closes #2397 from dguy/hotfix-npe-state-store
(cherry picked from commit 961ebca57fdd4f8f84df8cf26a835f07ea1718c9)
Signed-off-by: Guozhang Wang <[email protected]>
commit dbca4a3b60d6039558c6fec99c86400c2131f9e8
Author: Matthias J. Sax <[email protected]>
Date: 2017-01-18T19:55:23Z
MINOR: refactor streams system test class hierachy
Author: Matthias J. Sax <[email protected]>
Reviewers: Eno Thereska, Guozhang Wang
Closes #2392 from mjsax/minor-system-test-rework
(cherry picked from commit d8a77560c2fa2c209353e3ba2366ad3d4cfdf22c)
Signed-off-by: Guozhang Wang <[email protected]>
commit 866b33c16496ab654d59e69cb3e83587c52da673
Author: Colin P. Mccabe <[email protected]>
Date: 2017-01-18T22:45:57Z
KAFKA-4457; Add BrokerApiVersionsCommand
Author: Colin P. Mccabe <[email protected]>
Reviewers: Ewen Cheslack-Postava <[email protected]>, Apurva Mehta
<[email protected]>, Ismael Juma <[email protected]>
Closes #2184 from cmccabe/KAFKA-4457
(cherry picked from commit 4a6f2c6cc0647a08f016a2d712a01ec02630cf87)
Signed-off-by: Ismael Juma <[email protected]>
commit 5562d63e8a02fdddaabe066437fe7674d28d5d01
Author: Hojjat Jafarpour <[email protected]>
Date: 2017-01-18T23:53:17Z
KAFKA-4060; Follow-up: Throw exceptions when internal topics to create
already exist with unexpected number of partitions
Re-branched the trunk and applied the changes to the new branch to simplify
commit log.
Author: Hojjat Jafarpour <[email protected]>
Reviewers: Ismael Juma, Damian Guy, Eno Thereska, Guozhang Wang
Closes #2389 from
hjafarpour/KAFKA-4060-Remove-ZkClient-dependency-in-Kafka-Streams-followup-from-trunk
Address Ismael's comments upon merging
(cherry picked from commit 8e2cbae8b12cd71f99f13953b744163f68021323)
Signed-off-by: Guozhang Wang <[email protected]>
commit ae5591a3af45832d9a50e2aaa31b11c039d8d389
Author: Rajini Sivaram <[email protected]>
Date: 2017-01-19T16:35:39Z
KAFKA-4589; SASL/SCRAM documentation
Author: Rajini Sivaram <[email protected]>
Reviewers: Ismael Juma <[email protected]>, Gwen Shapira
<[email protected]>, Sriharsha Chintalapani <[email protected]>, Jun Rao
<[email protected]>
Closes #2369 from rajinisivaram/KAFKA-4589
(cherry picked from commit 666abafcc54f8cab64912355dba4c8ada8e44827)
Signed-off-by: Jun Rao <[email protected]>
commit 770bab39916e213443aff8f696c6097b551b9408
Author: Yuto Kawamura <[email protected]>
Date: 2017-01-19T17:47:34Z
KAFKA-4614; Forcefully unmap mmap of OffsetIndex to prevent long GC pause
Issue: https://issues.apache.org/jira/browse/KAFKA-4614
Fixes the problem that the broker threads suffered by long GC pause.
When GC thread collects mmap objects which were created for index files, it
unmaps memory mapping so kernel turns to delete a file physically. This work
may transparently read file's metadata from physical disk if it's not available
on cache.
This seems to happen typically when we're using G1GC, due to it's strategy
to left a garbage for a long time if other objects in the same region are still
alive.
See the link for the details.
Author: Yuto Kawamura <[email protected]>
Reviewers: Apurva Mehta <[email protected]>, Guozhang Wang
<[email protected]>, Ismael Juma <[email protected]>,
Closes #2352 from kawamuray/KAFKA-4614-force-munmap-for-index
(cherry picked from commit 5fc530bc483db145e0cba3b63a57d6d6a7c547f2)
Signed-off-by: Jun Rao <[email protected]>
commit 17dd4db29fe7f930ca1c5d0fb5687a48c364f2d7
Author: Vahid Hashemian <[email protected]>
Date: 2017-01-20T00:35:12Z
KAFKA-4547; Avoid incorrect reset of paused partitions to the committed
offsets
Author: Vahid Hashemian <[email protected]>
Reviewers: Jason Gustafson <[email protected]>
Closes #2341 from vahidhashemian/KAFKA-4547
commit 3e1abac4c7d298d47567d23d6390eae4cd9ac1f8
Author: Sandesh K <[email protected]>
Date: 2017-01-20T01:46:59Z
KAFKA-4432; Added support to supply custom message payloads to
perf-producer script.
Current implementation of ProducerPerformance creates static payload. This
is not very useful in testing compression or when you want to test with
production/custom payloads. So, we decided to add support for providing payload
file as an input to producer perf test script.
We made the following changes:
1. Added support to provide a payload file which can have the list of
payloads that you actually want to send.
2. Moved payload generation inside the send loop for cases when payload
file is provided.
Following are the changes to how the producer-performance is evoked:
1. You must provide "--record-size" or "--payload-file" but not both. This
is because, record size cannot be guaranteed when you are using custom events.
e.g. ./kafka-producer-perf-test.sh --topic test_topic --num-records
100000 --producer-props bootstrap.servers=127.0.0.1:9092 acks=0
buffer.memory=33554432 compression.type=gzip batch.size=10240 linger.ms=10
--throughput -1 --payload-file ./test_payloads --payload-delimiter ,
2. Earlier "--record-size" was a required config, now you must provide
exactly one of "--record-size" or "--payload-file". Providing both will result
in an error.
3. Support for an additional parameter "--payload-delimiter" has been added
which defaults to "\n"
Author: Sandesh K <[email protected]>
Reviewers: dan norwood <[email protected]>, Jun Rao <[email protected]>
Closes #2158 from SandeshKarkera/PerfProducerChanges
(cherry picked from commit a37bf5fffa4a607ded14ec055276d76f19df2d50)
Signed-off-by: Jun Rao <[email protected]>
commit 05237367d7a12f395c9b2a1c2b354e1564f21aae
Author: Xavier Léauté <[email protected]>
Date: 2017-01-20T19:07:57Z
MINOR: reduce verbosity of cache flushes
This log message tends to be extremely verbose when state stores are being
restored
Author: Xavier Léauté <[email protected]>
Reviewers: Guozhang Wang <[email protected]>
Closes #2412 from xvrl/reduce-verbosity
(cherry picked from commit 50e035dc341bb595decbac0851354a10c90b0e06)
Signed-off-by: Guozhang Wang <[email protected]>
commit c3f923cbf9f686bf42adda44d34a975167a13159
Author: Shikhar Bhushan <[email protected]>
Date: 2017-01-21T00:15:15Z
KAFKA-3209: KIP-66: more single message transforms
Renames `HoistToStruct` SMT to `HoistField`.
Adds the following SMTs:
`ExtractField`
`MaskField`
`RegexRouter`
`ReplaceField`
`SetSchemaMetadata`
`ValueToKey`
Adds HTML doc generation and updates to `connect.html`.
Author: Shikhar Bhushan <[email protected]>
Reviewers: Ewen Cheslack-Postava <[email protected]>
Closes #2374 from shikhar/more-smt
commit db57c1a0c7e057e19ec165d3cf9fa35d54b97c8f
Author: Matthias J. Sax <[email protected]>
Date: 2017-01-21T21:18:39Z
KAFKA-4060 and KAFKA-4476 follow up: remove unnecessary
InternalTopicManager branch and fixed one copartitioning bug
ZK removed reveal a bug in `StreamPartitionAssigner` but did not fix it
properly. This is a follow up bug fix.
Issue:
- If topic metadata is missing, `StreamPartitionAssigner` should not
create any affected tasks that consume topics with missing metadata.
- Depending downstream tasks should not be create either.
- For tasks that are not created, no store changelog topics (if any)
should get created
- For tasks that write output to not-yet existing internal repartitioning
topics, those repartitioning topics should not get created
Author: Matthias J. Sax <[email protected]>
Reviewers: Damian Guy, Guozhang Wang
Closes #2404 from mjsax/kafka-4060-zk-test-follow-up
(cherry picked from commit 0b99bea590842018e8e97e7fd1c71b1471db4d08)
Signed-off-by: Guozhang Wang <[email protected]>
commit e393c9928972c00677250ae2df3b4f061fdccd58
Author: Xavier Léauté <[email protected]>
Date: 2017-01-21T22:10:01Z
KAFKA-4672; fix source compatibility for lambda expressions
Variance changes introduced in KIP-100 cause compilation failures with
lambda expression in Java 8. To my knowledge this only affects the following
method
`KStreams.transform(TransformerSupplier<...>, String...)`
prior to the changes it was possible to write:
`streams.transform(MyTransformer::new)`
where `MyTransformer` extends `Transformer`
After the changes the Java compiler is unable to infer correct return types
for the lambda expressions. This change fixed this by reverting to invariant
return types for transformer suppliers.
please cherry-pick into 0.10.2.x
Author: Xavier Léauté <[email protected]>
Reviewers: Ismael Juma, Damian Guy, Guozhang Wang
Closes #2402 from xvrl/lambdas-oh-my
(cherry picked from commit 20e957c9b1f389621bb8ae9e9f5554b66cd92a4f)
Signed-off-by: Guozhang Wang <[email protected]>
commit f0449d3248801facca442d93552a829e8b8d06a7
Author: Eno Thereska <[email protected]>
Date: 2017-01-23T14:45:30Z
MINOR: Pass RecordingLevel to MetricConfig in the broker
This is a KIP-104/105 follow-up. Thanks to ijuma for pointing out.
Author: Eno Thereska <[email protected]>
Reviewers: Ismael Juma <[email protected]>
Closes #2350 from enothereska/minor-broker-level-config
(cherry picked from commit 1eb1e2f60ae006144757fc9fc10ab423c58970bb)
Signed-off-by: Ismael Juma <[email protected]>
commit afde87804211dfea2b9ee87c5f6470e8460dcd33
Author: Matthias J. Sax <[email protected]>
Date: 2017-01-23T17:34:52Z
HOTFIX: KAFKA-4060 and KAFKA-4476 follow up
Author: Matthias J. Sax <[email protected]>
Reviewers: Damian Guy, Guozhang Wang
Closes #2418 from mjsax/kafka-4060-zk-test-follow-up
(cherry picked from commit dd897bdb2e9cc8790d1e0494fb7867a5cd09ecc6)
Signed-off-by: Guozhang Wang <[email protected]>
commit 80ceb75d2acf5c3d8c21e03bf55e706c637d9fd9
Author: Hojjat Jafarpour <[email protected]>
Date: 2017-01-23T18:21:06Z
KAFKA-4060 Follow-up: update docs accordingly
Updated the docs with changes in KAFKA-4060.
Author: Hojjat Jafarpour <[email protected]>
Author: Hojjat Jafarpour <[email protected]>
Reviewers: Ismael Juma, Matthias J. Sax, Guozhang Wang
Closes #2377 from hjafarpour/KAFKA-4060-docs-update
(cherry picked from commit 62206de4e52dbec32b99baffa0ebf7f5303d7b7f)
Signed-off-by: Guozhang Wang <[email protected]>
commit 0f87991d5dbe8741953be04628810d76a9ee7434
Author: Guozhang Wang <[email protected]>
Date: 2017-01-23T18:45:45Z
KAFKA-4633; Always using regex pattern subscription in StreamThread
1. In StreamThread, always use subscribe(Pattern, ..) function in order to
avoid sending MetadataRequest with specific topic names and cause brokers to
possibly auto-create subscribed topics; the pattern is generated as
"topic-1|topic-2..|topic-n".
2. In ConsumerCoordinator, let the leader to refresh its metadata if the
generated assignment contains some topics that is not contained in the
subscribed topics; also in SubscriptionState, modified the verification for
regex subscription to against the regex pattern instead of the matched topics
since the returned assignment may contain some topics not yet created when
joining the group but existed after the rebalance; also modified some unit
tests in `KafkaConsumerTest` to accommodate the above changes.
3. Minor cleanup: changed String[] to List<String> to avoid overloaded
functions.
4. Minor cleanup: enforced strong typing in SinkNodeFactory and removed
unnecessary unchecked tags.
5. Minor cleanup: augmented unit test error message and fixed a potential
transient failure in KafkaStreamTest.
Author: Guozhang Wang <[email protected]>
Reviewers: Damian Guy <[email protected]>, Matthias J. Sax
<[email protected]>, Jason Gustafson <[email protected]>
Closes #2379 from guozhangwang/K4633-regex-pattern
(cherry picked from commit 3400d0c3cc07d5d3136bf9a19142b36cca93f92d)
Signed-off-by: Jason Gustafson <[email protected]>
commit 4218c02478a213935afbb2c2d661d8a29a5f29ee
Author: Matthias J. Sax <[email protected]>
Date: 2017-01-24T00:31:36Z
KAFKA-4671: Fix Streams window retention policy
Author: Matthias J. Sax <[email protected]>
Reviewers: Damian Guy, Eno Thereska, Guozhang Wang
Closes #2401 from mjsax/kafka-4671-window-retention-policy
(cherry picked from commit 79987590e3e96351ff75ce86718801ec605b2419)
Signed-off-by: Guozhang Wang <[email protected]>
commit 928f69ce883c8e30050d2e6f5e72e77e6740bdb9
Author: huxi <[email protected]>
Date: 2017-01-24T10:04:42Z
KAFKA-4576; Log segments close to max size break on fetch
`FileChannel.read` may not fill the destination buffer even if there
are enough bytes in the channel to do so. Add a couple of utility
methods that ensure this and use them from all the relevant places.
Author: huxi <[email protected]>
Author: amethystic <[email protected]>
Author: Ismael Juma <[email protected]>
Reviewers: Jun Rao <[email protected]>, Jason Gustafson
<[email protected]>, Ismael Juma <[email protected]>
Closes #2304 from amethystic/kafka4576_FileChannel_read
(cherry picked from commit 337f576f5979bf8924c5707b338cf4d3c76a53fd)
Signed-off-by: Ismael Juma <[email protected]>
commit b676f51dd54125b950fee35fe867057778866bcb
Author: Colin P. Mccabe <[email protected]>
Date: 2017-01-24T11:09:47Z
KAFKA-4688; Kafka 0.10.1.1 should be available in system tests
Author: Colin P. Mccabe <[email protected]>
Reviewers: Ewen Cheslack-Postava <[email protected]>, Ismael Juma
<[email protected]>
Closes #2424 from cmccabe/KAFKA-4688
(cherry picked from commit 567180605c0a8cf74b8d2cad52e2873c83065a95)
Signed-off-by: Ismael Juma <[email protected]>
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
