[GitHub] kafka pull request #4251: KAFKA-6253: Improve sink connector topic regex val...
GitHub user jklukas opened a pull request: https://github.com/apache/kafka/pull/4251 KAFKA-6253: Improve sink connector topic regex validation KAFKA-3073 added topic regex support for sink connectors. The addition requires that you only specify one of topics or topics.regex settings. This is being validated in one place, but not during submission of connectors. This PR adds validation at `AbstractHerder.validateConnectorConfig` and `WorkerConnector.initialize`. This adds a test of the new behavior to `AbstractHerderTest`. ### Committer Checklist (excluded from commit message) - [ ] Verify design and implementation - [ ] Verify test coverage and CI build status - [ ] Verify documentation (including upgrade notes) You can merge this pull request into a Git repository by running: $ git pull https://github.com/jklukas/kafka connect-topics-validation Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/4251.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #4251 commit 830f1b627f6ba2d76f72626fac0741380d03a010 Author: Jeff Klukas <j...@klukas.net> Date: 2017-11-22T17:42:56Z KAFKA-6253: Improve sink connector topic regex validation ---
[GitHub] kafka pull request #4151: [WIP] KIP-215: Add topic regex support for Connect...
GitHub user jklukas opened a pull request: https://github.com/apache/kafka/pull/4151 [WIP] KIP-215: Add topic regex support for Connect sinks I'm starting this WIP PR to prove out the concept of the KIP and identify some additional public interface changes that I had not previously anticipated would be necessary. You can merge this pull request into a Git repository by running: $ git pull https://github.com/jklukas/kafka connect-topics.regex Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/4151.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #4151 commit cb8e2615e9ef1bf78e3d44671123b7e7231ee5f6 Author: Jeff Klukas <j...@klukas.net> Date: 2017-10-28T01:10:49Z Add topics.regex option and ensure existing tests pass ---
[GitHub] kafka pull request #3249: MINOR: Fix docs; client no longer uses yammer metr...
GitHub user jklukas opened a pull request: https://github.com/apache/kafka/pull/3249 MINOR: Fix docs; client no longer uses yammer metrics Yammer metrics is no longer a dependency of the client, but the Monitoring section of the docs contradicts this. For quite some time, clients have been using an internal metrics system unrelated to Yammer/Dropwizard metrics. This contribution is my original work and I license the work to the project under the project's open source license. You can merge this pull request into a Git repository by running: $ git pull https://github.com/jklukas/kafka docs-no-more-yammer-metrics-in-client Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/3249.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3249 commit 607c1579e86000fc4f45914ebedc779cbb0812d5 Author: Jeff Klukas <j...@klukas.net> Date: 2017-06-06T18:51:24Z Fix docs; client no longer uses yammer metrics --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] kafka pull request #1486: KAFKA-3753: Add size() method to KeyValueStore int...
GitHub user jklukas opened a pull request: https://github.com/apache/kafka/pull/1486 KAFKA-3753: Add size() method to KeyValueStore interface See https://issues.apache.org/jira/browse/KAFKA-3753 This contribution is my original work and I license the work to the project under the project's open source license. cc @guozhangwang @kichristensen @ijuma You can merge this pull request into a Git repository by running: $ git pull https://github.com/jklukas/kafka kvstore-size Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/1486.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1486 commit c4ea7cd0ed4d822e65cee3fd283f81c29ce11d57 Author: Jeff Klukas <j...@klukas.net> Date: 2016-06-09T15:08:33Z Add size() method to KeyValueStore interface --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] kafka pull request #1481: MINOR: Pass absolute directory path to RocksDB.ope...
GitHub user jklukas opened a pull request: https://github.com/apache/kafka/pull/1481 MINOR: Pass absolute directory path to RocksDB.open The method `RocksDB.open` assumes an absolute file path. If a relative path is configured, it leads to an exception like the following: ``` org.apache.kafka.streams.errors.ProcessorStateException: Error opening store CustomerIdToUserIdLookup at location ./tmp/rocksdb/CustomerIdToUserIdLookup at org.rocksdb.RocksDB.open(Native Method) at org.rocksdb.RocksDB.open(RocksDB.java:183) at org.apache.kafka.streams.state.internals.RocksDBStore.openDB(RocksDBStore.java:214) at org.apache.kafka.streams.state.internals.RocksDBStore.openDB(RocksDBStore.java:165) at org.apache.kafka.streams.state.internals.RocksDBStore.init(RocksDBStore.java:170) at org.apache.kafka.streams.state.internals.MeteredKeyValueStore.init(MeteredKeyValueStore.java:85) at org.apache.kafka.test.KStreamTestDriver.(KStreamTestDriver.java:64) at org.apache.kafka.test.KStreamTestDriver.(KStreamTestDriver.java:50) at com.simple.estuary.transform.streaming.CartesianTransactionEnrichmentJobTest.testBuilder(CartesianTransactionEnrichmentJobTest.java:41) ``` Is there any risk to always fetching the absolute path as proposed here? Let me know if you think this requires a JIRA issue or a unit test. I started working on a unit test, but don't know of a great solution for writing out a file to a relative directory. This contribution is my original work and I license the work to the project under the project's open source license. You can merge this pull request into a Git repository by running: $ git pull https://github.com/jklukas/kafka rocksdb-abspath Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/1481.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1481 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] kafka pull request #1479: KAFKA-3711: Ensure a RecordingMap is passed to con...
GitHub user jklukas opened a pull request: https://github.com/apache/kafka/pull/1479 KAFKA-3711: Ensure a RecordingMap is passed to configured instances See https://issues.apache.org/jira/browse/KAFKA-3711 I've tested locally that this change does indeed resolve the warning I mention in the ticket: ``` org.apache.kafka.clients.consumer.ConsumerConfig: The configuration metric.dropwizard.registry = kafka-metrics was supplied but isn't a known config. ``` where `metric.dropwizard.registry` is a configuration value defined in a custom `MetricReporter` (https://github.com/SimpleFinance/kafka-dropwizard-reporter). With this change, the above warning no longer appears, as @ewencp predicted. This contribution is my original work and I license the work to the project under the project's open source license. You can merge this pull request into a Git repository by running: $ git pull https://github.com/jklukas/kafka abstractconfig-originals Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/1479.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1479 commit 5c7a41b3116bb65db3cc0ca5cb90521766d41fcb Author: Jeff Klukas <j...@klukas.net> Date: 2016-06-07T14:33:32Z Ensure a RecordingMap is passed to configured instances --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] kafka pull request: KAFKA-3701: Expose KafkaStreams Metrics in pub...
Github user jklukas closed the pull request at: https://github.com/apache/kafka/pull/1362 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] kafka pull request: MINOR: Allow Serdes subclasses to access Wrapp...
GitHub user jklukas opened a pull request: https://github.com/apache/kafka/pull/1382 MINOR: Allow Serdes subclasses to access WrapperSerde This PR loosens access restrictions on `WrapperSerde` to `protected` so that users can define a `Serdes` subclass that provides additional custom serde members following the same pattern as the parent class. This is my own work and is compatible with Kafka's license. You can merge this pull request into a Git repository by running: $ git pull https://github.com/jklukas/kafka expose-wrapperserde Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/1382.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1382 commit 264b8cb4e75aef423fbfca4743a0d9107c2c2a4d Author: Jeff Klukas <j...@klukas.net> Date: 2016-05-13T17:04:35Z MINOR: Allow Serdes subclasses to access WrapperSerde --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] kafka pull request: MINOR: Call Metrics.close() in KafkaStreams
GitHub user jklukas opened a pull request: https://github.com/apache/kafka/pull/1379 MINOR: Call Metrics.close() in KafkaStreams Initially proposed by @ijuma in https://github.com/apache/kafka/pull/1362#issuecomment-218293662 @mjsax commented: > StreamThread.close() should be extended to call metrics.close() (the class need a private member to reference the Metrics object, too) The `Metrics` instance is created in the `KafkaStreams` constructor and shared between all threads, so closing it within the threads doesn't seem like the right approach. This PR calls `Metrics.close()` in `KafkaStreams.close()` instead. cc @guozhangwang You can merge this pull request into a Git repository by running: $ git pull https://github.com/jklukas/kafka close-streams-metrics Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/1379.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1379 commit 1066244aafa165e2ac22c9f43d44b754aca3c6d4 Author: Jeff Klukas <j...@klukas.net> Date: 2016-05-10T20:51:46Z Call Metrics.close() in KafkaStreams --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] kafka pull request: MINOR: Expose KafkaStreams Metrics in public A...
GitHub user jklukas opened a pull request: https://github.com/apache/kafka/pull/1362 MINOR: Expose KafkaStreams Metrics in public API The Kafka clients expose their metrics registries through a `metrics` method presenting an unmodifiable collection (see [KafkaProducer's metrics() method](https://github.com/apache/kafka/blob/2790d0cb1fc6ade216f2c1bf2667a0ebb53d85e3/clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java#L609-L615)), but `KafkaStreams` does not expose its registry. This PR exposes an unmodifiable collection of metrics on `KafkaStreams` in direct analogy to `KafkaProducer`. This contribution is my original work and I license the work to the project under the project's open source license. cc @guozhangwang @miguno @mjsax You can merge this pull request into a Git repository by running: $ git pull https://github.com/jklukas/kafka expose-streams-metrics Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/1362.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1362 commit 1e414d41c47bc04feb498a57d46f20bd1f582012 Author: Jeff Klukas <j...@klukas.net> Date: 2016-05-10T20:51:46Z Expose KafkaStreams Metrics in public API --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] kafka pull request: MINOR: Fix some copy-pasted Javadoc in Streams...
GitHub user jklukas opened a pull request: https://github.com/apache/kafka/pull/1270 MINOR: Fix some copy-pasted Javadoc in StreamsConfig.java This contribution is my original work and I license the work to the Kafka project under the project's open source license. cc @guozhangwang @miguno @ymatsuda You can merge this pull request into a Git repository by running: $ git pull https://github.com/jklukas/kafka streams-doc-fix Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/1270.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1270 commit 7034f7d548cd49b35b94e9f4d23fd627d5b92362 Author: Jeff Klukas <j...@klukas.net> Date: 2016-04-26T13:47:50Z Fix some copy-pasted Javadoc in StreamsConfig.java --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
