[GitHub] kafka pull request: MINOR: small code optimizations in streams
GitHub user ymatsuda opened a pull request: https://github.com/apache/kafka/pull/1176 MINOR: small code optimizations in streams @guozhangwang You can merge this pull request into a Git repository by running: $ git pull https://github.com/ymatsuda/kafka optimize Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/1176.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1176 commit 9d4c0c2a67e070ebb7cb8a1de5644d3505958861 Author: Yasuhiro Matsuda <yasuh...@confluent.io> Date: 2016-04-01T21:50:35Z MINOR: small code optimizations in streams --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] kafka pull request: MINOR: a simple benchmark for Streams
GitHub user ymatsuda opened a pull request: https://github.com/apache/kafka/pull/1164 MINOR: a simple benchmark for Streams @guozhangwang @miguno You can merge this pull request into a Git repository by running: $ git pull https://github.com/ymatsuda/kafka perf Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/1164.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1164 commit 8549122435b71b41a9270d0dc2c70b863c0cd064 Author: Yasuhiro Matsuda <yasuh...@confluent.io> Date: 2016-03-29T22:15:02Z MINOR: a simple benchmark for Streams --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] kafka pull request: HOTFIX: RocksDBStore must clear dirty flags af...
GitHub user ymatsuda opened a pull request: https://github.com/apache/kafka/pull/1163 HOTFIX: RocksDBStore must clear dirty flags after flush @guozhangwang Without clearing the dirty flags, RocksDBStore will perform flush for every new record. This bug made the store performance painfully slower. You can merge this pull request into a Git repository by running: $ git pull https://github.com/ymatsuda/kafka clear_dirty_flag Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/1163.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1163 commit 53690e66d3a9bf564a9bb64fe53b92f9a14c9dd8 Author: Yasuhiro Matsuda <yasuh...@confluent.io> Date: 2016-03-29T19:09:05Z HOTFIX: RocksDBStore must clear dirty flags after flush --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] kafka pull request: HOTFIX: stop using batch write to rocksdb
Github user ymatsuda closed the pull request at: https://github.com/apache/kafka/pull/1161 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] kafka pull request: HOTFIX: stop using batch write to rocksdb
GitHub user ymatsuda opened a pull request: https://github.com/apache/kafka/pull/1161 HOTFIX: stop using batch write to rocksdb @guozhangwang Removing batch write to RocksDB because I observed segmentation faults in RocksDB's batch write. You can merge this pull request into a Git repository by running: $ git pull https://github.com/ymatsuda/kafka no_batch_write Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/1161.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1161 commit dab7985529a8e3ca106cb4d577b91e72597bfd83 Author: Yasuhiro Matsuda <yasuh...@confluent.io> Date: 2016-03-28T22:04:02Z HOTFIX: stop using batch write to rocksdb --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] kafka pull request: HOTFIX: set timestamp in SinkNode
GitHub user ymatsuda opened a pull request: https://github.com/apache/kafka/pull/1137 HOTFIX: set timestamp in SinkNode @guozhangwang Setting the timestamp in produced records in SinkNode. This forces the producer record's timestamp same as the context's timestamp. You can merge this pull request into a Git repository by running: $ git pull https://github.com/ymatsuda/kafka set_timestamp_in_sinknode Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/1137.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1137 commit bd5dd3efae0784261728afddc7acd86612bef610 Author: Yasuhiro Matsuda <yasuh...@confluent.io> Date: 2016-03-24T22:05:30Z HOTFIX: set timestamp in SinkNode --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] kafka pull request: MINOR: remove streams-smoke-test.sh
GitHub user ymatsuda opened a pull request: https://github.com/apache/kafka/pull/1125 MINOR: remove streams-smoke-test.sh @guozhangwang You can merge this pull request into a Git repository by running: $ git pull https://github.com/ymatsuda/kafka remove_smoketest_shell_script Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/1125.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1125 commit ec89c471bbb818b09093bad0639cf316a2a8e89d Author: Yasuhiro Matsuda <yasuh...@confluent.io> Date: 2016-03-23T21:46:37Z MINOR: remove streams-smoke-test.sh --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] kafka pull request: HOTFIX: fix NPE in changelogger
GitHub user ymatsuda opened a pull request: https://github.com/apache/kafka/pull/1124 HOTFIX: fix NPE in changelogger Fix NPE in StoreChangeLogger caused by a record out of window retention period. @guozhangwang You can merge this pull request into a Git repository by running: $ git pull https://github.com/ymatsuda/kafka logger_npe Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/1124.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1124 commit 3fc5093a7d236b0ecba008dfe890bee300c33e51 Author: Yasuhiro Matsuda <yasuh...@confluent.io> Date: 2016-03-23T21:17:53Z HOTFIX: fix NPE in changelogger --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] kafka pull request: MINOR: kstream/ktable counting method with def...
GitHub user ymatsuda opened a pull request: https://github.com/apache/kafka/pull/1065 MINOR: kstream/ktable counting method with default long serdes @guozhangwang @miguno You can merge this pull request into a Git repository by running: $ git pull https://github.com/ymatsuda/kafka count_serdes Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/1065.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1065 commit f9350c6068010901e538052d8c039eaeccdf229a Author: Yasuhiro Matsuda <yasuh...@confluent.io> Date: 2016-03-14T22:24:12Z MINOR: kstream/ktable counting method with default long serdes --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] kafka pull request: KAFKA-3395: prefix job id to internal topic na...
GitHub user ymatsuda opened a pull request: https://github.com/apache/kafka/pull/1062 KAFKA-3395: prefix job id to internal topic names @guozhangwang You can merge this pull request into a Git repository by running: $ git pull https://github.com/ymatsuda/kafka k3395 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/1062.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1062 commit 6bd08b21adaa8373349918cd0e249fa4222c8e43 Author: Yasuhiro Matsuda <yasuh...@confluent.io> Date: 2016-03-14T20:15:05Z KAFKA-3395: prefix job id to internal topic names --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] kafka pull request: MINOR: add AUTO_OFFSET_RESET_CONFIG to Streams...
GitHub user ymatsuda opened a pull request: https://github.com/apache/kafka/pull/985 MINOR: add AUTO_OFFSET_RESET_CONFIG to StreamsConfig, and remove TOTAL_RECORDS_TO_PROCESS @guozhangwang You can merge this pull request into a Git repository by running: $ git pull https://github.com/ymatsuda/kafka config_params Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/985.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #985 commit 4074465fa0032c242a066a7684748df7eb432e02 Author: Yasuhiro Matsuda <yasuh...@confluent.io> Date: 2016-02-29T22:47:04Z MINOR: add AUTO_OFFSET_RESET_CONFIG to StreamsConfig, remove TOTAL_RECORDS_TO_PROCESS commit 1ee31381eb2ed55aceb836fce214a2a5b3f3f5cc Author: Yasuhiro Matsuda <yasuh...@confluent.io> Date: 2016-02-29T22:47:14Z Merge branch 'trunk' of github.com:apache/kafka into config_params --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] kafka pull request: MINOR: enhance streams system test
GitHub user ymatsuda opened a pull request: https://github.com/apache/kafka/pull/966 MINOR: enhance streams system test @guozhangwang * add table aggregate to the system test * actually create change log partition replica You can merge this pull request into a Git repository by running: $ git pull https://github.com/ymatsuda/kafka enh_systest Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/966.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #966 commit 6c45b52f97e61282ffda141fc6b4c0f6ffcbf7f1 Author: Yasuhiro Matsuda <yasuh...@confluent.io> Date: 2016-02-24T15:01:13Z MINOR: enhance streams system test --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] kafka pull request: HOTFIX: fix consumer config for streams
GitHub user ymatsuda opened a pull request: https://github.com/apache/kafka/pull/959 HOTFIX: fix consumer config for streams @guozhangwang My bad. I removed ZOOKEEPER_CONNECT_CONFIG from consumer's config by mistake. It is needed by our own partition assigner running in consumers. You can merge this pull request into a Git repository by running: $ git pull https://github.com/ymatsuda/kafka hotfix3 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/959.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #959 commit b01d80d10548647e15f70c7fcaf34a0b8d8c5742 Author: Yasuhiro Matsuda <yasuh...@confluent.io> Date: 2016-02-23T22:47:55Z HOTFIX: fix consumer config for streams --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] kafka pull request: KAFKA-3245: config for changelog replication f...
GitHub user ymatsuda opened a pull request: https://github.com/apache/kafka/pull/948 KAFKA-3245: config for changelog replication factor @guozhangwang You can merge this pull request into a Git repository by running: $ git pull https://github.com/ymatsuda/kafka changelog_topic_replication Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/948.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #948 commit 9f3ec06214d6bdbad5833ffb3b68512ae9c58bbc Author: Yasuhiro Matsuda <yasuh...@confluent.io> Date: 2016-02-18T21:37:27Z change log replication commit ce5ebe42cdbc79a73fedccc5dbaf3d9c8d03597f Author: Yasuhiro Matsuda <yasuh...@confluent.io> Date: 2016-02-22T21:26:28Z Merge branch 'trunk' of github.com:apache/kafka into changelog_topic_replication --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] kafka pull request: HOTFIX: check offset limits in streamtask when...
GitHub user ymatsuda opened a pull request: https://github.com/apache/kafka/pull/947 HOTFIX: check offset limits in streamtask when recovering KTable store @guozhangwang You can merge this pull request into a Git repository by running: $ git pull https://github.com/ymatsuda/kafka hotfix2 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/947.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #947 commit 2329249b89149ec048ca1d172292db85baa93ab6 Author: Yasuhiro Matsuda <yasuh...@confluent.io> Date: 2016-02-22T21:24:45Z HOTFIX: check offset limits in streamtask when recovering KTable store --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] kafka pull request: MINOR: catch a commit failure due to rebalance
GitHub user ymatsuda opened a pull request: https://github.com/apache/kafka/pull/933 MINOR: catch a commit failure due to rebalance StreamThread should keep going after a commit was failed due to a group rebalance. Currently the thread just dies. @guozhangwang You can merge this pull request into a Git repository by running: $ git pull https://github.com/ymatsuda/kafka catch_commit_failure Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/933.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #933 commit fd5380725758458de8720a157fa7c443031f903d Author: Yasuhiro Matsuda <yasuh...@confluent.io> Date: 2016-02-18T21:54:40Z MINOR: catch a commit failure due to rebalance --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] kafka pull request: KAFKA-2802: kafka streams system tests
GitHub user ymatsuda opened a pull request: https://github.com/apache/kafka/pull/930 KAFKA-2802: kafka streams system tests You can merge this pull request into a Git repository by running: $ git pull https://github.com/ymatsuda/kafka streams_systest Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/930.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #930 commit d5c046707138db6db4535cbdfdd51ddb104b2325 Author: Yasuhiro Matsuda <yasuh...@confluent.io> Date: 2016-02-17T01:08:58Z system test commit 8d035d43ebff6470bdedd6bcb459d5380dec7be3 Author: Yasuhiro Matsuda <yasuh...@confluent.io> Date: 2016-02-17T21:02:23Z more info in log --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] kafka pull request: HOTFIX: make sure to go through all shutdown s...
GitHub user ymatsuda opened a pull request: https://github.com/apache/kafka/pull/928 HOTFIX: make sure to go through all shutdown steps You can merge this pull request into a Git repository by running: $ git pull https://github.com/ymatsuda/kafka shutdown Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/928.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #928 commit a3672f973517c8c3da12d2d80884db29eb210b2e Author: Yasuhiro Matsuda <yasuh...@confluent.io> Date: 2016-02-17T15:39:37Z HOTFIX: make sure to unlock state dir locks commit f2618838544cff6d36dd34ec0b62cf92a5b02c13 Author: Yasuhiro Matsuda <yasuh...@confluent.io> Date: 2016-02-17T16:01:46Z and more --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] kafka pull request: HOTFIX: release resources on abrupt terminatio...
Github user ymatsuda closed the pull request at: https://github.com/apache/kafka/pull/925 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] kafka pull request: HOTFIX: release resources on abrupt terminatio...
GitHub user ymatsuda opened a pull request: https://github.com/apache/kafka/pull/925 HOTFIX: release resources on abrupt termination of stream threads Currently the resources, such as the state dir locks, are not release when a stream thread is abruptly terminated. ```KakfaStreams.close()``` does not release them for the failed threads. You can merge this pull request into a Git repository by running: $ git pull https://github.com/ymatsuda/kafka close_failed_streamthread Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/925.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #925 commit aeb858eba6cdf0e345d3a0363261ce06a5e3ce42 Author: Yasuhiro Matsuda <yasuh...@confluent.io> Date: 2016-02-17T01:26:04Z HOTFIX: release resources on abrupt termination of stream threads --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] kafka pull request: MINOR: remove streams config params from produ...
GitHub user ymatsuda opened a pull request: https://github.com/apache/kafka/pull/906 MINOR: remove streams config params from producer/consumer configs Removing streams' specific config params from producer/consumer configs to reduce warning messages. You can merge this pull request into a Git repository by running: $ git pull https://github.com/ymatsuda/kafka clean_config Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/906.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #906 commit 77ce8051673050df151f00ef674df55fe98b869b Author: Yasuhiro Matsuda <yasuh...@confluent.io> Date: 2016-02-12T00:51:05Z MINOR: remove streams config params from producer/consumer configs --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] kafka pull request: MINOR: add retry to state dir locking
GitHub user ymatsuda opened a pull request: https://github.com/apache/kafka/pull/899 MINOR: add retry to state dir locking There is a possibility that the state directory locking fails when another stream thread is taking long to close all tasks. Simple retries should alleviate the problem. You can merge this pull request into a Git repository by running: $ git pull https://github.com/ymatsuda/kafka minor2 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/899.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #899 commit c211cae875bd41d35b897f54d33fe3827f18c326 Author: Yasuhiro Matsuda <yasuh...@confluent.io> Date: 2016-02-10T17:43:56Z MINOR: add retry to state dir locking --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] kafka pull request: KAFKA-3153: KStream,Type and Serialization
Github user ymatsuda closed the pull request at: https://github.com/apache/kafka/pull/794 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] kafka pull request: MINOR: catch an exception in rebalance and sto...
GitHub user ymatsuda opened a pull request: https://github.com/apache/kafka/pull/901 MINOR: catch an exception in rebalance and stop the stream thread You can merge this pull request into a Git repository by running: $ git pull https://github.com/ymatsuda/kafka minor3 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/901.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #901 commit 0343edc184721aaa4cf62b4e47859337be86c615 Author: Yasuhiro Matsuda <yasuh...@confluent.io> Date: 2016-02-10T22:21:04Z MINOR: catch an exception in rebalance and stop the stream thread commit 77087a1912a3fc6622a7bbbf63e61fb5fea80bd8 Author: Yasuhiro Matsuda <yasuh...@confluent.io> Date: 2016-02-10T22:21:09Z Merge branch 'trunk' of github.com:apache/kafka into minor3 commit 282b889f2c2808138c3a1d2a5231092d4216e35a Author: Yasuhiro Matsuda <yasuh...@confluent.io> Date: 2016-02-10T22:25:13Z msg --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] kafka pull request: HOTFIX: open window segments in order, add seg...
GitHub user ymatsuda opened a pull request: https://github.com/apache/kafka/pull/891 HOTFIX: open window segments in order, add segment id check in getSegment * During window store initialization, we have to open segments in the segment id order and update ```currentSegmentId```, otherwise cleanup won't work. * ```getSegment()``` should not create a segment and clean up old segments if the segment id is greater than ```currentSegmentId```. Segment maintenance should be driven not by query but only by data insertion. You can merge this pull request into a Git repository by running: $ git pull https://github.com/ymatsuda/kafka hotfix2 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/891.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #891 commit b64b84cdeeda65284e3d99ab8278383d1a2aec72 Author: Yasuhiro Matsuda <yasuh...@confluent.io> Date: 2016-02-09T18:05:38Z HOTFIX: open window segments in order --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] kafka pull request: HOTFIX: poll even when all partitions are paus...
GitHub user ymatsuda opened a pull request: https://github.com/apache/kafka/pull/893 HOTFIX: poll even when all partitions are paused. handle concurrent cleanup * We need to poll periodically even when all partitions are paused in order to respond to a possible rebalance promptly. * There is a race condition when two (or more) threads try to clean up the same state directory. One of the thread fails with FileNotFoundException. Thus the new code simply catches it and ignore. You can merge this pull request into a Git repository by running: $ git pull https://github.com/ymatsuda/kafka hotfix Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/893.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #893 commit 8210133bf15e47b5ef379b1a15b89c5dd8da6386 Author: Yasuhiro Matsuda <yasuh...@confluent.io> Date: 2016-02-09T23:06:23Z HOTFIX: poll even when all partitions are paused. handle concurrent cleanup --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] kafka pull request: MINOR: add setUncaughtExceptionHandler to Kafk...
GitHub user ymatsuda opened a pull request: https://github.com/apache/kafka/pull/894 MINOR: add setUncaughtExceptionHandler to KafkaStreams You can merge this pull request into a Git repository by running: $ git pull https://github.com/ymatsuda/kafka minor Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/894.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #894 commit d6222a81c8b9d9b9e47c4f139d783f3141c0ae5c Author: Yasuhiro Matsuda <yasuh...@confluent.io> Date: 2016-02-09T23:52:39Z MINOR: add setUncaughtExceptionHandler to KafkaStreams --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] kafka pull request: HOTFIX: fix NPE after standby task reassignmen...
GitHub user ymatsuda opened a pull request: https://github.com/apache/kafka/pull/889 HOTFIX: fix NPE after standby task reassignment Buffered records of change logs must be cleared upon reassignment of standby tasks. You can merge this pull request into a Git repository by running: $ git pull https://github.com/ymatsuda/kafka hotfix Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/889.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #889 commit ab86801a478cf0c971e90a9d6fa71f25786a4c4e Author: Yasuhiro Matsuda <yasuh...@confluent.io> Date: 2016-02-09T17:34:27Z HOTFIX: fix NPE after standby task reassignment --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] kafka pull request: MINOR: fix RocksDBStore range search
GitHub user ymatsuda opened a pull request: https://github.com/apache/kafka/pull/883 MINOR: fix RocksDBStore range search The range is inclusive according to KeyValueStore's java doc. You can merge this pull request into a Git repository by running: $ git pull https://github.com/ymatsuda/kafka minor Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/883.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #883 commit fda853faa3142b1025ecdd2edaaccff16d207084 Author: Yasuhiro Matsuda <yasuh...@confluent.io> Date: 2016-02-08T17:43:39Z MINOR: fix RocksDBStore range search --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] kafka pull request: HOTFIX: open window segments on init
GitHub user ymatsuda opened a pull request: https://github.com/apache/kafka/pull/886 HOTFIX: open window segments on init @guozhangwang A window store should open all existing segments. This is important for segment cleanup, and it also ensures that the first fetch() call returns the hits, the values in the search range. (previously, it missed the hits in fetch() immediately after initialization). You can merge this pull request into a Git repository by running: $ git pull https://github.com/ymatsuda/kafka hotfix3 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/886.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #886 commit 867963bebcc7be188743a355fec71076cd84cd34 Author: Yasuhiro Matsuda <yasuh...@confluent.io> Date: 2016-02-08T21:31:16Z HOTFIX: own dir for window, open segments on init commit a62e263b46b1828814d967f3ba214ec54ea5b25a Author: Yasuhiro Matsuda <yasuh...@confluent.io> Date: 2016-02-08T21:33:33Z Merge branch 'trunk' of github.com:apache/kafka into hotfix3 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] kafka pull request: HOTFIX: RecordCollector should send a record t...
GitHub user ymatsuda opened a pull request: https://github.com/apache/kafka/pull/887 HOTFIX: RecordCollector should send a record to the specified partition @guozhangwang You can merge this pull request into a Git repository by running: $ git pull https://github.com/ymatsuda/kafka hotfix4 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/887.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #887 commit 6a411fda878ac88ed94e9250b7fcf26c3b7531cb Author: Yasuhiro Matsuda <yasuh...@confluent.io> Date: 2016-02-08T22:51:26Z HOTFIX: RecordCollector should send a record to the specified partition --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] kafka pull request: HOTFIX: fix streams issues
GitHub user ymatsuda opened a pull request: https://github.com/apache/kafka/pull/877 HOTFIX: fix streams issues * RocksDBStore.putInternal should by pass logging. * StoreChangeLogger should not call context.recordCollector() when nothing to log * This is for standby tasks. In standby task, recordCollector() throws an exception. There should be nothing to log anyway. * fixed ConcurrentModificationException in StreamThread @guozhangwang You can merge this pull request into a Git repository by running: $ git pull https://github.com/ymatsuda/kafka hotfix2 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/877.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #877 commit 22484936e076b58279e19898b4da39f73ef7e120 Author: Yasuhiro Matsuda <yasuh...@confluent.io> Date: 2016-02-05T19:48:45Z HOTFIX: fix streams issues --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] kafka pull request: HOTFIX: fix partition ordering in assignment
GitHub user ymatsuda opened a pull request: https://github.com/apache/kafka/pull/868 HOTFIX: fix partition ordering in assignment workround partition ordering not preserved by the consumer group management. @guozhangwang You can merge this pull request into a Git repository by running: $ git pull https://github.com/ymatsuda/kafka partitionOrder Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/868.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #868 commit b964e7b0d76d7071d27926a3da684a9157abb522 Author: Yasuhiro Matsuda <yasuh...@confluent.io> Date: 2016-02-04T22:05:05Z HOTFIX: fix partition ordering in assignment --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] kafka pull request: MINOR: more info in error msg
GitHub user ymatsuda opened a pull request: https://github.com/apache/kafka/pull/873 MINOR: more info in error msg @guozhangwang You can merge this pull request into a Git repository by running: $ git pull https://github.com/ymatsuda/kafka hotfix Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/873.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #873 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] kafka pull request: HOTFIX: temp fix for ktable look up
GitHub user ymatsuda opened a pull request: https://github.com/apache/kafka/pull/864 HOTFIX: temp fix for ktable look up @guozhangwang Temporarily disabled state store access checking. You can merge this pull request into a Git repository by running: $ git pull https://github.com/ymatsuda/kafka fix_table_lookup Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/864.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #864 commit e25415c2082ec12b52ec22f4b774d73eee4f6526 Author: Yasuhiro Matsuda <yasuh...@confluent.io> Date: 2016-02-04T18:05:56Z HOTFIX: temp fix for ktable look up --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] kafka pull request: MINOR: some javadocs for kstream public api
GitHub user ymatsuda opened a pull request: https://github.com/apache/kafka/pull/844 MINOR: some javadocs for kstream public api @guozhangwang You can merge this pull request into a Git repository by running: $ git pull https://github.com/ymatsuda/kafka javadoc Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/844.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #844 commit e935112ba0c9750c412f72d89cdbdd9755fd4eb2 Author: Yasuhiro Matsuda <yasuh...@confluent.io> Date: 2016-02-01T20:49:52Z MINOR: some javadocs for kstream public api --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] kafka pull request: MINOR: removed obsolete class
GitHub user ymatsuda opened a pull request: https://github.com/apache/kafka/pull/843 MINOR: removed obsolete class @guozhangwang You can merge this pull request into a Git repository by running: $ git pull https://github.com/ymatsuda/kafka remove_unused Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/843.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #843 commit 74aeda08c64f67db79c0f7845d3ff6ea83890d50 Author: Yasuhiro Matsuda <yasuh...@confluent.io> Date: 2016-02-01T18:47:36Z MINOR: removed obsolete class --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] kafka pull request: MINOR: remove the init method from Serdes
GitHub user ymatsuda opened a pull request: https://github.com/apache/kafka/pull/834 MINOR: remove the init method from Serdes @guozhangwang You can merge this pull request into a Git repository by running: $ git pull https://github.com/ymatsuda/kafka remove_init_from_Serdes Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/834.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #834 commit 8f432f47e2aaf2358152bee8330b53b7d8bf0b5a Author: Yasuhiro Matsuda <yasuh...@confluent.io> Date: 2016-01-28T00:32:16Z MINOR: remove the init method from Serdes --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] kafka pull request: MINOR: jointest for windowed keys
GitHub user ymatsuda opened a pull request: https://github.com/apache/kafka/pull/814 MINOR: jointest for windowed keys @guozhangwang You can merge this pull request into a Git repository by running: $ git pull https://github.com/ymatsuda/kafka windowed_key_join_test Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/814.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #814 commit 6ea2caebf06eca6aec2d96e6ca11acd2516596ce Author: Yasuhiro Matsuda <yasuh...@confluent.io> Date: 2016-01-26T22:00:48Z MINOR: jointest for windowed keys --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] kafka pull request: MINOR: remove FilteredIterator
GitHub user ymatsuda opened a pull request: https://github.com/apache/kafka/pull/816 MINOR: remove FilteredIterator @guozhangwang removing an unused class, FilteredIterator, and its test. You can merge this pull request into a Git repository by running: $ git pull https://github.com/ymatsuda/kafka remove_obsolete_class Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/816.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #816 commit 42f3e3f2ed549b51077480a1dc5bfb046ec2e429 Author: Yasuhiro Matsuda <yasuh...@confluent.io> Date: 2016-01-27T00:31:30Z MINOR: remove FilteredIterator --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] kafka pull request: MINOR: add equals and hashCode to Windowed
GitHub user ymatsuda opened a pull request: https://github.com/apache/kafka/pull/808 MINOR: add equals and hashCode to Windowed @guozhangwang You can merge this pull request into a Git repository by running: $ git pull https://github.com/ymatsuda/kafka windowed_key Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/808.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #808 commit c0b8209904db63271c754ce3a8887417a6e13b90 Author: Yasuhiro Matsuda <yasuh...@confluent.io> Date: 2016-01-25T21:37:26Z MINOR: add equals and hashCode to Windowed --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] kafka pull request: WIP: KStream,Type and Serialization
GitHub user ymatsuda opened a pull request: https://github.com/apache/kafka/pull/794 WIP: KStream,Type and Serialization @guozhangwang You can merge this pull request into a Git repository by running: $ git pull https://github.com/ymatsuda/kafka type_and_serdes Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/794.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #794 commit baacda936bc9dd9102168e5c73711adea5b768a5 Author: Yasuhiro Matsuda <yasuh...@confluent.io> Date: 2016-01-19T19:20:23Z wip commit 1be936a9acfcae549511dec4fb218ff5cce60128 Author: Yasuhiro Matsuda <yasuh...@confluent.io> Date: 2016-01-19T19:28:13Z Merge branch 'trunk' of github.com:apache/kafka into type_and_serdes commit 7b8a5f8ff469fe4738fb6a48880379896971348f Author: Yasuhiro Matsuda <yasuh...@confluent.io> Date: 2016-01-19T19:40:41Z wip commit c42a18f331d532d22c31bee51cb984aa91b5ab5f Author: Yasuhiro Matsuda <yasuh...@confluent.io> Date: 2016-01-19T22:43:07Z wip commit 5f8638829d84bb52f7b7da42ef27d70b3c1761d9 Author: Yasuhiro Matsuda <yasuh...@confluent.io> Date: 2016-01-20T20:25:02Z wip --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] kafka pull request: KAFKA-3108: custom StreamParitioner for Window...
GitHub user ymatsuda opened a pull request: https://github.com/apache/kafka/pull/779 KAFKA-3108: custom StreamParitioner for Windowed key @guozhangwang When ```WindowedSerializer``` is specified in ```to(...)``` or ```through(...)``` for a key, we use ```WindowedStreamPartitioner```. You can merge this pull request into a Git repository by running: $ git pull https://github.com/ymatsuda/kafka partitioner Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/779.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #779 commit 509bdb2568931365a2a6f912588721213ea6f408 Author: Yasuhiro Matsuda <yasuh...@confluent.io> Date: 2016-01-14T23:20:37Z KAFKA-3108: custom StreamParitioner for Windowed key --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] kafka pull request: KAFKA-3016: phase-2. stream join implementatio...
GitHub user ymatsuda opened a pull request: https://github.com/apache/kafka/pull/737 KAFKA-3016: phase-2. stream join implementations @guozhangwang You can merge this pull request into a Git repository by running: $ git pull https://github.com/ymatsuda/kafka windowed_join2 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/737.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #737 commit 5494ec5b816a86d3414b84172385c46f4c2603c5 Author: Yasuhiro Matsuda <yasuh...@confluent.io> Date: 2016-01-06T17:39:35Z KAFKA-3016: phase-2. stream join implementations --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] kafka pull request: KAFKA-3016: phase-1. A local store for join wi...
GitHub user ymatsuda opened a pull request: https://github.com/apache/kafka/pull/726 KAFKA-3016: phase-1. A local store for join window @guozhangwang An implementation of local store for join window. This implementation uses "rolling" of RocksDB instances for timestamp based truncation. You can merge this pull request into a Git repository by running: $ git pull https://github.com/ymatsuda/kafka windowed_join Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/726.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #726 commit 87734776268d8f9d7315cc7552cdfb1fe86ecb69 Author: Yasuhiro Matsuda <yasuh...@confluent.io> Date: 2016-01-04T17:34:57Z join window store commit 096a83941f97af5da8192a71c8d5bb6e66130a45 Author: Yasuhiro Matsuda <yasuh...@confluent.io> Date: 2016-01-04T17:42:53Z Merge branch 'trunk' of github.com:apache/kafka into windowed_join --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] kafka pull request: MINOR: StreamThread performance optimization
GitHub user ymatsuda opened a pull request: https://github.com/apache/kafka/pull/680 MINOR: StreamThread performance optimization @guozhangwang You can merge this pull request into a Git repository by running: $ git pull https://github.com/ymatsuda/kafka perf Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/680.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #680 commit 2809cac7f1f735429cfa5e5cbe8dc1cd62b6b58b Author: Yasuhiro Matsuda <yasuh...@confluent.io> Date: 2015-12-15T20:06:09Z performance optimization --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] kafka pull request: KAFKA-2984: ktable sends old values when requi...
GitHub user ymatsuda opened a pull request: https://github.com/apache/kafka/pull/672 KAFKA-2984: ktable sends old values when required @guozhangwang At DAG level, `KTable<K,V>` sends (key, (new value, old value)) to down stream. This is done by wrapping the new value and the old value in an instance of `Change` class and sending it as a "value" part of the stream. The old value is omitted (set to null) by default for optimization. When any downstream processor needs to use the old value, the framework should enable it (see `KTableImpl.enableSendingOldValues()` and implementations of `KTableProcessorSupplier.enableSensingOldValues()`). NOTE: This is meant to be used by aggregation. But, if there is a use case like a SQL database trigger, we can add a new KTable method to expose this. You can merge this pull request into a Git repository by running: $ git pull https://github.com/ymatsuda/kafka trigger Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/672.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #672 commit 41bbef2c1ba6e170403f7b48fcb618bbe49d9b6c Author: Yasuhiro Matsuda <yasuh...@confluent.io> Date: 2015-12-11T22:41:55Z KAFKA-2984: KTable should send old values along with new values to downstreams commit 91a9fad6e729bf631d82a88db5bb6ec483ae2062 Author: Yasuhiro Matsuda <yasuh...@confluent.io> Date: 2015-12-14T18:08:15Z Merge branch 'trunk' of github.com:apache/kafka into trigger commit 7a1b689c594d2e859454c28a2367df793178d3b9 Author: Yasuhiro Matsuda <yasuh...@confluent.io> Date: 2015-12-14T18:08:50Z method names --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] kafka pull request: MINOR: test ktable state store creation
GitHub user ymatsuda opened a pull request: https://github.com/apache/kafka/pull/661 MINOR: test ktable state store creation @guozhangwang * a test for ktable state store creation You can merge this pull request into a Git repository by running: $ git pull https://github.com/ymatsuda/kafka more_ktable_test Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/661.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #661 commit 5a73d701b5794964360a0c5ddcc36cde6610a5c7 Author: Yasuhiro Matsuda <yasuh...@confluent.io> Date: 2015-12-10T19:22:29Z MINOR: more ktable tests --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] kafka pull request: HOTFIX: fix table-table outer join and left jo...
GitHub user ymatsuda opened a pull request: https://github.com/apache/kafka/pull/653 HOTFIX: fix table-table outer join and left join. more tests @guozhangwang * fixed bugs in table-table outer/left joins * added more tests You can merge this pull request into a Git repository by running: $ git pull https://github.com/ymatsuda/kafka join_tests Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/653.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #653 commit bc8bbd3c3662d79e93331a109f85bd7a168f45ad Author: Yasuhiro Matsuda <yasuh...@confluent.io> Date: 2015-12-09T21:29:52Z HOTFIX: fix table-table outer join and left join. more tests --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] kafka pull request: KAFKA-2962: stream-table table-table joins
GitHub user ymatsuda opened a pull request: https://github.com/apache/kafka/pull/644 KAFKA-2962: stream-table table-table joins @guozhangwang You can merge this pull request into a Git repository by running: $ git pull https://github.com/ymatsuda/kafka join_methods Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/644.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #644 commit 15804dc1b8a8d9cfeee685d66b64d5fb9f77989f Author: Yasuhiro Matsuda <yasuh...@confluent.io> Date: 2015-12-08T23:39:15Z stream-table table-table joins --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] kafka pull request: HOTFIX: fix ProcessorStateManager to use corre...
GitHub user ymatsuda opened a pull request: https://github.com/apache/kafka/pull/635 HOTFIX: fix ProcessorStateManager to use correct ktable partitions @guozhangwang * fix ProcessorStateManager to use correct ktable partitions * more ktable tests You can merge this pull request into a Git repository by running: $ git pull https://github.com/ymatsuda/kafka more_ktable_test Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/635.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #635 commit 6b5b287a88dccdd095be574e4e96c4ae3543a879 Author: Yasuhiro Matsuda <yasuh...@confluent.io> Date: 2015-12-04T23:16:36Z add more ktable test commit e7ba48810c594424b56f0b1cbec922acdd974d01 Author: Yasuhiro Matsuda <yasuh...@confluent.io> Date: 2015-12-07T17:28:18Z Merge branch 'trunk' of github.com:apache/kafka into more_ktable_test --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] kafka pull request: KAFKA-2856: add ktable
Github user ymatsuda closed the pull request at: https://github.com/apache/kafka/pull/604 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] kafka pull request: KAFKA-2856: add ktable
GitHub user ymatsuda opened a pull request: https://github.com/apache/kafka/pull/604 KAFKA-2856: add ktable @guozhangwang * added KTable API and impl * added standby support for KTable You can merge this pull request into a Git repository by running: $ git pull https://github.com/ymatsuda/kafka add_ktable Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/604.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #604 commit fd4f9998b52768d8a29baec7869167fca714a925 Author: Yasuhiro Matsuda <yasuh...@confluent.io> Date: 2015-11-30T18:04:21Z KAFKA-2856: add ktable --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] kafka pull request: KAFKA-2856: add ktable
GitHub user ymatsuda reopened a pull request: https://github.com/apache/kafka/pull/604 KAFKA-2856: add ktable @guozhangwang * added KTable API and impl * added standby support for KTable You can merge this pull request into a Git repository by running: $ git pull https://github.com/ymatsuda/kafka add_ktable Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/604.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #604 commit fd4f9998b52768d8a29baec7869167fca714a925 Author: Yasuhiro Matsuda <yasuh...@confluent.io> Date: 2015-11-30T18:04:21Z KAFKA-2856: add ktable commit cee5874274d275f532f8b98ea06bee52113e4dab Author: Yasuhiro Matsuda <yasuh...@confluent.io> Date: 2015-11-30T19:35:31Z ensure committing standby tasks --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] kafka pull request: MINOR: initialize Serdes with ProcessorContext
GitHub user ymatsuda opened a pull request: https://github.com/apache/kafka/pull/589 MINOR: initialize Serdes with ProcessorContext @guozhangwang You can merge this pull request into a Git repository by running: $ git pull https://github.com/ymatsuda/kafka init_serdes_with_procctx Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/589.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #589 commit 451b7be8bb2c405cebea77536c1d8d5710085507 Author: Yasuhiro Matsuda <yasuh...@confluent.io> Date: 2015-11-25T22:28:20Z MINOR: initialize Serdes with ProcessorContext --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] kafka pull request: MINOR: comments on KStream methods, and fix ge...
GitHub user ymatsuda opened a pull request: https://github.com/apache/kafka/pull/591 MINOR: comments on KStream methods, and fix generics @guozhangwang You can merge this pull request into a Git repository by running: $ git pull https://github.com/ymatsuda/kafka comments Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/591.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #591 commit 37ad9e4a839eb03d5bb508894403399288032223 Author: Yasuhiro Matsuda <yasuh...@confluent.io> Date: 2015-11-25T23:42:02Z MINOR: comments on KStream methods, and fix generics --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] kafka pull request: HOTFIX: fix StreamTask.close()
GitHub user ymatsuda opened a pull request: https://github.com/apache/kafka/pull/586 HOTFIX: fix StreamTask.close() @guozhangwang You can merge this pull request into a Git repository by running: $ git pull https://github.com/ymatsuda/kafka fix_streamtask_close Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/586.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #586 commit f74ac68eb1a092e7b1abaa0fbda7ef0e86405f77 Author: Yasuhiro Matsuda <yasuh...@confluent.io> Date: 2015-11-25T19:55:55Z HOTFIX: fix StreamTask.close() --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] kafka pull request: MINOR: change KStream processor names
GitHub user ymatsuda opened a pull request: https://github.com/apache/kafka/pull/587 MINOR: change KStream processor names @guozhangwang You can merge this pull request into a Git repository by running: $ git pull https://github.com/ymatsuda/kafka kstream_processor_names Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/587.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #587 commit e256d9c08c359b108f5a55194f2ae885b2bf091e Author: Yasuhiro Matsuda <yasuh...@confluent.io> Date: 2015-11-25T21:18:01Z MINOR: change KStream processor names --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] kafka pull request: MINOR: remove the group id from a restore cons...
GitHub user ymatsuda opened a pull request: https://github.com/apache/kafka/pull/543 MINOR: remove the group id from a restore consumer @guozhangwang A restore consumer does not belong to a consumer group. You can merge this pull request into a Git repository by running: $ git pull https://github.com/ymatsuda/kafka no_group_for_restore_consumer Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/543.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #543 commit 53ebad168a23faaf097adf6a72ddd78a1f933de6 Author: Yasuhiro Matsuda <yasuh...@confluent.io> Date: 2015-11-17T17:46:38Z remove the group id from a restore consumer --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] kafka pull request: MINOR: do not create a StandbyTask if there is...
GitHub user ymatsuda opened a pull request: https://github.com/apache/kafka/pull/535 MINOR: do not create a StandbyTask if there is no state store in the task @guozhangwang An optimization which may reduce unnecessary poll for standby tasks. You can merge this pull request into a Git repository by running: $ git pull https://github.com/ymatsuda/kafka remove_empty_standby_task Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/535.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #535 commit b24fc838fb8201e1d162904e4d9388c3057d493b Author: Yasuhiro Matsuda <yasuh...@confluent.io> Date: 2015-11-16T21:38:45Z do not create a StandbyTask if there is no state store in the task --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] kafka pull request: MINOR: add KStream merge operator
GitHub user ymatsuda opened a pull request: https://github.com/apache/kafka/pull/536 MINOR: add KStream merge operator @guozhangwang Added KStreamBuilder.merge(KStream...). You can merge this pull request into a Git repository by running: $ git pull https://github.com/ymatsuda/kafka kstream_merge_operator Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/536.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #536 commit 9fd01e9d9e2afa435816c903d32cd011c0796b11 Author: Yasuhiro Matsuda <yasuh...@confluent.io> Date: 2015-11-05T22:36:02Z kstream merge commit d62f1335628aaa2de41434c00b6870235e1f9d49 Author: Yasuhiro Matsuda <yasuh...@confluent.io> Date: 2015-11-16T21:46:06Z Merge branch 'trunk' of github.com:apache/kafka into kstream_merge_operator commit a7ce2995f15e6a3dbb35f77f47ba88317e4eee2a Author: Yasuhiro Matsuda <yasuh...@confluent.io> Date: 2015-11-16T21:57:14Z add test --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] kafka pull request: KAFKA-2811: add standby tasks
GitHub user ymatsuda opened a pull request: https://github.com/apache/kafka/pull/526 KAFKA-2811: add standby tasks @guozhangwang * added a new config param "num.standby.replicas" (the default value is 0). * added a new abstract class AbstractTask * added StandbyTask as a subclass of AbstractTask * modified StreamTask to a subclass of AbstractTask * StreamThread * standby tasks are created by calling StreamThread.addStandbyTask() from onPartitionsAssigned() * standby tasks are destroyed by calling StreamThread.removeStandbyTasks() from onPartitionRevoked() * In addStandbyTasks(), change log partitions are assigned to restoreConsumer. * In removeStandByTasks(), change log partitions are removed from restoreConsumer. * StreamThread polls change log records using restoreConsumer in the runLoop with timeout=0. * If records are returned, StreamThread calls StandbyTask.update and pass records to each standby tasks. You can merge this pull request into a Git repository by running: $ git pull https://github.com/ymatsuda/kafka standby_task Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/526.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #526 commit 4f015e107ede8b03cea9aeddd40c7f04bc930acc Author: Yasuhiro Matsuda <yasuh...@confluent.io> Date: 2015-11-13T18:51:20Z standby tasks commit ea8814e3d668da63847650718d107b9a1a47ceb9 Author: Yasuhiro Matsuda <yasuh...@confluent.io> Date: 2015-11-13T19:16:36Z cleanup --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] kafka pull request: KAFKA-2763: better stream task assignment
GitHub user ymatsuda opened a pull request: https://github.com/apache/kafka/pull/497 KAFKA-2763: better stream task assignment @guozhangwang When the rebalance happens each consumer reports the following information to the coordinator. * Client UUID (a unique id assigned to an instance of KafkaStreaming) * Task ids of previously running tasks * Task ids of valid local states on the client's state directory TaskAssignor does the following * Assign a task to a client which was running it previously. If there is no such client, assign a task to a client which has its valid local state. * Try to balance the load among stream threads. * A client may have more than one stream threads. The assignor tries to assign tasks to a client proportionally to the number of threads. You can merge this pull request into a Git repository by running: $ git pull https://github.com/ymatsuda/kafka task_assignment Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/497.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #497 commit 0e4cd31d3f9f055aa8db5917bcb30f1dbc3da984 Author: Yasuhiro Matsuda <yasuh...@confluent.io> Date: 2015-11-11T17:46:57Z better task assignment --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] kafka pull request: KAFKA-2727: Topology partial construction
GitHub user ymatsuda opened a pull request: https://github.com/apache/kafka/pull/411 KAFKA-2727: Topology partial construction @guozhangwang You can merge this pull request into a Git repository by running: $ git pull https://github.com/ymatsuda/kafka topology_partial_construction Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/411.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #411 commit e561254d9f3e6927a9e0d62c85d7144d936d678b Author: Yasuhiro Matsuda <yasuh...@confluent.io> Date: 2015-10-29T17:03:10Z partial construction of topology commit 3ae930c43e0f2f60caa183d7265d3e69442a3d96 Author: Yasuhiro Matsuda <yasuh...@confluent.io> Date: 2015-11-02T22:59:42Z Merge branch 'trunk' of github.com:apache/kafka into topology_partial_construction commit f984b3214e72c678ea6f45a564325eb212c4ccdf Author: Yasuhiro Matsuda <yasuh...@confluent.io> Date: 2015-11-02T23:13:31Z cleanup commit d170007d0df57af3afbe22c121aac0bd10dbeb7e Author: Yasuhiro Matsuda <yasuh...@confluent.io> Date: 2015-11-02T23:24:06Z test --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] kafka pull request: KAFKA-2707: make KStream processor names deter...
GitHub user ymatsuda opened a pull request: https://github.com/apache/kafka/pull/408 KAFKA-2707: make KStream processor names deterministic @guozhangwang You can merge this pull request into a Git repository by running: $ git pull https://github.com/ymatsuda/kafka kstream_processor_name Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/408.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #408 commit 4d26184b26fcf50633f26081d10748ce6db94f65 Author: Yasuhiro Matsuda <yasuh...@confluent.io> Date: 2015-10-30T16:55:52Z kstream processor name commit 9265d24ec7538052bc60fd51d7aa18f6cb7d6b66 Author: Yasuhiro Matsuda <yasuh...@confluent.io> Date: 2015-11-02T21:31:39Z Merge branch 'trunk' of github.com:apache/kafka into kstream_processor_name --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] kafka pull request: KAFKA-2706: make state stores first class citi...
GitHub user ymatsuda opened a pull request: https://github.com/apache/kafka/pull/387 KAFKA-2706: make state stores first class citizens in the processor topology * Added StateStoreSupplier * StateStore * Added init(ProcessorContext context) method * TopologyBuilder * Added addStateStore(StateStoreSupplier supplier, String... processNames) * Added connectProessorAndStateStores(String processorName, String... stateStoreNames) * This is for the case processors are not created when a store is added to the topology. (used by KStream) * KStream * add stateStoreNames to process(), transform(), transformValues(). * Refactored existing state stores to implement StateStoreSupplier @guozhangwang You can merge this pull request into a Git repository by running: $ git pull https://github.com/ymatsuda/kafka state_store_supplier Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/387.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #387 commit 97869c26fe503818dc57b62671f7d82450b002c1 Author: Yasuhiro Matsuda <yasuh...@confluent.io> Date: 2015-10-29T23:19:30Z KAFKA-2706: make state stores first class citizens in the processor topology --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] kafka pull request: HOTFIX: fix off-by-one stream offset commit
GitHub user ymatsuda opened a pull request: https://github.com/apache/kafka/pull/372 HOTFIX: fix off-by-one stream offset commit @guozhangwang You can merge this pull request into a Git repository by running: $ git pull https://github.com/ymatsuda/kafka commit_offset Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/372.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #372 commit dea0ad09d9d977d0d0d30ecc12d2b305daf6d55d Author: Yasuhiro Matsuda <yasuh...@confluent.io> Date: 2015-10-27T20:18:54Z HOTFIX: fix off-by-one stream offset commit --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] kafka pull request: HOTFIX: call consumer.poll() even when no task...
GitHub user ymatsuda opened a pull request: https://github.com/apache/kafka/pull/373 HOTFIX: call consumer.poll() even when no task is assigned StreamThread should keep calling consumer.poll() even when no task is assigned. This is necessary to get a task. @guozhangwang You can merge this pull request into a Git repository by running: $ git pull https://github.com/ymatsuda/kafka no_task Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/373.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #373 commit 0e9acf0a1d037e3ee0d4916cd515d09c47b77657 Author: Yasuhiro Matsuda <yasuh...@confluent.io> Date: 2015-10-27T20:45:29Z HOTFIX: call consumer.poll() even when no task is assigned --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] kafka pull request: HOTFIX: correct sourceNodes for kstream.throug...
GitHub user ymatsuda opened a pull request: https://github.com/apache/kafka/pull/374 HOTFIX: correct sourceNodes for kstream.through() @guozhangwang You can merge this pull request into a Git repository by running: $ git pull https://github.com/ymatsuda/kafka fix_through_operator Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/374.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #374 commit 3a18a5cd7c2a69e95be35d606c2642c47b28d13d Author: Yasuhiro Matsuda <yasuh...@confluent.io> Date: 2015-10-27T23:07:35Z HOTFIX: correct sourceNodes for kstream.through() --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] kafka pull request: KAFKA-2652: integrate new group protocol into ...
GitHub user ymatsuda opened a pull request: https://github.com/apache/kafka/pull/353 KAFKA-2652: integrate new group protocol into partition grouping @guozhangwang * added ```PartitionGrouper``` (abstract class) * This class is responsible for grouping partitions to form tasks. * Users may implement this class for custom grouping. * added ```DefaultPartitionGrouper``` * our default implementation of ```PartitionGrouper``` * added ```KafkaStreamingPartitionAssignor``` * We always use this as ```PartitionAssignor``` of stream consumers. * Actual grouping is delegated to ```PartitionGrouper```. * ```TopologyBuilder``` * added ```topicGroups()``` * This returns groups of related topics according to the topology * added ```copartitionSources(sourceNodes...)``` * This is used by DSL layer. It asserts the specified source nodes must be copartitioned. * added ```copartitionGroups()``` which returns groups of copartitioned topics * KStream layer * keep track of source nodes to determine copartition sources when steams are joined * source nodes are set to null when partitioning property is not preserved (ex. ```map()```, ```transform()```), and this indicates the stream is no longer joinable You can merge this pull request into a Git repository by running: $ git pull https://github.com/ymatsuda/kafka grouping Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/353.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #353 commit 708718c1be23fad25fa6206f665cbb619c1b5097 Author: Yasuhiro Matsuda <yasuh...@confluent.io> Date: 2015-10-19T19:38:06Z partition grouping commit d2bae046b5509022e2821a2c5eb08853d228e791 Author: Yasuhiro Matsuda <yasuh...@confluent.io> Date: 2015-10-19T20:19:54Z wip commit 86fa8110b23ee1992fbd19daa08c63a4b427448e Author: Yasuhiro Matsuda <yasuh...@confluent.io> Date: 2015-10-20T20:01:37Z long task id commit 4f4f9ac642ebe0eae33a5c8464309106e9239f2e Author: Yasuhiro Matsuda <yasuh...@confluent.io> Date: 2015-10-20T20:03:15Z Merge branch 'trunk' of github.com:apache/kafka into grouping commit e4ecf39b9ab0b0f4c915a4f43cfe771b1de69f7f Author: Yasuhiro Matsuda <yasuh...@confluent.io> Date: 2015-10-21T19:33:05Z joinability commit 37d72a691173a8fe878ac3d99e8973e72f5675c6 Author: Yasuhiro Matsuda <yasuh...@confluent.io> Date: 2015-10-21T19:33:48Z Merge branch 'trunk' of github.com:apache/kafka into grouping commit f68723bab83c3a3f1c15872f4f24bc932df8198f Author: Yasuhiro Matsuda <yasuh...@confluent.io> Date: 2015-10-22T18:21:31Z partition assignor commit 457cf270222139eae89750781d09abaa07120932 Author: Yasuhiro Matsuda <yasuh...@confluent.io> Date: 2015-10-22T18:21:40Z Merge branch 'trunk' of github.com:apache/kafka into grouping commit 13f3ad703960581229d511287f27345c567b5d3e Author: Yasuhiro Matsuda <yasuh...@confluent.io> Date: 2015-10-22T18:34:52Z complete undoing long taskid commit 98f3bcc1896fd159ccbbd37fc65b1d9d6f568bb9 Author: Yasuhiro Matsuda <yasuh...@confluent.io> Date: 2015-10-22T18:45:38Z fix a test --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] kafka pull request: MINOR: optimize unnecessary poll(0) away
GitHub user ymatsuda opened a pull request: https://github.com/apache/kafka/pull/315 MINOR: optimize unnecessary poll(0) away @guozhangwang This change aims to remove unnecessary ```consumer.poll(0)``` calls. * ```once``` after some partition is resumed * whenever the size of the top queue in any task is below ```BUFFERED_RECORDS_PER_PARTITION_CONFIG``` You can merge this pull request into a Git repository by running: $ git pull https://github.com/ymatsuda/kafka less_poll_zero Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/315.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #315 commit 8557db0f63410b033f1d3c93764aae74e3787a19 Author: Yasuhiro Matsuda <yasuh...@confluent.io> Date: 2015-10-15T16:31:37Z MINOR: optimize unnecessary poll(0) away --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] kafka pull request: MINOR: set up temp directories properly in Str...
GitHub user ymatsuda opened a pull request: https://github.com/apache/kafka/pull/317 MINOR: set up temp directories properly in StreamTaskTest @guozhangwang StreamTaskTest did not set up a temp directory for each test. This rarely caused interference between tests through state directory locking. You can merge this pull request into a Git repository by running: $ git pull https://github.com/ymatsuda/kafka fix_StreamTaskTest Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/317.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #317 commit 1366e8ab083bc2fcaa1ccd82568fcfc218f82d2f Author: Yasuhiro Matsuda <yasuh...@confluent.io> Date: 2015-10-15T18:28:16Z MINOR: set up temp directories properly in StreamTaskTest --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] kafka pull request: MINOR: flush record collector after local stat...
GitHub user ymatsuda opened a pull request: https://github.com/apache/kafka/pull/304 MINOR: flush record collector after local state flush @guozhangwang Fix the order of flushing. Undoing the change I did sometime ago. You can merge this pull request into a Git repository by running: $ git pull https://github.com/ymatsuda/kafka flush_order Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/304.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #304 commit e9238c31071426a3d82435415fbb3a31a9ca2ae8 Author: Yasuhiro Matsuda <yasuh...@confluent.io> Date: 2015-10-13T21:43:04Z MINOR: flush record collector after local state flush --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] kafka pull request: MINOR: putting back kstream stateful transform...
GitHub user ymatsuda opened a pull request: https://github.com/apache/kafka/pull/292 MINOR: putting back kstream stateful transform methods @guozhangwang * added back type safe stateful transform methods (kstream.transform() and kstream.transformValues()) * changed kstream.process() to void You can merge this pull request into a Git repository by running: $ git pull https://github.com/ymatsuda/kafka transform_method Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/292.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #292 commit 9d8654b9536115c25af7f58658248cde041a4646 Author: Yasuhiro Matsuda <yasuh...@confluent.io> Date: 2015-10-09T21:36:34Z MINOR: putting back kstream stateful transform methods commit 7ccbf05a5087ad77d312add0e46b4369284dc131 Author: Yasuhiro Matsuda <yasuh...@confluent.io> Date: 2015-10-09T22:04:39Z MINOR: putting back kstream stateful transform methods --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] kafka pull request: MINOR: typing ProcessorDef
GitHub user ymatsuda opened a pull request: https://github.com/apache/kafka/pull/289 MINOR: typing ProcessorDef @guozhangwang This code change properly types ProcessorDef. This also makes KStream.process() typesafe. You can merge this pull request into a Git repository by running: $ git pull https://github.com/ymatsuda/kafka typing_ProcessorDef Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/289.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #289 commit 753bbca942278177b29b70ac03db33e4ae321b5f Author: Yasuhiro Matsuda <yasuh...@confluent.io> Date: 2015-10-08T23:04:57Z MINOR: typing ProcessorDef --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
