[GitHub] kafka pull request #1472: KAFKA-3561: Auto create through topic for KStream ...

2016-06-05 Thread dguy
GitHub user dguy opened a pull request: https://github.com/apache/kafka/pull/1472 KAFKA-3561: Auto create through topic for KStream aggregation and join [WIP] @guozhangwang @enothereska @mjsax @miguno If you get a chance can you please take a look at this. I've done

[GitHub] kafka pull request #1453: KAFKA-3561: Auto create through topic for KStream ...

2016-06-04 Thread dguy
Github user dguy closed the pull request at: https://github.com/apache/kafka/pull/1453 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature

[GitHub] kafka pull request: KAFKA-3561: Auto create through topic for KStream aggreg...

2016-06-01 Thread dguy
GitHub user dguy opened a pull request: https://github.com/apache/kafka/pull/1453 KAFKA-3561: Auto create through topic for KStream aggregation and join [WIP] @guozhangwang can you please take a look at this? It is not close to being done but i'd like some feedback to ensure i'm

[GitHub] kafka pull request #1534: MINOR: update streams.html with KStream API change...

2016-06-21 Thread dguy
GitHub user dguy opened a pull request: https://github.com/apache/kafka/pull/1534 MINOR: update streams.html with KStream API changes @mjsax @guozhangwang You can merge this pull request into a Git repository by running: $ git pull https://github.com/dguy/kafka update-streams

[GitHub] kafka pull request #1525: MINOR: speed up streams integration tests

2016-06-20 Thread dguy
GitHub user dguy opened a pull request: https://github.com/apache/kafka/pull/1525 MINOR: speed up streams integration tests ... by a) merging some for startup/shutdown efficiency. b) use independent state dirs. c) remove some tests that are covered elsewhere

[GitHub] kafka pull request #1549: KAFKA-3896: Unstable test KStreamRepartitionJoinTe...

2016-06-24 Thread dguy
GitHub user dguy opened a pull request: https://github.com/apache/kafka/pull/1549 KAFKA-3896: Unstable test KStreamRepartitionJoinTest.shouldCorrectlyRepartitionOnJoinOperations @ijuma i checked the cases where this test has failed and it seems to always be on the verification

[GitHub] kafka pull request #1565: KAFKA-3912: Query local state stores

2016-06-28 Thread dguy
GitHub user dguy opened a pull request: https://github.com/apache/kafka/pull/1565 KAFKA-3912: Query local state stores @guozhangwang @enothereska please review You can merge this pull request into a Git repository by running: $ git pull https://github.com/dguy/kafka kafka-3912

[GitHub] kafka pull request: MINOR: add unit test for KGroupedTable.count

2016-04-22 Thread dguy
GitHub user dguy opened a pull request: https://github.com/apache/kafka/pull/1255 MINOR: add unit test for KGroupedTable.count You can merge this pull request into a Git repository by running: $ git pull https://github.com/dguy/kafka kgroupedtable-count-test Alternatively

[GitHub] kafka pull request: KAFKA-3629 KStreamImpl.to(...) throws NPE when...

2016-04-27 Thread dguy
GitHub user dguy opened a pull request: https://github.com/apache/kafka/pull/1272 KAFKA-3629 KStreamImpl.to(...) throws NPE when the value SerDe is null @guozhangwang You can merge this pull request into a Git repository by running: $ git pull https://github.com/dguy/kafka

[GitHub] kafka pull request #1649: KAFKA-3929: Add prefix for underlying clients conf...

2016-07-21 Thread dguy
GitHub user dguy opened a pull request: https://github.com/apache/kafka/pull/1649 KAFKA-3929: Add prefix for underlying clients configs in StreamConfig Add prefixes for consumer and producer configs to StreamsConfig, but be backward compatible. You can merge this pull request

[GitHub] kafka pull request #1679: MINOR: replace reference to HoppingWindows in stre...

2016-07-29 Thread dguy
GitHub user dguy opened a pull request: https://github.com/apache/kafka/pull/1679 MINOR: replace reference to HoppingWindows in streams.html HoppingWindows was removed prior to the 0.10.0 release. I've updated the doc to refer to the correct TimeWindows You can merge this pull

[GitHub] kafka pull request #1682: HOTFIX: non-unique state.dirs in integration tests...

2016-07-29 Thread dguy
GitHub user dguy opened a pull request: https://github.com/apache/kafka/pull/1682 HOTFIX: non-unique state.dirs in integration tests causing build to hang Three Streams Integration tests were using the same directory for the state.dir config. This was causing the build to hang when

[GitHub] kafka pull request #1682: HOTFIX: non-unique state.dirs in integration tests...

2016-07-30 Thread dguy
Github user dguy closed the pull request at: https://github.com/apache/kafka/pull/1682 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature

[GitHub] kafka pull request #1679: MINOR: replace reference to HoppingWindows in stre...

2016-07-30 Thread dguy
Github user dguy closed the pull request at: https://github.com/apache/kafka/pull/1679 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature

[GitHub] kafka pull request #1693: HOTFIX: start embedded kafka in KafkaStreamsTest t...

2016-08-02 Thread dguy
GitHub user dguy opened a pull request: https://github.com/apache/kafka/pull/1693 HOTFIX: start embedded kafka in KafkaStreamsTest to avoid hanging The KafkaStreamsTest can occasionally hang if the test doesn't run fast enough. This is due to there being no brokers available

[GitHub] kafka pull request #1699: MINOR: rename StateStoreProvider.getStores() -> St...

2016-08-03 Thread dguy
GitHub user dguy opened a pull request: https://github.com/apache/kafka/pull/1699 MINOR: rename StateStoreProvider.getStores() -> StateStoreProvider.stores() Rename StateStoreProvider.getStores(...) to StateStoreProvider.stores(...) as this is consistent with the naming of ot

[GitHub] kafka pull request #1628: KAFKA-3812 State store locking is incorrect

2016-07-17 Thread dguy
GitHub user dguy opened a pull request: https://github.com/apache/kafka/pull/1628 KAFKA-3812 State store locking is incorrect Move all state directory creation/locking/unlocking/cleaning to a single class. Don't release the channel until the lock is released. Refactor code to make

[GitHub] kafka pull request #1633: KAFKA-3855: Guard race conditions in TopologyBuild...

2016-07-18 Thread dguy
GitHub user dguy opened a pull request: https://github.com/apache/kafka/pull/1633 KAFKA-3855: Guard race conditions in TopologyBuilder Mark all public `TopologyBuilder` methods as synchronized as they can modify data-structures and these methods could be called from multiple

[GitHub] kafka pull request #1641: HOTFIX: fix compilation error due to merge of KAFK...

2016-07-19 Thread dguy
GitHub user dguy opened a pull request: https://github.com/apache/kafka/pull/1641 HOTFIX: fix compilation error due to merge of KAFKA-3812 Merge of KAFKA-3812 caused a compilation error in StreamThreadStateStoreProviderTest You can merge this pull request into a Git repository

[GitHub] kafka pull request #1640: KAFKA-3740: Enable configuration of RocksDBStores

2016-07-19 Thread dguy
GitHub user dguy opened a pull request: https://github.com/apache/kafka/pull/1640 KAFKA-3740: Enable configuration of RocksDBStores Add new config StreamsConfig.ROCKSDB_CONFIG_SETTER_CLASS_CONFIG to enable advanced RocksDB users to override default RocksDB configuration You

[GitHub] kafka pull request #1587: MINOR: fix generics in Windows.segments and Window...

2016-07-05 Thread dguy
GitHub user dguy opened a pull request: https://github.com/apache/kafka/pull/1587 MINOR: fix generics in Windows.segments and Windows.until `Windows.segments(...)` and `Windows.until(...)` currently aren't returning the `Window` with its type param `W`. This causes the generic type

[GitHub] kafka pull request #1600: KAFKA-3942: Change IntegrationTestUtils.purgeLocal...

2016-07-09 Thread dguy
GitHub user dguy opened a pull request: https://github.com/apache/kafka/pull/1600 KAFKA-3942: Change IntegrationTestUtils.purgeLocalStreamsState to use java.io.tmpdir It was previously only deleting files/folders where the path started with /tmp. Changed it to delete from

[GitHub] kafka pull request #1583: MINOR: fix generics in KStream.groupBy(...)

2016-07-04 Thread dguy
Github user dguy closed the pull request at: https://github.com/apache/kafka/pull/1583 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature

[GitHub] kafka pull request #1584: MINOR: fix generics in KStream.groupBy(...)

2016-07-04 Thread dguy
GitHub user dguy opened a pull request: https://github.com/apache/kafka/pull/1584 MINOR: fix generics in KStream.groupBy(...) The `KStream.groupBy(..)` calls don't change the value, only the key, so they don't need the type param `V1` as the new stream will always be of type

[GitHub] kafka pull request #1583: MINOR: fix generics in KStream.groupBy(...)

2016-07-04 Thread dguy
GitHub user dguy opened a pull request: https://github.com/apache/kafka/pull/1583 MINOR: fix generics in KStream.groupBy(...) The type param `V1` in the `KStream.groupBy(...)` methods is not needed. @guozhangwang You can merge this pull request into a Git repository

[GitHub] kafka pull request #1711: KAFKA-3936: Validate parameters as early as possib...

2016-08-08 Thread dguy
GitHub user dguy opened a pull request: https://github.com/apache/kafka/pull/1711 KAFKA-3936: Validate parameters as early as possible Added non null checks to parameters supplied via the DSL and `TopologyBuilder` You can merge this pull request into a Git repository by running

[GitHub] kafka pull request #1570: KAFKA-3914: Global discovery of state stores

2016-06-30 Thread dguy
Github user dguy closed the pull request at: https://github.com/apache/kafka/pull/1570 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature

[GitHub] kafka pull request #1576: KAFKA-3914: Global discovery of state stores

2016-06-30 Thread dguy
GitHub user dguy opened a pull request: https://github.com/apache/kafka/pull/1576 KAFKA-3914: Global discovery of state stores @guozhangwang @enothereska @mjsax @miguno please take a look. A few things that need to be clarified 1. I've added

[GitHub] kafka pull request #1570: KAFKA-3914: Global discovery of state stores

2016-06-29 Thread dguy
GitHub user dguy opened a pull request: https://github.com/apache/kafka/pull/1570 KAFKA-3914: Global discovery of state stores @guozhangwang @enothereska please take a look. A few things that need to be clarified 1. I've added `StreamsConfig.USER_ENDPOINT_CONFIG

[GitHub] kafka pull request #1742: KAFKA-4015: Add new cleanup.policy, compact_and_de...

2016-08-16 Thread dguy
GitHub user dguy opened a pull request: https://github.com/apache/kafka/pull/1742 KAFKA-4015: Add new cleanup.policy, compact_and_delete Added compact_and_delete cleanup.policy to LogConfig. Updated LogCleaner.CleanerThread to also run deletion for any topics configured

[GitHub] kafka pull request #2477: MINOR: Change rocksdb logging to error level

2017-02-01 Thread dguy
GitHub user dguy opened a pull request: https://github.com/apache/kafka/pull/2477 MINOR: Change rocksdb logging to error level According to the java-doc: https://github.com/facebook/rocksdb/blob/master/java/src/main/java/org/rocksdb/Logger.java#L31 the rocksdb logging level should

[GitHub] kafka pull request #2471: KAFKA-4317: Checkpoint State Stores on commit/flus...

2017-01-31 Thread dguy
GitHub user dguy opened a pull request: https://github.com/apache/kafka/pull/2471 KAFKA-4317: Checkpoint State Stores on commit/flush Currently the checkpoint file is deleted at state store initialization and it is only ever written again during a clean shutdown. This can result

[GitHub] kafka pull request #2486: KAFKA-4724: Clean up of state directories can poss...

2017-02-02 Thread dguy
GitHub user dguy opened a pull request: https://github.com/apache/kafka/pull/2486 KAFKA-4724: Clean up of state directories can possibly remove stores that are about to be used by another thread Delay the cleanup of state directories that are not locked and not owned

[GitHub] kafka pull request #2516: MINOR: add GlobalKTable doc to streams.html

2017-02-08 Thread dguy
GitHub user dguy opened a pull request: https://github.com/apache/kafka/pull/2516 MINOR: add GlobalKTable doc to streams.html Update streams.html with GlobalKTable docs You can merge this pull request into a Git repository by running: $ git pull https://github.com/dguy/kafka

[GitHub] kafka pull request #2421: HOTFIX: Fix cmd line argument order in streams sys...

2017-01-23 Thread dguy
GitHub user dguy opened a pull request: https://github.com/apache/kafka/pull/2421 HOTFIX: Fix cmd line argument order in streams system tests Fix the ordering of cmd line arguments passed to the system tests. You can merge this pull request into a Git repository by running

[GitHub] kafka pull request #2411: [WIP] KAFKA-4677: Avoid unnecessary task movement ...

2017-01-23 Thread dguy
Github user dguy closed the pull request at: https://github.com/apache/kafka/pull/2411 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature

[GitHub] kafka pull request #2429: KAFKA-4677: Avoid unnecessary task movement across...

2017-01-24 Thread dguy
GitHub user dguy opened a pull request: https://github.com/apache/kafka/pull/2429 KAFKA-4677: Avoid unnecessary task movement across threads during rebalance Makes task assignment more sticky by preferring to assign tasks to clients that had previously had the task as active task

[GitHub] kafka pull request #2421: HOTFIX: Fix cmd line argument order in streams sys...

2017-01-24 Thread dguy
Github user dguy closed the pull request at: https://github.com/apache/kafka/pull/2421 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature

[GitHub] kafka pull request #2430: MINOR: close state store in CachingSessionStoreTes...

2017-01-24 Thread dguy
GitHub user dguy opened a pull request: https://github.com/apache/kafka/pull/2430 MINOR: close state store in CachingSessionStoreTest close the sessions store in `@After` to release rocksdb resources. You can merge this pull request into a Git repository by running: $ git

[GitHub] kafka pull request #2444: KAFKA-4645: Improve test coverage of ProcessorTopo...

2017-01-26 Thread dguy
GitHub user dguy opened a pull request: https://github.com/apache/kafka/pull/2444 KAFKA-4645: Improve test coverage of ProcessorTopology the toString method prints the topology, but had no tests making sure it works and/or doesn't cause exceptions You can merge this pull request

[GitHub] kafka pull request #2447: KAFKA-4646: Improve test coverage AbstractProcesso...

2017-01-26 Thread dguy
GitHub user dguy opened a pull request: https://github.com/apache/kafka/pull/2447 KAFKA-4646: Improve test coverage AbstractProcessorContext Exception paths in `register()`, `topic()`, `partition()`, `offset()`, and `timestamp()`, were not covered by any existing tests You can

[GitHub] kafka pull request #2442: KAFKA-4642: Improve test coverage of ProcessorStat...

2017-01-26 Thread dguy
GitHub user dguy opened a pull request: https://github.com/apache/kafka/pull/2442 KAFKA-4642: Improve test coverage of ProcessorStateManager Most of the exception paths weren't covered. Now they are. You can merge this pull request into a Git repository by running: $ git pull

[GitHub] kafka pull request #2448: KAFKA-4644: Improve test coverage of StreamsPartit...

2017-01-26 Thread dguy
GitHub user dguy opened a pull request: https://github.com/apache/kafka/pull/2448 KAFKA-4644: Improve test coverage of StreamsPartitionAssignor Some exception paths not previously covered. Extracted `ensureCopartitioning` into a static class. You can merge this pull request

[GitHub] kafka pull request #2452: KAFKA-4649: Improve test coverage GlobalStateManag...

2017-01-26 Thread dguy
GitHub user dguy opened a pull request: https://github.com/apache/kafka/pull/2452 KAFKA-4649: Improve test coverage GlobalStateManagerImpl Add coverage for exception paths in `initialize()` You can merge this pull request into a Git repository by running: $ git pull https

[GitHub] kafka pull request #2450: KAFKA-4647: Improve test coverage of GlobalStreamT...

2017-01-26 Thread dguy
GitHub user dguy opened a pull request: https://github.com/apache/kafka/pull/2450 KAFKA-4647: Improve test coverage of GlobalStreamThread Add a test to ensure a `StreamsException` is thrown when an exception other than `StreamsException` is caught You can merge this pull request

[GitHub] kafka pull request #2451: KAFKA-4640: Improve test coverage StreamTask

2017-01-26 Thread dguy
GitHub user dguy opened a pull request: https://github.com/apache/kafka/pull/2451 KAFKA-4640: Improve test coverage StreamTask Provide test coverage for exception paths in: `schedule()`, `closeTopology()`, and `punctuate()` You can merge this pull request into a Git repository

[GitHub] kafka pull request #2560: KAFKA-4494: Reduce startup and rebalance time

2017-02-16 Thread dguy
GitHub user dguy opened a pull request: https://github.com/apache/kafka/pull/2560 KAFKA-4494: Reduce startup and rebalance time Replace one-by-one initialization of state stores with bulk initialization. You can merge this pull request into a Git repository by running: $ git

[GitHub] kafka pull request #2594: MINOR: add code quality checks to checkstyle.xml. ...

2017-02-24 Thread dguy
GitHub user dguy opened a pull request: https://github.com/apache/kafka/pull/2594 MINOR: add code quality checks to checkstyle.xml. Also add suppressions Add code quality/complexity checks to checkstyle You can merge this pull request into a Git repository by running: $ git

[GitHub] kafka pull request #2535: MINOR: don't throw CommitFailedException during su...

2017-02-10 Thread dguy
GitHub user dguy opened a pull request: https://github.com/apache/kafka/pull/2535 MINOR: don't throw CommitFailedException during suspendTasksAndState You can merge this pull request into a Git repository by running: $ git pull https://github.com/dguy/kafka minor-commit

[GitHub] kafka pull request #2547: MINOR: add session windows doc to streams.html

2017-02-14 Thread dguy
GitHub user dguy opened a pull request: https://github.com/apache/kafka/pull/2547 MINOR: add session windows doc to streams.html You can merge this pull request into a Git repository by running: $ git pull https://github.com/dguy/kafka session-window-doc Alternatively you

[GitHub] kafka pull request #2527: MINOR: update README with how to run code coverage...

2017-02-09 Thread dguy
GitHub user dguy opened a pull request: https://github.com/apache/kafka/pull/2527 MINOR: update README with how to run code coverage on module. add reportCoverage at subProject level You can merge this pull request into a Git repository by running: $ git pull https

[GitHub] kafka pull request #2523: MINOR: fix integer overflow in simple benchmark MB...

2017-02-09 Thread dguy
GitHub user dguy opened a pull request: https://github.com/apache/kafka/pull/2523 MINOR: fix integer overflow in simple benchmark MB/sec calculation You can merge this pull request into a Git repository by running: $ git pull https://github.com/dguy/kafka minor-streams-bench

[GitHub] kafka pull request #2532: MINOR: update KafkaStreams.metadataForKey(...) jav...

2017-02-10 Thread dguy
GitHub user dguy opened a pull request: https://github.com/apache/kafka/pull/2532 MINOR: update KafkaStreams.metadataForKey(...) javadoc Add a note to `KafkaStreams.metadataForKey(String, K, Serializer)` to point out that in the case of a Window Store the Serializer should still

[GitHub] kafka pull request #2525: KAFKA-4484: Set more conservative default values o...

2017-02-09 Thread dguy
GitHub user dguy opened a pull request: https://github.com/apache/kafka/pull/2525 KAFKA-4484: Set more conservative default values on RocksDB for memory usage Lowered the default RocksDB settings for the block cache and write buffers You can merge this pull request into a Git

[GitHub] kafka pull request #2359: KAFKA-3452: follow up - fix state store restoratio...

2017-01-16 Thread dguy
Github user dguy closed the pull request at: https://github.com/apache/kafka/pull/2359 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature

[GitHub] kafka pull request #2385: MINOR: remove unused constructor param from Proces...

2017-01-16 Thread dguy
GitHub user dguy opened a pull request: https://github.com/apache/kafka/pull/2385 MINOR: remove unused constructor param from ProcessorStateManager Remove applicationId parameter as it is no longer used. You can merge this pull request into a Git repository by running: $ git

[GitHub] kafka pull request #2396: MINOR: Fix javadoc typos in KStream#process

2017-01-18 Thread dguy
GitHub user dguy opened a pull request: https://github.com/apache/kafka/pull/2396 MINOR: Fix javadoc typos in KStream#process interface for `Processor` in comments incorrectly had `transform` rather than `process`. You can merge this pull request into a Git repository by running

[GitHub] kafka pull request #2397: HOTFIX: ChangeLoggingKeyValueStore.name() returns ...

2017-01-18 Thread dguy
GitHub user dguy opened a pull request: https://github.com/apache/kafka/pull/2397 HOTFIX: ChangeLoggingKeyValueStore.name() returns null This class doesn't need to override this method as it is handled appropriately by the super class You can merge this pull request into a Git

[GitHub] kafka pull request #2411: [WIP] KAFKA-4677: Avoid unnecessary task movement ...

2017-01-20 Thread dguy
GitHub user dguy opened a pull request: https://github.com/apache/kafka/pull/2411 [WIP] KAFKA-4677: Avoid unnecessary task movement across threads of the same process during rebalance You can merge this pull request into a Git repository by running: $ git pull https

[GitHub] kafka pull request #2609: KAFKA-4677: [Follow Up] add optimization to Sticky...

2017-02-28 Thread dguy
GitHub user dguy opened a pull request: https://github.com/apache/kafka/pull/2609 KAFKA-4677: [Follow Up] add optimization to StickyTaskAssignor for rolling rebounce Detect when a rebalance has happened due to one or more existing nodes bouncing. Keep assignment of previous active

[GitHub] kafka pull request #2627: [WIP]: add a consumer leave group config

2017-03-02 Thread dguy
GitHub user dguy opened a pull request: https://github.com/apache/kafka/pull/2627 [WIP]: add a consumer leave group config Add a new config to ConsumerConfig, `leave.group.on.close`. When set to `false` then the `AbstractCoordinator` will not send the `LeaveGroupRequest`. Default

[GitHub] kafka pull request #2611: MINOR: improve MinTimestampTrackerTest and fix NPE...

2017-02-28 Thread dguy
GitHub user dguy opened a pull request: https://github.com/apache/kafka/pull/2611 MINOR: improve MinTimestampTrackerTest and fix NPE when null element removed You can merge this pull request into a Git repository by running: $ git pull https://github.com/dguy/kafka testing

[GitHub] kafka pull request #1819: WIP: add context to some exceptions

2016-09-02 Thread dguy
GitHub user dguy opened a pull request: https://github.com/apache/kafka/pull/1819 WIP: add context to some exceptions KafkaExceptions currently thrown from within StreamThread/StreamTask currently bubble up without any additional context. This makes it hard to figure out where

[GitHub] kafka pull request #1824: KAFKA-4123: RocksDBStore can be marked as open bef...

2016-09-05 Thread dguy
GitHub user dguy opened a pull request: https://github.com/apache/kafka/pull/1824 KAFKA-4123: RocksDBStore can be marked as open before it has been initialized Mark the store as open after the DB has been restored from the changelog. Only add the store to the map

[GitHub] kafka pull request #1811: MINOR: rephrase alpha quality kafka streams wordin...

2016-09-01 Thread dguy
GitHub user dguy opened a pull request: https://github.com/apache/kafka/pull/1811 MINOR: rephrase alpha quality kafka streams wording Rephrase 'alpha quality' wording in Streams section of api.html. Couple of other minor fixes in streams.html You can merge this pull request

[GitHub] kafka pull request #1833: MINOR: fix transient QueryableStateIntegration tes...

2016-09-07 Thread dguy
GitHub user dguy opened a pull request: https://github.com/apache/kafka/pull/1833 MINOR: fix transient QueryableStateIntegration test failure The verification in verifyGreaterOrEqual was incorrect. It was failing when a new key was found. Set the TimeWindow to a large value so

[GitHub] kafka pull request #1804: KAFKA-4104: Queryable state metadata is sometimes ...

2016-08-31 Thread dguy
GitHub user dguy opened a pull request: https://github.com/apache/kafka/pull/1804 KAFKA-4104: Queryable state metadata is sometimes invalid If the thread or process is not the coordinator the Cluster instance in StreamPartitionAssignor will always be null. This builds an instance

[GitHub] kafka pull request #1845: HOTFIX: throw exception if the Cluster object has ...

2016-09-12 Thread dguy
GitHub user dguy opened a pull request: https://github.com/apache/kafka/pull/1845 HOTFIX: throw exception if the Cluster object has not been intialized in StreamMetadataState During rebalance operations the Cluster object gets set to Cluster.empty(). This can result in NPEs when

[GitHub] kafka pull request #1840: MINOR: catch InvalidStateStoreException in Queryab...

2016-09-09 Thread dguy
GitHub user dguy opened a pull request: https://github.com/apache/kafka/pull/1840 MINOR: catch InvalidStateStoreException in QueryableStateIntegrationTest A couple of the tests may transiently fail in QueryableStateIntegrationTest as they are not catching InvalidStateStoreException

[GitHub] kafka pull request #1862: KAFKA-4175: Can't have StandbyTasks in KafkaStream...

2016-09-15 Thread dguy
GitHub user dguy opened a pull request: https://github.com/apache/kafka/pull/1862 KAFKA-4175: Can't have StandbyTasks in KafkaStreams where NUM_STREAM_THREADS_CONFIG > 1 standby tasks should be assigned per consumer not per process You can merge this pull request into a

[GitHub] kafka pull request #1861: HOTFIX: fix KafkaStreams SmokeTest

2016-09-15 Thread dguy
GitHub user dguy opened a pull request: https://github.com/apache/kafka/pull/1861 HOTFIX: fix KafkaStreams SmokeTest Set the NUM_STREAM_THREADS_CONFIG = 1 in SmokeTestClient as we get locking issues when we have NUM_STREAM_THREADS_CONFIG > 1 and we have Standby Tasks,

[GitHub] kafka pull request #1970: KAFKA-4253: Fix Kafka Stream thread shutting down ...

2016-10-05 Thread dguy
GitHub user dguy opened a pull request: https://github.com/apache/kafka/pull/1970 KAFKA-4253: Fix Kafka Stream thread shutting down process ordering Changed the ordering in `StreamThread.shutdown` 1. commitAll (we need to commit so that any cached data is flushed through

[GitHub] kafka pull request #1986: HOTFIX: move restoreConsumer.assign() to shutdownT...

2016-10-07 Thread dguy
GitHub user dguy opened a pull request: https://github.com/apache/kafka/pull/1986 HOTFIX: move restoreConsumer.assign() to shutdownTasksAndState restoreConsumer.assign(..) in removeStandbyTasks was logging an (ignorable) exception due to the restoreConsumer being closed. Moved

[GitHub] kafka pull request #1888: MINOR: remove unused code from InternalTopicManage...

2016-09-20 Thread dguy
GitHub user dguy opened a pull request: https://github.com/apache/kafka/pull/1888 MINOR: remove unused code from InternalTopicManager Remove isValidCleanupPolicy and related fields as they are never used. You can merge this pull request into a Git repository by running: $ git

[GitHub] kafka pull request #1891: MINOR: add javadoc comment to PersistenKeyValueFac...

2016-09-20 Thread dguy
GitHub user dguy opened a pull request: https://github.com/apache/kafka/pull/1891 MINOR: add javadoc comment to PersistenKeyValueFactory.enableCaching missing javadoc on public API method PersistenKeyValueFactory.enableCaching You can merge this pull request into a Git repository

[GitHub] kafka pull request #1879: HOTFIX: logic in QuerybaleStateIntegrationTest.sho...

2016-09-19 Thread dguy
GitHub user dguy opened a pull request: https://github.com/apache/kafka/pull/1879 HOTFIX: logic in QuerybaleStateIntegrationTest.shouldBeAbleToQueryState incorrect The logic in `verifyCanGetByKey` was incorrect. It was ``` windowState.size() < keys.len

[GitHub] kafka pull request #1792: KAFKA-3595: window stores use compact,delete confi...

2016-08-26 Thread dguy
GitHub user dguy opened a pull request: https://github.com/apache/kafka/pull/1792 KAFKA-3595: window stores use compact,delete config for changelogs changelogs of window stores now configure cleanup.policy=compact,delete with retention.ms set to window maintainMs

[GitHub] kafka pull request #1918: [WIP] allow Processors to write to State Stores du...

2016-09-27 Thread dguy
GitHub user dguy opened a pull request: https://github.com/apache/kafka/pull/1918 [WIP] allow Processors to write to State Stores during Processor.init() If a Processor writes to a State Store in the init and 0 records are processed before `commit` is called

[GitHub] kafka pull request #1917: KAFKA-4223: RocksDBStore should close all open ite...

2016-09-27 Thread dguy
GitHub user dguy opened a pull request: https://github.com/apache/kafka/pull/1917 KAFKA-4223: RocksDBStore should close all open iterators on close Keep track of open Rocks DB iterators. When a store is closed, close all open iterators. You can merge this pull request into a Git

[GitHub] kafka pull request #1920: HOTFIX: fix npe in StreamsMetadataState when onCha...

2016-09-27 Thread dguy
GitHub user dguy opened a pull request: https://github.com/apache/kafka/pull/1920 HOTFIX: fix npe in StreamsMetadataState when onChange has not been called If some StreamsMetadataState methods are called before the onChange method is called a NullPointerException was being thrown

[GitHub] kafka pull request #1916: MINOR: add test to make sure ProcessorStateManager...

2016-09-27 Thread dguy
GitHub user dguy opened a pull request: https://github.com/apache/kafka/pull/1916 MINOR: add test to make sure ProcessorStateManager can handle State Stores with logging disabled Adding the test so we know that the State Stores with logging disabled or without a topic don't throw

[GitHub] kafka pull request #2109: MINOR: fix incorrect logging in StreamThread

2016-11-07 Thread dguy
GitHub user dguy opened a pull request: https://github.com/apache/kafka/pull/2109 MINOR: fix incorrect logging in StreamThread Fix incorrect logging when unable to create an active task. The output was: Failed to create an active task %s: It should have the taskId. You can

[GitHub] kafka pull request #2051: KAFKA-4311: Multi layer cache eviction causes forw...

2016-10-21 Thread dguy
GitHub user dguy opened a pull request: https://github.com/apache/kafka/pull/2051 KAFKA-4311: Multi layer cache eviction causes forwarding to incorrect ProcessorNode Given a topology like the one below. If a record arriving in `tableOne` causes a cache eviction, it will trigger

[GitHub] kafka pull request #2166: KAFKA-3452: Support session windows

2016-11-24 Thread dguy
GitHub user dguy opened a pull request: https://github.com/apache/kafka/pull/2166 KAFKA-3452: Support session windows Add support for SessionWindows based on design detailed in https://cwiki.apache.org/confluence/display/KAFKA/KIP-94+Session+Windows. This includes refactoring

[GitHub] kafka pull request #2084: KAFKA-4361: Streams does not respect user configs ...

2016-11-01 Thread dguy
GitHub user dguy opened a pull request: https://github.com/apache/kafka/pull/2084 KAFKA-4361: Streams does not respect user configs for "default" params Enable user provided consumer and producer configs to override the streams default configs. You can merge this pu

[GitHub] kafka pull request #2092: MINOR: remove commented out code and System.out.pr...

2016-11-02 Thread dguy
GitHub user dguy opened a pull request: https://github.com/apache/kafka/pull/2092 MINOR: remove commented out code and System.out.println Remove commented out code and System.out.println from KTableKTableJoinIntegrationTest You can merge this pull request into a Git repository

[GitHub] kafka pull request #2097: KAFKA-4366: KafkaStreams.close() blocks indefinite...

2016-11-03 Thread dguy
GitHub user dguy opened a pull request: https://github.com/apache/kafka/pull/2097 KAFKA-4366: KafkaStreams.close() blocks indefinitely Added `timeout` and `timeUnit` to `KafkaStreams.close(..)`. Now do close on a thread and `join` that thread with the provided `timeout

[GitHub] kafka pull request #2103: KAFKA-4379: Remove caching of dirty and removed ke...

2016-11-04 Thread dguy
GitHub user dguy opened a pull request: https://github.com/apache/kafka/pull/2103 KAFKA-4379: Remove caching of dirty and removed keys from StoreChangeLogger The `StoreChangeLogger` currently keeps a cache of dirty and removed keys and will batch the changelog records such that we

[GitHub] kafka pull request #2024: HOTFIX: fix npe in NamedCache if evict is called w...

2016-10-13 Thread dguy
GitHub user dguy opened a pull request: https://github.com/apache/kafka/pull/2024 HOTFIX: fix npe in NamedCache if evict is called when the cache is empty If evict is called on a NamedCache and the cache is empty an NPE is thrown. This was reported on the user list from a developer

[GitHub] kafka pull request #2001: KAFKA-4283: records deleted from CachingKeyValueSt...

2016-10-10 Thread dguy
GitHub user dguy opened a pull request: https://github.com/apache/kafka/pull/2001 KAFKA-4283: records deleted from CachingKeyValueStore still appear in range and all queries Records that are deleted/removed from the CachingKeyValueStore shouldn't appear in range and all queries

[GitHub] kafka pull request #2226: KAFKA-4492: java.lang.IllegalStateException: Attem...

2016-12-07 Thread dguy
GitHub user dguy opened a pull request: https://github.com/apache/kafka/pull/2226 KAFKA-4492: java.lang.IllegalStateException: Attempting to put a clean entry for key... into NamedCache The NamedCache wasn't correctly dealing with its re-entrant nature. This would result

[GitHub] kafka pull request #2254: KAFKA-4537: StreamPartitionAssignor incorrectly ad...

2016-12-14 Thread dguy
GitHub user dguy opened a pull request: https://github.com/apache/kafka/pull/2254 KAFKA-4537: StreamPartitionAssignor incorrectly adds standby partitions to the partitionsByHostState map If a KafkaStreams app is using Standby Tasks then `StreamPartitionAssignor` will add

[GitHub] kafka pull request #2253: KAFKA-4534: StreamPartitionAssignor only ever upda...

2016-12-14 Thread dguy
GitHub user dguy opened a pull request: https://github.com/apache/kafka/pull/2253 KAFKA-4534: StreamPartitionAssignor only ever updates the partitionsByHostState and metadataWithInternalTopics on first assignment. `partitionsByHostState` and `metadataWithInternalTopics` need

[GitHub] kafka pull request #2255: KAFKA-4539: NPE in StreamThread when creating Stan...

2016-12-14 Thread dguy
GitHub user dguy opened a pull request: https://github.com/apache/kafka/pull/2255 KAFKA-4539: NPE in StreamThread when creating StandbyTasks Tasks that don't have any `StateStore`s wont have a `StandbyTask`, so `createStandbyTask` can return `null`. We need to check

[GitHub] kafka pull request #2253: KAFKA-4534: StreamPartitionAssignor only ever upda...

2016-12-14 Thread dguy
Github user dguy closed the pull request at: https://github.com/apache/kafka/pull/2253 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature

[GitHub] kafka pull request #2256: KAFKA-4534: StreamPartitionAssignor only ever upda...

2016-12-14 Thread dguy
GitHub user dguy opened a pull request: https://github.com/apache/kafka/pull/2256 KAFKA-4534: StreamPartitionAssignor only ever updates the partitionsByHostState and metadataWithInternalTopics on first assignment partitionsByHostState and metadataWithInternalTopics need

[GitHub] kafka pull request #2266: KAFKA-4540: Suspended tasks that are not assigned ...

2016-12-16 Thread dguy
GitHub user dguy opened a pull request: https://github.com/apache/kafka/pull/2266 KAFKA-4540: Suspended tasks that are not assigned to the StreamThread need to be closed before new active and standby tasks are created During `onPartitionsAssigned` first close, and remove, any

[GitHub] kafka pull request #2249: KAFKA-4473: RecordCollector should handle retriabl...

2016-12-13 Thread dguy
GitHub user dguy opened a pull request: https://github.com/apache/kafka/pull/2249 KAFKA-4473: RecordCollector should handle retriable exceptions more strictly The `RecordCollectorImpl` currently drops messages on the floor if an exception is non-null in the producer callback

[GitHub] kafka pull request #2244: KAFKA-4490: Add Global Table support to Kafka Stre...

2016-12-12 Thread dguy
GitHub user dguy opened a pull request: https://github.com/apache/kafka/pull/2244 KAFKA-4490: Add Global Table support to Kafka Streams Add Global Tables to KafkaStreams. Global Tables are fully replicated once-per instance of KafkaStreams. A single thread is used to update them

[GitHub] kafka pull request #2250: KAFKA-4532: StateStores can be connected to the wr...

2016-12-13 Thread dguy
GitHub user dguy opened a pull request: https://github.com/apache/kafka/pull/2250 KAFKA-4532: StateStores can be connected to the wrong source topic resulting in incorrect metadata returned from Interactive Queries When building a topology with tables and StateStores

  1   2   3   >