[GitHub] storm issue #2152: STORM-2639: Kafka Spout incorrectly computes numCommitted...

[askprasanna]

Github user askprasanna commented on the issue:

https://github.com/apache/storm/pull/2152
  
Created a new JIRA and updated PR title as recommended.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm issue #2220: STORM-2638: Add in maximum object size for Pacemaker Thri...

[vesense]

Github user vesense commented on the issue:

https://github.com/apache/storm/pull/2220
  
Looks good to me. It would be better if we can apply new code style to the 
changes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm issue #2208: [STORM-2627] The annotation of storm.zookeeper.topology.a...

[liu-zhaokun]

Github user liu-zhaokun commented on the issue:

https://github.com/apache/storm/pull/2208
  
@HeartSaVioR 
I have modified this PR according to your opinion,and it has passed all the 
test.Could you help me merge it?Thank you very much.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: Connection refused

[Li Wang]

Make sure the nimbus is running properly. 

Sent from my iPhone

> On 18 Jul 2017, at 09:16, sam mohel  wrote:
> 
> i'm facing problem with submitting topology in distributed mode
> storm-0.10.2  zookeeper-3.4.6
> 
> Exception in thread "main" java.lang.RuntimeException:
> org.apache.thrift7.transport.TTransportException:
> java.net.ConnectException: Connection refused (Connection refused)
> at
> backtype.storm.security.auth.TBackoffConnect.retryNext(TBackoffConnect.java:59)
> at
> backtype.storm.security.auth.TBackoffConnect.doConnectWithRetry(TBackoffConnect.java:51)
> at
> backtype.storm.security.auth.ThriftClient.reconnect(ThriftClient.java:103)
> at backtype.storm.security.auth.ThriftClient.(ThriftClient.java:72)
> at backtype.storm.utils.DRPCClient.(DRPCClient.java:44)
> 
> what should i check for fixing this problem ?

Connection refused

[sam mohel]

i'm facing problem with submitting topology in distributed mode
storm-0.10.2  zookeeper-3.4.6

Exception in thread "main" java.lang.RuntimeException:
org.apache.thrift7.transport.TTransportException:
java.net.ConnectException: Connection refused (Connection refused)
at
backtype.storm.security.auth.TBackoffConnect.retryNext(TBackoffConnect.java:59)
at
backtype.storm.security.auth.TBackoffConnect.doConnectWithRetry(TBackoffConnect.java:51)
at
backtype.storm.security.auth.ThriftClient.reconnect(ThriftClient.java:103)
at backtype.storm.security.auth.ThriftClient.(ThriftClient.java:72)
at backtype.storm.utils.DRPCClient.(DRPCClient.java:44)

what should i check for fixing this problem ?

[GitHub] storm issue #2208: [STORM-2627] The annotation of storm.zookeeper.topology.a...

[liu-zhaokun]

Github user liu-zhaokun commented on the issue:

https://github.com/apache/storm/pull/2208
  
@HeartSaVioR 
Thanks for your reply.I will remove it from Config.java.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm issue #2208: [STORM-2627] The annotation of storm.zookeeper.topology.a...

[HeartSaVioR]

Github user HeartSaVioR commented on the issue:

https://github.com/apache/storm/pull/2208
  
I think the best way to avoid misleading is just removing the configuration 
and set the value as constant. We don't need to provide an option to user while 
the value is actually fixed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm issue #2208: [STORM-2627] The annotation of storm.zookeeper.topology.a...

[liu-zhaokun]

Github user liu-zhaokun commented on the issue:

https://github.com/apache/storm/pull/2208
  
@harshach 
Thanks for your reply.You can see StormSubmitter.java,line 
91.STORM_ZOOKEEPER_TOPOLOGY_AUTH_SCHEME is should always be set to digest.It 
can't be and won't be other value.So,I think we should declare  
“"storm.zookeeper.topology.auth.scheme” should always be set to digest.We 
shouldn't mislead others.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm issue #2219: [STORM-2637] fix ClassCastException in logviewer get-log-...

[Ethanlm]

Github user Ethanlm commented on the issue:

https://github.com/apache/storm/pull/2219
  
Close this PR.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request #2219: [STORM-2637] fix ClassCastException in logviewer g...

[Ethanlm]

Github user Ethanlm closed the pull request at:

https://github.com/apache/storm/pull/2219


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm issue #2152: [STORM-2505] OffsetManager should account for offset void...

[hmcl]

Github user hmcl commented on the issue:

https://github.com/apache/storm/pull/2152
  
@askprasanna let's just create a new JIRA and associate it with the 
STORM-2505 JIRA - it's a quick thing and will take care of this clean. 

Another minor detail we have a script to find orphan pull requests, 
malformed pull requests, etc. The pull request description `[STORM-2505] 
OffsetManager...` does not conform with the name convention. Can you please 
rename it to follow the pattern `STORM-2505: OffsetManager...`. Thanks.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm issue #2152: [STORM-2505] OffsetManager should account for offset void...

[HeartSaVioR]

Github user HeartSaVioR commented on the issue:

https://github.com/apache/storm/pull/2152
  
@askprasanna 
Maybe we would need to associate this to a new issue to track your effort 
(given that assignee for STORM-2505 is not you and it can't be overwritten), or 
remove the issue at all. Except test code I think it's fairly small change 
which doesn't strictly require filing an issue.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm issue #2219: [STORM-2637] fix ClassCastException in logviewer get-log-...

[Ethanlm]

Github user Ethanlm commented on the issue:

https://github.com/apache/storm/pull/2219
  
Thanks. I will submit a patch against 1.x branch.  I am not very familiar 
with logviewer clojure implementation yet. But I will try and see what I can 
do. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm issue #2219: [STORM-2637] fix ClassCastException in logviewer get-log-...

[HeartSaVioR]

Github user HeartSaVioR commented on the issue:

https://github.com/apache/storm/pull/2219
  
Thanks for the patch. I just would like to see this patch against 1.x 
branch rather than master, given that we should review and apply #2204 to 
remove Clojure implementation. 
Btw, if you had (or have) a chance to read logviewer clojure 
implementation, I would like to ask a favor with reviewing #2204.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request #2219: [STORM-2637] fix ClassCastException in logviewer g...

[HeartSaVioR]

Github user HeartSaVioR commented on a diff in the pull request:

https://github.com/apache/storm/pull/2219#discussion_r127842992
  
--- Diff: storm-core/src/clj/org/apache/storm/daemon/logviewer.clj ---
@@ -311,7 +311,7 @@
 
 (defn get-log-user-group-whitelist [fname]
   (let [wl-file (ServerConfigUtils/getLogMetaDataFile fname)
-m (clojurify-structure (Utils/readYamlFile wl-file))]
+m (clojurify-structure (Utils/readYamlFile (.toString wl-file)))]
--- End diff --

Maybe better to use `getAbsolutePath` or `getCanonicalPath` to clearly 
represent path. I used `getAbsolutePath` from #2204 but I can change it if 
others claim that `getCanonicalPath` is more preferred.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request #2147: STORM-2538: New kafka spout emits duplicate tuples

[hmcl]

Github user hmcl closed the pull request at:

https://github.com/apache/storm/pull/2147


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm issue #2147: STORM-2538: New kafka spout emits duplicate tuples

[hmcl]

Github user hmcl commented on the issue:

https://github.com/apache/storm/pull/2147
  
Closed in favor of #2151 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm issue #2147: STORM-2538: New kafka spout emits duplicate tuples

[harshach]

Github user harshach commented on the issue:

https://github.com/apache/storm/pull/2147
  
@hmcl  @srdo I don't think we need this given this PR  
https://github.com/apache/storm/pull/2151 makes manual assignment as default.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm issue #2151: STORM-2542: Remove KafkaConsumer.subscribe API option, ma...

[harshach]

Github user harshach commented on the issue:

https://github.com/apache/storm/pull/2151
  
+1. Thanks @srdo this looks great.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm issue #2155: STORM-2548: Simplify KafkaSpoutConfig to avoid duplicatin...

[harshach]

Github user harshach commented on the issue:

https://github.com/apache/storm/pull/2155
  
still +1 .


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm issue #2217: [1.x-branch] [STORM-2505] OffsetManager should account fo...

[harshach]

Github user harshach commented on the issue:

https://github.com/apache/storm/pull/2217
  
+1


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm issue #2215: STORM-2548: Simplify KafkaSpoutConfig (1.x)

[harshach]

Github user harshach commented on the issue:

https://github.com/apache/storm/pull/2215
  
+1


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request #2215: STORM-2548: Simplify KafkaSpoutConfig (1.x)

[harshach]

Github user harshach commented on a diff in the pull request:

https://github.com/apache/storm/pull/2215#discussion_r127821694
  
--- Diff: 
external/storm-kafka-client/src/main/java/org/apache/storm/kafka/spout/SerializableDeserializer.java
 ---
@@ -16,10 +16,14 @@
 package org.apache.storm.kafka.spout;
 
 import java.io.Serializable;
+import org.apache.kafka.clients.consumer.ConsumerConfig;
 import org.apache.kafka.common.serialization.Deserializer;
 
 /**
  * @param  The type this deserializer deserializes to.
+ * @deprecated Avoid using this class. Use {@link 
KafkaSpoutConfig.Builder#setProp(java.lang.String, java.lang.Object) } with
+ * {@link ConsumerConfig#KEY_DESERIALIZER_CLASS_CONFIG} and {@link 
ConsumerConfig#VALUE_DESERIALIZER_CLASS_CONFIG} instead
  */
-public interface SerializableDeserializer extends Deserializer, 
Serializable { 
+@Deprecated
--- End diff --

why are we adding this back?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm issue #2147: STORM-2538: New kafka spout emits duplicate tuples

[srdo]

Github user srdo commented on the issue:

https://github.com/apache/storm/pull/2147
  
@hmcl I am strongly in favor of option 1, since no one has provided a use 
case for/argued in favor of the subscribe API based partition assignment code. 
Ideally we'll deprecate (1.x) and remove (2.x) subscribe API support as well, 
rather than leaving it as an unsupported option.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm issue #2208: [STORM-2627] The annotation of storm.zookeeper.topology.a...

[harshach]

Github user harshach commented on the issue:

https://github.com/apache/storm/pull/2208
  
@liu-zhaokun I think the comment there meant to say by default it will be 
"No Authentication". I.e Its users responsibility to set to digest in a secure 
clusters. But since the default settings for non-secure the comment looks ok to 
me.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm issue #2218: STORM-2614: Enhance stateful windowing to persist the win...

[roshannaik]

Github user roshannaik commented on the issue:

https://github.com/apache/storm/pull/2218
  
Sounds like a checkpoint mechanism limited to windowed bolts. Could you 
provide a design document for this ?  


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request #2220: STORM-2638: Add in maximum object size for Pacemak...

[knusbaum]

GitHub user knusbaum opened a pull request:

https://github.com/apache/storm/pull/2220

STORM-2638: Add in maximum object size for Pacemaker Thrift objects



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/knusbaum/incubator-storm STORM-2638

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/storm/pull/2220.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2220


commit e6465b5dc5c7e31e0b92a18de578a13d05a5abf2
Author: Kyle Nusbaum 
Date:   2017-07-17T19:40:15Z

Adding in maximum/minimum object size




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm issue #2209: [STORM-2622] Add owner resource summary on storm UI

[harshach]

Github user harshach commented on the issue:

https://github.com/apache/storm/pull/2209
  
+1


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request #2219: [STORM-2637] fix ClassCastException in logviewer g...

[Ethanlm]

GitHub user Ethanlm opened a pull request:

https://github.com/apache/storm/pull/2219

[STORM-2637] fix ClassCastException in logviewer 
get-log-user-group-whitelist func…

In get-log-user-group-whitelist function, the getLogMetaDataFile function 
returns a File while the readYamlFile function requires a String.

see https://issues.apache.org/jira/browse/STORM-2637

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/Ethanlm/storm STORM-2637

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/storm/pull/2219.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2219


commit f636fdbab9aec58bc27da3b5b8c7a1110266c05a
Author: Ethan Li 
Date:   2017-07-17T19:02:21Z

fix ClassCastException in logviewer get-log-user-group-whitelist function




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm issue #2147: STORM-2538: New kafka spout emits duplicate tuples

[hmcl]

Github user hmcl commented on the issue:

https://github.com/apache/storm/pull/2147
  
@srdo @harshach @arunmahadevan in face of the Manual Partition assignment 
related [fixes](https://github.com/apache/storm/pull/2151/commits), there are 
two options. I believe in the email list we agreed to go with option #1, but I 
would like to make sure.

1 - We disregard and close this PR. If we do so, we are implicitly telling 
the user that if they decide to use dynamic partition assignment they are at 
their own responsibility and it won't work with multiple parallelism. Currently 
the only implementation we support is `ManualPartitionSubscription`
2 - We improve this code to make it work for the dynamic case.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request #2151: STORM-2542: Remove KafkaConsumer.subscribe API opt...

[hmcl]

Github user hmcl commented on a diff in the pull request:

https://github.com/apache/storm/pull/2151#discussion_r127773597
  
--- Diff: docs/storm-kafka-client.md ---
@@ -240,12 +240,9 @@ streams.  If you are doing this for Trident a value 
must be in the List returned
 otherwise trident can throw exceptions.
 
 
-### Manual Partition Control (ADVANCED)
+### Manual Partition Assigment (ADVANCED)
 
-By default Kafka will automatically assign partitions to the current set 
of spouts.  It handles lots of things, but in some cases you may want to 
manually assign the partitions.
-This can cause less churn in the assignments when spouts go down and come 
back up, but it can result in a lot of issues if not done right.  This can all 
be handled by subclassing
-Subscription and we have a few implementations that you can look at for 
examples on how to do this.  ManualPartitionNamedSubscription and 
ManualPartitionPatternSubscription.  Again
-please be careful when using these or implementing your own.
+By default the KafkaSpout instancs will be assigned partitions by round 
robin assignment. If you need to customize this assignment, you can implement 
the `ManualPartitioner` interface. The implementation can be passed to the 
`ManualPartitionSubscription` constructor, and the `Subscription` can then be 
set in the `KafkaSpoutConfig` via the `KafkaSpoutConfig.Builder` constructor. 
Please take care when supplying a custom implementation, since an incorrect 
`ManualPartitioner` implementation could leave some partitions unread, or 
concurrently read by multiple spout instances. See the 
`RoundRobinManualPartitioner` for an example of how to implement this 
functionality.
--- End diff --

By default the KafkaSpout instances will be assigned partitions using a 
round robin strategy. If you need to customize partitions assignment, you must 
implement the `ManualPartitioner` interface. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request #2150: STORM-2541: Fix storm-kafka-client manual subscrip...

[hmcl]

Github user hmcl commented on a diff in the pull request:

https://github.com/apache/storm/pull/2150#discussion_r127769417
  
--- Diff: 
external/storm-kafka-client/src/main/java/org/apache/storm/kafka/spout/TopicFilter.java
 ---
@@ -0,0 +1,38 @@
+/*
+ * Copyright 2017 The Apache Software Foundation.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *  http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.storm.kafka.spout;
+
+import java.io.Serializable;
+import java.util.List;
+import org.apache.kafka.clients.consumer.KafkaConsumer;
+import org.apache.kafka.common.TopicPartition;
+
+public interface TopicFilter extends Serializable {
+
+/**
+ * Get the Kafka TopicPartitions passed by this filter. 
+ * @param consumer The Kafka consumer to use to read the list of 
existing partitions
+ * @return The Kafka partitions passed by this filter.
+ */
+List getFilteredTopicPartitions(KafkaConsumer 
consumer);
+
+/**
+ * @return A human-readable string representing the subscribed topics.
--- End diff --

NIT: representing the topics that pass the filter


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm issue #2150: STORM-2541: Fix storm-kafka-client manual subscription no...

[hmcl]

Github user hmcl commented on the issue:

https://github.com/apache/storm/pull/2150
  
+1. @harshach @priyank5485 can you please take one final look. If you don't 
have any objection, I suggest that we merge this patch in the next day or so.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm issue #2155: STORM-2548: Simplify KafkaSpoutConfig to avoid duplicatin...

[hmcl]

Github user hmcl commented on the issue:

https://github.com/apache/storm/pull/2155
  
+1. @harshach @priyank5485 can you please take one final look. If you don't 
have any objection, I suggest that we merge this patch in the next day or so.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request #2218: STORM-2614: Enhance stateful windowing to persist ...

[arunmahadevan]

GitHub user arunmahadevan opened a pull request:

https://github.com/apache/storm/pull/2218

STORM-2614: Enhance stateful windowing to persist the window state

Right now the tuples in window are stored in memory. This limits the usage 
to windows
that fit in memory. Also the source tuples cannot be acked until the window 
expiry.
By persisting the window transparently in the state backend and 
caching/iterating them as needed,
we could support larger windows and also support windowed bolts with 
user/application state.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/arunmahadevan/storm STORM-2614

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/storm/pull/2218.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2218


commit 376d3120a5075d9687f63a587dca5e040a5aa6eb
Author: Arun Mahadevan 
Date:   2017-06-28T16:13:26Z

STORM-2614: Enhance stateful windowing to persist the window state

Right now the tuples in window are stored in memory. This limits the usage 
to windows
that fit in memory. Also the source tuples cannot be acked until the window 
expiry.
By persisting the window transparently in the state backend and 
caching/iterating them as needed,
we could support larger windows and also support windowed bolts with 
user/application state.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm issue #2199: [STORM-2201] Add dynamic scheduler configuration loading

[Ethanlm]

Github user Ethanlm commented on the issue:

https://github.com/apache/storm/pull/2199
  
Hi @HeartSaVioR @revans2, could you please review this when you get a 
chance? Thanks very much!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm issue #2209: [STORM-2622] Add owner resource summary on storm UI

[Ethanlm]

Github user Ethanlm commented on the issue:

https://github.com/apache/storm/pull/2209
  
Added some unit tests


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm issue #2152: [STORM-2505] OffsetManager should account for offset void...

[askprasanna]

Github user askprasanna commented on the issue:

https://github.com/apache/storm/pull/2152
  
Squashed the commits. Also opened https://github.com/apache/storm/pull/2217 
for 1.x branch


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request #2217: [1.x-branch] [STORM-2505] OffsetManager should acc...

[askprasanna]

GitHub user askprasanna opened a pull request:

https://github.com/apache/storm/pull/2217

[1.x-branch] [STORM-2505] OffsetManager should account for offset voids 
when retur…

…ning the count of committed offsets

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/askprasanna/storm 1.x-2505

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/storm/pull/2217.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2217


commit b661c3b2f0ef529ca33c3ba142f9440f39b8ef4f
Author: Prasanna Ranganathan 
Date:   2017-06-07T12:33:09Z

[STORM-2505] OffsetManager should account for offset voids when returning 
the count of committed offsets




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request #2152: [STORM-2505] OffsetManager should account for offs...

[askprasanna]

Github user askprasanna commented on a diff in the pull request:

https://github.com/apache/storm/pull/2152#discussion_r127695589
  
--- Diff: 
external/storm-kafka-client/src/test/java/org/apache/storm/kafka/spout/KafkaSpoutCommitTest.java
 ---
@@ -0,0 +1,146 @@
+/*
+ * Copyright 2017 The Apache Software Foundation.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *  http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.storm.kafka.spout;
+
+import static 
org.apache.storm.kafka.spout.builders.SingleTopicKafkaSpoutConfiguration.getKafkaSpoutConfigBuilder;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertTrue;
+import static org.mockito.Matchers.*;
+import static org.mockito.Mockito.*;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+import org.apache.kafka.clients.consumer.*;
+import org.apache.kafka.common.TopicPartition;
+import 
org.apache.storm.kafka.spout.builders.SingleTopicKafkaSpoutConfiguration;
+import org.apache.storm.kafka.spout.internal.KafkaConsumerFactory;
+import org.apache.storm.spout.SpoutOutputCollector;
+import org.apache.storm.task.TopologyContext;
+import org.apache.storm.utils.Time;
+import org.apache.storm.utils.Time.SimulatedTime;
+import org.junit.Test;
+import org.mockito.ArgumentCaptor;
+import org.mockito.Captor;
+import org.mockito.InOrder;
+import org.mockito.MockitoAnnotations;
+
+public class KafkaSpoutCommitTest {
+
+private final long offsetCommitPeriodMs = 2_000;
+private final TopologyContext contextMock = 
mock(TopologyContext.class);
+private final SpoutOutputCollector collectorMock = 
mock(SpoutOutputCollector.class);
+private final Map conf = new HashMap<>();
+private final TopicPartition partition = new 
TopicPartition(SingleTopicKafkaSpoutConfiguration.TOPIC, 1);
+private KafkaConsumer consumerMock;
+private KafkaSpout spout;
+private KafkaSpoutConfig spoutConfig;
+
+@Captor
+private ArgumentCaptor> 
commitCapture;
+
+private void setupSpout(Set assignedPartitions) {
+MockitoAnnotations.initMocks(this);
+spoutConfig = getKafkaSpoutConfigBuilder(-1)
+.setOffsetCommitPeriodMs(offsetCommitPeriodMs)
+.build();
+
+consumerMock = mock(KafkaConsumer.class);
+KafkaConsumerFactory consumerFactory = 
(kafkaSpoutConfig) -> consumerMock;
+
+//Set up a spout listening to 1 topic partition
+spout = new KafkaSpout<>(spoutConfig, consumerFactory);
+
+spout.open(conf, contextMock, collectorMock);
+spout.activate();
+
+ArgumentCaptor rebalanceListenerCapture 
= ArgumentCaptor.forClass(ConsumerRebalanceListener.class);
+verify(consumerMock).subscribe(anyCollection(), 
rebalanceListenerCapture.capture());
+
+//Assign partitions to the spout
+ConsumerRebalanceListener consumerRebalanceListener = 
rebalanceListenerCapture.getValue();
+consumerRebalanceListener.onPartitionsAssigned(assignedPartitions);
+}
+
+@Test
+public void testCommitSuccessWithOffsetVoids() {
+//Verify that the commit logic can handle offset voids
+try (SimulatedTime simulatedTime = new SimulatedTime()) {
+setupSpout(Collections.singleton(partition));
+Map>> 
records = new HashMap<>();
+List> recordsForPartition = new 
ArrayList<>();
+// Offsets emitted are 0,1,2,3,4,...,8,9
--- End diff --

will tweak the comment.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Release Planning for 1.1.1 and others?

[Jungtaek Lim]

Hi devs,

We released Storm 1.1.0 at the late March, and we didn't have short plan
for next release.
(We discussed Storm 2.0.0, and I think we're getting closer, a bit slowly
but steady.)

While we still have some issues and pull requests for storm-kafka-client
which I think should be included to 1.0.1, but maybe we can start planning
1.1.1, having epic issue for release as we did for prior releases, and also
other versions as well (1.2.0 or 2.0.0).

Unless I'm missing something, looks like there're no critical bugs except
storm-kafka-client for 1.1.1 candidate. So I think we could roll out 1.1.1
fairly soon after sorting out current issue on storm-kafka-client. I'm not
familiar with storm-kafka-client module hence I'd like to rely on Hugo or
Stig or other committers/contributors to sort out, but may spend some time
to be familiar and participate if its progress is going to be slow.

Looking forward to hear your opinions.

Thanks,
Jungtaek Lim (HeartSaVioR)

[GitHub] storm issue #2208: [STORM-2627] The annotation of storm.zookeeper.topology.a...

[liu-zhaokun]

Github user liu-zhaokun commented on the issue:

https://github.com/apache/storm/pull/2208
  
@haitaoyao 
I am so sorry to bother you.Do you have time to help me review it?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm issue #2207: [STORM-2626] Provided a template for drpc-auth-acl.yaml

[liu-zhaokun]

Github user liu-zhaokun commented on the issue:

https://github.com/apache/storm/pull/2207
  
@harshach 
I am so sorry to bother you.Do you have time to help me review it?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---