[GitHub] storm pull request #1942: STORM-2358: Update storm hdfs spout to remove spec...

[pasalkarsachin1]

Github user pasalkarsachin1 commented on a diff in the pull request:

https://github.com/apache/storm/pull/1942#discussion_r101702796
  
--- Diff: 
external/storm-hdfs/src/main/java/org/apache/storm/hdfs/spout/HDFSSpout.java ---
@@ -105,87 +105,87 @@
 
   private String configKey = Configs.DEFAULT_HDFS_CONFIG_KEY; // key for 
hdfs Kerberos configs
 
-  public HdfsSpout() {
+  public HDFSSpout() {
   }
 
-  public HdfsSpout setHdfsUri(String hdfsUri) {
+  public HDFSSpout setHdfsUri(String hdfsUri) {
 this.hdfsUri = hdfsUri;
 return this;
   }
 
-  public HdfsSpout setReaderType(String readerType) {
+  public HDFSSpout setReaderType(Class readerType) {
--- End diff --

Done please review


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request #1942: STORM-2358: Update storm hdfs spout to remove spec...

[pasalkarsachin1]

Github user pasalkarsachin1 commented on a diff in the pull request:

https://github.com/apache/storm/pull/1942#discussion_r101699216
  
--- Diff: 
external/storm-hdfs/src/main/java/org/apache/storm/hdfs/spout/HDFSSpout.java ---
@@ -105,87 +105,87 @@
 
   private String configKey = Configs.DEFAULT_HDFS_CONFIG_KEY; // key for 
hdfs Kerberos configs
 
-  public HdfsSpout() {
+  public HDFSSpout() {
   }
 
-  public HdfsSpout setHdfsUri(String hdfsUri) {
+  public HDFSSpout setHdfsUri(String hdfsUri) {
 this.hdfsUri = hdfsUri;
 return this;
   }
 
-  public HdfsSpout setReaderType(String readerType) {
+  public HDFSSpout setReaderType(Class readerType) {
--- End diff --

Sure I will deprecate this. However adding deprecation will add more code


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request #1942: STORM-2358: Update storm hdfs spout to remove spec...

[arunmahadevan]

Github user arunmahadevan commented on a diff in the pull request:

https://github.com/apache/storm/pull/1942#discussion_r101698782
  
--- Diff: 
external/storm-hdfs/src/main/java/org/apache/storm/hdfs/spout/HDFSSpout.java ---
@@ -722,19 +703,10 @@ private FileReader createFileReader(Path file)
*/
   private FileReader createFileReader(Path file, String offset)
   throws IOException {
-if ( readerType.equalsIgnoreCase(Configs.SEQ) ) {
-  return new SequenceFileReader(this.hdfs, file, conf, offset);
-}
-if ( readerType.equalsIgnoreCase(Configs.TEXT) ) {
-  return new TextFileReader(this.hdfs, file, conf, offset);
-}
-
 try {
-  Class clsType = Class.forName(readerType);
-  Constructor constructor = 
clsType.getConstructor(FileSystem.class, Path.class, Map.class, String.class);
+  Constructor constructor = 
readerType.getConstructor(FileSystem.class, Path.class, Map.class, 
String.class);
--- End diff --

Not sure what you mean by "get rid of specific implementations". I see that 
you are mainly cleaning up and refactoring the code and so its better if you do 
it with this patch.  


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request #1942: STORM-2358: Update storm hdfs spout to remove spec...

[pasalkarsachin1]

Github user pasalkarsachin1 commented on a diff in the pull request:

https://github.com/apache/storm/pull/1942#discussion_r101698425
  
--- Diff: 
external/storm-hdfs/src/main/java/org/apache/storm/hdfs/spout/HDFSSpout.java ---
@@ -722,19 +703,10 @@ private FileReader createFileReader(Path file)
*/
   private FileReader createFileReader(Path file, String offset)
   throws IOException {
-if ( readerType.equalsIgnoreCase(Configs.SEQ) ) {
-  return new SequenceFileReader(this.hdfs, file, conf, offset);
-}
-if ( readerType.equalsIgnoreCase(Configs.TEXT) ) {
-  return new TextFileReader(this.hdfs, file, conf, offset);
-}
-
 try {
-  Class clsType = Class.forName(readerType);
-  Constructor constructor = 
clsType.getConstructor(FileSystem.class, Path.class, Map.class, String.class);
+  Constructor constructor = 
readerType.getConstructor(FileSystem.class, Path.class, Map.class, 
String.class);
--- End diff --

Same code is being there before my changes, I can create JIRA for this. ATM 
I am just trying to get rid of specific implementations


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request #1942: STORM-2358: Update storm hdfs spout to remove spec...

[pasalkarsachin1]

Github user pasalkarsachin1 commented on a diff in the pull request:

https://github.com/apache/storm/pull/1942#discussion_r101698420
  
--- Diff: 
external/storm-hdfs/src/main/java/org/apache/storm/hdfs/spout/HDFSSpout.java ---
@@ -409,9 +409,14 @@ public void open(Map conf, TopologyContext context, 
SpoutOutputCollector collect
 
 // Reader type config
 if ( readerType==null && conf.containsKey(Configs.READER_TYPE) ) {
-  readerType = conf.get(Configs.READER_TYPE).toString();
+   String className = (String) conf.get(Configs.READER_TYPE);
--- End diff --

It was already there so I updated as per requirement. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request #1942: STORM-2358: Update storm hdfs spout to remove spec...

[arunmahadevan]

Github user arunmahadevan commented on a diff in the pull request:

https://github.com/apache/storm/pull/1942#discussion_r101696141
  
--- Diff: 
external/storm-hdfs/src/main/java/org/apache/storm/hdfs/spout/HDFSSpout.java ---
@@ -722,19 +703,10 @@ private FileReader createFileReader(Path file)
*/
   private FileReader createFileReader(Path file, String offset)
   throws IOException {
-if ( readerType.equalsIgnoreCase(Configs.SEQ) ) {
-  return new SequenceFileReader(this.hdfs, file, conf, offset);
-}
-if ( readerType.equalsIgnoreCase(Configs.TEXT) ) {
-  return new TextFileReader(this.hdfs, file, conf, offset);
-}
-
 try {
-  Class clsType = Class.forName(readerType);
-  Constructor constructor = 
clsType.getConstructor(FileSystem.class, Path.class, Map.class, String.class);
+  Constructor constructor = 
readerType.getConstructor(FileSystem.class, Path.class, Map.class, 
String.class);
--- End diff --

This assumes a ctor with expected number and type of params is available 
and tries to invoke it via reflection. Adding an init/open with required params 
in the `FileReader` is much cleaner IMO and avoids having to check and throw 
exceptions at runtime.  


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request #1942: STORM-2358: Update storm hdfs spout to remove spec...

[arunmahadevan]

Github user arunmahadevan commented on a diff in the pull request:

https://github.com/apache/storm/pull/1942#discussion_r101697143
  
--- Diff: 
external/storm-hdfs/src/main/java/org/apache/storm/hdfs/spout/HDFSSpout.java ---
@@ -105,87 +105,87 @@
 
   private String configKey = Configs.DEFAULT_HDFS_CONFIG_KEY; // key for 
hdfs Kerberos configs
 
-  public HdfsSpout() {
+  public HDFSSpout() {
   }
 
-  public HdfsSpout setHdfsUri(String hdfsUri) {
+  public HDFSSpout setHdfsUri(String hdfsUri) {
 this.hdfsUri = hdfsUri;
 return this;
   }
 
-  public HdfsSpout setReaderType(String readerType) {
+  public HDFSSpout setReaderType(Class readerType) {
--- End diff --

1. How about backward compatibility ? Changing the signature will break 
existing clients. May be you can deprecate the earlier method and add a new one 
like `setFileReader(Class reader)`

2. How would one initialize a `HDFSSpout` via Flux? Earlier since the 
method accepted a String it was pretty trivial. Try it out and add some 
example, or add an additional method like `setFileReader(String className)` 
that takes the class name.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request #1942: STORM-2358: Update storm hdfs spout to remove spec...

[arunmahadevan]

Github user arunmahadevan commented on a diff in the pull request:

https://github.com/apache/storm/pull/1942#discussion_r101696530
  
--- Diff: 
external/storm-hdfs/src/main/java/org/apache/storm/hdfs/spout/HDFSSpout.java ---
@@ -409,9 +409,14 @@ public void open(Map conf, TopologyContext context, 
SpoutOutputCollector collect
 
 // Reader type config
 if ( readerType==null && conf.containsKey(Configs.READER_TYPE) ) {
-  readerType = conf.get(Configs.READER_TYPE).toString();
+   String className = (String) conf.get(Configs.READER_TYPE);
--- End diff --

What is the need for this when the spout accepts the readerType via 
`setReaderType ` ?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm issue #1939: STORM-1363: TridentKafkaState should handle null values f...

[pasalkarsachin1]

Github user pasalkarsachin1 commented on the issue:

https://github.com/apache/storm/pull/1939
  
Can someone take a look at this?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm issue #1831: STORM-2249: Make Distribution Scripts Put Examples to the...

[HeartSaVioR]

Github user HeartSaVioR commented on the issue:

https://github.com/apache/storm/pull/1831
  
Also could you check how big binary dist (tar.gz / zip) will be? It's 
already over 200MB so IMHO it would be better to reduce/restrict the size of 
binary dist.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: [DISCUSS] Release Storm 1.1.0

[Jungtaek Lim]

Apache Zeppelin provides 700 MB of full set and 272 MB of netinst via
Apache download page so there should be ways to do it, but for me 200MB is
already too huge.

I have been wondering why we package external connectors, which other
projects don't.

When we didn't make uber jar for connectors from previous versions, only
adding jar shouldn't work which makes packaged jars useless.
Since we start to make uber jar for some connectors, and 1.1.0 will
introduce "adding jars at the topology submission step" so it's not
useless, but still inconvenient way to use. Moreover, in this case we can
still use new feature "adding maven artifacts at the topology submission
step".
It's only useful for who has the node for submission which blocks Maven
central or even internet.

If we want to have a discussion about this, I'll initiate separate
discussion thread.

- Jungtaek Lim (HeartSaVioR)

2017년 2월 17일 (금) 오전 6:00, P. Taylor Goetz 님이 작성:

> I have a release candidate for 1.1.0 built, but I can’t stage the
> convenience binaries because they are now over the ASF subversion size
> limit of 200MB.
>
> I’ll work with infra to figure out a solution.
>
> -Taylor
>
> > On Feb 16, 2017, at 9:46 AM, Jungtaek Lim  wrote:
> >
> > Now all issues on the Storm 1.1.0 epic got resolved. :)
> >
> > There're still some bugfix pull requests for storm-kafka-client, but
> > releasing Storm 1.1.0 has been tremendously dragged (I initiated this at
> > Sep. 2016) so unless they're critical, I'm +1 on starting release phase
> on
> > Storm 1.1.0 ASAP.
> >
> > - Jungtaek Lim
> >
> > 2017년 2월 14일 (화) 오전 11:41, Harsha Chintalapani 님이 작성:
> >
> > STORM-2340 is more of a feature . Auto-commit mode in storm-kafka used
> > rarely and most users
> > run the kafka spout with ackers and get at-least once guarantee.  If its
> > going to longer to address the PR reviews
> > I am +1 on moving this out of Storm 1.1.0. We already quite a few patches
> > storm-kafka-client and 1.1.0 release brings in lot of improvements
> > and bug-fixes.
> > -Harsha
> >
> > On Wed, Feb 8, 2017 at 6:15 PM Jungtaek Lim  wrote:
> >
> >> There seems some pull requests for bugfix/improvement on
> >> storm-kafka-client, and some authors in PRs are not availble for now.
> >> (waiting 7 days)
> >>
> >> If we plan to get 1.1.1 out soon (say 1 month later or even closer) we
> can
> >> postpone, but if not, it might be better to coordinate these things ASAP
> >> and include to 1.1.0.
> >>
> >> There seems to be other small PRs, but nothing seems critical so it
> would
> >> be OK to not wait for merging.
> >>
> >> - Jungtaek Lim
> >>
> >> 2017년 2월 9일 (목) 오전 6:48, P. Taylor Goetz 님이 작성:
> >>
> >> Right now we’re down to 1 open issue on the 1.1.0 release epic:
> STORM-2250
> >> which is under active review/discussion.
> >>
> >> Assuming that is mergeable in the near future, are there any other open
> >> issues that should be considered for this release?
> >>
> >> -Taylor
> >>
> >>
> >>> On Feb 2, 2017, at 4:48 PM, P. Taylor Goetz  wrote:
> >>>
> >>> Thanks for putting this list together Jungtaek. I added a few to the
> 1.1
> >> release epic that I think are important. Feel free to do the same.
> >>>
> >>> Looks like we have a few to go, but there are pull requests for them.
> >> It’s mostly just a matter of reviews and review responses, so I think we
> >> are close.
> >>>
> >>> -Taylor
> >>>
>  On Feb 2, 2017, at 1:41 AM, Jungtaek Lim  wrote:
> 
>  Seems like there're not blockers for 1.1.0, but some pull requests are
>  worth to check.
>  There're pending pull requests for storm-kafka-client waited on
> >> STORM-2225.
>  Given that STORM-2225 is now merged, we might need to take a look at.
> 
>  *- reviewing*
> 
>  [storm-core]
> 
> > STORM-2324 : Fix deployment failure if resources directory is missing
> >> in
>  topology jar
>  (master) https://github.com/apache/storm/pull/1908
>  (1.x) https://github.com/apache/storm/pull/1898
> 
> > STORM-2321 Handle blobstore zk key deletion in KeySequenceNumber
>  (master) https://github.com/apache/storm/pull/1904
>  (1.x) https://github.com/apache/storm/pull/1905
> 
>  [storm-kafka]
> 
> > STORM-2270 Kafka spout should consume from latest when ZK partition
>  commit offset bigger than the latest offset
>  (1.x) https://github.com/apache/storm/pull/1851
> 
>  [storm-kafka-client]
> 
> > STORM-2281: Running Multiple Kafka Spouts (Trident) Throws Illegal
> >> State
>  Exception
>  (1.x) https://github.com/apache/storm/pull/1902
> 
> > STORM-2315 Storm kafka client does not commit offsets when ack is
> >> disabled
>  (1.x) https://github.com/apache/storm/pull/1891
> 
> > fix: KafkaSpout is blocked in AutoCommitMode
>  (master) https://github.com/apache/storm/pull/1863
> 
> 

Re: [DISCUSS] Release Storm 1.1.0

[P. Taylor Goetz]

I have a release candidate for 1.1.0 built, but I can’t stage the convenience 
binaries because they are now over the ASF subversion size limit of 200MB.

I’ll work with infra to figure out a solution.

-Taylor

> On Feb 16, 2017, at 9:46 AM, Jungtaek Lim  wrote:
> 
> Now all issues on the Storm 1.1.0 epic got resolved. :)
> 
> There're still some bugfix pull requests for storm-kafka-client, but
> releasing Storm 1.1.0 has been tremendously dragged (I initiated this at
> Sep. 2016) so unless they're critical, I'm +1 on starting release phase on
> Storm 1.1.0 ASAP.
> 
> - Jungtaek Lim
> 
> 2017년 2월 14일 (화) 오전 11:41, Harsha Chintalapani 님이 작성:
> 
> STORM-2340 is more of a feature . Auto-commit mode in storm-kafka used
> rarely and most users
> run the kafka spout with ackers and get at-least once guarantee.  If its
> going to longer to address the PR reviews
> I am +1 on moving this out of Storm 1.1.0. We already quite a few patches
> storm-kafka-client and 1.1.0 release brings in lot of improvements
> and bug-fixes.
> -Harsha
> 
> On Wed, Feb 8, 2017 at 6:15 PM Jungtaek Lim  wrote:
> 
>> There seems some pull requests for bugfix/improvement on
>> storm-kafka-client, and some authors in PRs are not availble for now.
>> (waiting 7 days)
>> 
>> If we plan to get 1.1.1 out soon (say 1 month later or even closer) we can
>> postpone, but if not, it might be better to coordinate these things ASAP
>> and include to 1.1.0.
>> 
>> There seems to be other small PRs, but nothing seems critical so it would
>> be OK to not wait for merging.
>> 
>> - Jungtaek Lim
>> 
>> 2017년 2월 9일 (목) 오전 6:48, P. Taylor Goetz 님이 작성:
>> 
>> Right now we’re down to 1 open issue on the 1.1.0 release epic: STORM-2250
>> which is under active review/discussion.
>> 
>> Assuming that is mergeable in the near future, are there any other open
>> issues that should be considered for this release?
>> 
>> -Taylor
>> 
>> 
>>> On Feb 2, 2017, at 4:48 PM, P. Taylor Goetz  wrote:
>>> 
>>> Thanks for putting this list together Jungtaek. I added a few to the 1.1
>> release epic that I think are important. Feel free to do the same.
>>> 
>>> Looks like we have a few to go, but there are pull requests for them.
>> It’s mostly just a matter of reviews and review responses, so I think we
>> are close.
>>> 
>>> -Taylor
>>> 
 On Feb 2, 2017, at 1:41 AM, Jungtaek Lim  wrote:
 
 Seems like there're not blockers for 1.1.0, but some pull requests are
 worth to check.
 There're pending pull requests for storm-kafka-client waited on
>> STORM-2225.
 Given that STORM-2225 is now merged, we might need to take a look at.
 
 *- reviewing*
 
 [storm-core]
 
> STORM-2324 : Fix deployment failure if resources directory is missing
>> in
 topology jar
 (master) https://github.com/apache/storm/pull/1908
 (1.x) https://github.com/apache/storm/pull/1898
 
> STORM-2321 Handle blobstore zk key deletion in KeySequenceNumber
 (master) https://github.com/apache/storm/pull/1904
 (1.x) https://github.com/apache/storm/pull/1905
 
 [storm-kafka]
 
> STORM-2270 Kafka spout should consume from latest when ZK partition
 commit offset bigger than the latest offset
 (1.x) https://github.com/apache/storm/pull/1851
 
 [storm-kafka-client]
 
> STORM-2281: Running Multiple Kafka Spouts (Trident) Throws Illegal
>> State
 Exception
 (1.x) https://github.com/apache/storm/pull/1902
 
> STORM-2315 Storm kafka client does not commit offsets when ack is
>> disabled
 (1.x) https://github.com/apache/storm/pull/1891
 
> fix: KafkaSpout is blocked in AutoCommitMode
 (master) https://github.com/apache/storm/pull/1863
 
> STORM-2250: Kafka Spout Refactoring to Increase Modularity and
>> Testability
 (master) https://github.com/apache/storm/pull/1832
 
> STORM-2014: Put logic around dropping messages into RetryService,
>> remove
 maxRetry setting from new KafkaSpout
 (master) https://github.com/apache/storm/pull/1605
 
> fix NullPointException with acked.get(rtp)
 (master) https://github.com/apache/storm/pull/1807
 
 [storm-sql]
 
> STORM-1443 [Storm SQL] Support customizing parallelism in StormSQL
 https://github.com/apache/storm/pull/1739
 
 *- pending*
 
 [storm-kafka-client]
 
> STORM-2296 Kafka spout no dup on leader changes
 (1.0.x) https://github.com/apache/storm/pull/1873
 (1.x) https://github.com/apache/storm/pull/1888
 
 [storm-sql]
 
> STORM-2148 [Storm SQL] Trident mode: back to code generate and compile
 Trident topology
 https://github.com/apache/storm/pull/1743
 
 Thanks,
 Jungtaek Lim (HeartSaVioR)
 
 2017년 2월 2일 (목) 오전 8:14, Harsha Chintalapani 님이 작성:
 
> Trying to check the status 

Re: storm stops recieving events suddenly

[S G]

Hi Ganesh,

You can try enabling
http://storm.apache.org/releases/1.0.0/Eventlogging.html if you are using
1.x version of storm.
You can also try using
http://storm.apache.org/releases/1.0.0/dynamic-log-level-settings.html to
debug the solr-bolt

-SG


On Thu, Feb 16, 2017 at 1:22 AM, Ganesh Kumar J 
wrote:

> Hi,
>
>  We are facing problem in apache storm frequently after few hours of
> continuous message receive. we are using architecture like Kafka---> storm
> --> solr.
>
> Storm continuously received message from Kafka but after 1 day of
> continuousness receiving message storm suddenly stops receiving.
>
> when we check the log in the worker.log file we don't find any useful
> information there. we are getting frequency below trace after stops
> receiving the message from Kafka.
>
>
> 2017-02-15 18:00:47.500 o.a.s.k.ZkCoordinator [INFO] Task [12/12]
> Refreshing partition manager connections
> 2017-02-15 18:00:47.502 o.a.s.k.ZkCoordinator [INFO] Task [6/12]
> Refreshing partition manager connections
> 2017-02-15 18:00:47.502 o.a.s.k.ZkCoordinator [INFO] Task [3/12]
> Refreshing partition manager connections
> 2017-02-15 18:00:47.504 o.a.s.k.ZkCoordinator [INFO] Task [6/12]
> Refreshing partition manager connections
> 2017-02-15 18:00:47.508 o.a.s.k.DynamicBrokersReader [INFO] Read partition
> info from zookeeper:
>
> Please help us.
>
>
> Thanks & Regards,
> J.Ganesh Kumar.
>
> Le informazioni contenute in questo messaggio di posta elettronica sono
> riservate, rivolte esclusivamente al destinatario e non comportano alcun
> vincolo ne' creano obblighi per la societa' mittente, salvo cio' che non
> sia espressamente previsto da un precedente accordo.
> Ogni altra persona diversa dal destinatario non puo' copiare o consegnare
> il presente messaggio o parte dello stesso a terzi ne' trattare in alcun
> modo i dati contenuti.
> La informiamo che l' utilizzo non autorizzato del messaggio o dei suoi
> allegati potrebbe costituire reato.
> Grazie per la collaborazione.
>
>
> The information in this e-mail is confidential only addressed to the
> addressee and it is neither binding on this Company nor to be understood as
> creating any obligation on its part except where provided for an agreement.
> Anyone different from the addressee (who received this e-mail) cannot copy
> or deliver the message or part of it to a third party, nor deal in any way
> with the information in this e-mail. Furthermore, the unauthorized use of
> this message or any attachment could be an offence.
> Thank you for your cooperation.
>

Re: Cassandra spout

[Hugo Da Cruz Louro]

There is a JIRA filed for this: https://issues.apache.org/jira/browse/STORM-2362

SG, can you please add any requirements that you may have in mind to the JIRA.

Thanks,
Hugo

On Feb 16, 2017, at 8:48 AM, Jungtaek Lim 
> wrote:

For me, I'm OK to adopt the Spout implementation if it can be used for
general purpose, and also can meet at-least-once semantic. I didn't add
Redis spout (using pubsub or list) because it can't meet at-least-once
semantic.

Do you have any sketches or design doc for Cassandra spout?

Thanks,
Jungtaek Lim (HeartSaVioR)

2017년 2월 16일 (목) 오전 3:22, S G 
>님이 작성:

Hi,

We need Cassandra spout to read from Cassandra and write to other systems.

But I could not find any such thing documented at:
http://storm.apache.org/releases/1.0.3/storm-cassandra.html

Is there a reason why there exists only Cassandra Bolt?
Would Cassandra Spout be a welcome addition to the codebase?

Thanks
SG

Re: Cassandra spout

[Jungtaek Lim]

For me, I'm OK to adopt the Spout implementation if it can be used for
general purpose, and also can meet at-least-once semantic. I didn't add
Redis spout (using pubsub or list) because it can't meet at-least-once
semantic.

Do you have any sketches or design doc for Cassandra spout?

Thanks,
Jungtaek Lim (HeartSaVioR)

2017년 2월 16일 (목) 오전 3:22, S G 님이 작성:

> Hi,
>
> We need Cassandra spout to read from Cassandra and write to other systems.
>
> But I could not find any such thing documented at:
> http://storm.apache.org/releases/1.0.3/storm-cassandra.html
>
> Is there a reason why there exists only Cassandra Bolt?
> Would Cassandra Spout be a welcome addition to the codebase?
>
> Thanks
> SG
>

storm stops recieving events suddenly

[Ganesh Kumar J]

Hi,

 We are facing problem in apache storm frequently after few hours of 
continuous message receive. we are using architecture like Kafka---> storm --> 
solr.

Storm continuously received message from Kafka but after 1 day of 
continuousness receiving message storm suddenly stops receiving.

when we check the log in the worker.log file we don't find any useful 
information there. we are getting frequency below trace after stops receiving 
the message from Kafka.


2017-02-15 18:00:47.500 o.a.s.k.ZkCoordinator [INFO] Task [12/12] Refreshing 
partition manager connections
2017-02-15 18:00:47.502 o.a.s.k.ZkCoordinator [INFO] Task [6/12] Refreshing 
partition manager connections
2017-02-15 18:00:47.502 o.a.s.k.ZkCoordinator [INFO] Task [3/12] Refreshing 
partition manager connections
2017-02-15 18:00:47.504 o.a.s.k.ZkCoordinator [INFO] Task [6/12] Refreshing 
partition manager connections
2017-02-15 18:00:47.508 o.a.s.k.DynamicBrokersReader [INFO] Read partition info 
from zookeeper:

Please help us.


Thanks & Regards,
J.Ganesh Kumar.

Le informazioni contenute in questo messaggio di posta elettronica sono 
riservate, rivolte esclusivamente al destinatario e non comportano alcun 
vincolo ne' creano obblighi per la societa' mittente, salvo cio' che non sia 
espressamente previsto da un precedente accordo.
Ogni altra persona diversa dal destinatario non puo' copiare o consegnare il 
presente messaggio o parte dello stesso a terzi ne' trattare in alcun modo i 
dati contenuti.
La informiamo che l' utilizzo non autorizzato del messaggio o dei suoi allegati 
potrebbe costituire reato.
Grazie per la collaborazione.


The information in this e-mail is confidential only addressed to the addressee 
and it is neither binding on this Company nor to be understood as creating any 
obligation on its part except where provided for an agreement.
Anyone different from the addressee (who received this e-mail) cannot copy or 
deliver the message or part of it to a third party, nor deal in any way with 
the information in this e-mail. Furthermore, the unauthorized use of this 
message or any attachment could be an offence.
Thank you for your cooperation.

[GitHub] storm issue #1939: STORM-1363: TridentKafkaState should handle null values f...

[pasalkarsachin1]

Github user pasalkarsachin1 commented on the issue:

https://github.com/apache/storm/pull/1939
  
@HeartSaVioR  I did my best. Can you take a look at it now


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm issue #1924: STORM-2343: New Kafka spout can stop emitting tuples if m...

[hmcl]

Github user hmcl commented on the issue:

https://github.com/apache/storm/pull/1924
  
@srdo Thanks for the note. I will take a look later today.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm issue #1939: STORM-1363: TridentKafkaState should handle null values f...

[HeartSaVioR]

Github user HeartSaVioR commented on the issue:

https://github.com/apache/storm/pull/1939
  
@pasalkarsachin1 
Indentation is still off. Could you please check the diff on PR?
https://github.com/apache/storm/pull/1939/files


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm issue #1944: [STORM-2365] Support for specifying output stream in even...

[HeartSaVioR]

Github user HeartSaVioR commented on the issue:

https://github.com/apache/storm/pull/1944
  
+1


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: [DISCUSS] Release Storm 1.1.0

[Jungtaek Lim]

Now all issues on the Storm 1.1.0 epic got resolved. :)

There're still some bugfix pull requests for storm-kafka-client, but
releasing Storm 1.1.0 has been tremendously dragged (I initiated this at
Sep. 2016) so unless they're critical, I'm +1 on starting release phase on
Storm 1.1.0 ASAP.

- Jungtaek Lim

2017년 2월 14일 (화) 오전 11:41, Harsha Chintalapani 님이 작성:

STORM-2340 is more of a feature . Auto-commit mode in storm-kafka used
rarely and most users
run the kafka spout with ackers and get at-least once guarantee.  If its
going to longer to address the PR reviews
I am +1 on moving this out of Storm 1.1.0. We already quite a few patches
storm-kafka-client and 1.1.0 release brings in lot of improvements
and bug-fixes.
-Harsha

On Wed, Feb 8, 2017 at 6:15 PM Jungtaek Lim  wrote:

> There seems some pull requests for bugfix/improvement on
> storm-kafka-client, and some authors in PRs are not availble for now.
> (waiting 7 days)
>
> If we plan to get 1.1.1 out soon (say 1 month later or even closer) we can
> postpone, but if not, it might be better to coordinate these things ASAP
> and include to 1.1.0.
>
> There seems to be other small PRs, but nothing seems critical so it would
> be OK to not wait for merging.
>
> - Jungtaek Lim
>
> 2017년 2월 9일 (목) 오전 6:48, P. Taylor Goetz 님이 작성:
>
> Right now we’re down to 1 open issue on the 1.1.0 release epic: STORM-2250
> which is under active review/discussion.
>
> Assuming that is mergeable in the near future, are there any other open
> issues that should be considered for this release?
>
> -Taylor
>
>
> > On Feb 2, 2017, at 4:48 PM, P. Taylor Goetz  wrote:
> >
> > Thanks for putting this list together Jungtaek. I added a few to the 1.1
> release epic that I think are important. Feel free to do the same.
> >
> > Looks like we have a few to go, but there are pull requests for them.
> It’s mostly just a matter of reviews and review responses, so I think we
> are close.
> >
> > -Taylor
> >
> >> On Feb 2, 2017, at 1:41 AM, Jungtaek Lim  wrote:
> >>
> >> Seems like there're not blockers for 1.1.0, but some pull requests are
> >> worth to check.
> >> There're pending pull requests for storm-kafka-client waited on
> STORM-2225.
> >> Given that STORM-2225 is now merged, we might need to take a look at.
> >>
> >> *- reviewing*
> >>
> >> [storm-core]
> >>
> >>> STORM-2324 : Fix deployment failure if resources directory is missing
> in
> >> topology jar
> >> (master) https://github.com/apache/storm/pull/1908
> >> (1.x) https://github.com/apache/storm/pull/1898
> >>
> >>> STORM-2321 Handle blobstore zk key deletion in KeySequenceNumber
> >> (master) https://github.com/apache/storm/pull/1904
> >> (1.x) https://github.com/apache/storm/pull/1905
> >>
> >> [storm-kafka]
> >>
> >>> STORM-2270 Kafka spout should consume from latest when ZK partition
> >> commit offset bigger than the latest offset
> >> (1.x) https://github.com/apache/storm/pull/1851
> >>
> >> [storm-kafka-client]
> >>
> >>> STORM-2281: Running Multiple Kafka Spouts (Trident) Throws Illegal
> State
> >> Exception
> >> (1.x) https://github.com/apache/storm/pull/1902
> >>
> >>> STORM-2315 Storm kafka client does not commit offsets when ack is
> disabled
> >> (1.x) https://github.com/apache/storm/pull/1891
> >>
> >>> fix: KafkaSpout is blocked in AutoCommitMode
> >> (master) https://github.com/apache/storm/pull/1863
> >>
> >>> STORM-2250: Kafka Spout Refactoring to Increase Modularity and
> Testability
> >> (master) https://github.com/apache/storm/pull/1832
> >>
> >>> STORM-2014: Put logic around dropping messages into RetryService,
> remove
> >> maxRetry setting from new KafkaSpout
> >> (master) https://github.com/apache/storm/pull/1605
> >>
> >>> fix NullPointException with acked.get(rtp)
> >> (master) https://github.com/apache/storm/pull/1807
> >>
> >> [storm-sql]
> >>
> >>> STORM-1443 [Storm SQL] Support customizing parallelism in StormSQL
> >> https://github.com/apache/storm/pull/1739
> >>
> >> *- pending*
> >>
> >> [storm-kafka-client]
> >>
> >>> STORM-2296 Kafka spout no dup on leader changes
> >> (1.0.x) https://github.com/apache/storm/pull/1873
> >> (1.x) https://github.com/apache/storm/pull/1888
> >>
> >> [storm-sql]
> >>
> >>> STORM-2148 [Storm SQL] Trident mode: back to code generate and compile
> >> Trident topology
> >> https://github.com/apache/storm/pull/1743
> >>
> >> Thanks,
> >> Jungtaek Lim (HeartSaVioR)
> >>
> >> 2017년 2월 2일 (목) 오전 8:14, Harsha Chintalapani 님이 작성:
> >>
> >>> Trying to check the status on this release of 1.1.0. Are we going to
do
> >>> this release anytime soon?
> >>>
> >>>
> >>> On Fri, Jan 13, 2017 at 7:50 PM S G  wrote:
> >>>
>  Not sure if its a little late to include for the 1.1.0 and 1.0.3
> releases
>  now, but can we consider using zookeeper 3.4.9 for the future
versions
> as
>  3.4.9 brings in a lot of stability improvements (
> 

[GitHub] storm issue #1939: STORM-1363: TridentKafkaState should handle null values f...

[pasalkarsachin1]

Github user pasalkarsachin1 commented on the issue:

https://github.com/apache/storm/pull/1939
  
@HeartSaVioR  Done


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm issue #1939: STORM-1363: TridentKafkaState should handle null values f...

[HeartSaVioR]

Github user HeartSaVioR commented on the issue:

https://github.com/apache/storm/pull/1939
  
@pasalkarsachin1 
We don't use 8 spaces for indentation from other java files. If the 
indentation spaces was 4, it should be 4 unless we set brand new style guide. 
Code style is important as much as code change.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm issue #1938: STORM-2360: Storm-Hive: Thrift version mismatch with stor...

[HeartSaVioR]

Github user HeartSaVioR commented on the issue:

https://github.com/apache/storm/pull/1938
  
+1


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm issue #1924: STORM-2343: New Kafka spout can stop emitting tuples if m...

[srdo]

Github user srdo commented on the issue:

https://github.com/apache/storm/pull/1924
  
@hmcl I think this is ready for another look when you get a chance.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm issue #1944: [STORM-2365] Support for specifying output stream in even...

[satishd]

Github user satishd commented on the issue:

https://github.com/apache/storm/pull/1944
  
+1 LGTM


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request #1944: [STORM-2365] Support for specifying output stream ...

[arunmahadevan]

GitHub user arunmahadevan opened a pull request:

https://github.com/apache/storm/pull/1944

[STORM-2365] Support for specifying output stream in event hubs spout



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/arunmahadevan/storm STORM-2365-master

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/storm/pull/1944.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1944


commit 83fe7805f36f2dd9c85619f7e7d5af9e90b6dd6e
Author: Arun Mahadevan 
Date:   2017-02-15T18:20:49Z

[STORM-2365] Support for specifying output stream in event hubs spout




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---