Re: Storm 2.0 Streaming API

Hi Satheesh, That’s because lambas are not serializable. An easy workaround is to do something like this. https://dzone.com/articles/how-and-why-to-serialialize-lambdas However, you should try to understand the reason why lambas are not serializable, and if doing this workaround won’t have any

Re: New Committer/PMC Member: Erik Weathers

Congrats & Welcome! > On Feb 22, 2018, at 2:45 PM, Jungtaek Lim wrote: > > Welcome Erik! Congrats! > > -Jungtaek Lim (HeartSaVioR) > > 2018년 2월 23일 (금) 오전 7:05, Stig Rohde Døssing 님이 작성: > >> Congratulations Erik. Happy to see you join. >> >>

Re: [VOTE] Release Apache Storm 1.2.1 (rc1)

+1 (binding) > On Feb 16, 2018, at 1:01 PM, P. Taylor Goetz wrote: > > This is a call to vote on releasing Apache Storm 1.2.1 (rc1) > > NOTE: The primary purpose of this release is to remove an LGPL-licensed > binary that was inadvertently included in the 1.2.0 binary

Re: [DISCUSS] consider EOL for version lines

+1 to maintain 3 version lines. Let’s properly announce that in our portal and users list such that users know what’s coming. Agree with focusing on 2.0 which has a lot of improvements, rather than 1.x, x >= 3. > On Feb 13, 2018, at 10:43 AM, Alexandre Vermeerbergen >

Re: [CANCELED] [VOTE] Release Apache Storm 1.2.0 (rc2)

Agree that release should include Netty upgrade (STORM-2918) > On Feb 1, 2018, at 3:39 AM, Satish Duggana wrote: > > -1 (binding) to include STORM-2918 > > > Thanks, > Satish. > > On Thu, Feb 1, 2018 at 2:25 PM,

Re: [VOTE] Release Apache Storm 1.1.2 (rc2)

Agree that release should include Netty upgrade (STORM-2918) > On Feb 1, 2018, at 3:39 AM, Satish Duggana wrote: > > -1 (binding) to include STORM-2918 > > > Thanks, > Satish. > > > On Thu, Feb 1, 2018 at 2:23 PM,

Re: [CANCELED] [VOTE] Release Apache Storm 1.0.6 (rc1)

ate their connectors >> to >>>>> bahir so it is also for Apache Flink. >>>>> I admit we may want to keep some connectors in core even we split out >>>>> connectors, since moving to another repo. makes connectors hard to be >>> in >>>&g

Re: [DISCUSS] Replace storm-kafka-client on 1.1.x-branch / 1.0.x-branch with 1.x-branch

+1 to replace storm-kafka-client in 1.0.x branch. Hugo > On Jan 31, 2018, at 11:03 AM, Stig Rohde Døssing > wrote: > > +1 to replace storm-kafka-client in 1.0.x branch. Breaking semantic > versioning is really nasty, but I think it is the lesser evil in this case. > >

Re: [DISCUSS] Decouple Storm core and connectors

I think that the bahir approach makes sense for connectors that don’t fall into the "first class support” category. I am in favor of moving such lower adoption connectors and have the interested communities support them with the most suitable release cycle. Connectors that are idle, such as

Re: [CANCELED] [VOTE] Release Apache Storm 1.0.6 (rc1)

I am in favor of not incubating storm-kafka-client but rather keep it as part of the storm project. We can consider supporting a separate branch, but before we agree to go that route I would like to hear lessons learned from community members that have been part of similar transitions in other

Re: [Discuss] Automating tracking of contributors

+1 > On Dec 22, 2017, at 7:35 PM, Jungtaek Lim wrote: > > +2017 (given that we may miss the year :) ) > > I think I've raised same discussion (exactly same approach: Spark solution) > for several times. I'd like to automate as many as possible, instead of > requiring us to

[Question] [storm-kafka-client] - RoundRobinManualPartitioner

Hi, can someone clarify what is the reasoning behind having totalTaskCount+1 in the following line of code? Is it possible for totalTaskCount to be zero?

Re: [DISCUSS] Drop standalone mode of Storm SQL

+1. It would just be good that once you move onto streams API this is easy to test. It is highly important that we can quickly run simple tests form the CLI and IDE. Without that it is very hard to fix bugs, test new features during early developments, and come up with ways to reproduce users

Re: [Discuss] Release Storm 1.2.0

testing, I can run any Storm 1.2.0 preview on my pre-production (well stressed) environment. Is there some URL from where I can download such "snapshot" binaries? Best regards, Alexandre Vermeerbergen PS: By the way, I have no interest in the metrics-related enhancements you mentionn

Re: [Discuss] Release Storm 1.2.0

ome binaries for testing, I can run any Storm 1.2.0 >>>> preview on my pre-production (well stressed) environment. Is there some >>> URL >>>> from where I can download such "snapshot" binaries? >>>> >>>> >>>> Best regar

Re: [Discuss] Release Storm 1.2.0

I am also in agreement that we should not delay the 1.2.0 release for too long, but in order to release it a few things need to be kept in mind: - If we want to avoid releasing 1.3.0 soon or at all, that means that we should include in 1.2.0 as many important features (e.g. metricsV2) as

Re: [DISCUSS] Release Storm 1.0.5 / 1.1.2

I am +1 to releasing 1.1.2 right away. I am in the middle of one review but I will finish it in the next day, such that we can get this merged soon. However, we need to hold onto releasing 1.2.0 until some of the changes for ProcessingGuarantee that got in this

Re: The #storm-user IRC channel

Hi, I believe that we never reached a consensus on this thread. Can we make a decision on this ? I have changed my position to NOT support a channel at all. The reason is because all the user/dev discussion should go through the email list, and it is very easy to reach out to a particular

Re: [Discuss] - Output format of CLI command 'storm list'

Hi everyone, No one replied to this thread. If no one opposes I am going to submit a patch with these changes. Thanks, Hugo > On Aug 13, 2017, at 5:39 PM, Hugo Da Cruz Louro <hlo...@hortonworks.com> > wrote: > > Currently the CLI command 'storm list’ presents the

Re: Question: ways to handle inactive pull requests

Agree. Question is what is the reasonable time frame for a response, and how many times one should reach out to a person asking for a response. > On Sep 14, 2017, at 8:43 AM, Stig Rohde Døssing > wrote: > > I agree, if the original author is not responding it seems

[Discuss] - Output format of CLI command 'storm list'

Currently the CLI command 'storm list’ presents the output as follows: Topology_nameStatus Num_tasks Num_workers Uptime_secs --- wct ACTIVE 28 32581 Is there any technical

Re: [Discussion]: Storm Improvemement Proposal (SIP) to discuss changes

I am +1 with having a bit more structure in the process (SIP), but being cautious about limiting the overhead. I like the SIP idea, but I agree with Bobby’s point of - "what if you missed SIP and don’t agree with something during PR phase". For that I suggest that SIP is an iterative process -

Re: [DISCUSS] Ideas for resolving storm-drpc-server compilation issue on IDE

I have been following this discussion thread as part of the storm-core-ui migration. I would like to bring up a couple of points: * The names of the packages "storm-client" and "storm-server" are a bit misleading to me. Isn’t what we really mean here "storm-workers" and "storm-daemons” ? Even

Re: [DISCUSS] Remove CHANGELOG file

I am +1 for discontinuing CHANGELOG. However, using JIRA to compile this info will only work if contributors are very disciplined and consistent updating JIRA. That leads to the question, is it any easier to maintain JIRA consistent then it is to keep CHANGELOG consistent? A clean and

Re: The #storm-user IRC channel

+1 removing links to IRC channel. > On Jul 20, 2017, at 9:27 AM, Bobby Evans wrote: > > I agree if it is not being use then lets remove it. > > > - Bobby > > > On Thursday, July 20, 2017, 9:57:57 AM CDT, Jungtaek Lim > wrote: > > I don't

[DISCUSS] [Storm-Kafka] - Maintenance, Branch Support, and Deprecation Plans for storm-kafka and storm-kaka-client

Hi, The goal of this email is to summarize and unify the discussion started across several email threads (Storm 2.0 Roadmap, 1.1.1 Release

Re: Release Planning for 1.1.1 and others?

Hi Jungtaek, Currently there are 2 critical bugs and no blocker bugs. https://issues.apache.org/jira/browse/STORM-2342- I still need to investigate it. It may or may not be a bug. It is a bit surprising that such basic functionality is not working.

Re: New Committer/PMC Member: Stig Rohde Døssing

Welcome Stig. Looking forward to collaborating with you. Hugo > On Jul 14, 2017, at 11:44 AM, P. Taylor Goetz wrote: > > Please join me in congratulating the latest Committer and PMC Member, Stig > Rohde Døssing. > > Stig has been very active contributing patches to

Re: Few observations related to KafkaSpout implementation (1.1.0)

@chandan #1 was designed to be single threaded. There are several reasons for that. 1.1 KafkaConsumer is single threaded. If you have multiple threads attempting to poll from it, you will get a ConcurrentModificationException("KafkaConsumer is not safe for multi-threaded access”); 1.2

Re: What is the difference between UNCOMMITTED_EARLIEST and UNCOMMITTED_LATEST ?

> > 2017-06-29 1:06 GMT+08:00 Hugo Da Cruz Louro <hlo...@hortonworks.com>: > >> Your KafkaConsumer instance (which boils down to your KafkaSpout) can be >> in one of two states: >> >> 1 - Has committed to Kafka >> Here, EARLIEST fetches from the

Re: [DISCUSS] Storm 2.0 Roadmap

the new one converges. > > Best regards, > Alexandre Vermeerbergen > > > 2017-06-28 19:40 GMT+02:00 P. Taylor Goetz <ptgo...@gmail.com>: > >> >>> On Jun 28, 2017, at 1:16 PM, Hugo Da Cruz Louro <hlo...@hortonworks.com> >> wrote:

Re: [DISCUSS] Storm 2.0 Roadmap

https://issues.apache.org/jira/browse/STORM-2153 On Mar 30, 2017, at 9:09 PM, Jungtaek Lim <kabh...@gmail.com<mailto:kabh...@gmail.com>> wrote: Maybe related to or not, but would we want to create a new branch "1.1.x-branch", and make "1.x-branch"

Re: What is the difference between UNCOMMITTED_EARLIEST and UNCOMMITTED_LATEST ?

Your KafkaConsumer instance (which boils down to your KafkaSpout) can be in one of two states: 1 - Has committed to Kafka Here, EARLIEST fetches from the first offset. LATEST fetches from the last offset. UNCOMMITTED_EARLIEST and UNCOMMITTED_LATEST will fetch from the last committed offset -

Re: Use of the KafkaConsumer.subscribe API in the storm-kafka-client spout

pattern since builder.build() is supposed to create > the object. I personally don’t think immutability of KafkaSpoutConfig is that > important here. I would rather get rid of the builder and have it with one or > two constructors with some setXXX methods. Let me know what you guys thin

Re: Use of the KafkaConsumer.subscribe API in the storm-kafka-client spout

like YARN, Mesos, or AWS facilities, or as part of a stream processing framework). In this case there is no need for Kafka to detect the failure and reassign the partition since the consuming process will be restarted on another machine." Manual assignment is the right way to go. -Harsha On Jun 9, 201

Re: [Discussion]: Storm Improvemement Proposal (SIP) to discuss changes

+1 for KIP equivalent for Storm. We should probably call them something like Storm Feature/Improvement Proposals. Well documented features could help new contributors to more quickly as they would have the context to start that feature. Unless one knows the product well, it is hard to have

Re: Use of the KafkaConsumer.subscribe API in the storm-kafka-client spout

+1 for simplifying KafkaSpoutConfig. Too many constructors and too many methods.. I am not sure it’s justifiable to have any methods that simply set KafkaConsumer properties. All of these properties should just go in a Map, which is what KafkaConsumer receives, and what was

Re: [VOTE] Java Code Style Standard for Apache Storm.

My vote is in the following order: [1] Google Java Style Guide [2] Sun Java Code Conventions [3] HBase style [4] Hadoop style, which is described as java but with 2 spaces instead of 4 Thanks, Hugo > On Apr 26, 2017, at 12:03 PM, P. Taylor Goetz wrote: > > [1 ] Google Java

Re: [DISCUSS] Code Style

Hi, We should just follow the Google Java Style Guide. There are many opinions on this subject, but I conjecture that this guide, which is very similar to the original Sun Java Code

Re: Recording - Storm & Kafka Meetup on April 20th 2017

Hi, In the comments section of the Meetup page you can find links for all the slides. https://www.meetup.com/Apache-Storm-Apache-Kafka/events/238975416/ Thank you for your interest. Hugo On Apr 24, 2017, at 10:19 AM, Aditya Desai > wrote: Hi Harsha

[DISCUSS] - Reconcile Multiple SECURITY.md Files in Master and 1.x-Branch

Hi, I noticed that we have two SECURITY.md files with different contents in both Master and 1.x-branch. Master https://github.com/apache/storm/blob/master/SECURITY.md https://github.com/apache/storm/blob/master/docs/SECURITY.md 1.x-branch

Re: [DISCUSS] Storm 2.0 Roadmap

+1 for finishing the porting to Java ahead of anything else - it will be a significant milestone. I have a JIRA assigned concerning to the porting. I will work on it for the 2.0 release. it’s a priority to guarantee no performance regressions. As part of this endeavor, explore an automated (or

Re: [DISCUSS] Move non-connectors modules to out of external

tory, so we need to be careful about what the > destination for those components is. > > -Taylor > >> On Mar 24, 2017, at 3:11 PM, Hugo Da Cruz Louro <hlo...@hortonworks.com> >> wrote: >> >> +1 non-connectors to top level >> +1 to renaming external t

Re: [DISCUSS] Move non-connectors modules to out of external

+1 non-connectors to top level +1 to renaming external to connectors As for storm-kaka, if we are already touching the external modules, all the modules should be a submodule of a parent module called storm-kafka. I don’t think we should have 3 parent modules as we currently have (storm-kafka,

Re: [VOTE] Release Apache Storm 1.1.0 (RC3)

+1 (binding) - unzipped .zip sources and binaries - mvn clean install in unzipped sources - created uber jar for storm-solr and storm-kafka-client - tested storm-solr and storm-kafka-client in local cluster mode and remode cluster mode The issue

Re: [VOTE] Release Apache Storm 1.1.0 (RC3)

Since this is an external module connector perhaps we can get this fix onto a minor release instead of being a blocker. I will update the JIRA accordingly. > On Mar 23, 2017, at 8:58 AM, Hugo Da Cruz Louro <hlo...@hortonworks.com> > wrote: > > -1 (non binding) > > T

Re: [VOTE] Release Apache Storm 1.1.0 (RC3)

-1 (non binding) This blocker bug was just reported - https://issues.apache.org/jira/browse/STORM-2432 Here is the fix: https://github.com/apache/storm/pull/2027 I consider that this is a blocker bug because if the user choses UNCOMITTED_LATEST first poll strategy, no data gets polled at all.

Re: [CANCELED] [VOTE] Release Apache Storm 1.1.0 (RC2)

Roshan, does this PR and JIRA address the missing jars problem that you mentioned. I had created it in December 2016, but there is an ongoing discussion if we should indeed put the jars in the examples

Re: [VOTE] Release Apache Storm 1.1.0 (RC1)

t; > On Wednesday, February 22, 2017, 11:56:56 AM CST, P. Taylor Goetz > <ptgo...@gmail.com> wrote:What are other’s thoughts on including the patch > Hugo mentioned below? Cut another RC, or proceed with the current one? > > -Taylor > >> On Feb 22, 2017, at 12:41 PM,

Re: [VOTE] Release Apache Storm 1.1.0 (RC1)

+1 (non binding) I found out that the Storm Kafka Client Test Topologies were throwing SerializationException. I have submitted two patches for master and 1.x-branch . It would be good if we could merge

Re: Cassandra spout

There is a JIRA filed for this: https://issues.apache.org/jira/browse/STORM-2362 SG, can you please add any requirements that you may have in mind to the JIRA. Thanks, Hugo On Feb 16, 2017, at 8:48 AM, Jungtaek Lim > wrote: For me, I'm OK to adopt

Re: [VOTE] Release Apache Storm 1.0.3 (rc1)

+1 (non binding) - unzipped .zip sources and binaries - mvn clean install in unzipped sources - created uber jar for storm-solr and storm-kafka-client - tested storm-solr and storm-kafka-client in local cluster mode and remode cluster mode > On Feb 3, 2017, at 1:17 AM, Satish Duggana

Re: [DISCUSS] Adopt pull request template

Squashing guidelines, we've gone through this discussion and I > think this should be left to author discretion as they would understand > what commits are more important. > Also as reviewers one can suggest to squash the commits if additional > commits doesn't make sense. > > Thanks, &

Re: [DISCUSS] Adopt pull request template

I am strongly +1. In addition to the templates suggested by Jungtaek, in the past I suggested the template bellow to a different project. It is a bit simpler than the ones listed, as I think it may be too demanding to ask people to categorize in the git message the type of contribution the

Re: New Kafka spout doesn't support seek to given offset

Hi Sachin, The 2nd case can likely handled with the committed offset, which is covered by UNCOMMITTED_EARLIEST or UNCOMMITTED_LATEST. The 1st case it may make sense but even if you give the start offset, since Kafka polls a certain number of bytes, and not specifically a number of records, it

Re: [DISCUSS] Prioritizing works in progress

Lambdas can also lead to clean code reuse. It is part of evolving to adopt new JDK functionalities, and I think that we should do it. It is expected that contributors starting with a new technology/paradigm may not always implement code that follows perfect practices. However, we should look

Re: Integration between python and java in storm

Sam, You can take a look at the WordCountTopology example. The split sentence logic is in Python. This should be a good reference. Hugo On Dec 16, 2016, at 3:35 PM, sam

Request to create `storm-pmml` JIRA Component field

Hi, Can someone with permissions to create Components on JIRA please do so. It would be helpful to tag the JIRAs associated with PMML. Thanks, Hugo

Re: [BROKEN BUILD] - FIXED with Latest JDK 8

;> wrote: My guess is this bug: http://bugs.java.com/bugdatabase/view_bug.do?bug_id=8080645 <http://bugs.java.com/bugdatabase/view_bug.do?bug_id=8080645> I’m on Java(TM) SE Runtime Environment (build 1.8.0_112-b16) and it compiles for me. -Taylor On Dec 2, 2016, at 2:07

Re: [BROKEN BUILD] - Master not compiling for me

ch just fine. What JDK version are you using? -Taylor On Dec 2, 2016, at 1:54 PM, Hugo Da Cruz Louro <hlo...@hortonworks.com<mailto:hlo...@hortonworks.com>> wrote: [ERROR] /Users/hlouro/Apache/Dev/GitHub/hmcl/storm-apache/storm-core/src/jvm/org/apache/storm/daemon/nimbus/Nimbus.

[BROKEN BUILD] - Master not compiling for me

[ERROR] /Users/hlouro/Apache/Dev/GitHub/hmcl/storm-apache/storm-core/src/jvm/org/apache/storm/daemon/nimbus/Nimbus.java:[2297,23] incompatible types: inferred type does not conform to upper bound(s) inferred: java.util.Map upper bound(s):

Re: [ANNOUNCE] New Storm Commiter/PMC Member: Xin Wang

Congratulations Xing! Keep up the good work. Best, Hugo > On Dec 1, 2016, at 6:17 AM, Bobby Evans wrote: > > Congratulations. > - Bobby > > On Thursday, December 1, 2016, 4:23:16 AM CST, Satish Duggana > wrote:Congratulations Xin!! > >

Re: [DISCUSS] Feature Branch for Apache Beam Runner

I somehow missed this email … I would like to contribute to this effort as well. Please keep me posted. Thanks. > On Oct 19, 2016, at 8:51 AM, Satish Duggana wrote: > > +1, waiting for that. :) > Currently,there are API changes going on in Beam. It seem they plan to

Request write access to Storm's wiki page on Confluent

Hi, I would like to request write access to the wiki page on Confluent. I am currently registered but I don't have writing access. I have a couple of benchmarks and performance analysis that I would like to share with the

Re: New Storm Commiter/PMC Member: Satish Duggana

Congratulations Satish. > On Aug 10, 2016, at 6:31 AM, Bobby Evans wrote: > > Congratulations > - Bobby > >On Tuesday, August 9, 2016 5:05 PM, Jungtaek Lim wrote: > > > Congrats Satish! > > Jungtaek Lim (HeartSaVioR) > > On

Re: STORM JIRA being spammed - CANNOT comment on JIRA (as a result of the counter spam measures)

ng the commiters ACL very well either. >> Are there any commiter/PMC out there that want to be added to the >> committer/PMC roles. There really isn't much different between the >> roles, but I am happy to adjust it if needed. >> - Bobby >> >> On Thursday, May 12,

STORM JIRA being spammed - CANNOT comment on JIRA (as a result of the counter spam measures)

Hi, I understand that there is a wave of spam going on, but the measure that was taken, which limits active contributors to comment on JIRA is not very practical. I have several open JIRA issues that I am working on, or that I am interested in contributing to, that I cannot comment on right

Re: [DISCUSSION] Version lines of 1.x

+1 (non-binding) or maintaining 1.x branch as it is the base for all the package name change and lots of new features. If we have to discontinue a branch, I would also favor discontinuing 0.10.x . Hugo > On May 9, 2016, at 8:06 AM, Bobby Evans wrote: > > +1 for

Re: [VOTE] Release Apache Storm 0.10.1 (rc2)

+1 (non binding) - Found a very minor bug in the storm-solr test topology, for which there is a very simple workaround and by NO means a blocker for the release. - unzipped .zip sources and binaries - mvn clean install in unzipped sources - created uber jar for storm-solr - tested storm-solr in

Re: [VOTE] Release Apache Storm 1.0.1 (rc3)

+1 (non binding) - Found a very minor bug in the storm-solr test topology, for which there is a very simple workaround and by NO means a blocker for the release. - unzipped .zip sources and binaries - mvn clean install in unzipped sources - created uber jar for storm-solr and storm-kafka-client -

Re: [Discussion] rolling upgrade of storm topologies with 1.0 release

offsets are stored outside of > storm.zookeeper.root so we can document saying clear the > storm.zookeeper.root node and not touch any other nodes in > the zookeeper. > > -Harsha > > On Fri, Apr 15, 2016, at 11:17 AM, Hugo Da Cruz Louro wr

Re: [Discussion] rolling upgrade of storm topologies with 1.0 release

Agree with Jungtaek. With the new KafkaSpout we should not have to worry about preserving offsets, as it’s done by Kafka. We should encourage our customers to move onto the new API. I am going to add more documentation to facilitate the adoption of the new Kafka Spout. Hugo > On Apr 14,

Re: [RESULT] [VOTE] Release Apache Storm 1.0.0 (rc2)

+1 for rapid releases as well… even if only to add more examples, documentation, and more testing to new features. > On Apr 13, 2016, at 10:38 PM, Zhuo Liu wrote: > > Agree with Abhishek. > +1 for rapid releases (btw, STORM-1696's fixes for both 1.x and master

Re: [DISCUSS] Java REST Framework adoption

I also vote on DropWizard > On Feb 23, 2016, at 2:29 PM, Parth Brahmbhatt > wrote: > > +1 on DropWizard. > > On 2/23/16, 2:02 PM, "Harsha" wrote: > >> -1 on spring boot or anything related to spring. >> This api is intended to be very simple

[ARTICLE] - Benchmarking Streaming Computation Engines at Yahoo!

Interesting article from Bobby Evans, Apache Storm committer. http://revans2.tumblr.com/post/135322597930/benchmarking-streaming-computation-engines-at

Re: Welcome new Apache Storm Committers/PMC Members

Congrats everyone. Looking forward to working with you! > On Dec 8, 2015, at 2:54 PM, Aaron.Dossett wrote: > > Thanks, everyone, and congratulations to the other new committers as well! > > On 12/8/15, 4:12 PM, "임정택" wrote: > >> Congratulation!

Re: [DISCUSS] 1.0 Release (was Re: [DISCUSS] Initial 0.11.0 Release)

+1 as well. I support moving to the org.apache.storm package as early as possible and I am OK with storm-compat. My only concern with using storm-compat is if we are going to have to support it forever, or plan on dropping it after a certain release. Backwards compatibility is a valid concern

Re: [Discusson] Storm System Tests

I a agree with these three levels of testing, and that at the very least we should keep unit tests separated from the system/integration tests. One huge advantage would be to quickly run unit tests that hopefully are more predictable, and thus avoid intermittent test fails. Bobby just

Re: [Discusson] Storm System Tests

e: Hugo, I would love to see that happen. It has been on my list for a while, but I have never found the time to do it. I personally am +1 on this, hopefully we can do this quickly in preparation for a 0.11.0 release. - Bobby On Thursday, November 5, 2015 1:37 PM, Hugo Da Cruz Louro <h