Re: [VOTE] Java Code Style Standard for Apache Storm.

[Kyle Nusbaum]

[1] Google Java Style Guide

[2] Sun Java Code 
Conventions

[3] HBase 
style
 (sorry all I could find was an eclipse XML config)

[4] Hadoop style which is 
described as java but with 2 spaces instead of 4.
Thanks,
-- Kyle

On Thursday, April 27, 2017, 3:51:25 AM CDT, Julien Nioche 
 wrote:Non-binding : [1 ] Google Java Style Guide

On 26 April 2017 at 19:50, Bobby Evans  wrote:

> We would like to adopt a code style standard for Apache Storm.  Please
> rank the following with 1 being the most desired and 4 being the least
> desired (5 if you have a write in choice).  This is not an official vote as
> per the ByLaws, but we will probably go with whichever wins the most 1
> votes (unless it is really close and then we may go with some STeVe like
> ranking, but I want to avoid it because it is hard and I might get the math
> wrong) This is open to everyone so please vote.
>
>
> [ ] Google Java Style Guide io/styleguide/javaguide.html>
>
> [ ] Sun Java Code Conventions codeconventions-150003.pdf>
>
> [ ] HBase style support/hbase_eclipse_formatter.xml> (sorry all I could find was an
> eclipse XML config)
>
> [ ] Hadoop style
> which is described as java but with 2 spaces instead of 4.
>
> [ ] Other (Please specify)
>
>
> I apologize if the formatting is bad I am forced to use a corporate
> sponsored mail client that somehow messed up plain text formatting.  I
> don't know how.
>
>
> - Bobby




-- 

*Open Source Solutions for Text Engineering*

http://www.digitalpebble.com
http://digitalpebble.blogspot.com/
#digitalpebble 

Re: [DISCUSS] Code Style

[Kyle Nusbaum]

+1 for what Hugo, Taylor and Bobby said.


Thanks,
-- Kyle

On Wednesday, April 26, 2017, 9:45:08 AM CDT, Bobby Evans 
<ev...@yahoo-inc.com.INVALID> wrote:I really don't care what style we use so 
long as we have one and it is enforced.  Checkstyle is highly configurable so 
just about any style we pick we can enforce with it.  Most Java IDEs that I 
know of either use the checkstyle format itself or at least support a 
checkstyle plugin to let you know if you are off.
I have my own opinions about this but to avoid too much further discussion (aka 
religious war) I would propose that we have a simple vote on what standard 
people what.  So far I have heard
Google Java Style Guide<https://google.github.io/styleguide/javaguide.html>Sun 
Java Code 
Conventions<http://www.oracle.com/technetwork/java/codeconventions-150003.pdf> 
HBase 
style<https://github.com/apache/hbase/blob/master/dev-support/hbase_eclipse_formatter.xml>
 (sorry all I could find was an eclipse XML config)Hadoop 
style<https://wiki.apache.org/hadoop/CodeReviewChecklist> which is described as 
java but with 2 spaces instead of 4.

If no one has any objections I will send out a new thread to vote on this.  I 
will probably also include a write in option too.


- Bobby

On Wednesday, April 26, 2017, 2:23:46 AM CDT, Satish Duggana 
<satish.dugg...@gmail.com> wrote:+1 to what Hugo and Taylor said. Lets go with 
google java style guide for
now and modify any of those rules if it is really required.

On Wed, Apr 26, 2017 at 5:23 AM, P. Taylor Goetz <ptgo...@gmail.com> wrote:

> Code convention discussions can get pretty heated/religious in nature,
> which we should seek to avoid. I'm largely in alignment with Hugo on this:
> Start with a good base like the Google or Sun/Oracle style guides, and
> establish consensus on where we feel a desire to differ.
>
> Rather than haggle over certain individual conventions, I propose we
> choose an existing style guide to base ours on and tweak it as we see fit.
> I have definite opinions about java code style, but I'll hold off on
> sharing them for now.
>
> Let's pick a starting point first. I believe the google and sun/oracle
> guides are both good starting points. Let's also try not to drag this out
> too much. I've seen similar conversations absorb more time and effort than
> it would take to update the associated codebase.
>
> -Taylor
>
> > On Apr 25, 2017, at 7:13 PM, Hugo Da Cruz Louro <hlo...@hortonworks.com>
> wrote:
> >
> > Hi,
> >
> > We should just follow the Google Java Style Guide<https://google.github.
> io/styleguide/javaguide.html>. There are many opinions on this subject,
> but I conjecture that this guide, which is very similar to the original Sun
> Java Code Conventions<http://www.oracle.com/technetwork/java/
> codeconventions-150003.pdf>  has a lot of research and reasoning behind
> it. It’s widely adopted, and I think that we should just follow it. The
> only exception I would really make is the column limit. I would change 100
> to 120 or 130, but not longer than that. Very long lines makes the code
> really hard to read.
> >
> > There are already existing check style, IntelliJ, and even Git hooks
> enforcing/validating this style, so adopting it would make it easier to
> incorporate it in the dev tools.
> >
> > Regardless of the variants to the aforementioned style, I do not think
> that we should use names starting with underscore _ in any circumstance.
> Not only it is not a Java convention, as it does not add anything to the
> code. The reason behind using _ was before the widespread uses of IDE’s, to
> mark class fields. IDEs do that with smart coloring.
> >
> > Thanks,
> > Hugo
> >
> > On Apr 25, 2017, at 3:50 PM, Jungtaek Lim <kabh...@gmail.com<mailto:kabh
> w...@gmail.com>> wrote:
> >
> > I would like to review the style guide of other projects, like HBase, and
> > so.
> >
> > Btw, IMHO, I don't like using underscore as prefix for fields. We're
> using
> > Java and the expression "class member variable" is from C++, and also
> > underscore style came from C++. We need to avoid mixing other languages'
> > style as well.
> >
> > Thanks,
> > Jungtaek Lim (HeartSaVioR)
> >
> > 2017년 4월 26일 (수) 오전 7:10, Kyle Nusbaum <knusb...@yahoo-inc.com.
> invalid<mailto:knusb...@yahoo-inc.com.invalid>>님이
> > 작성:
> >
> > Now that most of our code is in Java, I think it might be time to revisit
> > the issue of having some official and enforced code style.
> > I don't have very strong feelings about most of it, but here is what I
> was
> > thinking as a start, since I've seen this style quite a bit, a

[DISCUSS] Code Style

[Kyle Nusbaum]

Now that most of our code is in Java, I think it might be time to revisit the 
issue of having some official and enforced code style.
I don't have very strong feelings about most of it, but here is what I was 
thinking as a start, since I've seen this style quite a bit, and I've found it 
makes code pretty easy to read:
1. Indentation is 4 spaces per level (no tabs)
2. All class member variables begin with underscore3. No wildcard imports4. if 
/ else / for / etc. always get curly braces.
There are obviously tons of other things, and we can get as picky as we want, 
but I think enforcing at least these rules would go a long way to making the 
code more consistent.

Thoughts?

Thanks,
-- Kyle

Re: [DISCUSS] breaking changes in 2.x

[Kyle Nusbaum]

On Wednesday, November 9, 2016, 7:23:09 AM CST, Harsha Chintalapani 
 wrote:> If we want users to upgrade to new version, the 
rolling upgrade is a major
> decision factor. As a community, we need to look API updates or breaking
> changes much more diligently.
Within a major version, I agree. APIs should be as stable as possible within a 
version release.

> I agree to an extent we shouldn't limiting ourselves with rolling upgrade.
> But having announced rolling-upgrade in 0.10 and then not supporting it in
> 1.x and now in 2.x. In User's point of view, Storm is not rolling
> upgradable although we shipped a release stating that rolling upgrade is
> supported and in follow-up release we taken that off.
The user would be correct. Storm would not be rolling-upgradable *between major 
versions.*I don't see how it's possible to develop and improve a project if it 
must remain perpetually backwards compatible, so I think it's necessary to 
reject compatibility as a *primary* goal.
Eventually (hopefully) we'll arrive at an API that we're happy with that we 
don't feel like we need to change.Then we can claim rolling upgrades across 
major version numbers.

> Does these API changes are critical and worth breaking rolling upgrade?
My position is that we don't want to limit ourselves to "critical" API changes. 
This will stick us with an inferior API that we can't evolve.It's accepting the 
long-term pain of inconsistent API or old baggage to avoid the short-term pain 
of relaunching or updating topologies when you do a major version upgrade.
Storm is not at the place in its life where it has stopped evolving, and I 
don't want to stifle its development.

Re: [DISCUSS] breaking changes in 2.x

[Kyle Nusbaum]

I worry that making it a priority to have rolling upgrades between major 
versions significantly restricts the kinds of changes that we can make, 
including some kinds of changes that a major version increment is supposed to 
mark. I'm not really in support of trying to do that.
If we can't make changes that break compatibility now, when can we make them? 
Can changes like that ever be made? I don't know that it's good to limit 
ourselves like that.
Trying to accommodate rolling upgrades is a good idea, but I'm not sure about 
rejecting code changes across major versions to support them. 2.x represents a 
large shift in the project, and I expect once the translation, etc. is done, 
things will calm down and APIs will become more stable, allowing more of our 
future releases to be rolling even across major versions. I'd rather get these 
kinds of changes out of the way now in the 2.x release than cart along the 
vestiges of 1.x from now on.
What do others think about this?
-- Kyle

On Monday, November 7, 2016, 3:10:08 AM CST, Bobby Evans 
 wrote:Let's distinguish between wire 
compatibility changes and API compatibility changes, along with impact to 
workers vs impact to clients.
For 3) splitting the classpath up for each daemon wire compatibility is not 
impacted, but we are potentially removing a bunch of APIs from the worker and 
client classpath.  Most of these where shaded and users should not be impacted 
by them being removed, but a few like servlet-api-2.5.jar are likely to be 
removed.  So yes the impact here would likely be very small.
However on the client side if a topology wants to include LocalCluster (like we 
do in a lot of examples) the topology jar will get a lot bigger.  LocalCluster 
needs access to nimbus, supervisor, and drpc server.  These would not be on the 
worker classpath any more and would then need to be packaged into the topology 
jar to make LocalCluster work.  In production I would expect LocalCluster to be 
used by tests and not be included like we do in a lot of examples.  This is 
more of a shift for how we expect users to interact with the LocalCluster.
For 1) NodeInfo.port depending on how we do it, it could break wire 
compatibility and API compatibility (which is what I would prefer).  We could 
play some games to maintain compatibility, but for me it is enough of a pain 
that I am not sure it is worth it.  However this is not likely to impact 
workers because they don't use these APIs directly.  It might impact clients 
but only if they have custom code to profile their topologies.  IF they use the 
build in CLI/UI it would not be impacted.
For 2) nocamel would break API compatibility, but not wire compatibility. This 
is not likely to impact workers because like with 1 workers don't really 
interact with nimbus directly so it would not be a problem.  Old clients 
running with older versions of storm would continue to work, but any custom 
client code (think what gets run by storm jar) would need to be 
recompiled/modified to be able to run on against a storm-2.x client.

- Bobby

Re: [DISCUSS] Provision for dead-letter topic in storm

[Kyle Nusbaum]

It seems to me that this can be solved by allowing a user to attach some 
arbitrary data to a call to fail(), which is passed to the spout.
So there would be an override for fail in IOutputCollector which takes both the 
Tuple input and also some object to give to the spout. The spout's fail method 
would now accept an object as a second argument.

The spout can then decide what to do about the failure based on the content of 
the object.

This makes it generic, possibly useful for other things like reporting, etc. I 
only looked at the relevant code briefly, but it looks like it would also be 
relatively simple to implement. -- Kyle 

On Tuesday, September 27, 2016 12:06 PM, Tech Id  
wrote:
 

 Any more thoughts on this?
Seems like a useful feature for all the spouts/bolts.

On Wed, Sep 21, 2016 at 9:09 AM, S G  wrote:

> Thank you Aaron.
>
> We use Kafka and JMS spouts and several bolts - Elastic-Search, Solr,
> Cassandra, Couchbase and HDFS in different scenarios and need to have the
> dead letter functionality for almost all these scenarios.
> Locally we have this functionality almost ready for writing dead-letters to
> Solr or Kafka.
> I will try to contribute the same to Storm as a PR and we can then look
> into adding the failing tuple as well. I agree adding the failing tuple
> would be somewhat more complicated.
>
>
> On Tue, Sep 20, 2016 at 4:34 PM, Aaron Niskodé-Dossett 
> wrote:
>
> > I like the idea, especially if it can be implemented as generically as
> > possible. Ideally we could "dead letter" both the original tuple and the
> > tuple that itself failed. Intervening transformations could have changed
> > the original tuple. I realize that's adds a lot of complexity to your
> idea
> > and may not be feasible.
> > On Tue, Sep 20, 2016 at 1:15 AM S G  wrote:
> >
> > > Hi,
> > >
> > > I want to gather some thoughts on a suggestion to provide a dead-letter
> > > functionality common to all spouts/bolts.
> > >
> > > Currently, if any spout / bolt reports a failure, it is retried by the
> > > spout.
> > > For a single bolt-failure in a large ADG, this retry logic can cause
> > > several perfectly successful component to replay and yet the Tuple
> could
> > > fail exactly at the same bolt on retry.
> > >
> > > This is fine usually (if the failure was temporary, say due to a
> network
> > > glitch) but sometimes, the message is bad enough such that it should
> not
> > be
> > > retried but at the same time important enough that its failure should
> not
> > > be ignored.
> > >
> > > Example: ElasticSearch-bolt receiving bytes from Kafka-Spout.
> > >
> > > Most of the times, it is able to deserialize the bytes correctly but
> > > sometimes a badly formatted message fails to deserialize. For such
> cases,
> > > neither Kafka-Spout should retry nor ES-bolt should report a success.
> It
> > > should however be reported to the user somehow that a badly serialized
> > > message entered the stream.
> > >
> > > For cases like temporary network glitch, the Tuple should be retried.
> > >
> > > So the proposal is to implement a dead-letter topic as:
> > >
> > > 1) Add a new method *failWithoutRetry(Tuple, Exception)* in the
> > collector.
> > > Bolts will begin using it once its available for use.
> > >
> > > 2) Provide the ability to *configure a dead-letter data-store in the
> > > spout* for
> > > failed messages reported by #1 above.
> > >
> > >
> > > The configurable data-store should support kafka, solr and redis to
> > > begin-with (Plus the option to implement one's own and dropping a jar
> > file
> > > in the classpath).
> > >
> > > Such a feature should benefit all the spouts as:
> > >
> > > 1) Topologies will not block replaying the same doomed-to-fail tuples.
> > > 2) Users can set alerts on dead-letters and find out easily actual
> > problems
> > > in their topologies rather than analyze all failed tuples only to find
> > that
> > > they failed because of a temporary network glitch.
> > > 3) Since the entire Tuple is put into the dead-letter, all the data is
> > > available for retrying after fixing the topology code.
> > >
> > > Please share your thoughts if you think it can benefit storm in a
> generic
> > > way.
> > >
> > > Thx,
> > > SG
> > >
> >
>

   

Re: classpath of worker

[Kyle Nusbaum]

This is intentional. We don't want user code to preempt any of storm's 
libraries. Storm shades most of the libraries it uses, so you should be able to 
bundle the libraries you want in with your topology jar.
For guava specifically, storm-core does not include it. You may just package it 
in your jar and use it.
 -- Kyle 

On Friday, September 2, 2016 2:53 AM, jinhong lu  
wrote:
 

 Hi, I found this about storm worker class path:

protected String getWorkerClassPath(String stormJar, Map stormConf) {
    List topoClasspath = new ArrayList<>();
    Object object = stormConf.get(Config.TOPOLOGY_CLASSPATH);

    if (object instanceof List) {
        topoClasspath.addAll((List) object);
    } else if (object instanceof String) {
        topoClasspath.add((String) object);
    }
    LOG.debug("topology specific classpath is {}", object);

    String classPath = Utils.workerClasspath();
    String classAddPath = Utils.addToClasspath(classPath, 
Arrays.asList(stormJar));
    return Utils.addToClasspath(classAddPath, topoClasspath);
}


According to this:
    String classAddPath = Utils.addToClasspath(classPath, 
Arrays.asList(stormJar));
user’s code will be load behind storm/lib, and storm/extlib.

Is it right?
and if I want another version of jar(for example, guava.jar), how can I do that?


Thanks,
lujinhong


   

[jira] [Created] (STORM-2085) Remove guava from storm-core pom.

[Kyle Nusbaum (JIRA)]

Kyle Nusbaum created STORM-2085:
---

 Summary: Remove guava from storm-core pom.
 Key: STORM-2085
 URL: https://issues.apache.org/jira/browse/STORM-2085
 Project: Apache Storm
  Issue Type: Improvement
Reporter: Kyle Nusbaum
Assignee: Kyle Nusbaum


Guava is not used by storm-core. Remove it as a dependency from the pom.xml



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Closed] (STORM-436) Clean up Clojure namespace declarations

[Kyle Nusbaum (JIRA)]

 [ 
https://issues.apache.org/jira/browse/STORM-436?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kyle Nusbaum closed STORM-436.
--
Resolution: Won't Fix

> Clean up Clojure namespace declarations
> ---
>
> Key: STORM-436
> URL: https://issues.apache.org/jira/browse/STORM-436
> Project: Apache Storm
>  Issue Type: Improvement
>  Components: storm-core
>Affects Versions: 0.9.2-incubating
>Reporter: Daniel Compton
>Priority: Minor
>  Labels: newbie
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> Some of the Clojure namespace declarations in the storm project are messy and 
> non-idiomatic. 
> https://github.com/apache/incubator-storm/blob/master/storm-core/src/clj/backtype/storm/ui/core.clj#L18-L38
>  is a good example of this.
> There are a few things I'd like to improve:
> 1. Coalesce multiple use/require/import's into a single use/require/import.
> 2. Order imports use, require, import.
> 3. Optionally, replacing use with some mix of
> * [... :refer :all]
> * Referring just the vars that are used
> * Qualifying the namespace imports
> Is there a reason why the namespaces were done in the way they have been? 
> What would be the preferred way to do 3?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Closed] (STORM-588) Executor-Level Rebalance Mechanism

[Kyle Nusbaum (JIRA)]

 [ 
https://issues.apache.org/jira/browse/STORM-588?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kyle Nusbaum closed STORM-588.
--
Resolution: Won't Fix

I'm going to close this and the corresponding pull request. This has been 
sitting for a long time, and I don't think this method fits with our current 
goals for storm. 

> Executor-Level Rebalance Mechanism
> --
>
> Key: STORM-588
> URL: https://issues.apache.org/jira/browse/STORM-588
> Project: Apache Storm
>  Issue Type: Improvement
>  Components: storm-core
>Affects Versions: 0.10.0, 0.9.3-rc2
>Reporter: troy ding
>Assignee: troy ding
>
> I. The motivation
> The current rebalance mechanism is implemented on the worker level. When 
> rebalance operation is triggered (e.g. by adding/removing a worker), storm 
> kills all the workers with different assignment. It means the rebalance 
> operation has to kill certain running workers and launches them according to 
> the new assignment. The advantage of the mechanism is the simplicity of the 
> implementation, but possibly incurs _huge_ overhead. Actually, the restarting 
> latency is usually more than one second, making the system almost impossible 
> to recover under high incoming data stream rate. No system administrator 
> dares to call rebalance, especially when the system is overloaded! To bring 
> back the real benefits of rebalancing operation, we believe it is important 
> to address the following problems:
> *1. Resource wastage and additional initialization cost*: In most cases, the 
> changes on worker’s assignment (if not killed) only affect a small fraction 
> of running executors on it. Only part of them needs to be migrated or 
> created, while the remaining can keep running on the same worker. The current 
> implementation, however, forcefully restarts all the executors, and calls 
> unnecessary initializations (i.e. call Bolt.prepare() and Spout.prepare()) to 
> most of the running tasks. It not only wastes the computation resources of 
> unaffected executors, but also amplifies the initialization costs under 
> certain condition, e.g. index load in the bolt.
> *2. Restarting workers causes avoidable in-memory data loss*: Currently, a 
> supervisor uses “kill -9” command to kill its correspondent worker. 
> Consequently, all the tasks on this worker have no chance to save the task 
> data. The running states of the workers, including important information when 
> resuming its duty, are simply lost, potentially causing unnecessary 
> recomputation on the states.
> *3. JVM restart cost, long duration and lost of HotSpot optimizations*: 
> Restarting a JVM involves a long initialization procedure, and loses all the 
> runtime optimizations available for the application byte-code. As far as we 
> know, the HotSpot JVM is capable of detecting the performance-critical 
> sections in the code and dynamically translates the Java byte codes of these 
> hot spots into native machine code. In particular, tasks that are CPU-bound 
> can greatly benefit from this feature. If we directly kill the worker, all 
> the advantages of these features are lost.
> II. Proposed solutions
> 1. At the supervisor side:
> The current supervisor implementation periodically calls the “sync-processes” 
> function to check whether a live worker should be killed: (1) the mapping 
> relationship between the worker and the topology has changed (e.g. this 
> worker is re-assigned to another topology or the serving topology is killed); 
> (2) the worker’s assignment has updated (e.g. the parallelism of some bolts 
> increases/decreases). 
> In order to reuse the worker’s JVM instance as much as possible, we propose 
> that we do not kill the workers mentioned in condition (2), but only kill 
> those that do not belong to the topology anymore (condition (1)).
> 2. At the worker side: 
> Because of the reuse of the JVM instance, workers needs to periodically 
> synchronize its assigned executors. To achieve this, a new thread which is 
> similar to the existing “refresh-connections” is launched, to kill the 
> non-existing executors, and to start newly assigned ones. Note that, in 
> practice, the “refresh-connections“ threads already retrieves the assignment 
> information from the ZK, and this information can be shared with this new 
> thread, which reduce the load of the ZK.
> Due to the change of the binging from the running executors to the worker, 
> re-routing tuple is also required. To fulfill this prepose, we need to 
> rewrite the following two functions, “transfer-local-fn” and “transfer-fn” 
> (note the rewrite is compul

Re: Too many machine mails

[Kyle Nusbaum]

+1 for Taylor's proposed solution.
 -- Kyle 

On Friday, August 12, 2016 4:57 AM, Jungtaek Lim <kabh...@gmail.com> wrote:
 

 IMHO, all committers/PMCs need to be noticed for JIRA and Github activities.
(Sometimes pull requests are submitted without having associated JIRA issue
as I did it yesterday. Actually I don't strictly think committers/PMCs need
to be noticed for all of pull request comments, but at least need to be
noticed for open/close pull request activities.)

So separating list is only for contributors which would like to hear about
community activities but don't want to see the details (issue level),
and reducing
duplication should be handled even though we can move JIRA / Github
activities out of dev@ list.

I assume that adding JIRA work log doesn't send notification. If it is,
that's enough for me. Separating list is optional.

- Jungtaek Lim (HeartSaVioR)



2016년 8월 12일 (금) 오전 6:20, Kyle Nusbaum <knusb...@yahoo-inc.com.invalid>님이
작성:

> If this other mailing list gets notified of all github activity (all
> comments, etc.), is that sufficient for being "archived" on ASF hardware?
> I'm assuming the ASF is hosting their own mail servers.
> I'd much rather have all github activity go to a mailing list for archive
> than go to jira and end up in the mailing list 4 times anyway.
> -- Kyle
>
>    On Thursday, August 11, 2016 3:54 PM, P. Taylor Goetz <
> ptgo...@gmail.com> wrote:
>
>
>  We don’t need a formal vote if we have a general consensus.
> This is an issue I’d like to see fixed since it drives me nuts and is
> amplified by the number of mailing lists I’m subscribed to.
> The requirement to link github pull request comments, etc. to JIRA
> originates from the ASF policy that all artifacts of the decision-making
> process (email, issues, etc.) be archived on ASF-controlled hardware. The
> linking of github activity to JIRA partly addressed that, but it’s not
> optimal (e.g. What happens when a PR isn’t linked to a JIRA?).
> Personally, I want to get notified of all JIRA and github activity, but
> what I don’t want is 4 copies of every notification. I have some email
> filters to help alleviate that, but it’s not a good solution.
> One short-term solution for those who don’t care about JIRA/GitHub traffic
> would be to setup issues@storm.a.o and funnel all generated mail there
> (that’s just an INFRA ticket followed by some patience). That won’t
> alleviate the issue of duplicates for those that subscribe to that list.
> Another option that’s available now (it wasn’t in the past) is to switch
> over to github issues and stop using JIRA. Then people could set their own
> notification preferences in github. The downside is to close issues, it
> would require a git commit (the same mechanism we use today to close
> abandoned pull requests). Another downside is that issues would be
> read-only, so we wouldn’t be able to add labels, etc.
> Actually as I was typing this I was also chatting with INFRA about it. And
> it turns out you can funnel github activity into the work log for a JIRA,
> rather than comments. That will eliminate the duplicate emails.
> So it seems the solution might be:
> 1. Funnel github activity to the associated JIRA work log.2. setup
> issues@storm.a.o. and route JIRA notification to that.
> That would keep dev@ clear of the machines. ;) How does that sound?
> -Taylor
>
>
> On Aug 11, 2016, at 3:08 PM, Kyle Nusbaum <knusb...@yahoo-inc.com.INVALID>
> wrote:
> I'm pro-getting-rid-of-github-messages-on-jira as well, but that's less
> annoying to me personally than the mails. It's also not clear what a better
> solution for keeping jira and github "linked" is at this point.
> As far as what notifications come through, once it's on its own list I
> don't care if everything comes through.
> Do we need to call an official vote or something to actually get this
> moving? I'm not sure what the procedure is for setting up mailing lists.
>  -- Kyle
>
>    On Thursday, August 11, 2016 9:18 AM, Jungtaek Lim <kabh...@gmail.com>
> wrote:
>
>
>  First of all we need to define which things are annoying. Belows are some
> which are mentioned one or more people,
>
> 1. Duplicated notifications per comment (You can receive 2 mails from dev@
> + 1 mails from github up to condition (you're an author, you're watching,
> you're mentioned, etc) + occasionally 1 empty change mail from dev -> up to
> 4 mails)
> 2. Copied comments from JIRA issue (with or without notification)
>
> and also need to define which things should be notified
>
> a. open pull request and close pull request
> b. only open pull request (linking github pull request and notified by
> changing status of issue - we can have 'patch available' status for that)
&

Re: Too many machine mails

[Kyle Nusbaum]

If this other mailing list gets notified of all github activity (all comments, 
etc.), is that sufficient for being "archived" on ASF hardware? I'm assuming 
the ASF is hosting their own mail servers.
I'd much rather have all github activity go to a mailing list for archive than 
go to jira and end up in the mailing list 4 times anyway.
-- Kyle 

On Thursday, August 11, 2016 3:54 PM, P. Taylor Goetz <ptgo...@gmail.com> 
wrote:
 

 We don’t need a formal vote if we have a general consensus.
This is an issue I’d like to see fixed since it drives me nuts and is amplified 
by the number of mailing lists I’m subscribed to.
The requirement to link github pull request comments, etc. to JIRA originates 
from the ASF policy that all artifacts of the decision-making process (email, 
issues, etc.) be archived on ASF-controlled hardware. The linking of github 
activity to JIRA partly addressed that, but it’s not optimal (e.g. What happens 
when a PR isn’t linked to a JIRA?).
Personally, I want to get notified of all JIRA and github activity, but what I 
don’t want is 4 copies of every notification. I have some email filters to help 
alleviate that, but it’s not a good solution.
One short-term solution for those who don’t care about JIRA/GitHub traffic 
would be to setup issues@storm.a.o and funnel all generated mail there (that’s 
just an INFRA ticket followed by some patience). That won’t alleviate the issue 
of duplicates for those that subscribe to that list.
Another option that’s available now (it wasn’t in the past) is to switch over 
to github issues and stop using JIRA. Then people could set their own 
notification preferences in github. The downside is to close issues, it would 
require a git commit (the same mechanism we use today to close abandoned pull 
requests). Another downside is that issues would be read-only, so we wouldn’t 
be able to add labels, etc.
Actually as I was typing this I was also chatting with INFRA about it. And it 
turns out you can funnel github activity into the work log for a JIRA, rather 
than comments. That will eliminate the duplicate emails.
So it seems the solution might be:
1. Funnel github activity to the associated JIRA work log.2. setup 
issues@storm.a.o. and route JIRA notification to that.
That would keep dev@ clear of the machines. ;) How does that sound?
-Taylor


On Aug 11, 2016, at 3:08 PM, Kyle Nusbaum <knusb...@yahoo-inc.com.INVALID> 
wrote:
I'm pro-getting-rid-of-github-messages-on-jira as well, but that's less 
annoying to me personally than the mails. It's also not clear what a better 
solution for keeping jira and github "linked" is at this point.
As far as what notifications come through, once it's on its own list I don't 
care if everything comes through.
Do we need to call an official vote or something to actually get this moving? 
I'm not sure what the procedure is for setting up mailing lists.
 -- Kyle 

    On Thursday, August 11, 2016 9:18 AM, Jungtaek Lim <kabh...@gmail.com> 
wrote:


 First of all we need to define which things are annoying. Belows are some
which are mentioned one or more people,

1. Duplicated notifications per comment (You can receive 2 mails from dev@
+ 1 mails from github up to condition (you're an author, you're watching,
you're mentioned, etc) + occasionally 1 empty change mail from dev -> up to
4 mails)
2. Copied comments from JIRA issue (with or without notification)

and also need to define which things should be notified

a. open pull request and close pull request
b. only open pull request (linking github pull request and notified by
changing status of issue - we can have 'patch available' status for that)
c. no we should receive all of comments (just need to reduce duplicated
things)

- Jungtaek Lim (HeartSaVioR)


2016년 8월 11일 (목) 오후 10:52, Bobby Evans <ev...@yahoo-inc.com.invalid>님이 작성:


Yes lets have a separate firehouse/commit/whatever mailing list that if
people really want all of that data they can see it all.  That way it is
archived in ASF infra.  I do see value in having JIRA and GITHUB linked,
I'm not sure if there is a better way to link the two right now though.  If
someone does have experience with this type of thing and can make a better
solution I think we can talk to INFRA about adopting/supporting those
changes. - Bobby

    On Thursday, August 11, 2016 8:41 AM, Aditya Desai <adity...@usc.edu>
wrote:


  Please reduce the number of emails. I am getting many many emails in
recent
days and spam my inbox.

On Thu, Aug 11, 2016 at 2:41 AM, Erik Weathers <
eweath...@groupon.com.invalid> wrote:


I will state again (as I've done on prior email threads) that I find no
value in spamming the JIRA issues like this, and that I strongly believe
that this behavior is in fact detrimental since it obscures the actual
comments on the issue itself.  The proposed solution of just moving the
destination of the JIRA emails to a different list than
dev@storm.apa

Re: Too many machine mails

[Kyle Nusbaum]

I'm pro-getting-rid-of-github-messages-on-jira as well, but that's less 
annoying to me personally than the mails. It's also not clear what a better 
solution for keeping jira and github "linked" is at this point.
As far as what notifications come through, once it's on its own list I don't 
care if everything comes through.
Do we need to call an official vote or something to actually get this moving? 
I'm not sure what the procedure is for setting up mailing lists.
 -- Kyle 

On Thursday, August 11, 2016 9:18 AM, Jungtaek Lim <kabh...@gmail.com> 
wrote:
 

 First of all we need to define which things are annoying. Belows are some
which are mentioned one or more people,

1. Duplicated notifications per comment (You can receive 2 mails from dev@
+ 1 mails from github up to condition (you're an author, you're watching,
you're mentioned, etc) + occasionally 1 empty change mail from dev -> up to
4 mails)
2. Copied comments from JIRA issue (with or without notification)

and also need to define which things should be notified

a. open pull request and close pull request
b. only open pull request (linking github pull request and notified by
changing status of issue - we can have 'patch available' status for that)
c. no we should receive all of comments (just need to reduce duplicated
things)

- Jungtaek Lim (HeartSaVioR)


2016년 8월 11일 (목) 오후 10:52, Bobby Evans <ev...@yahoo-inc.com.invalid>님이 작성:

> Yes lets have a separate firehouse/commit/whatever mailing list that if
> people really want all of that data they can see it all.  That way it is
> archived in ASF infra.  I do see value in having JIRA and GITHUB linked,
> I'm not sure if there is a better way to link the two right now though.  If
> someone does have experience with this type of thing and can make a better
> solution I think we can talk to INFRA about adopting/supporting those
> changes. - Bobby
>
>    On Thursday, August 11, 2016 8:41 AM, Aditya Desai <adity...@usc.edu>
> wrote:
>
>
>  Please reduce the number of emails. I am getting many many emails in
> recent
> days and spam my inbox.
>
> On Thu, Aug 11, 2016 at 2:41 AM, Erik Weathers <
> eweath...@groupon.com.invalid> wrote:
>
> > I will state again (as I've done on prior email threads) that I find no
> > value in spamming the JIRA issues like this, and that I strongly believe
> > that this behavior is in fact detrimental since it obscures the actual
> > comments on the issue itself.  The proposed solution of just moving the
> > destination of the JIRA emails to a different list than
> > dev@storm.apache.org
> > doesn't solve that root problem.
> >
> > I want to be able to read a JIRA issue without having to skim over dozens
> > and dozens of auto-appended code review messages.  I truly cannot
> > understand why this isn't an annoyance for others.  I could be really
> > snarky and reformat this email to have a bunch of random stuff in between
> > every sentence to make my point, but I hope this sentence suffices to
> prove
> > it?
> >
> > Though I must acknowledge your point Jungtaek  that there is some Apache
> > policy that all code review comments need to be archived into some apache
> > system.  Maybe we can use the attachment functionality of JIRA instead of
> > making these separate comments on the JIRA issue?  I'm not sure how the
> > integration is set up right now, that seems feasible.
> >
> > - Erik
> >
> > On Thu, Aug 11, 2016 at 2:08 AM, Matthias J. Sax <mj...@apache.org>
> wrote:
> >
> > > I like the idea of have one more mailing list to reduce load on
> dev-list.
> > >
> > > -Matthias
> > >
> > > On 08/11/2016 11:07 AM, Jungtaek Lim wrote:
> > > > I remember that Taylor stated that all github comments should be
> copied
> > > to
> > > > somewhere Apache infra, and it's Apache JIRA for us.
> > > >
> > > > It seems to make sense but I'm curious other projects respect this
> > rule.
> > > I
> > > > also subscribed dev list of Kafka, Zeppelin, Flink, HBase, Spark
> > > (although
> > > > I barely see them) but no project is sending mail per each comment.
> > Some
> > > of
> > > > them copy github comments to JIRA issue but no notification, and
> others
> > > > doesn't even copy comments to JIRA issue.
> > > > (You can check this with dev mailing list archive, too.)
> > > >
> > > > I'm in favor of reducing simple notification mails. Personally I saw
> > most
> > > > of Storm dev. mails so I'm fine to keep mailing as it is (with some
> > > > annoying 'empty' notification)

Too many machine mails

[Kyle Nusbaum]

There seems to be a surplus of automatically-generated emails on the dev 
mailing list.
Github and Apache's Jira constantly send mails to the dev list.

I'm not sure that anyone finds these useful. Even if they do, I wonder if its 
better to move them to a separate list. It's possible that everyone has email 
filters employed to sort this out, but if every subscriber has the same filters 
employed, it might indicate the need for a separate list. -- Kyle

[jira] [Resolved] (STORM-1954) Large Trident topologies can cause memory issues due to DefaultResourceDeclarer object reading config

[Kyle Nusbaum (JIRA)]

 [ 
https://issues.apache.org/jira/browse/STORM-1954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kyle Nusbaum resolved STORM-1954.
-
   Resolution: Fixed
Fix Version/s: 2.0.0

> Large Trident topologies can cause memory issues due to 
> DefaultResourceDeclarer object reading config
> -
>
> Key: STORM-1954
> URL: https://issues.apache.org/jira/browse/STORM-1954
> Project: Apache Storm
>  Issue Type: Bug
>    Reporter: Kyle Nusbaum
> Fix For: 2.0.0
>
>
> DefaultResourceDeclarer objects each read the full config on instantiation. 
> We've seen this causing slow startup and OOMs for large Trident topologies. 
> The config should be static in DefaultResourceDeclarer.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (STORM-1916) Add ability for worker-first classpath

[Kyle Nusbaum (JIRA)]

 [ 
https://issues.apache.org/jira/browse/STORM-1916?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kyle Nusbaum resolved STORM-1916.
-
   Resolution: Fixed
Fix Version/s: 2.0.0

> Add ability for worker-first classpath
> --
>
> Key: STORM-1916
> URL: https://issues.apache.org/jira/browse/STORM-1916
> Project: Apache Storm
>  Issue Type: Improvement
>        Reporter: Kyle Nusbaum
>    Assignee: Kyle Nusbaum
> Fix For: 2.0.0
>
>
> Debugging in multi-tenant environments can be difficult, and having the 
> ability to override the classpath can be helpful. We want a topology config 
> that specifies an overriding classpath for a given topology. This config is 
> disabled by default since users overriding the classpath is dangerous in a 
> multi-tenant environment. The config can be temporarily enabled on Nimbus to 
> allow submission of topologies with overridden classpaths. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (STORM-1913) Additions and Improvements for Trident RAS API

[Kyle Nusbaum (JIRA)]

 [ 
https://issues.apache.org/jira/browse/STORM-1913?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kyle Nusbaum resolved STORM-1913.
-
Resolution: Fixed

> Additions and Improvements for Trident RAS API
> --
>
> Key: STORM-1913
> URL: https://issues.apache.org/jira/browse/STORM-1913
> Project: Apache Storm
>  Issue Type: Improvement
>        Reporter: Kyle Nusbaum
>    Assignee: Kyle Nusbaum
> Fix For: 2.0.0
>
>
> Trident's RAS API does not honor the following config values:
> {code}
> topology.component.resources.onheap.memory.mb
> topology.component.resources.offheap.memory.mb
> topology.component.cpu.pcore.percent
> {code}
> Trident does not receive the user's config as part of its builder API, so it 
> does not know the value of these. Instead of altering the existing API (we 
> want to remain backwards-compatible), add some new methods for dealing with 
> this.
> There is also currently no way to set the master coord spouts' resources. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (STORM-1913) Additions and Improvements for Trident RAS API

[Kyle Nusbaum (JIRA)]

 [ 
https://issues.apache.org/jira/browse/STORM-1913?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kyle Nusbaum updated STORM-1913:

Fix Version/s: 2.0.0

> Additions and Improvements for Trident RAS API
> --
>
> Key: STORM-1913
> URL: https://issues.apache.org/jira/browse/STORM-1913
> Project: Apache Storm
>  Issue Type: Improvement
>        Reporter: Kyle Nusbaum
>    Assignee: Kyle Nusbaum
> Fix For: 2.0.0
>
>
> Trident's RAS API does not honor the following config values:
> {code}
> topology.component.resources.onheap.memory.mb
> topology.component.resources.offheap.memory.mb
> topology.component.cpu.pcore.percent
> {code}
> Trident does not receive the user's config as part of its builder API, so it 
> does not know the value of these. Instead of altering the existing API (we 
> want to remain backwards-compatible), add some new methods for dealing with 
> this.
> There is also currently no way to set the master coord spouts' resources. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Closed] (STORM-1913) Additions and Improvements for Trident RAS API

[Kyle Nusbaum (JIRA)]

 [ 
https://issues.apache.org/jira/browse/STORM-1913?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kyle Nusbaum closed STORM-1913.
---
Resolution: Fixed

> Additions and Improvements for Trident RAS API
> --
>
> Key: STORM-1913
> URL: https://issues.apache.org/jira/browse/STORM-1913
> Project: Apache Storm
>  Issue Type: Improvement
>        Reporter: Kyle Nusbaum
>    Assignee: Kyle Nusbaum
>
> Trident's RAS API does not honor the following config values:
> {code}
> topology.component.resources.onheap.memory.mb
> topology.component.resources.offheap.memory.mb
> topology.component.cpu.pcore.percent
> {code}
> Trident does not receive the user's config as part of its builder API, so it 
> does not know the value of these. Instead of altering the existing API (we 
> want to remain backwards-compatible), add some new methods for dealing with 
> this.
> There is also currently no way to set the master coord spouts' resources. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Reopened] (STORM-1913) Additions and Improvements for Trident RAS API

[Kyle Nusbaum (JIRA)]

 [ 
https://issues.apache.org/jira/browse/STORM-1913?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kyle Nusbaum reopened STORM-1913:
-

> Additions and Improvements for Trident RAS API
> --
>
> Key: STORM-1913
> URL: https://issues.apache.org/jira/browse/STORM-1913
> Project: Apache Storm
>  Issue Type: Improvement
>        Reporter: Kyle Nusbaum
>    Assignee: Kyle Nusbaum
> Fix For: 2.0.0
>
>
> Trident's RAS API does not honor the following config values:
> {code}
> topology.component.resources.onheap.memory.mb
> topology.component.resources.offheap.memory.mb
> topology.component.cpu.pcore.percent
> {code}
> Trident does not receive the user's config as part of its builder API, so it 
> does not know the value of these. Instead of altering the existing API (we 
> want to remain backwards-compatible), add some new methods for dealing with 
> this.
> There is also currently no way to set the master coord spouts' resources. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-1949) Backpressure can cause spout to stop emitting and stall topology

[Kyle Nusbaum (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15371017#comment-15371017
 ] 

Kyle Nusbaum commented on STORM-1949:
-

I'm okay with polling ZK for changes as a fix (for now) so we can keep 
backpressure on by default. There are obviously issues with the design of 
backpressure in general, but polling shouldn't make these issues significantly 
worse. I don't think the load on ZK will increase appreciably from a few nodes 
polling. The load problem is inherent to other parts of backpressure design.

We could also turn it off by default, but it's nice to have a flow-control 
mechanism that works automatically without needing the user to perform acking. 
It might be some time until we have a better system in place. 

> Backpressure can cause spout to stop emitting and stall topology
> 
>
> Key: STORM-1949
> URL: https://issues.apache.org/jira/browse/STORM-1949
> Project: Apache Storm
>  Issue Type: Bug
>Reporter: Roshan Naik
>
> Problem can be reproduced by this [Word count 
> topology|https://github.com/hortonworks/storm/blob/perftopos1.x/examples/storm-starter/src/jvm/org/apache/storm/starter/perf/FileReadWordCountTopo.java]
> within a IDE.
> I ran it with 1 spout instance, 2 splitter bolt instances, 2 counter bolt 
> instances.
> The problem is more easily reproduced with WC topology as it causes an 
> explosion of tuples due to splitting a sentence tuple into word tuples. As 
> the bolts have to process more tuples than the  spout is producing, spout 
> needs to operate slower.
> The amount of time it takes for the topology to stall can vary.. but 
> typically under 10 mins. 
> *My theory:*  I suspect there is a race condition in the way ZK is being 
> utilized to enable/disable back pressure. When congested (i.e pressure 
> exceeds high water mark), the bolt's worker records this congested situation 
> in ZK by creating a node. Once the congestion is reduced below the low water 
> mark, it deletes this node. 
> The spout's worker has setup a watch on the parent node, expecting a callback 
> whenever there is change in the child nodes. On receiving the callback the 
> spout's worker lists the parent node to check if there are 0 or more child 
> nodes it is essentially trying to figure out the nature of state change 
> in ZK to determine whether to throttle or not. Subsequently  it setsup 
> another watch in ZK to keep an eye on future changes.
> When there are multiple bolts, there can be rapid creation/deletion of these 
> ZK nodes. Between the time the worker receives a callback and sets up the 
> next watch.. many changes may have undergone in ZK which will go unnoticed by 
> the spout. 
> The condition that the bolts are no longer congested may not get noticed as a 
> result of this.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (STORM-1954) Large Trident topologies can cause memory issues due to DefaultResourceDeclarer object reading config

[Kyle Nusbaum (JIRA)]

 [ 
https://issues.apache.org/jira/browse/STORM-1954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kyle Nusbaum updated STORM-1954:

Summary: Large Trident topologies can cause memory issues due to 
DefaultResourceDeclarer object reading config  (was: Large Trident topologies 
can cause memory issues due to Node object reading config)

> Large Trident topologies can cause memory issues due to 
> DefaultResourceDeclarer object reading config
> -
>
> Key: STORM-1954
> URL: https://issues.apache.org/jira/browse/STORM-1954
> Project: Apache Storm
>  Issue Type: Bug
>    Reporter: Kyle Nusbaum
>
> DefaultResourceDeclarer objects each read the full config on instantiation. 
> We've seen this causing slow startup and OOMs for large Trident topologies. 
> The config should be static in DefaultResourceDeclarer.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (STORM-1954) Large Trident topologies can cause memory issues due to Node object reading config

[Kyle Nusbaum (JIRA)]

 [ 
https://issues.apache.org/jira/browse/STORM-1954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kyle Nusbaum updated STORM-1954:

Description: 
DefaultResourceDeclarer objects each read the full config on instantiation. 
We've seen this causing slow startup and OOMs for large Trident topologies. 

The config should be static in DefaultResourceDeclarer.

  was:
Trident Node objects each read the full config on instantiation. We've seen 
this causing slow startup and OOMs for large Trident topologies. 

The config should be static in Node.


> Large Trident topologies can cause memory issues due to Node object reading 
> config
> --
>
> Key: STORM-1954
> URL: https://issues.apache.org/jira/browse/STORM-1954
> Project: Apache Storm
>  Issue Type: Bug
>    Reporter: Kyle Nusbaum
>
> DefaultResourceDeclarer objects each read the full config on instantiation. 
> We've seen this causing slow startup and OOMs for large Trident topologies. 
> The config should be static in DefaultResourceDeclarer.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (STORM-1954) Large Trident topologies can cause memory issues due to Node object reading config

[Kyle Nusbaum (JIRA)]

Kyle Nusbaum created STORM-1954:
---

 Summary: Large Trident topologies can cause memory issues due to 
Node object reading config
 Key: STORM-1954
 URL: https://issues.apache.org/jira/browse/STORM-1954
 Project: Apache Storm
  Issue Type: Bug
Reporter: Kyle Nusbaum


Trident Node objects each read the full config on instantiation. We've seen 
this causing slow startup and OOMs for large Trident topologies. 

The config should be static in Node.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (STORM-1916) Add ability for worker-first classpath

[Kyle Nusbaum (JIRA)]

Kyle Nusbaum created STORM-1916:
---

 Summary: Add ability for worker-first classpath
 Key: STORM-1916
 URL: https://issues.apache.org/jira/browse/STORM-1916
 Project: Apache Storm
  Issue Type: Improvement
Reporter: Kyle Nusbaum
Assignee: Kyle Nusbaum


Debugging in multi-tenant environments can be difficult, and having the ability 
to override the classpath can be helpful. We want a topology config that 
specifies an overriding classpath for a given topology. This config is disabled 
by default since users overriding the classpath is dangerous in a multi-tenant 
environment. The config can be temporarily enabled on Nimbus to allow 
submission of topologies with overridden classpaths. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (STORM-1913) Additions and Improvements for Trident RAS API

[Kyle Nusbaum (JIRA)]

Kyle Nusbaum created STORM-1913:
---

 Summary: Additions and Improvements for Trident RAS API
 Key: STORM-1913
 URL: https://issues.apache.org/jira/browse/STORM-1913
 Project: Apache Storm
  Issue Type: Improvement
Reporter: Kyle Nusbaum
Assignee: Kyle Nusbaum


Trident's RAS API does not honor the following config values:
{code}
topology.component.resources.onheap.memory.mb
topology.component.resources.offheap.memory.mb
topology.component.cpu.pcore.percent
{code}

Trident does not receive the user's config as part of its builder API, so it 
does not know the value of these. Instead of altering the existing API (we want 
to remain backwards-compatible), add some new methods for dealing with this.

There is also currently no way to set the master coord spouts' resources. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (STORM-1912) Additions and Improvements for Trident RAS API

[Kyle Nusbaum (JIRA)]

Kyle Nusbaum created STORM-1912:
---

 Summary: Additions and Improvements for Trident RAS API
 Key: STORM-1912
 URL: https://issues.apache.org/jira/browse/STORM-1912
 Project: Apache Storm
  Issue Type: Improvement
Reporter: Kyle Nusbaum
Assignee: Kyle Nusbaum


Trident's RAS API does not honor the following config values:
{code}
topology.component.resources.onheap.memory.mb
topology.component.resources.offheap.memory.mb
topology.component.cpu.pcore.percent
{code}

Trident does not receive the user's config as part of its builder API, so it 
does not know the value of these. Instead of altering the existing API (we want 
to remain backwards-compatible), add some new methods for dealing with this.

There is also currently no way to set the master coord spouts' resources. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (STORM-1764) Pacemaker is throwing some stack traces

[Kyle Nusbaum (JIRA)]

Kyle Nusbaum created STORM-1764:
---

 Summary: Pacemaker is throwing some stack traces
 Key: STORM-1764
 URL: https://issues.apache.org/jira/browse/STORM-1764
 Project: Apache Storm
  Issue Type: Bug
Reporter: Kyle Nusbaum
Assignee: Kyle Nusbaum


Pacemaker throws a handful of stack traces when the occasional weird thing 
happens. This fixes an IndexOutOfBounds exception, a very strange 
IllegalStateException, and a NullPointerException.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (STORM-1611) port org.apache.storm.pacemaker.pacemaker to java

[Kyle Nusbaum (JIRA)]

 [ 
https://issues.apache.org/jira/browse/STORM-1611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kyle Nusbaum resolved STORM-1611.
-
   Resolution: Fixed
Fix Version/s: 2.0.0

> port org.apache.storm.pacemaker.pacemaker to java
> -
>
> Key: STORM-1611
> URL: https://issues.apache.org/jira/browse/STORM-1611
> Project: Apache Storm
>  Issue Type: New Feature
>Reporter: John Fang
>Assignee: John Fang
> Fix For: 2.0.0
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (STORM-1673) log4j2/worker.xml refers old package of LoggerMetricsConsumer

[Kyle Nusbaum (JIRA)]

 [ 
https://issues.apache.org/jira/browse/STORM-1673?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kyle Nusbaum updated STORM-1673:

Assignee: Jungtaek Lim  (was: Kyle Nusbaum)

> log4j2/worker.xml refers old package of LoggerMetricsConsumer
> -
>
> Key: STORM-1673
> URL: https://issues.apache.org/jira/browse/STORM-1673
> Project: Apache Storm
>  Issue Type: Bug
>  Components: storm-core
>Affects Versions: 1.0.0, 2.0.0
>Reporter: Jungtaek Lim
>Assignee: Jungtaek Lim
>
> We changed package path from 'backtype.storm' to 'org.apache.storm'. Source 
> codes seem to moved properly, but missed log4j2 configuration, so metric log 
> is logged into worker log file, not metrics file.
> It should be simple patch so I'd like to include this as 1.0.0. If we don't 
> want to include any bugfixes I'm OK to remove epic.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (STORM-1667) Log the IO Exception when deleting worker pid dir

[Kyle Nusbaum (JIRA)]

 [ 
https://issues.apache.org/jira/browse/STORM-1667?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kyle Nusbaum resolved STORM-1667.
-
Resolution: Fixed

> Log the IO Exception when deleting worker pid dir
> -
>
> Key: STORM-1667
> URL: https://issues.apache.org/jira/browse/STORM-1667
> Project: Apache Storm
>  Issue Type: Bug
>  Components: storm-core
>Affects Versions: 1.0.0
>Reporter: Zhuo Liu
>Assignee: Zhuo Liu
> Fix For: 1.0.0
>
>
> There is a race condition since shutdown-worker is called by both 
> sync-processes and synchronize-supervisor in supervisor. One pid directory 
> might have been already deleted when rmr-as-user tries to delete it. 
> Therefore, we should not let the FileNotFoundException get thrown and crash 
> the supervisor.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (STORM-1673) log4j2/worker.xml refers old package of LoggerMetricsConsumer

[Kyle Nusbaum (JIRA)]

 [ 
https://issues.apache.org/jira/browse/STORM-1673?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kyle Nusbaum reassigned STORM-1673:
---

Assignee: Kyle Nusbaum  (was: Jungtaek Lim)

> log4j2/worker.xml refers old package of LoggerMetricsConsumer
> -
>
> Key: STORM-1673
> URL: https://issues.apache.org/jira/browse/STORM-1673
> Project: Apache Storm
>  Issue Type: Bug
>  Components: storm-core
>Affects Versions: 1.0.0, 2.0.0
>Reporter: Jungtaek Lim
>    Assignee: Kyle Nusbaum
>
> We changed package path from 'backtype.storm' to 'org.apache.storm'. Source 
> codes seem to moved properly, but missed log4j2 configuration, so metric log 
> is logged into worker log file, not metrics file.
> It should be simple patch so I'd like to include this as 1.0.0. If we don't 
> want to include any bugfixes I'm OK to remove epic.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Reopened] (STORM-1581) Repair github links in the storm documentation

[Kyle Nusbaum (JIRA)]

 [ 
https://issues.apache.org/jira/browse/STORM-1581?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kyle Nusbaum reopened STORM-1581:
-
  Assignee: Kyle Nusbaum  (was: Abhishek Agarwal)

> Repair github links in the storm documentation
> --
>
> Key: STORM-1581
> URL: https://issues.apache.org/jira/browse/STORM-1581
> Project: Apache Storm
>  Issue Type: Bug
>  Components: documentation
>Affects Versions: 1.0.0
>Reporter: Abhishek Agarwal
>    Assignee: Kyle Nusbaum
> Fix For: 1.0.0
>
>
> Java classes have been migrated to org.apache.storm and the github links are 
> now broken in the documentation.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (STORM-1581) Repair github links in the storm documentation

[Kyle Nusbaum (JIRA)]

 [ 
https://issues.apache.org/jira/browse/STORM-1581?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kyle Nusbaum resolved STORM-1581.
-
Resolution: Fixed

> Repair github links in the storm documentation
> --
>
> Key: STORM-1581
> URL: https://issues.apache.org/jira/browse/STORM-1581
> Project: Apache Storm
>  Issue Type: Bug
>  Components: documentation
>Affects Versions: 1.0.0
>Reporter: Abhishek Agarwal
>    Assignee: Kyle Nusbaum
> Fix For: 1.0.0
>
>
> Java classes have been migrated to org.apache.storm and the github links are 
> now broken in the documentation.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (STORM-1668) Flux silently fails while setting a non-existent property.

[Kyle Nusbaum (JIRA)]

 [ 
https://issues.apache.org/jira/browse/STORM-1668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kyle Nusbaum resolved STORM-1668.
-
Resolution: Fixed

> Flux silently fails while setting a non-existent property.
> --
>
> Key: STORM-1668
> URL: https://issues.apache.org/jira/browse/STORM-1668
> Project: Apache Storm
>  Issue Type: Bug
>  Components: Flux
>Affects Versions: 1.0.0, 2.0.0
>Reporter: Priyank Shah
>Assignee: P. Taylor Goetz
> Fix For: 1.0.0
>
>
> Currently, if a yaml file has a property with a name that does not exist on 
> the java topology component object then flux silently fails by logging a 
> message and does not throw an exception. This needs to be changed so that 
> flux throws the exception failing topology submission so that user can take 
> corrective action.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Closed] (STORM-1581) Repair github links in the storm documentation

[Kyle Nusbaum (JIRA)]

 [ 
https://issues.apache.org/jira/browse/STORM-1581?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kyle Nusbaum closed STORM-1581.
---
   Resolution: Fixed
Fix Version/s: 1.0.0

Fixed when [~revans2] moved the docs to SVN.

> Repair github links in the storm documentation
> --
>
> Key: STORM-1581
> URL: https://issues.apache.org/jira/browse/STORM-1581
> Project: Apache Storm
>  Issue Type: Bug
>  Components: documentation
>Affects Versions: 1.0.0
>Reporter: Abhishek Agarwal
>Assignee: Abhishek Agarwal
> Fix For: 1.0.0
>
>
> Java classes have been migrated to org.apache.storm and the github links are 
> now broken in the documentation.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Proposal on concentrating release 1.0.0

[Kyle Nusbaum]

STORM-1595 is closed now. I wasn't able to reproduce it again.
 -- Kyle 

On Monday, March 28, 2016 8:28 PM, Jungtaek Lim  wrote:
 

 Thanks Bobby and Taylor,

I intended to initiate discussion about handling holding issues (STORM-1560
 and STORM-1595
) since those issues are
no progress on it and cannot reproduce for now but can hold releasing 1.0.0.
Since STORM-1560 is moved out of 1.0.0, we only left STORM-1595
, which is marked as
'Major'.
Would we like to move this out of 1.0.0, too?

Regarding STORM-1632 ,
I'm also OK to choose either option you provided.
Though we already moved issue out of 1.0.0, conversation becomes getting
longer because of instability of benchmark numbers.
(I would like to lend a hand to Bobby since Bobby showed amazing details
when evaluating performance.)
But one thing we seems have consensus is that event logger is a feature
which could be OK to disable by default. (UI should be reflected)
I asked Harsha who filed the issue to see needs or use cases of event
logger, and Harsha said he guess it would be likely to be used with dev.
cluster and rarely used on production.

Seems like there's some progress to resolve on issue today, so I'd like to
follow.

Thanks,
Jungtaek Lim (HeartSaVioR)

2016년 3월 29일 (화) 오전 6:15, P. Taylor Goetz 님이 작성:

> FYI, I pulled STORM-1560 from the 1.0 release epic, and marked all issues
> with active pull requests as “in progress”.
>
> -Taylor
>
> > On Mar 28, 2016, at 4:18 PM, P. Taylor Goetz  wrote:
> >
> > I would like to be able to initiate a release VOTE this week. We are
> very, very close.
> >
> > Pretty much everything under STORM-1491 has been addressed, and should
> be able to be merged soon.
> >
> > The two outliers are:
> >
> > STORM-1560 (Reported by me. I’m ready to close this, since I’ve not been
> able to reproduce it reliably.)
> > STORM-1595 (Reported by Kyle. Still waiting for information about how to
> reproduce this.)
> >
> > Aside from that is the debate over disabling the event logger by default
> (STORM-1632). Yes, there is a performance hit, especially in a
> single-node/single-worker configuration. In a multi-node/multi-worker
> configuration, that hit is significantly reduced.
> >
> > For STORM-1632 I see two options:
> >
> > 1. Ship with event logging enabled by default so the UI doesn’t appear
> broken by default.
> > 2. Update the UI so the user knows when feature is disabled, and give
> clear instructions on how to enable it. Ship with event logging disabled by
> default.
> >
> > I support either option. We could even ship 1.0 with option #1, and
> follow with a 1.0.1 release with option 2 implemented. We could even
> release 1.0 with a documented “known issue” that for best performance,
> users should disable event logging in production.
> >
> > -Taylor
> >
> >
> >
> >> On Mar 28, 2016, at 3:50 PM, Bobby Evans 
> wrote:
> >>
> >> I would be happy to see a 1.0 release sooner rather then later.  If
> there are some issues that are blocking it being released that you don't
> feel should be blocking it I am happy to join in that conversation.
> >> - Bobby
> >>
> >>  On Friday, March 25, 2016 11:55 PM, Jungtaek Lim 
> wrote:
> >>
> >>
> >> Hi devs,
> >>
> >> I guess it's been two months after creating issue for releasing 1.0.0 (
> >> STORM-1491 ), which I
> >> expected it's in progress of releasing so it can be resolved in several
> >> weeks.
> >>
> >> Porting works for preparing 2.0.0 are in progress simultaneously.
> >> I think two track strategy is great, but publishing releases
> continuously
> >> can make community more active and prevents specific release having
> >> too-many changes. 1.0.0 is one of the example, and I suspect its reason
> to
> >> drag releasing 0.10.0 (including beta) too long.
> >>
> >> I'd like to propose that we have some moments to concentrate on
> releasing
> >> 1.0.0, and back to work.
> >> Concentration includes discussion about lowering issues' priority,
> >> excluding issues from epic (which means out of 1.0.0), assigning issues
> >> which are remaining but not resolved yet.
> >>
> >> What do you think?
> >>
> >> Thanks,
> >> Jungtaek Lim (HeartSaVioR)
> >>
> >>
> >
>
>

  

[jira] [Closed] (STORM-1595) 'Fail' messages get stuck somewhere

[Kyle Nusbaum (JIRA)]

 [ 
https://issues.apache.org/jira/browse/STORM-1595?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kyle Nusbaum closed STORM-1595.
---
Resolution: Cannot Reproduce

> 'Fail' messages get stuck somewhere 
> 
>
> Key: STORM-1595
> URL: https://issues.apache.org/jira/browse/STORM-1595
> Project: Apache Storm
>  Issue Type: Bug
>Affects Versions: 1.0.0, 2.0.0
>    Reporter: Kyle Nusbaum
> Attachments: screenshot-1.png
>
>
> 'Fail' acks seem to be getting stuck somewhere between the acker and the 
> spout. 
> After a long time - sometimes multiple minutes - the fails show up in the 
> spout.
> I tested this on master and 1.x-branch and it occurs in both places.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-1595) 'Fail' messages get stuck somewhere

[Kyle Nusbaum (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1595?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15216581#comment-15216581
 ] 

Kyle Nusbaum commented on STORM-1595:
-

[~ptgoetz] I'm not able to reproduce this anymore. Going to close it.


> 'Fail' messages get stuck somewhere 
> 
>
> Key: STORM-1595
> URL: https://issues.apache.org/jira/browse/STORM-1595
> Project: Apache Storm
>  Issue Type: Bug
>Affects Versions: 1.0.0, 2.0.0
>    Reporter: Kyle Nusbaum
> Attachments: screenshot-1.png
>
>
> 'Fail' acks seem to be getting stuck somewhere between the acker and the 
> spout. 
> After a long time - sometimes multiple minutes - the fails show up in the 
> spout.
> I tested this on master and 1.x-branch and it occurs in both places.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Closed] (STORM-1497) Repo Branches Cleanup

[Kyle Nusbaum (JIRA)]

 [ 
https://issues.apache.org/jira/browse/STORM-1497?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kyle Nusbaum closed STORM-1497.
---
Resolution: Fixed

> Repo Branches Cleanup 
> --
>
> Key: STORM-1497
> URL: https://issues.apache.org/jira/browse/STORM-1497
> Project: Apache Storm
>  Issue Type: Improvement
>        Reporter: Kyle Nusbaum
>    Assignee: Kyle Nusbaum
>
> I've identified a bunch of branches that I think can go away:
> Already merged:
> STORM-633
> STORM-648
> STORM-651
> STORM-1153-V1
> better-trident-spouts
> experimental
> clojure1.3
> clojure1.4
> executors
> gettuples
> kryo-2.16
> mesos
> new-resource-scheduler
> nimbus-ha-branch
> scheduler
> security
> transactional-spout
> ui-url
> Branches not merged, but I think we can delete:
> STORM-1040 (harschach)
> PR_736 (ptgoetz)
> nimbus-ha (nathanmarz)
> debug (do not want)  (nathanmarz)
> iso-scheduler (not merged but VERY old) (nathanmarz)
> jarjar (not merged but VERY old) (nathanmarz)
> statespout (not merged but VERY old) (nathanmarz)
> std-redirect (not merged but VERY old) (nathanmarz)
> superclojure (not merged but VERY old) (nathanmarz)
> thunk (VERY old, don't want) (sritchie)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Closed] (STORM-1340) Use Travis-CI build matrix to improve test execution times

[Kyle Nusbaum (JIRA)]

 [ 
https://issues.apache.org/jira/browse/STORM-1340?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kyle Nusbaum closed STORM-1340.
---
Resolution: Won't Fix

> Use Travis-CI build matrix to improve test execution times
> --
>
> Key: STORM-1340
> URL: https://issues.apache.org/jira/browse/STORM-1340
> Project: Apache Storm
>  Issue Type: Improvement
>        Reporter: Kyle Nusbaum
>    Assignee: Kyle Nusbaum
>
> Travis-CI provides a 'build-matrix' that we can use to parallelize the unit 
> tests of various components in storm.
> It also provides a caching mechanism that can reduce the overhead of starting 
> up a test container.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (STORM-1653) BYLAWS has disappeared

[Kyle Nusbaum (JIRA)]

Kyle Nusbaum created STORM-1653:
---

 Summary: BYLAWS has disappeared
 Key: STORM-1653
 URL: https://issues.apache.org/jira/browse/STORM-1653
 Project: Apache Storm
  Issue Type: Bug
Reporter: Kyle Nusbaum


When moving asf-site to svn, BYLAWS (and probably some other stuff) 
disappeared. 

Someone needs to compare the previous docs with what's on SVN now and fill in 
any missing spots.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Closed] (STORM-1616) Add RAS API for Trident

[Kyle Nusbaum (JIRA)]

 [ 
https://issues.apache.org/jira/browse/STORM-1616?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kyle Nusbaum closed STORM-1616.
---
   Resolution: Fixed
Fix Version/s: 1.0.0

> Add RAS API for Trident
> ---
>
> Key: STORM-1616
> URL: https://issues.apache.org/jira/browse/STORM-1616
> Project: Apache Storm
>  Issue Type: Bug
>        Reporter: Kyle Nusbaum
>    Assignee: Kyle Nusbaum
> Fix For: 1.0.0
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (STORM-1595) 'Fail' messages get stuck somewhere

[Kyle Nusbaum (JIRA)]

 [ 
https://issues.apache.org/jira/browse/STORM-1595?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kyle Nusbaum updated STORM-1595:

Attachment: (was: Screen Shot 2016-03-02 at 3.14.25 PM.png)

> 'Fail' messages get stuck somewhere 
> 
>
> Key: STORM-1595
> URL: https://issues.apache.org/jira/browse/STORM-1595
> Project: Apache Storm
>  Issue Type: Bug
>Affects Versions: 1.0.0, 2.0.0
>    Reporter: Kyle Nusbaum
> Attachments: screenshot-1.png
>
>
> 'Fail' acks seem to be getting stuck somewhere between the acker and the 
> spout. 
> After a long time - sometimes multiple minutes - the fails show up in the 
> spout.
> I tested this on master and 1.x-branch and it occurs in both places.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (STORM-1595) 'Fail' messages get stuck somewhere

[Kyle Nusbaum (JIRA)]

 [ 
https://issues.apache.org/jira/browse/STORM-1595?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kyle Nusbaum updated STORM-1595:

Attachment: screenshot-1.png

> 'Fail' messages get stuck somewhere 
> 
>
> Key: STORM-1595
> URL: https://issues.apache.org/jira/browse/STORM-1595
> Project: Apache Storm
>  Issue Type: Bug
>Affects Versions: 1.0.0, 2.0.0
>    Reporter: Kyle Nusbaum
> Attachments: screenshot-1.png
>
>
> 'Fail' acks seem to be getting stuck somewhere between the acker and the 
> spout. 
> After a long time - sometimes multiple minutes - the fails show up in the 
> spout.
> I tested this on master and 1.x-branch and it occurs in both places.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (STORM-1595) 'Fail' messages get stuck somewhere

[Kyle Nusbaum (JIRA)]

 [ 
https://issues.apache.org/jira/browse/STORM-1595?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kyle Nusbaum updated STORM-1595:

Attachment: Screen Shot 2016-03-02 at 3.14.25 PM.png

> 'Fail' messages get stuck somewhere 
> 
>
> Key: STORM-1595
> URL: https://issues.apache.org/jira/browse/STORM-1595
> Project: Apache Storm
>  Issue Type: Bug
>Affects Versions: 1.0.0, 2.0.0
>    Reporter: Kyle Nusbaum
> Attachments: Screen Shot 2016-03-02 at 3.14.25 PM.png
>
>
> 'Fail' acks seem to be getting stuck somewhere between the acker and the 
> spout. 
> After a long time - sometimes multiple minutes - the fails show up in the 
> spout.
> I tested this on master and 1.x-branch and it occurs in both places.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (STORM-1595) 'Fail' messages get stuck somewhere

[Kyle Nusbaum (JIRA)]

Kyle Nusbaum created STORM-1595:
---

 Summary: 'Fail' messages get stuck somewhere 
 Key: STORM-1595
 URL: https://issues.apache.org/jira/browse/STORM-1595
 Project: Apache Storm
  Issue Type: Bug
Affects Versions: 1.0.0, 2.0.0
Reporter: Kyle Nusbaum


'Fail' acks seem to be getting stuck somewhere between the acker and the spout. 

After a long time - sometimes multiple minutes - the fails show up in the spout.
I tested this on master and 1.x-branch and it occurs in both places.




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (STORM-1571) Improvment Kafka Spout Time Metric

[Kyle Nusbaum (JIRA)]

 [ 
https://issues.apache.org/jira/browse/STORM-1571?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kyle Nusbaum resolved STORM-1571.
-
   Resolution: Fixed
Fix Version/s: 2.0.0

> Improvment Kafka Spout Time Metric
> --
>
> Key: STORM-1571
> URL: https://issues.apache.org/jira/browse/STORM-1571
> Project: Apache Storm
>  Issue Type: Improvement
>  Components: storm-kafka
>Affects Versions: 0.9.3, 0.9.4, 0.9.5, 0.9.6
> Environment: Mac 10.11.1 JDK 1.8.0_40
>Reporter: darion yaphet
>Assignee: darion yaphet
>Priority: Minor
> Fix For: 2.0.0
>
> Attachments: 0001-Improvment-Kafka-Spout-Time-Metric.patch, 
> 0002-Update-time-interval-counting-on-TridentKafkaEmitter.patch
>
>
> Use System.currentTimeMillis() to calculation time interval is better than 
> System.nanoTime() 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (STORM-1226) Port backtype.storm.util to java

[Kyle Nusbaum (JIRA)]

 [ 
https://issues.apache.org/jira/browse/STORM-1226?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kyle Nusbaum reassigned STORM-1226:
---

Assignee: Kyle Nusbaum  (was: Reza Farivar)

> Port backtype.storm.util to java
> 
>
> Key: STORM-1226
> URL: https://issues.apache.org/jira/browse/STORM-1226
> Project: Apache Storm
>  Issue Type: New Feature
>  Components: storm-core
>Reporter: Robert Joseph Evans
>Assignee: Kyle Nusbaum
>  Labels: java-migration, jstorm-merger
> Fix For: 2.0.0
>
>
> Port backtype.storm.util from clojure to java.  In as many instances as 
> possible the same interface should be maintained, and calls to clojure 
> functions in the rest of the code should be replaces with calls to the 
> corresponding java code.
> Some similar functions can be found at  
> https://github.com/apache/storm/blob/jstorm-import/jstorm-core/src/main/java/com/alibaba/jstorm/utils/JStormUtils.java
> Although they are not identical.
> For function callbacks we may need to evaluate adding in appropriate callback 
> interfaces instead.  Please try to avoid using clojure internal java classes 
> unless necessary.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (STORM-1226) Port backtype.storm.util to java

[Kyle Nusbaum (JIRA)]

 [ 
https://issues.apache.org/jira/browse/STORM-1226?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kyle Nusbaum resolved STORM-1226.
-
   Resolution: Fixed
Fix Version/s: 2.0.0

> Port backtype.storm.util to java
> 
>
> Key: STORM-1226
> URL: https://issues.apache.org/jira/browse/STORM-1226
> Project: Apache Storm
>  Issue Type: New Feature
>  Components: storm-core
>Reporter: Robert Joseph Evans
>Assignee: Kyle Nusbaum
>  Labels: java-migration, jstorm-merger
> Fix For: 2.0.0
>
>
> Port backtype.storm.util from clojure to java.  In as many instances as 
> possible the same interface should be maintained, and calls to clojure 
> functions in the rest of the code should be replaces with calls to the 
> corresponding java code.
> Some similar functions can be found at  
> https://github.com/apache/storm/blob/jstorm-import/jstorm-core/src/main/java/com/alibaba/jstorm/utils/JStormUtils.java
> Although they are not identical.
> For function callbacks we may need to evaluate adding in appropriate callback 
> interfaces instead.  Please try to avoid using clojure internal java classes 
> unless necessary.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (STORM-1538) Exception being thrown after Utils conversion to java

[Kyle Nusbaum (JIRA)]

 [ 
https://issues.apache.org/jira/browse/STORM-1538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kyle Nusbaum updated STORM-1538:

Description: 
1. TTransportException thrown when trying to visit topology page in UI:

When trying to access the topology page on the storm ui, exceptions are thrown.

Visiting a topology page displays a TTransportException.

The cause can be seen in nimbus.log:

java.lang.IllegalArgumentException: Don't know how to create ISeq from: 
clojure.core$_PLUS_
at clojure.lang.RT.seqFrom(RT.java:528)
at clojure.lang.RT.seq(RT.java:509)
at clojure.core$seq__4128.invoke(core.clj:137)
at clojure.core.protocols$seq_reduce.invoke(protocols.clj:26)
at clojure.core.protocols$fn__6500.invoke(protocols.clj:83)
at 
clojure.core.protocols$fn__6452$G__6447__6465.invoke(protocols.clj:13)
at clojure.core$reduce.invoke(core.clj:6515)
at 
org.apache.storm.stats$agg_pre_merge_topo_page_bolt.invoke(stats.clj:564)
at 
org.apache.storm.stats$agg_topo_exec_stats_STAR_.invoke(stats.clj:744)
at org.apache.storm.stats$fn__1257.invoke(stats.clj:792)
at clojure.lang.MultiFn.invoke(MultiFn.java:243)
at clojure.core$partial$fn__4529.invoke(core.clj:2501)
at clojure.core.protocols$fn__6523.invoke(protocols.clj:167)
at 
clojure.core.protocols$fn__6478$G__6473__6487.invoke(protocols.clj:19)
at clojure.core.protocols$seq_reduce.invoke(protocols.clj:31)
at clojure.core.protocols$fn__6506.invoke(protocols.clj:101)
at 
clojure.core.protocols$fn__6452$G__6447__6465.invoke(protocols.clj:13)
at clojure.core$reduce.invoke(core.clj:6519)
at org.apache.storm.stats$aggregate_topo_stats.invoke(stats.clj:874)
at org.apache.storm.stats$agg_topo_execs_stats.invoke(stats.clj:1037)
at 
org.apache.storm.daemon.nimbus$fn__4817$exec_fn__2064__auto__$reify__4846.getTopologyPageInfo(nimbus.clj:2105)
at 
org.apache.storm.generated.Nimbus$Processor$getTopologyPageInfo.getResult(Nimbus.java:3800)
at 
org.apache.storm.generated.Nimbus$Processor$getTopologyPageInfo.getResult(Nimbus.java:3784)
at 
org.apache.storm.thrift.ProcessFunction.process(ProcessFunction.java:39)
at 
org.apache.storm.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
at 
org.apache.storm.security.auth.SimpleTransportPlugin$SimpleWrapProcessor.process(SimpleTransportPlugin.java:158)
at 
org.apache.storm.thrift.server.AbstractNonblockingServer$FrameBuffer.invoke(AbstractNonblockingServer.java:518)
at org.apache.storm.thrift.server.Invocation.run(Invocation.java:18)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)


2. Exception thrown in supervisor: 

java.lang.ClassCastException: java.lang.Integer cannot be cast to 
java.lang.String
at 
org.apache.storm.daemon.supervisor$create_artifacts_link.invoke(supervisor.clj:1083)
at 
org.apache.storm.daemon.supervisor$fn__7494.invoke(supervisor.clj:1180)
at clojure.lang.MultiFn.invoke(MultiFn.java:251)
at 
org.apache.storm.daemon.supervisor$get_valid_new_worker_ids$iter__7081__7085$fn__7086.invoke(supervisor.clj:393)
at clojure.lang.LazySeq.sval(LazySeq.java:40)
at clojure.lang.LazySeq.seq(LazySeq.java:49)
at clojure.lang.RT.seq(RT.java:507)
at clojure.core$seq__4128.invoke(core.clj:137)
at clojure.core$dorun.invoke(core.clj:3009)
at clojure.core$doall.invoke(core.clj:3025)
at 
org.apache.storm.daemon.supervisor$get_valid_new_worker_ids.invoke(supervisor.clj:380)
at 
org.apache.storm.daemon.supervisor$sync_processes.invoke(supervisor.clj:447)
at clojure.core$partial$fn__4527.invoke(core.clj:2492)
at org.apache.storm.event$event_manager$fn__6781.invoke(event.clj:40)
at clojure.lang.AFn.run(AFn.java:22)
at java.lang.Thread.run(Thread.java:745)

3. Exception in UI:

java.lang.ClassCastException: java.lang.String cannot be cast to 
java.lang.Number
at org.apache.storm.ui.core$worker_log_link.invoke(core.clj:150)
at org.apache.storm.ui.core$get_error_json.invoke(core.clj:531)
at org.apache.storm.ui.core$fn__11747.invoke(core.clj:553)
at clojure.lang.MultiFn.invoke(MultiFn.java:238)
at clojure.core$partial$fn__4529.invoke(core.clj:2500)
at clojure.core$map$fn__4553.invoke(core.clj:2624)
at clojure.lang.LazySeq.sval(LazySeq.java:40)
at clojure.lang.LazySeq.seq(LazySeq.java:49)
at clojure.lang.RT.seq(RT.java:507)
at clojure.lang.SeqIterator.hasNext(SeqIterator.java:38)

  was:
1. TTransportException thrown when trying to visit topology page in UI:

When trying

Re: [DISCUSSION] Repo cleanup

[Kyle Nusbaum]

Great. 

I quit deleting branches from the ASF repo before I got too far because I 
noticed github wasn't mirroring the deletes. 

Taylor, do you have access to any special tool to do stuff like this, or should 
I file a ticket with infra?

I deleted the following branches from the ASF repo before stopping:
STORM-633
STORM-648
STORM-651
STORM-1153-V1


-- Kyle 

On Sunday, January 24, 2016 5:52 PM, Jungtaek Lim <kabh...@gmail.com> wrote:
 

 +1 for cleanup.

- Jungtaek Lim (HeartSaVioR)

2016년 1월 23일 (토) 오전 7:49, Kyle Nusbaum <knusb...@yahoo-inc.com.invalid>님이
작성:

> Please check out this jira and let me know what you all think. I won't
> take any action until I get some responses here.
>
> Even if I get positive reaction here, I'm going to leave it up to the
> owners of the unmerged branches to take care of them. I will only delete
> merged branches.
> https://issues.apache.org/jira/browse/STORM-1497
>  -- Kyle

  

Re: [DISCUSSION] Restructure Storm documentation

[Kyle Nusbaum]

https://issues.apache.org/jira/browse/STORM-1502 -- Kyle 

On Sunday, January 24, 2016 4:29 PM, Jungtaek Lim <kabh...@gmail.com> wrote:
 

 +1 for having version specific docs.

We move docs to asf-site branch, and now we have only one REST API doc. for
1.0.0 and later versions.

It would also great to describe available versions for each API, but just
having whole copy may be easier.
Same things may be applied to new features.

- Jungtaek Lim (HeartSaVioR)

2016년 1월 25일 (월) 오전 12:19, Matthias J. Sax <mj...@apache.org>님이 작성:

> +1 for having documentation for older releases on the website and
> JavaDocs for each version, too.
>
> Btw: in the Flink project this process is automated completely. I am not
> sure exactly how, but could figure it out. However, the documentation is
> not part of the project website itself but hosted at ci.apache.org
>
> Having this automated, is very nice for people how are using the current
> Snapshot version, as the new docs get available very soon when something
> changes.
>
> -Matthias
>
>
> On 01/22/2016 11:59 PM, Nathan Marz wrote:
> > At the very least, the Javadocs should be available by version. This is
> > something I used to do but looks like we forgot to keep doing that after
> > the transition to Apache. Maintaining other docs (tutorials, etc.) by
> > version is more difficult as those are rarely updated at the time of
> > release.
> >
> > On Fri, Jan 22, 2016 at 2:01 PM, Bobby Evans <ev...@yahoo-inc.com>
> wrote:
> >
> >> It doesn't have to be Taylor cutting releases.  The only major
> requirement
> >> around that is that the PMC votes on the release.
> >>  - Bobby
> >>
> >>    On Friday, January 22, 2016 3:48 PM, Kyle Nusbaum
> >> <knusb...@yahoo-inc.com.INVALID> wrote:
> >>
> >>
> >>  Yep, That's precisely what I was thinking.
> >>
> >> I don't really see a problem with the process being manual. It won't be
> >> *too* much work, and we do releases infrequently enough that I don't
> see it
> >> as a burden. A small helper script would probably be trivial to write.
> >>
> >> Of course, Taylor is the one cutting the releases, so I'll defer to him
> on
> >> the automated/manual issue. -- Kyle
> >>
> >>    On Friday, January 22, 2016 3:45 PM, P. Taylor Goetz <
> >> ptgo...@gmail.com> wrote:
> >>
> >>
> >>  I’m definitely open to improving the process such that we can have
> >> version-specific documentation, and finding a way to automate updating
> the
> >> asf-site branch during the release process. I’m also okay if that
> process
> >> is somewhat manual.
> >>
> >> I’ve thought about it a little but haven’t really come with a process.
> >>
> >> Ideally we’d do something that would do a snapshot of the docs at
> release
> >> time and create a subdirectory in the asf-site website (e.g.
> “1.0.0-docs”).
> >>
> >> I’m open to suggestions.
> >>
> >> -Taylor
> >>
> >>> On Jan 22, 2016, at 4:25 PM, Kyle Nusbaum <knusb...@yahoo-inc.com
> .INVALID>
> >> wrote:
> >>>
> >>> The new website is awesome.
> >>>
> >>> Tt would be great to keep tabs on documentation for different versions
> >> of Storm and host those different versions on the site.
> >>>
> >>> I don't care too much for having all the documentation in its own
> >> branch. I would suggest that each version branch of Storm keeps its own
> >> version of the docs -- or keeps any modifications to the docs, if not
> the
> >> entire collection, in order to keep the common parts in sync -- and that
> >> these docs get merged into the asf-site branch in their own version
> >> directory as part of the release process.
> >>> Please let me know what you think and I'll file Jira issues as
> >> necessary.-- Kyle
> >>
> >>
> >>
> >>
> >>
> >>
> >
> >
> >
>
>

  

[jira] [Created] (STORM-1502) Create per-version document scheme

[Kyle Nusbaum (JIRA)]

Kyle Nusbaum created STORM-1502:
---

 Summary: Create per-version document scheme
 Key: STORM-1502
 URL: https://issues.apache.org/jira/browse/STORM-1502
 Project: Apache Storm
  Issue Type: Improvement
Reporter: Kyle Nusbaum


We want to modify our documentation scheme such that each version of storm has 
its own set of documentation published to the apache storm website. 

General consensus was that at least javadocs should be available per-version, 
but it is okay if parts of the process require manual intervention.

The full conversation about it can be found here:
http://mail-archives.apache.org/mod_mbox/storm-dev/201601.mbox/browser





--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[DISCUSSION] Restructure Storm documentation

[Kyle Nusbaum]

The new website is awesome. 

Tt would be great to keep tabs on documentation for different versions of Storm 
and host those different versions on the site. 

I don't care too much for having all the documentation in its own branch. I 
would suggest that each version branch of Storm keeps its own version of the 
docs -- or keeps any modifications to the docs, if not the entire collection, 
in order to keep the common parts in sync -- and that these docs get merged 
into the asf-site branch in their own version directory as part of the release 
process.
Please let me know what you think and I'll file Jira issues as necessary.-- Kyle

Re: [DISCUSSION] Restructure Storm documentation

[Kyle Nusbaum]

Yep, That's precisely what I was thinking. 

I don't really see a problem with the process being manual. It won't be *too* 
much work, and we do releases infrequently enough that I don't see it as a 
burden. A small helper script would probably be trivial to write.

Of course, Taylor is the one cutting the releases, so I'll defer to him on the 
automated/manual issue. -- Kyle 

On Friday, January 22, 2016 3:45 PM, P. Taylor Goetz <ptgo...@gmail.com> 
wrote:
 

 I’m definitely open to improving the process such that we can have 
version-specific documentation, and finding a way to automate updating the 
asf-site branch during the release process. I’m also okay if that process is 
somewhat manual.

I’ve thought about it a little but haven’t really come with a process.

Ideally we’d do something that would do a snapshot of the docs at release time 
and create a subdirectory in the asf-site website (e.g. “1.0.0-docs”).

I’m open to suggestions.

-Taylor

> On Jan 22, 2016, at 4:25 PM, Kyle Nusbaum <knusb...@yahoo-inc.com.INVALID> 
> wrote:
> 
> The new website is awesome.
> 
> Tt would be great to keep tabs on documentation for different versions of 
> Storm and host those different versions on the site.
> 
> I don't care too much for having all the documentation in its own branch. I 
> would suggest that each version branch of Storm keeps its own version of the 
> docs -- or keeps any modifications to the docs, if not the entire collection, 
> in order to keep the common parts in sync -- and that these docs get merged 
> into the asf-site branch in their own version directory as part of the 
> release process.
> Please let me know what you think and I'll file Jira issues as necessary.-- 
> Kyle


  

[jira] [Commented] (STORM-1257) port backtype.storm.zookeeper to java

[Kyle Nusbaum (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15112905#comment-15112905
 ] 

Kyle Nusbaum commented on STORM-1257:
-

Is this actively being worked on? If not I will pick this issue up.

> port backtype.storm.zookeeper to java
> -
>
> Key: STORM-1257
> URL: https://issues.apache.org/jira/browse/STORM-1257
> Project: Apache Storm
>  Issue Type: New Feature
>  Components: storm-core
>Reporter: Robert Joseph Evans
>Assignee: Basti Liu
>  Labels: java-migration, jstorm-merger
>
> A wrapper around zookeeper/curator.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (STORM-1497) Repo Branches Cleanup

[Kyle Nusbaum (JIRA)]

Kyle Nusbaum created STORM-1497:
---

 Summary: Repo Branches Cleanup 
 Key: STORM-1497
 URL: https://issues.apache.org/jira/browse/STORM-1497
 Project: Apache Storm
  Issue Type: Improvement
Reporter: Kyle Nusbaum
Assignee: Kyle Nusbaum


I've identified a bunch of branches that I think can go away:

Already merged:
STORM-633
STORM-648
STORM-651
STORM-1153-V1
better-trident-spouts
experimental
clojure1.3
clojure1.4
executors
gettuples
kryo-2.16
mesos
new-resource-scheduler
nimbus-ha-branch
scheduler
security
transactional-spout
ui-url

Branches not merged, but I think we can delete:
STORM-1040 (harschach)
PR_736 (ptgoetz)
nimbus-ha (nathanmarz)
debug (do not want)  (nathanmarz)
iso-scheduler (not merged but VERY old) (nathanmarz)
jarjar (not merged but VERY old) (nathanmarz)
statespout (not merged but VERY old) (nathanmarz)
std-redirect (not merged but VERY old) (nathanmarz)
superclojure (not merged but VERY old) (nathanmarz)
thunk (VERY old, don't want) (sritchie)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[DISCUSSION] Repo cleanup

[Kyle Nusbaum]

Please check out this jira and let me know what you all think. I won't take any 
action until I get some responses here. 

Even if I get positive reaction here, I'm going to leave it up to the owners of 
the unmerged branches to take care of them. I will only delete merged branches.
https://issues.apache.org/jira/browse/STORM-1497
 -- Kyle

[jira] [Created] (STORM-1467) Switch apache-rat plugin off by default, but enable for Travis-CI

[Kyle Nusbaum (JIRA)]

Kyle Nusbaum created STORM-1467:
---

 Summary: Switch apache-rat plugin off by default, but enable for 
Travis-CI
 Key: STORM-1467
 URL: https://issues.apache.org/jira/browse/STORM-1467
 Project: Apache Storm
  Issue Type: Bug
Reporter: Kyle Nusbaum


apache-rat is super annoying when developing. We should change it so that it 
doesn't run all the time, but can be manually triggered and runs in CI.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (STORM-1466) Move the org.apache.thrift7 namespace to something correct/sensible

[Kyle Nusbaum (JIRA)]

 [ 
https://issues.apache.org/jira/browse/STORM-1466?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kyle Nusbaum reassigned STORM-1466:
---

Assignee: Kyle Nusbaum

> Move the org.apache.thrift7 namespace to something correct/sensible
> ---
>
> Key: STORM-1466
> URL: https://issues.apache.org/jira/browse/STORM-1466
> Project: Apache Storm
>  Issue Type: Bug
>        Reporter: Kyle Nusbaum
>    Assignee: Kyle Nusbaum
>
> Thrift is still shaded to org.apache.thrift7 even though we are not using 
> Thrift 7.
> We should also add something for temporary backwards compatibility.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (STORM-1466) Move the org.apache.thrift7 namespace to something correct/sensible

[Kyle Nusbaum (JIRA)]

 [ 
https://issues.apache.org/jira/browse/STORM-1466?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kyle Nusbaum updated STORM-1466:

Description: 
Thrift is still shaded to org.apache.thrift7 even though we are not using 
Thrift 7.

We should also add something for temporary backwards compatibility.

  was:Thrift is still shaded to org.apache.thrift7 even though we are not using 
Thrift 7.


> Move the org.apache.thrift7 namespace to something correct/sensible
> ---
>
> Key: STORM-1466
> URL: https://issues.apache.org/jira/browse/STORM-1466
> Project: Apache Storm
>  Issue Type: Bug
>        Reporter: Kyle Nusbaum
>
> Thrift is still shaded to org.apache.thrift7 even though we are not using 
> Thrift 7.
> We should also add something for temporary backwards compatibility.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (STORM-1466) Move the org.apache.thrift7 namespace to something correct/sensible

[Kyle Nusbaum (JIRA)]

Kyle Nusbaum created STORM-1466:
---

 Summary: Move the org.apache.thrift7 namespace to something 
correct/sensible
 Key: STORM-1466
 URL: https://issues.apache.org/jira/browse/STORM-1466
 Project: Apache Storm
  Issue Type: Bug
Reporter: Kyle Nusbaum


Thrift is still shaded to org.apache.thrift7 even though we are not using 
Thrift 7.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Java 8 for Storm 2.x?

[Kyle Nusbaum]

I agree with Derek.
There are still a lot of people using Java 7. However, if you could make it 
sort of Java 8 compatible so that when we do move to Java 8 we can do so with 
relative ease.
 -- Kyle 

On Monday, January 11, 2016 4:46 PM, Derek Dagit 
 wrote:
 

 I am not sure it makes sense to move to Java 8 merely because of the 
clojure->java translation, but it might be good timing so I would be OK with it.

 -- 
Derek


- Original Message -
From: Reza Farivar 
To: Dev 
Sent: Monday, January 11, 2016 4:40 PM
Subject: Java 8 for Storm 2.x?

I have started work on translating the util.clj to java (STORM-1226). I see 
some instances when translating the functional behavior of clojure to Java 
requires code that is already part of java 8.
For instance, there are many cases where a predicate function is passed around 
(e.g. in find-first). I can go ahead and implement a Predicate interface and 
use that, but Java 8 has already exactly that functionality implemented. 
Would it make sense to move to Java 8 for the post-clojure branch 2.x?
--Reza 


  

[jira] [Created] (STORM-1432) Spurious failure in storm-kafka test

[Kyle Nusbaum (JIRA)]

Kyle Nusbaum created STORM-1432:
---

 Summary: Spurious failure in storm-kafka test
 Key: STORM-1432
 URL: https://issues.apache.org/jira/browse/STORM-1432
 Project: Apache Storm
  Issue Type: Bug
Reporter: Kyle Nusbaum
Assignee: Kyle Nusbaum


ExponentialBackoffMsgRetryManagerTest.testMaxBackoff() fails sometimes, I think 
due to too small times. It can miss the mark on slow machines.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (STORM-1376) ZK Becoming deadlocked with zookeeper_state_factory

[Kyle Nusbaum (JIRA)]

 [ 
https://issues.apache.org/jira/browse/STORM-1376?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kyle Nusbaum reassigned STORM-1376:
---

Assignee: Kyle Nusbaum  (was: Sanket Reddy)

> ZK Becoming deadlocked with zookeeper_state_factory
> ---
>
> Key: STORM-1376
> URL: https://issues.apache.org/jira/browse/STORM-1376
> Project: Apache Storm
>  Issue Type: Bug
>  Components: storm-core
>Affects Versions: 0.11.0
>Reporter: Daniel Schonfeld
>    Assignee: Kyle Nusbaum
>Priority: Blocker
> Fix For: 0.11.0
>
>
> Since the introduction of blobstore and pacemaker we've noticed that when 
> using nimbus with the new zookeeper_state_factory backing cluster state 
> module, some of our ZK nodes become unresponsive and show and increasing 
> amounts of outstanding requests (STAT 4-letter command).
> Terminating storm supervisors and nimbus usually gets zookeeper to realize 
> after a few minutes those connections are dead and to become responsive 
> again.  In some extreme cases we have to kill that ZK nodes and bring it back 
> up.
> Our topologies ran across ~10 supervisor nodes with each having about 
> ~400-500 executors. 
> I mention the amount of executors cause I am not sure if someone made each 
> executor by mistake start sending heartbeats instead of each worker and that 
> might possibly be the reason for this slow down.
> Final note.  If someone can jot a few ideas of why this might be happening 
> i'd be more than happy to dig further in the storm code and submit a PR 
> myself.  But I need some hint or direction of where to go with this...



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (STORM-1393) Sort out storm.log.dir configure and add documentation about logs

[Kyle Nusbaum (JIRA)]

 [ 
https://issues.apache.org/jira/browse/STORM-1393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kyle Nusbaum resolved STORM-1393.
-
Resolution: Fixed

> Sort out storm.log.dir configure and add documentation about logs
> -
>
> Key: STORM-1393
> URL: https://issues.apache.org/jira/browse/STORM-1393
> Project: Apache Storm
>  Issue Type: Improvement
>  Components: storm-core
>Affects Versions: 0.11.0
>Reporter: Zhuo Liu
>Assignee: Zhuo Liu
>Priority: Minor
> Fix For: 0.11.0
>
>
> Currently, we have reorganized logs in STORM-901 and STORM-1387, it is 
> preferable for us to document the changes out for avoiding confusion to 
> users. 
> Also, the util/LOG-DIR and the way supervisor to get storm.log.dir in 
> worker-launch is inaccurate since it does not take the storm-conf into 
> account. We should fix it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (STORM-1381) Client side topology submission hook.

[Kyle Nusbaum (JIRA)]

 [ 
https://issues.apache.org/jira/browse/STORM-1381?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kyle Nusbaum resolved STORM-1381.
-
Resolution: Fixed

> Client side topology submission hook.
> -
>
> Key: STORM-1381
> URL: https://issues.apache.org/jira/browse/STORM-1381
> Project: Apache Storm
>  Issue Type: New Feature
>  Components: storm-core
>Affects Versions: 0.11.0
>Reporter: Parth Brahmbhatt
>Assignee: Parth Brahmbhatt
>Priority: Trivial
> Fix For: 0.11.0
>
>
> A client side hook is suppose to be invoked when a user submits the topology 
> using TopologySubmitter. We already have nimbus side hook for all the 
> topology actions however those are good if users don't want to actually 
> inspect the topology being submitted or the classes that makes up the 
> topology (spouts and bolts) as on nimbus side these classes are not available 
> in class path. 
> As a concrete example, in hortonworks we wanted to integrate storm with atlas 
> to provide complete lineage of data even when it passes through a storm 
> topology. Atlas needed to actually look inside the topology components (i.e. 
> kafka spout to figure out what topic the data is being pulled from, or hbase 
> bolt to figure out which cluster and what table data is being pushed into.) 
> to give a meaningful lineage. We originally proposed that they use the server 
> side hook but with that they had to download the user uploaded jar and add it 
> to the class path dynamically or spin a new jvn whose output will then be 
> read by the atlas integration hook. 
> The client side hook is suppose to make it easy when the topology itself 
> needs to be examined. We are using this in our internal repo for atlas 
> integration.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (STORM-1395) Move JUnit dependency to top-level pom

[Kyle Nusbaum (JIRA)]

 [ 
https://issues.apache.org/jira/browse/STORM-1395?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kyle Nusbaum resolved STORM-1395.
-
   Resolution: Fixed
Fix Version/s: 0.11.0

> Move JUnit dependency to top-level pom
> --
>
> Key: STORM-1395
> URL: https://issues.apache.org/jira/browse/STORM-1395
> Project: Apache Storm
>  Issue Type: Bug
>Affects Versions: 0.11.0
>Reporter: P. Taylor Goetz
>Assignee: P. Taylor Goetz
> Fix For: 0.11.0
>
>
> The switch to profiles for running different types of tests requires that 
> certain submodules explicitly include the JUnit dependency. Moving it to the 
> top level pom will allow all submodules to inherit it.
> Background: the build was failing in my environment because storm-redis did 
> not have the dependency.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (STORM-1372) Update BlobStore Documentation - Follow up STORM-876

[Kyle Nusbaum (JIRA)]

 [ 
https://issues.apache.org/jira/browse/STORM-1372?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kyle Nusbaum resolved STORM-1372.
-
   Resolution: Fixed
Fix Version/s: 0.11.0

> Update BlobStore Documentation - Follow up STORM-876
> 
>
> Key: STORM-1372
> URL: https://issues.apache.org/jira/browse/STORM-1372
> Project: Apache Storm
>  Issue Type: Story
>Reporter: Sanket Reddy
>Priority: Minor
> Fix For: 0.11.0
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (STORM-1376) ZK Becoming deadlocked with zookeeper_state_factory

[Kyle Nusbaum (JIRA)]

 [ 
https://issues.apache.org/jira/browse/STORM-1376?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kyle Nusbaum updated STORM-1376:

Assignee: Sanket Reddy  (was: Kyle Nusbaum)

> ZK Becoming deadlocked with zookeeper_state_factory
> ---
>
> Key: STORM-1376
> URL: https://issues.apache.org/jira/browse/STORM-1376
> Project: Apache Storm
>  Issue Type: Bug
>  Components: storm-core
>Affects Versions: 0.11.0
>Reporter: Daniel Schonfeld
>Assignee: Sanket Reddy
>Priority: Blocker
> Fix For: 0.11.0
>
>
> Since the introduction of blobstore and pacemaker we've noticed that when 
> using nimbus with the new zookeeper_state_factory backing cluster state 
> module, some of our ZK nodes become unresponsive and show and increasing 
> amounts of outstanding requests (STAT 4-letter command).
> Terminating storm supervisors and nimbus usually gets zookeeper to realize 
> after a few minutes those connections are dead and to become responsive 
> again.  In some extreme cases we have to kill that ZK nodes and bring it back 
> up.
> Our topologies ran across ~10 supervisor nodes with each having about 
> ~400-500 executors. 
> I mention the amount of executors cause I am not sure if someone made each 
> executor by mistake start sending heartbeats instead of each worker and that 
> might possibly be the reason for this slow down.
> Final note.  If someone can jot a few ideas of why this might be happening 
> i'd be more than happy to dig further in the storm code and submit a PR 
> myself.  But I need some hint or direction of where to go with this...



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (STORM-1376) ZK Becoming deadlocked with zookeeper_state_factory

[Kyle Nusbaum (JIRA)]

 [ 
https://issues.apache.org/jira/browse/STORM-1376?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kyle Nusbaum resolved STORM-1376.
-
   Resolution: Fixed
Fix Version/s: 0.11.0

> ZK Becoming deadlocked with zookeeper_state_factory
> ---
>
> Key: STORM-1376
> URL: https://issues.apache.org/jira/browse/STORM-1376
> Project: Apache Storm
>  Issue Type: Bug
>  Components: storm-core
>Affects Versions: 0.11.0
>Reporter: Daniel Schonfeld
>Assignee: Sanket Reddy
>Priority: Blocker
> Fix For: 0.11.0
>
>
> Since the introduction of blobstore and pacemaker we've noticed that when 
> using nimbus with the new zookeeper_state_factory backing cluster state 
> module, some of our ZK nodes become unresponsive and show and increasing 
> amounts of outstanding requests (STAT 4-letter command).
> Terminating storm supervisors and nimbus usually gets zookeeper to realize 
> after a few minutes those connections are dead and to become responsive 
> again.  In some extreme cases we have to kill that ZK nodes and bring it back 
> up.
> Our topologies ran across ~10 supervisor nodes with each having about 
> ~400-500 executors. 
> I mention the amount of executors cause I am not sure if someone made each 
> executor by mistake start sending heartbeats instead of each worker and that 
> might possibly be the reason for this slow down.
> Final note.  If someone can jot a few ideas of why this might be happening 
> i'd be more than happy to dig further in the storm code and submit a PR 
> myself.  But I need some hint or direction of where to go with this...



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (STORM-1388) Fix url and email links in README file

[Kyle Nusbaum (JIRA)]

 [ 
https://issues.apache.org/jira/browse/STORM-1388?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kyle Nusbaum resolved STORM-1388.
-
   Resolution: Fixed
Fix Version/s: 0.11.0

> Fix url and email links in README file
> --
>
> Key: STORM-1388
> URL: https://issues.apache.org/jira/browse/STORM-1388
> Project: Apache Storm
>  Issue Type: Bug
>  Components: documentation
>Reporter: Xin Wang
>Assignee: Xin Wang
>Priority: Minor
> Fix For: 0.11.0
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (STORM-1397) Merge conflict from Pacemaker merge

[Kyle Nusbaum (JIRA)]

Kyle Nusbaum created STORM-1397:
---

 Summary: Merge conflict from Pacemaker merge
 Key: STORM-1397
 URL: https://issues.apache.org/jira/browse/STORM-1397
 Project: Apache Storm
  Issue Type: Bug
Reporter: Kyle Nusbaum


When Pacemaker was merged, some calls got missed in nimbus.clj, supervisor.clj, 
and worker.clj.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (STORM-1397) Merge conflict from Pacemaker merge

[Kyle Nusbaum (JIRA)]

 [ 
https://issues.apache.org/jira/browse/STORM-1397?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kyle Nusbaum updated STORM-1397:

Description: When Pacemaker was merged, some calls got missed in 
nimbus.clj, supervisor.clj, worker.clj and executor.clj.  (was: When Pacemaker 
was merged, some calls got missed in nimbus.clj, supervisor.clj, and 
worker.clj.)

> Merge conflict from Pacemaker merge
> ---
>
> Key: STORM-1397
> URL: https://issues.apache.org/jira/browse/STORM-1397
> Project: Apache Storm
>  Issue Type: Bug
>        Reporter: Kyle Nusbaum
>
> When Pacemaker was merged, some calls got missed in nimbus.clj, 
> supervisor.clj, worker.clj and executor.clj.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (STORM-1397) Merge conflict from Pacemaker merge

[Kyle Nusbaum (JIRA)]

 [ 
https://issues.apache.org/jira/browse/STORM-1397?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kyle Nusbaum updated STORM-1397:

Description: 
When Pacemaker was merged, some calls got missed in nimbus.clj, supervisor.clj, 
worker.clj and executor.clj.

This merge error prevents nimbus from spawning separate read/write zookeeper 
clients since the ClusterState ends up with daemonType == DaemonType.UNKNOWN. 

  was:When Pacemaker was merged, some calls got missed in nimbus.clj, 
supervisor.clj, worker.clj and executor.clj.


> Merge conflict from Pacemaker merge
> ---
>
> Key: STORM-1397
> URL: https://issues.apache.org/jira/browse/STORM-1397
> Project: Apache Storm
>  Issue Type: Bug
>        Reporter: Kyle Nusbaum
>
> When Pacemaker was merged, some calls got missed in nimbus.clj, 
> supervisor.clj, worker.clj and executor.clj.
> This merge error prevents nimbus from spawning separate read/write zookeeper 
> clients since the ClusterState ends up with daemonType == DaemonType.UNKNOWN. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (STORM-1359) Change kryo links from google code to github

[Kyle Nusbaum (JIRA)]

 [ 
https://issues.apache.org/jira/browse/STORM-1359?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kyle Nusbaum resolved STORM-1359.
-
   Resolution: Fixed
Fix Version/s: 0.11.0

> Change kryo links from google code to github
> 
>
> Key: STORM-1359
> URL: https://issues.apache.org/jira/browse/STORM-1359
> Project: Apache Storm
>  Issue Type: Bug
>  Components: documentation
>Reporter: Xin Wang
>Assignee: Xin Wang
>Priority: Minor
> Fix For: 0.11.0
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (STORM-1385) Divide by zero exception in stats.clj

[Kyle Nusbaum (JIRA)]

 [ 
https://issues.apache.org/jira/browse/STORM-1385?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kyle Nusbaum resolved STORM-1385.
-
   Resolution: Fixed
Fix Version/s: 0.11.0

> Divide by zero exception in stats.clj
> -
>
> Key: STORM-1385
> URL: https://issues.apache.org/jira/browse/STORM-1385
> Project: Apache Storm
>  Issue Type: Bug
>Reporter: Sanket Reddy
>Assignee: Sanket Reddy
>Priority: Minor
> Fix For: 0.11.0
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (STORM-1340) Use Travis-CI build matrix to improve test execution times

[Kyle Nusbaum (JIRA)]

Kyle Nusbaum created STORM-1340:
---

 Summary: Use Travis-CI build matrix to improve test execution times
 Key: STORM-1340
 URL: https://issues.apache.org/jira/browse/STORM-1340
 Project: Apache Storm
  Issue Type: Improvement
Reporter: Kyle Nusbaum
Assignee: Kyle Nusbaum






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (STORM-1340) Use Travis-CI build matrix to improve test execution times

[Kyle Nusbaum (JIRA)]

 [ 
https://issues.apache.org/jira/browse/STORM-1340?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kyle Nusbaum updated STORM-1340:

Description: 
Travis-CI provides a 'build-matrix' that we can use to parallelize the unit 
tests of various components in storm.
It also provides a caching mechanism that can reduce the overhead of starting 
up a test container.

> Use Travis-CI build matrix to improve test execution times
> --
>
> Key: STORM-1340
> URL: https://issues.apache.org/jira/browse/STORM-1340
> Project: Apache Storm
>  Issue Type: Improvement
>        Reporter: Kyle Nusbaum
>    Assignee: Kyle Nusbaum
>
> Travis-CI provides a 'build-matrix' that we can use to parallelize the unit 
> tests of various components in storm.
> It also provides a caching mechanism that can reduce the overhead of starting 
> up a test container.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (STORM-885) Heartbeat Server (Pacemaker)

[Kyle Nusbaum (JIRA)]

 [ 
https://issues.apache.org/jira/browse/STORM-885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kyle Nusbaum resolved STORM-885.

   Resolution: Fixed
Fix Version/s: 0.11.0

> Heartbeat Server (Pacemaker)
> 
>
> Key: STORM-885
> URL: https://issues.apache.org/jira/browse/STORM-885
> Project: Apache Storm
>  Issue Type: Improvement
>  Components: storm-core
>Reporter: Robert Joseph Evans
>Assignee: Kyle Nusbaum
> Fix For: 0.11.0
>
>
> Large highly connected topologies and large clusters write a lot of data into 
> ZooKeeper.  The heartbeats, that make up the majority of this data, do not 
> need to be persisted to disk.  Pacemaker is intended to be a secure 
> replacement for storing the heartbeats without changing anything within the 
> heartbeats.  In the future as more metrics are added in, we may want to look 
> into switching it over to look more like Heron, where a metrics server is 
> running for each node/topology.  And can be used to aggregate/per-aggregate 
> them in a more scalable manor.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (STORM-885) Heartbeat Server (Pacemaker)

[Kyle Nusbaum (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15011663#comment-15011663
 ] 

Kyle Nusbaum edited comment on STORM-885 at 11/18/15 6:52 PM:
--

[~LongdaFeng]
I disagree with your assessment of this patch.

Addressing each of your numbered points:
1. I'm not sure what you mean by this. Will you please clarify?
2. Pacemaker may become a bottleneck in the system, yes. But in our internal 
testing, Nimbus itself becomes a bottleneck before Pacemaker. Furthermore, it 
is *not* very hard to do extension. I have a prototype here for Pacemaker HA 
that only adds a small number of lines of code, which I am planning on pushing 
out in the relatively near future.
3. The logic looks fairly straight-forward to me, and the pacemaker server file 
itself is only 239 lines long, with nearly 100 of those being the license, 
imports, and statistics helpers. The client is only 125 lines long. 

Pacemaker is also an optional component, and it's ready now. 
I see the benefit of a 'TopologyMaster', but I don't see the benefit in 
blocking the merge of Pacemaker at this point.


was (Author: knusbaum):
[~LongdaFeng]
I disagree with your assessment of this patch.

1. I'm not sure what you mean by this. Will you please clarify?
2. Pacemaker may become a bottleneck in the system, yes. But in our internal 
testing, Nimbus itself becomes a bottleneck before Pacemaker. Furthermore, it 
is *not* very hard to do extension. I have a prototype here for Pacemaker HA 
that only adds a small number of lines of code, which I am planning on pushing 
out in the relatively near future.
3. The logic looks fairly straight-forward to me, and the pacemaker server file 
itself is only 239 lines long, with nearly 100 of those being the license, 
imports, and statistics helpers. The client is only 125 lines long. 

Pacemaker is also an optional component, and it's ready now. 
I see the benefit of a 'TopologyMaster', but I don't see the benefit in 
blocking the merge of Pacemaker at this point.

> Heartbeat Server (Pacemaker)
> 
>
> Key: STORM-885
> URL: https://issues.apache.org/jira/browse/STORM-885
> Project: Apache Storm
>  Issue Type: Improvement
>  Components: storm-core
>Reporter: Robert Joseph Evans
>Assignee: Kyle Nusbaum
>
> Large highly connected topologies and large clusters write a lot of data into 
> ZooKeeper.  The heartbeats, that make up the majority of this data, do not 
> need to be persisted to disk.  Pacemaker is intended to be a secure 
> replacement for storing the heartbeats without changing anything within the 
> heartbeats.  In the future as more metrics are added in, we may want to look 
> into switching it over to look more like Heron, where a metrics server is 
> running for each node/topology.  And can be used to aggregate/per-aggregate 
> them in a more scalable manor.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (STORM-885) Heartbeat Server (Pacemaker)

[Kyle Nusbaum (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15011663#comment-15011663
 ] 

Kyle Nusbaum edited comment on STORM-885 at 11/18/15 7:05 PM:
--

[~LongdaFeng]
I disagree with your assessment of this patch.

Addressing each of your numbered points:
1. I'm not sure what you mean by this. Will you please clarify?
2. Pacemaker may become a bottleneck in the system, yes. But in our internal 
testing, Nimbus itself becomes a bottleneck before Pacemaker. Furthermore, it 
is *not* very hard to do extension. I have a prototype here for Pacemaker HA 
that only adds a small number of lines of code, which I am planning on pushing 
out in the relatively near future.
3. The logic looks fairly straight-forward to me, and the pacemaker server file 
itself is only 239 lines long, with nearly 100 of those being the license, 
imports, and statistics helpers. The client is only 125 lines long. 

Pacemaker is also an optional component, and it's ready now. 
A TopologyMaster also does not reduce the total amount of data written to 
ZooKeeper, whereas Pacemaker does.


was (Author: knusbaum):
[~LongdaFeng]
I disagree with your assessment of this patch.

Addressing each of your numbered points:
1. I'm not sure what you mean by this. Will you please clarify?
2. Pacemaker may become a bottleneck in the system, yes. But in our internal 
testing, Nimbus itself becomes a bottleneck before Pacemaker. Furthermore, it 
is *not* very hard to do extension. I have a prototype here for Pacemaker HA 
that only adds a small number of lines of code, which I am planning on pushing 
out in the relatively near future.
3. The logic looks fairly straight-forward to me, and the pacemaker server file 
itself is only 239 lines long, with nearly 100 of those being the license, 
imports, and statistics helpers. The client is only 125 lines long. 

Pacemaker is also an optional component, and it's ready now. 
I see the benefit of a 'TopologyMaster', but I don't see the benefit in 
blocking the merge of Pacemaker at this point.

> Heartbeat Server (Pacemaker)
> 
>
> Key: STORM-885
> URL: https://issues.apache.org/jira/browse/STORM-885
> Project: Apache Storm
>  Issue Type: Improvement
>  Components: storm-core
>Reporter: Robert Joseph Evans
>Assignee: Kyle Nusbaum
>
> Large highly connected topologies and large clusters write a lot of data into 
> ZooKeeper.  The heartbeats, that make up the majority of this data, do not 
> need to be persisted to disk.  Pacemaker is intended to be a secure 
> replacement for storing the heartbeats without changing anything within the 
> heartbeats.  In the future as more metrics are added in, we may want to look 
> into switching it over to look more like Heron, where a metrics server is 
> running for each node/topology.  And can be used to aggregate/per-aggregate 
> them in a more scalable manor.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (STORM-885) Heartbeat Server (Pacemaker)

[Kyle Nusbaum (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15011663#comment-15011663
 ] 

Kyle Nusbaum edited comment on STORM-885 at 11/18/15 6:51 PM:
--

[~LongdaFeng]
I disagree with your assessment of this patch.

1. I'm not sure what you mean by this. Will you please clarify?
2. Pacemaker may become a bottleneck in the system, yes. But in our internal 
testing, Nimbus itself becomes a bottleneck before Pacemaker. Furthermore, it 
is *not* very hard to do extension. I have a prototype here for Pacemaker HA 
that only adds a small number of lines of code, which I am planning on pushing 
out in the relatively near future.
3. The logic looks fairly straight-forward to me, and the pacemaker server file 
itself is only 239 lines long, with nearly 100 of those being the license, 
imports, and statistics helpers. The client is only 125 lines long. 

Pacemaker is also an optional component, and it's ready now. 
I see the benefit of a 'TopologyMaster', but I don't see the benefit in 
blocking the merge of Pacemaker at this point.


was (Author: knusbaum):
I disagree with your assessment of this patch.

1. I'm not sure what you mean by this. Will you please clarify?
2. Pacemaker may become a bottleneck in the system, yes. But in our internal 
testing, Nimbus itself becomes a bottleneck before Pacemaker. Furthermore, it 
is *not* very hard to do extension. I have a prototype here for Pacemaker HA 
that only adds a small number of lines of code, which I am planning on pushing 
out in the relatively near future.
3. The logic looks fairly straight-forward to me, and the pacemaker server file 
itself is only 239 lines long, with nearly 100 of those being the license, 
imports, and statistics helpers. The client is only 125 lines long. 

Pacemaker is also an optional component, and it's ready now. 
I see the benefit of a 'TopologyMaster', but I don't see the benefit in 
blocking the merge of Pacemaker at this point.

> Heartbeat Server (Pacemaker)
> 
>
> Key: STORM-885
> URL: https://issues.apache.org/jira/browse/STORM-885
> Project: Apache Storm
>  Issue Type: Improvement
>  Components: storm-core
>Reporter: Robert Joseph Evans
>Assignee: Kyle Nusbaum
>
> Large highly connected topologies and large clusters write a lot of data into 
> ZooKeeper.  The heartbeats, that make up the majority of this data, do not 
> need to be persisted to disk.  Pacemaker is intended to be a secure 
> replacement for storing the heartbeats without changing anything within the 
> heartbeats.  In the future as more metrics are added in, we may want to look 
> into switching it over to look more like Heron, where a metrics server is 
> running for each node/topology.  And can be used to aggregate/per-aggregate 
> them in a more scalable manor.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-885) Heartbeat Server (Pacemaker)

[Kyle Nusbaum (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15011663#comment-15011663
 ] 

Kyle Nusbaum commented on STORM-885:


I disagree with your assessment of this patch.

1. I'm not sure what you mean by this. Will you please clarify?
2. Pacemaker may become a bottleneck in the system, yes. But in our internal 
testing, Nimbus itself becomes a bottleneck before Pacemaker. Furthermore, it 
is *not* very hard to do extension. I have a prototype here for Pacemaker HA 
that only adds a small number of lines of code, which I am planning on pushing 
out in the relatively near future.
3. The logic looks fairly straight-forward to me, and the pacemaker server file 
itself is only 239 lines long, with nearly 100 of those being the license, 
imports, and statistics helpers. The client is only 125 lines long. 

Pacemaker is also an optional component, and it's ready now. 
I see the benefit of a 'TopologyMaster', but I don't see the benefit in 
blocking the merge of Pacemaker at this point.

> Heartbeat Server (Pacemaker)
> 
>
> Key: STORM-885
> URL: https://issues.apache.org/jira/browse/STORM-885
> Project: Apache Storm
>  Issue Type: Improvement
>  Components: storm-core
>Reporter: Robert Joseph Evans
>Assignee: Kyle Nusbaum
>
> Large highly connected topologies and large clusters write a lot of data into 
> ZooKeeper.  The heartbeats, that make up the majority of this data, do not 
> need to be persisted to disk.  Pacemaker is intended to be a secure 
> replacement for storing the heartbeats without changing anything within the 
> heartbeats.  In the future as more metrics are added in, we may want to look 
> into switching it over to look more like Heron, where a metrics server is 
> running for each node/topology.  And can be used to aggregate/per-aggregate 
> them in a more scalable manor.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [DISCUSS] 1.0 Release (was Re: [DISCUSS] Initial 0.11.0 Release)

[Kyle Nusbaum]

I may have misunderstood. 

I'm okay with cutting 1.0 if after we merge JStorm we go to 2.0. 
I thought I read in one of these threads the proposal to move to 1.1 
afterwards, but I cannot find it now. 

 -- Kyle 


 On Tuesday, November 17, 2015 5:00 PM, Kyle Nusbaum 
<knusb...@yahoo-inc.com.INVALID> wrote:
   

 I would prefer to wait until the JStorm code is merged to move to 1.0, and 
keep the current planned release as 0.11 
My concern is that there will be significant functionality and possibly 
stability changes involved, and it feels more natural to have major changes be 
across a
major version release.

Rather than release 1.0 and then immediately make major changes to the project, 
I would propose we release 0.11, then do the freeze and merge the JStorm code, 
and the first release after that can be 1.0. -- Kyle 


    On Tuesday, November 17, 2015 1:57 PM, Bobby Evans 
<ev...@yahoo-inc.com.INVALID> wrote:
  

 I have a pull request up now for these changes.  I can run both new and old 
jars, but you have to use a new client when submitting an old jar or it will 
not work.

https://github.com/apache/storm/pull/889
 - Bobby 


    On Tuesday, November 17, 2015 9:43 AM, Bobby Evans 
<ev...@yahoo-inc.com.INVALID> wrote:
  

 I have a patch that is close, but like I said on the JIRA 
https://issues.apache.org/jira/browse/STORM-1202 we are not going to be able to 
make a storm-compat.jar work.  Instead I have a binary that uses the shade code 
to rewrite the jar before it runs to match the new namespace.  I am just doing 
some cleanup on the code and testing now before I put up the pull request 
either today or tomorrow.  It too will be something that we can remove for a 
2.0 release.
 - Bobby 


    On Monday, November 16, 2015 7:23 PM, Harsha <st...@harsha.io> wrote:
  

 +1 on Bobby's suggestion on moving the packages to storm-compat and have
it part of lib folder. 
Moving 1.0 with org.apache.storm will make it easier in the future
rather than wait for 2.0 release we should make 
this change now and in 2.0 we can remove the storm-compat jar.

Thanks,
Harsha


On Thu, Nov 12, 2015, at 07:15 AM, Bobby Evans wrote:
> I agree that having the old package names for a 1.0 release is a little
> odd, but I also am nervous about breaking backwards compatibility for our
> customers in a very significant way.  The upgrade for us from 0.9.x to
> 0.10.x has gone fairly smoothly.  Most customers read the announcement
> and recompiled their code against 0.10, but followed our instructions so
> that their topologies could run on both 0.9.x and 0.10.x.  Having the
> ability for a topology to run on both an old cluster and a new cluster is
> very important for us, because of failover.  If we want to minimize
> downtime we need to have the ability to fail a topology over from one
> cluster to another, usually running on the other side of the
> country/world.  For stability reasons we do not want to upgrade both
> clusters at once, so we need to have confidence that a topology can run
> on both clusters.  Maintaining two versions of a topology is a huge pain
> as well.
> Perhaps what we can do for 1.0 is to move all of the packages to
> org.apache.storm, but provide a storm-compat package that will still have
> the old user facing APIs in it, that are actually (for the most part)
> subclasses/interfaces of the org.apache versions.  I am not sure this
> will work perfectly, but I think I will give it a try.
>  - Bobby 
> 
> 
>      On Thursday, November 12, 2015 9:04 AM, Aaron. Dossett
>      <aaron.doss...@target.com> wrote:
>    
> 
>  As a user/developer this sounds great.  The only item that gives me
>  pause
> is #2.  Still seeing backtype.* package names would be unexpected (for
> me)
> for a 1.0 release.  That small reservation aside, I am +1 (non-binding).
> 
> On 11/11/15, 2:45 PM, "임정택" <kabh...@gmail.com> wrote:
> 
> >+1
> >
> >Jungtaek Lim (HeartSaVioR)
> >
> >2015-11-12 7:21 GMT+09:00 P. Taylor Goetz <ptgo...@gmail.com>:
> >
> >> Changing subject in order to consolidate discussion of a 1.0 release in
> >> one thread (there was some additional discussion in the thread regarding
> >> the JStorm code merge).
> >>
> >> I just want to make sure I’m accurately capturing the sentiment of the
> >> community with regard to a 1.0 release. Please correct me if my summary
> >> seems off-base or jump in with an opinion.
> >>
> >> In summary:
> >>
> >> 1. What we have been calling “0.11.0” will become the Storm 1.0 release.
> >> 2. We will NOT be migrating package names for this release (i.e.
> >> “backtype.storm” —> “org.apache.storm”).
> >> 3. Post 1.0 release we will go into feature freeze for core
&

Re: [DISCUSS] 1.0 Release (was Re: [DISCUSS] Initial 0.11.0 Release)

[Kyle Nusbaum]

I would prefer to wait until the JStorm code is merged to move to 1.0, and keep 
the current planned release as 0.11 
My concern is that there will be significant functionality and possibly 
stability changes involved, and it feels more natural to have major changes be 
across a
major version release.

Rather than release 1.0 and then immediately make major changes to the project, 
I would propose we release 0.11, then do the freeze and merge the JStorm code, 
and the first release after that can be 1.0. -- Kyle 


 On Tuesday, November 17, 2015 1:57 PM, Bobby Evans 
 wrote:
   

 I have a pull request up now for these changes.  I can run both new and old 
jars, but you have to use a new client when submitting an old jar or it will 
not work.

https://github.com/apache/storm/pull/889
 - Bobby 


    On Tuesday, November 17, 2015 9:43 AM, Bobby Evans 
 wrote:
  

 I have a patch that is close, but like I said on the JIRA 
https://issues.apache.org/jira/browse/STORM-1202 we are not going to be able to 
make a storm-compat.jar work.  Instead I have a binary that uses the shade code 
to rewrite the jar before it runs to match the new namespace.  I am just doing 
some cleanup on the code and testing now before I put up the pull request 
either today or tomorrow.  It too will be something that we can remove for a 
2.0 release.
 - Bobby 


    On Monday, November 16, 2015 7:23 PM, Harsha  wrote:
  

 +1 on Bobby's suggestion on moving the packages to storm-compat and have
it part of lib folder. 
Moving 1.0 with org.apache.storm will make it easier in the future
rather than wait for 2.0 release we should make 
this change now and in 2.0 we can remove the storm-compat jar.

Thanks,
Harsha


On Thu, Nov 12, 2015, at 07:15 AM, Bobby Evans wrote:
> I agree that having the old package names for a 1.0 release is a little
> odd, but I also am nervous about breaking backwards compatibility for our
> customers in a very significant way.  The upgrade for us from 0.9.x to
> 0.10.x has gone fairly smoothly.  Most customers read the announcement
> and recompiled their code against 0.10, but followed our instructions so
> that their topologies could run on both 0.9.x and 0.10.x.  Having the
> ability for a topology to run on both an old cluster and a new cluster is
> very important for us, because of failover.  If we want to minimize
> downtime we need to have the ability to fail a topology over from one
> cluster to another, usually running on the other side of the
> country/world.  For stability reasons we do not want to upgrade both
> clusters at once, so we need to have confidence that a topology can run
> on both clusters.  Maintaining two versions of a topology is a huge pain
> as well.
> Perhaps what we can do for 1.0 is to move all of the packages to
> org.apache.storm, but provide a storm-compat package that will still have
> the old user facing APIs in it, that are actually (for the most part)
> subclasses/interfaces of the org.apache versions.  I am not sure this
> will work perfectly, but I think I will give it a try.
>  - Bobby 
> 
> 
>      On Thursday, November 12, 2015 9:04 AM, Aaron. Dossett
>       wrote:
>    
> 
>  As a user/developer this sounds great.  The only item that gives me
>  pause
> is #2.  Still seeing backtype.* package names would be unexpected (for
> me)
> for a 1.0 release.  That small reservation aside, I am +1 (non-binding).
> 
> On 11/11/15, 2:45 PM, "임정택"  wrote:
> 
> >+1
> >
> >Jungtaek Lim (HeartSaVioR)
> >
> >2015-11-12 7:21 GMT+09:00 P. Taylor Goetz :
> >
> >> Changing subject in order to consolidate discussion of a 1.0 release in
> >> one thread (there was some additional discussion in the thread regarding
> >> the JStorm code merge).
> >>
> >> I just want to make sure I’m accurately capturing the sentiment of the
> >> community with regard to a 1.0 release. Please correct me if my summary
> >> seems off-base or jump in with an opinion.
> >>
> >> In summary:
> >>
> >> 1. What we have been calling “0.11.0” will become the Storm 1.0 release.
> >> 2. We will NOT be migrating package names for this release (i.e.
> >> “backtype.storm” —> “org.apache.storm”).
> >> 3. Post 1.0 release we will go into feature freeze for core
> >>functionality
> >> to facilitate the JStorm merge.
> >> 4. During the feature freeze only fixes for high priority bugs in core
> >> functionality will be accepted (no new features).
> >> 5. During the feature freeze, enhancements to “external” modules can be
> >> accepted.
> >> 6. We will stop using the “beta” flag in favor of purely numeric version
> >> numbers. Stable vs. non-stable (development) releases can be indicated
> >>on
> >> the download page.
> >>
> >> Do we all agree?
> >>
> >> -Taylor
> >>
> >> > On Nov 11, 2015, at 4:10 PM, P. Taylor Goetz 
> >>wrote:
> >> >
> >> >
> >> >> On Nov 11, 2015, at 

[jira] [Assigned] (STORM-1196) Upgrade to thrift 0.9.3

[Kyle Nusbaum (JIRA)]

 [ 
https://issues.apache.org/jira/browse/STORM-1196?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kyle Nusbaum reassigned STORM-1196:
---

Assignee: Kyle Nusbaum

> Upgrade to thrift 0.9.3
> ---
>
> Key: STORM-1196
> URL: https://issues.apache.org/jira/browse/STORM-1196
> Project: Apache Storm
>  Issue Type: Improvement
>        Reporter: Kyle Nusbaum
>    Assignee: Kyle Nusbaum
>
> Thrift 0.9.3 has been released.
> Upgrade allows us to get rid of the pesky date changes in all the thrift 
> files.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Closed] (STORM-400) Thrift upgrade to 0.9.1

[Kyle Nusbaum (JIRA)]

 [ 
https://issues.apache.org/jira/browse/STORM-400?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kyle Nusbaum closed STORM-400.
--

> Thrift upgrade to 0.9.1
> ---
>
> Key: STORM-400
> URL: https://issues.apache.org/jira/browse/STORM-400
> Project: Apache Storm
>  Issue Type: Dependency upgrade
>  Components: storm-core
>Reporter: Kishor Patil
>Assignee: Kishor Patil
>  Labels: master, thrift
> Fix For: 0.10.0
>
>
> Upgrade Thrift to 0.9.1



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Reopened] (STORM-400) Thrift upgrade to 0.9.1

[Kyle Nusbaum (JIRA)]

 [ 
https://issues.apache.org/jira/browse/STORM-400?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kyle Nusbaum reopened STORM-400:


> Thrift upgrade to 0.9.1
> ---
>
> Key: STORM-400
> URL: https://issues.apache.org/jira/browse/STORM-400
> Project: Apache Storm
>  Issue Type: Dependency upgrade
>  Components: storm-core
>Reporter: Kishor Patil
>Assignee: Kishor Patil
>  Labels: master, thrift
> Fix For: 0.10.0
>
>
> Upgrade Thrift to 0.9.1



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (STORM-400) Thrift upgrade to 0.9.1

[Kyle Nusbaum (JIRA)]

 [ 
https://issues.apache.org/jira/browse/STORM-400?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kyle Nusbaum resolved STORM-400.

Resolution: Fixed

> Thrift upgrade to 0.9.1
> ---
>
> Key: STORM-400
> URL: https://issues.apache.org/jira/browse/STORM-400
> Project: Apache Storm
>  Issue Type: Dependency upgrade
>  Components: storm-core
>Reporter: Kishor Patil
>Assignee: Kishor Patil
>  Labels: master, thrift
> Fix For: 0.10.0
>
>
> Upgrade Thrift to 0.9.1



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (STORM-1197) Migrate Travis-CI to container-based builds.

[Kyle Nusbaum (JIRA)]

Kyle Nusbaum created STORM-1197:
---

 Summary: Migrate Travis-CI to container-based builds.
 Key: STORM-1197
 URL: https://issues.apache.org/jira/browse/STORM-1197
 Project: Apache Storm
  Issue Type: Improvement
Reporter: Kyle Nusbaum


http://docs.travis-ci.com/user/migrating-from-legacy/?utm_source=legacy-notice_medium=banner_campaign=legacy-upgrade



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (STORM-1197) Migrate Travis-CI to container-based builds.

[Kyle Nusbaum (JIRA)]

 [ 
https://issues.apache.org/jira/browse/STORM-1197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kyle Nusbaum updated STORM-1197:

Description: http://docs.travis-ci.com/user/migrating-from-legacy/  (was: 
http://docs.travis-ci.com/user/migrating-from-legacy/?utm_source=legacy-notice_medium=banner_campaign=legacy-upgrade)

> Migrate Travis-CI to container-based builds.
> 
>
> Key: STORM-1197
> URL: https://issues.apache.org/jira/browse/STORM-1197
> Project: Apache Storm
>  Issue Type: Improvement
>        Reporter: Kyle Nusbaum
>
> http://docs.travis-ci.com/user/migrating-from-legacy/



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (STORM-1196) Upgrade to thrift 0.9.3

[Kyle Nusbaum (JIRA)]

Kyle Nusbaum created STORM-1196:
---

 Summary: Upgrade to thrift 0.9.3
 Key: STORM-1196
 URL: https://issues.apache.org/jira/browse/STORM-1196
 Project: Apache Storm
  Issue Type: Improvement
Reporter: Kyle Nusbaum


Thrift 0.9.3 has been released.

Upgrade allows us to get rid of the pesky date changes in all the thrift files.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Closed] (STORM-1197) Migrate Travis-CI to container-based builds.

[Kyle Nusbaum (JIRA)]

 [ 
https://issues.apache.org/jira/browse/STORM-1197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kyle Nusbaum closed STORM-1197.
---
Resolution: Invalid

> Migrate Travis-CI to container-based builds.
> 
>
> Key: STORM-1197
> URL: https://issues.apache.org/jira/browse/STORM-1197
> Project: Apache Storm
>  Issue Type: Improvement
>        Reporter: Kyle Nusbaum
>
> http://docs.travis-ci.com/user/migrating-from-legacy/



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

github releases

[Kyle Nusbaum]

https://github.com/apache/storm/releases

There are releases for both 0.10.0 and 0.9.6 listed here. We should probably 
avoid putting things like that up until the actual releases are out.I've 
answered a couple of questions about it for confused people.

 -- Kyle

[jira] [Created] (STORM-1124) 'schema' in ui core should be 'scheme'

[Kyle Nusbaum (JIRA)]

Kyle Nusbaum created STORM-1124:
---

 Summary: 'schema' in ui core should be 'scheme'
 Key: STORM-1124
 URL: https://issues.apache.org/jira/browse/STORM-1124
 Project: Apache Storm
  Issue Type: Bug
Reporter: Kyle Nusbaum
Assignee: Kyle Nusbaum






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (STORM-1059) Upgrade Storm to use Clojure 1.7.0

[Kyle Nusbaum (JIRA)]

Kyle Nusbaum created STORM-1059:
---

 Summary: Upgrade Storm to use Clojure 1.7.0
 Key: STORM-1059
 URL: https://issues.apache.org/jira/browse/STORM-1059
 Project: Apache Storm
  Issue Type: Improvement
Reporter: Kyle Nusbaum
Priority: Minor






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (STORM-1059) Upgrade Storm to use Clojure 1.7.0

[Kyle Nusbaum (JIRA)]

 [ 
https://issues.apache.org/jira/browse/STORM-1059?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kyle Nusbaum reassigned STORM-1059:
---

Assignee: Kyle Nusbaum

> Upgrade Storm to use Clojure 1.7.0
> --
>
> Key: STORM-1059
> URL: https://issues.apache.org/jira/browse/STORM-1059
> Project: Apache Storm
>  Issue Type: Improvement
>        Reporter: Kyle Nusbaum
>    Assignee: Kyle Nusbaum
>Priority: Minor
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-885) Heartbeat Server (Pacemaker)

[Kyle Nusbaum (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14601537#comment-14601537
 ] 

Kyle Nusbaum commented on STORM-885:


[~kabhwan] Yes, pacemaker is mostly implemented. Planning on creating a pull 
request soon.

 Heartbeat Server (Pacemaker)
 

 Key: STORM-885
 URL: https://issues.apache.org/jira/browse/STORM-885
 Project: Apache Storm
  Issue Type: Improvement
Reporter: Robert Joseph Evans
Assignee: Kyle Nusbaum

 Large highly connected topologies and large clusters write a lot of data into 
 ZooKeeper.  The heartbeats, that make up the majority of this data, do not 
 need to be persisted to disk.  Pacemaker is intended to be a secure 
 replacement for storing the heartbeats without changing anything within the 
 heartbeats.  In the future as more metrics are added in, we may want to look 
 into switching it over to look more like Heron, where a metrics server is 
 running for each node/topology.  And can be used to aggregate/per-aggregate 
 them in a more scalable manor.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)