Re: [VOTE] Release Apache Storm 1.2.2 (rc2)

[Priyank Shah]

Is it possible to include https://issues.apache.org/jira/browse/STORM-2988  in 
1.2.2 release? Seems like a legit bug and a quick fix. 

On 5/7/18, 8:45 AM, "P. Taylor Goetz"  wrote:

Canceling in order to include STORM-3059 in this release.

-Taylor

> On May 3, 2018, at 11:42 AM, P. Taylor Goetz  wrote:
> 
> This is a call to vote on releasing Apache Storm 1.2.2 (rc2)
> 
> Full list of changes in this release:
> 
> 
https://dist.apache.org/repos/dist/dev/storm/apache-storm-1.2.2-rc2/RELEASE_NOTES.html
> 
> The tag/commit to be voted upon is v1.2.2:
> 
> 
https://git-wip-us.apache.org/repos/asf?p=storm.git;a=tree;h=7cb19fb3befa65e5ff9e5e02f38e16de865982a9;hb=e001672cf0ea59fe6989b563fb6bbb450fe8e7e5
> 
> The source archive being voted upon can be found here:
> 
> 
https://dist.apache.org/repos/dist/dev/storm/apache-storm-1.2.2-rc2/apache-storm-1.2.2-src.tar.gz
> 
> Other release files, signatures and digests can be found here:
> 
> https://dist.apache.org/repos/dist/dev/storm/apache-storm-1.2.2-rc2/
> 
> The release artifacts are signed with the following key:
> 
> 
https://git-wip-us.apache.org/repos/asf?p=storm.git;a=blob_plain;f=KEYS;hb=22b832708295fa2c15c4f3c70ac0d2bc6fded4bd
> 
> The Nexus staging repository for this release is:
> 
> https://repository.apache.org/content/repositories/orgapachestorm-1062
> 
> Please vote on releasing this package as Apache Storm 1.2.2.
> 
> When voting, please list the actions taken to verify the release.
> 
> This vote will be open for 72 hours or until at least 3 PMC members vote 
+1.
> 
> [ ] +1 Release this package as Apache Storm 1.2.2
> [ ]  0 No opinion
> [ ] -1 Do not release this package because...
> 
> Thanks to everyone who contributed to this release.
> 
> -Taylor

Re: Verification about my observation for current State implementation

[Priyank Shah]

Jungtaek,

I read about rebalancing to increase the parallelism a long time back. Trying 
to remember and see if it answers your question of "N tasks 1 executor". I 
think the use case for rebalancing parallelism was for users who are 
anticipating an increase in hardware capacity like adding cores to their worker 
node. In this case they can leverage these extra cores by rebalancing with 
higher number of executors instead of killing and re-deploying the topology. 
However, if they had the number of tasks = number of executors when they first 
submitted the topology they would not be able to leverage that since the 
precondition is "number of tasks >= number of executors". 

On 2/20/18, 4:40 PM, "Jungtaek Lim"  wrote:

> You can only change the parallelism (number of executors). AFAIK there is
no option to change the number of tasks and maybe I am missing if it
changed recently.

Yeah looks like I misunderstand the feature. It addresses number of
executors.

I'm wondering the benefit of having N tasks : 1 executor. It even makes me
confusing continuously, and I don't find beneficial use cases for that.
Most of things would be IO bound processing, and should be covered with
asynchronous processing. I hope we just simplify this one.

Another perspective of rebalance: suppose end users provide parallelism
bigger than current task count. If task count is not changed, rebalance is
not working as expected from users. If task count is changed, we are
breaking precondition of above statement.

We should take into consideration that end users modify topology code and
resubmit. In this case task count can be (and should be) changed, and we
should support state resharding in this case. Without resharding each task
fails to load all the KVs for keys routing to the task. It doesn't seem to
the issue about whether supporting dynamic task change or not.

- Jungtaek Lim (HeartSaVioR)

ps. btw, internal structure of Storm topology is not capable of scaling
task counts without restructuring since task id is increased sequentially
across components. I suspect it would block elasticity so that should be
also the thing to address.

2018년 2월 21일 (수) 오전 8:33, Arun Mahadevan 님이 작성:

> >>
> >>1. Right now users need to vary the database/table name in the state
> >provider config per topology.
> >
> >Redis Cluster doesn't support multi database and it is uncommon usage for
> >Redis to use even tens of databases, so end users can't do the needed
> thing
> >with Redis state backend. For HBase state backend they can do, but
> creating
> >table requires administrator permission of HBase, which someone may not
> >want to open to multiple users. So I feel including topology name in
> >namespace looks like "mandatory" rather than "good to have".
>
>
> I think this should be a fairly straightforward change and we should do 
it.
>
> >>
> >>2. As of now storm only supports rebalancing the tasks (task count
> >remains the same).
> >
> >That is not correct. We are still missed to provide the feature in UI 
(I'm
> >not familiar with front-end so I had in mind but I didn't work on that)
> but
> >you can rebalance with different parallelism of component via cli.
> >
> >Below is the rebalance command provided via cli. Quoting command line
> >client document[1]:
> >
> >rebalance
> >Syntax: storm rebalance topology-name [-w wait-time-secs] [-n
> >new-num-workers] [-e component=parallelism]*
>
>
> You can only change the parallelism (number of executors). AFAIK there is
> no option to change the number of tasks and maybe I am missing if it
> changed recently. As long as the number of tasks are fixed, we don’t have
> to worry about re-sharding the keys since the namespaces are at task 
level.
> Autoscaling may or may not change this design (we may only allow 
increasing
> or reducing the parallelism and not the tasks) and it requires more 
thought
> on how it would affect the user managed state at core spout/bolt level
> (outside of the higher level abstractions) and so IMO, we can tackle it
> later.
>
> Thanks,
> Arun
>
>
>
> On 2/20/18, 2:57 PM, "Jungtaek Lim"  wrote:
>
> >> 1. Right now users need to vary the database/table name in the state
> >provider config per topology.
> >
> >Redis Cluster doesn't support multi database and it is uncommon usage for
> >Redis to use even tens of databases, so end users can't do the needed
> thing
> >with Redis state backend. For HBase state backend they can do, but
> creating
> >table requires administrator permission of HBase, which someone may not
> >want to open to multiple users. So I feel including topology name in
> >namespace looks like "

Re: [DISCUSS] consider EOL for version lines

[Priyank Shah]

+1 to maintaining 3 version lines as suggested by Jungtaek.

On 2/13/18, 9:51 AM, "Arun Iyer on behalf of Arun Mahadevan" 
 wrote:

+1 to maintain 3 version lines.

I think the next focus should be 2.0.0 than 1.3.0.




On 2/12/18, 11:40 PM, "Jungtaek Lim"  wrote:

>Hi devs,
>
>I've noticed that we are providing 4 different version lines (1.1.x, 1.0.x,
>0.10.x, 0.9.x) in download page, and I expect we will add one more for
>1.2.0. Moreover, we have one more develop version line (2.0.0 - master)
>which most of development happens there.
>
>Recently we're releasing 3 version lines (1.0.6 / 1.1.2 / 1.2.0)
>simultaneously and it took heavy effort to track all the RCs and verify all
>of them. I guess release manager would take more overhead of releasing, and
>it doesn't make sense for me if we continue maintaining all of them.
>
>Ideally I'd like to propose maintaining three version lines: 2.0.0 (next
>major) / 1.3.0 (next minor - may not happen) / 1.2.1 (next bugfix) and
>making others EOL (that respects semantic versioning and even other
>projects tend to maintain only two version lines), but if someone feels too
>aggressive, I propose at least we explicitly announce EOL to 0.x version
>lines and get rid of any supports (downloads) for them.
>
>Would like to hear your opinion.
>
>Thanks,
>Jungtaek Lim (HeartSaVioR)

Re: [VOTE] Release Apache Storm 1.2.0 (rc1)

[Priyank Shah]

Release looks good overall except for complete latency and capacity.

Verified the md5, sha and asc
Unzipped and untared the source and binary releases.
Built from source and ran daemons from storm-dist binary release after packing 
from the same.
Submitted FastWordCount and RollingTopWords
Played around with UI and tried different operations like Deactivate, Activate, 
Kill, LogViewer, Search log, etc, . Everything works ok.

However I noticed complete latency for FastWordCount to be more than 4 seconds 
which seems higher. Also,  capacity for RollingTopWords (was verifying for 
Jungtaek) seemed to be very high.
Running 1.1.1 to compare the same numbers for both the topologies. Will update 
with my results. 


On 1/24/18, 12:03 PM, "Alexandre Vermeerbergen"  
wrote:

Hello Storm developers,

One of my team member (Noureddine Chatti) found that the regression in
"Assigned Memory (MB)" columns for our topologies with Storm 1.2.0rc1,
where the 65 MB value is displayed regardless of our actual
topologies's memory setting:

We define our workers's max memory heap using this option (as
documented in Hortonworks documentation here:

https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.2/bk_storm-component-guide/content/config-storm-settings.html)

worker.childopts: "-Xmx2048m"

Until Storm 1.1.0, this value is displayed in "Assigned Memory (MB)"
columns by Nimbus UI.
In Storm 1.2.0 rc1, the 65 MB value is displayed regardless of this setting.

He also found that using the following setting:

topology.worker.max.heap.size.mb: 2048.

then the displayed "Assigned Memory (MB)" columns by Nimbus UI are in
line with the value (i.e. they show 2048MB).

Questions:
1. This is change of behavior from Storm 1.1.0 to Storm 1.2.0rc1
international? any related JIRA who would help understanding the
rationale?
2. Is the maximum heap size defined using worker.childopts actually
used to setup Worker's max heap size?
3. What's the best practice for setting worker's memory consumption  :
is topology.worker.max.heap.size.mb now mandatory, or is the use of
-Xmx in worker.childopts still supported?
4. Could be there other differences from Storm 1.1.0 and Storm
1.2.0rc1 which could explain why we get very weird statistics in
Nimbus UI for Capacity & Latency for our topologies?

Best regards,
Alexandre Vermeerbergen

Re: [DISCUSS] Drop standalone mode of Storm SQL

[Priyank Shah]

+1  (NB). As Arun mentioned, it will be good to run test it by running 
topologies in local mode.

On 12/5/17, 10:07 AM, "Arun Iyer on behalf of Arun Mahadevan" 
 wrote:

+1, I don’t see much use for standalone mode other than for testing. 

Assume we can use the storm-sql in local mode to run topologies locally 
without deploying to cluster ?

Thanks,
Arun






On 12/4/17, 10:53 PM, "Jungtaek Lim"  wrote:

>Hi devs,
>
>We have been exposing "standalone mode" of Storm SQL which leverages Storm
>SQL in a JVM process rather than composing topology and run.
>At a start we implemented both standalone and trident modes with same
>approach, but while we improved Storm SQL by leveraging more features on
>Calcite, we addressed only trident mode, and now twos are diverged.
>
>I guess there is likely no actual user on standalone mode since its classes
>are exposed but we didn't document it. I know a case, but the source codes
>on standalone mode code are migrated to the project (and modified to
>conform to the project) and the project no longer depends on Storm SQL.
>
>If we all don't have any other case, how about dropping it and only
>concentrate to trident mode?
>(Btw, I'm trying to replace the backend on Storm SQL from Trident to
>Streams API, which may make the mode name obsolete, but after dropping
>standalone mode we don't even need the name for mode since there will be
>only one mode.)
>
>Thanks,
>Jungtaek Lim (HeartSaVioR)

Re: [Discuss] Release Storm 1.2.0

[Priyank Shah]

Regarding getTopicsString, its being used in storm-kafka-monitor 
https://github.com/apache/storm/blob/1.x-branch/external/storm-kafka-client/src/main/java/org/apache/storm/kafka/spout/KafkaSpout.java#L626
 and returning [topic_name] for one topic would not work. If we change it back 
to original format we should make sure we pass the correct format from that 
line of code.

On 11/20/17, 9:40 AM, "generalbas@gmail.com on behalf of Stig Rohde 
Døssing"  wrote:

Happy to hear that it's at least running for you now :) Eager to hear your
results.

Regarding the getTopicsString format, the Subscription interface doesn't
specify the string format (and it was never intended to be parsed in this
way). In my opinion the format is an implementation detail, so we should be
free to change it at any time.

2017-11-20 15:01 GMT+01:00 Alexandre Vermeerbergen :

> Hello All,
>
> Good news today: I found & solved what was preventing my topologies from
> consuming since I had upgraded from storm-kafka-client 1.1.0 to
> storm-kafka-client.1.2.0-lastestsnapopfromstig.
>
> The reasons from our own BasicKafkaSpout class, which is our homebrewed
> Kafka spout based on same configuration settings as the official
> storm-kafka-client Kafka spout.
>
> The issue came from the fact that, at least storm-kafka-client 1.1.0
> doesn't exposes a way to get the list of consumed topics, so we had to
> parse the value returned by KafkaSpoutConfig V>.getSubscription().getTopicsString() in order to extract the list of
> topics.
>
> Here's where the issue lies:
>
> public BasicKafkaSpout(KafkaSpoutConfig config) {
> this.kafkaBrokers = (String) config.getKafkaProps().get(
> ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG);
> this.consumerId = config.getConsumerGroupId();
> logger.debug("topics are: {}", config.getSubscription()
> .getTopicsString());
> String topicsStr = config.getSubscription().getTopicsString();
> this.topics = topicsStr.substring(1, topicsStr.length() -
> 1).split(",");
>
> => Bug here with storm-kafka-client 1.2.0 snapshot, because if
> there is only one topic,
>the value of topicsStr which was "[mytopic]" with 1.1.0 is now
> "mytopic"
>
> and here's a fixed (and ugly) version which works with both 1.1.0 & 1.2.0
> snapshot of the same code snippet:
>
> public BasicKafkaSpout(KafkaSpoutConfig config) {
> this.kafkaBrokers = (String) config.getKafkaProps().get(
> ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG);
> this.consumerId = config.getConsumerGroupId();
> logger.debug("topics are: {}", config.getSubscription()
> .getTopicsString());
> String topicsStr = config.getSubscription().getTopicsString();
> if (topicsStr.startsWith("[") && topicsStr.endsWith("]")) {
> this.topics = topicsStr.substring(1, topicsStr.length() -
> 1).split(",");
> }
> else {
> this.topics = topicsStr.split(",");
> }
>
> Maybe it would be good to either restore this string representation as it
> was in 1.1.0, or if not your intention, they document the change.
>
> Now I guess your question is : "why on earth did you implement your own
> Kafka Spout?"
>
> The answer is simple: we were waiting for 1.2.0 which contains a fix to
> storm-kafka-client to that Spout statistics are visible in Storm UI when
> this spout is used in autocommit mode. Our own spout was design on purpose
> to have same config as torm-kafka-client so as to make switch from our
> implementation to official storm-kafka-client-1.2.0 as simple as simple 
(as
> a matter of fact, it's a simple property to change on our side). We
> couldn't wait for 1.2.0 because we had to switch to Kafka 0.10.2.0 as soon
> as possible.
>
> Next steps for me related to Storm 1.2.0 preview tests:
> * Try our topologies with the official storm-kafka-client-1.2.0 in order 
to
> test non-regression
> * Try our topologies with the official storm-kafka-client-1.2.0 on our
> pre-production in order to test non-regression of performances.
>
> Hope it helps, stay tuned :)
>
> Best regards,
> Alexandre Vermeerbergen
>
>
> 2017-11-20 8:49 GMT+01:00 Stig Rohde Døssing :
>
> > Alexandre,
> >
> > Could you also post the BasicKafkaSpout source file? I'm curious what
> it's
> > doing.
> >
> > 2017-11-20 7:50 GMT+01:00 Alexandre Vermeerbergen <
> > avermeerber...@gmail.com>
> > :
> >
> > > Hello Jungtaek,
> > >
> > > OK I will activate these traces, but since we need to capture the
> Spouts'
> > > initializat

Re: [VOTE] Release Apache Storm 1.0.5 (rc1)

[Priyank Shah]

+1 (non-binding)

Verified md5 and sha
Successfully extracted source and distributions
Built source “mvn clean install” successfully
Launched  daemons(nimbus, supervisor, ui, logviewer) successfully from 
storm-dist after packaging
Ran FastWordCountTopology 
Played around with UI
Checked worker logs and other daemon logs 
Brought down the daemons without issues



On 9/7/17, 11:45 AM, "generalbas@gmail.com on behalf of Stig Rohde Døssing" 
 wrote:

* Extracted binary zip and built storm-starter using the included example
code and the Nexus staging repo.
* After working around https://issues.apache.org/jira/browse/STORM-2451 (I
forgot to pull it into 1.0.x after the 1.0.4 release, and don't want to
delay this release for that issue), started Nimbus, one supervisor and UI
daemons.
* Verified storm-local is no longer relative to the directory where storm
nimbus is executed.
* Deployed WordCountTopology and verified no errors in the worker logs, and
that the topology produces word counts in the log.
* Clicked around a bit in UI for the topology and components
* Killed the topology.

+1 (binding)

2017-09-07 10:56 GMT+02:00 Alexandre Vermeerbergen :

> Hello,
>
> (non binding)
>
> [x]  0 No opinion
>
> => we are currently running with 1.1.0, waiting for include
> https://issues.apache.org/jira/browse/STORM-2648 fix was unfortunately
> shifted in 1.2.0 (although lack of acks for KafkaSpout when it's running 
in
> autocommit mode sounds more like a bug than a feature-thing), so we skip
> testing 1.0.5
>
> Best regards,
> Alexandre Vermeerbergen
>
>
> 2017-09-07 2:38 GMT+02:00 Jungtaek Lim :
>
> > +1 (binding)
> >
> > > source
> >
> > - verify file (signature, MD5, SHA)
> > -- source, tar.gz : OK
> > -- source, zip : OK
> >
> > - extract file
> > -- source, tar.gz : OK
> > -- source, zip : OK
> >
> > - diff-ing extracted files between tar.gz and zip : OK
> >
> > - build source with JDK 7
> > -- source, tar.gz : OK
> >
> > - build source dist
> > -- source, tar.gz : OK
> >
> > - build binary dist
> > -- source, tar.gz : OK
> >
> > > binary
> >
> > - verify file (signature, MD5, SHA)
> > -- binary, tar.gz : OK
> > -- binary, zip : OK
> >
> > - extract file
> > -- source, tar.gz : OK
> > -- source, zip : OK
> >
> > - diff-ing extracted files between tar.gz and zip : OK
> >
> > - launch daemons : OK
> >
> > - run RollingTopWords (local) : OK
> >
> > - run RollingTopWords (remote) : OK
> >   - activate / deactivate / rebalance / kill : OK
> >   - logviewer (worker dir, daemon dir) : OK
> >   - change log level : OK
> >   - thread dump, heap dump, restart worker :
> >   - log search : OK
> >
> > Most critical issue in 1.0.5 has been verified: upgrade from Storm 1.0.4
> to
> > Storm 1.0.5 RC1 succeeds, though I've not tried RU. I tested this via
> > shutting down 1.0.4 cluster and starting 1.0.5 cluster which two 
clusters
> > are sharing storm-local directory and ZK path.
> >
> > Thanks,
> > Jungtaek Lim (HeartSaVioR)
> >
> > 2017년 9월 7일 (목) 오전 3:29, P. Taylor Goetz 님이 작성:
> >
> > > This is a call to vote on releasing Apache Storm 1.0.5 (rc1)
> > >
> > > Full list of changes in this release:
> > >
> > >
> > > https://dist.apache.org/repos/dist/dev/storm/apache-storm-1.
> > 0.5-rc1/RELEASE_NOTES.html
> > >
> > > The tag/commit to be voted upon is v1.0.5:
> > >
> > >
> > > https://git-wip-us.apache.org/repos/asf?p=storm.git;a=tree;h=
> > 239fd1522114c4928ed99b4ab5bcbf027540d5e7;hb=
> a445446e46e2228fabb83368cc25cf
> > 5422a2c400
> > >
> > > The source archive being voted upon can be found here:
> > >
> > >
> > > https://dist.apache.org/repos/dist/dev/storm/apache-storm-1.
> > 0.5-rc1/apache-storm-1.0.5-src.tar.gz
> > >
> > > Other release files, signatures and digests can be found here:
> > >
> > > https://dist.apache.org/repos/dist/dev/storm/apache-storm-1.0.5-rc1/
> > >
> > > The release artifacts are signed with the following key:
> > >
> > >
> > > https://git-wip-us.apache.org/repos/asf?p=storm.git;a=blob_
> > plain;f=KEYS;hb=22b832708295fa2c15c4f3c70ac0d2bc6fded4bd
> > >
> > > The Nexus staging repository for this release is:
> > >
> > > https://repository.apache.org/content/repositories/orgapachestorm-1051
> > >
> > > Please vote on releasing this package as Apache Storm 1.0.5.
> > >
> > > When voting, please list the actions taken to verify the release.
> > >
> > > This vote will be open for at least 72 hours.
> > >
> > > [ ] +1 Release this package as Apache Storm 1.0.5
> > > 

Re: [VOTE] Release Apache Storm 1.1.1 (rc1)

[Priyank Shah]

+1 (NB)

Verified signatures and checksums. 
Ran units tests on sources
Installed binaries and started daemons
Submitted sample topology and did manual testing by playing with UI

On 7/25/17, 3:36 PM, "Jungtaek Lim"  wrote:

I believe I ported back complete set of bugfixes to the 1.1.x-branch when I
created 1.1.x-branch.

1.x-branch has some new features after 1.1.0, even also has backward
incompatible change: Redis state changed to binary. I included migration
script on it but still don't think it is bugfix.

Taylor, which patches (only bugfixes) are important and not ported back to
1.1.x-branch? If they're clearly about bugfix and possible to be ported
back, isn't it better to do that?

If they're not bugfixes but have feeling that we should include the release
ASAP, let's enumerate in another discussion shortly and apply them to
1.x-branch when consensus has been made.

- Jungtaek Lim (HeartSaVioR)
On Wed, 26 Jul 2017 at 5:17 AM Stig Rohde Døssing 
wrote:

> I ran into https://issues.apache.org/jira/browse/STORM-2659, which is a
> regression compared to 1.0.x, but not compared to 1.1.0. I think it would
> be nice to get fixed.
>
> 2017-07-25 20:59 GMT+02:00 Bobby Evans :
>
> > We could do a 1.1.2 release sooner if needed.  Technically any committer
> > can call for a release at any point in time.  If there is a reason to do
> a
> > release (like an important fix for a critical component) then we can do
> it.
> >
> >
> > - Bobby
> >
> >
> > On Tuesday, July 25, 2017, 1:48:07 PM CDT, Alexandre Vermeerbergen <
> > avermeerber...@gmail.com> wrote:
> >
> > Hi,
> >
> > I guess 1.1.2 is going to be a few months away from now, so we'll have 
to
> > go with our own basic Kafka 0.10 Spout in the meantime...
> >
> > You can discard my previous vote, we'd need to at least download 1.1.1 
rc
> > and give it a try to make an objective vote.
> >
> > Regards,
> > Alexandre Vermeerbergen
> >
> > 2017-07-25 19:38 GMT+02:00 P. Taylor Goetz :
> >
> > > Hi Alexandre,
> > >
> > > STORM-2648 couldn’t be included because there is no patch available 
for
> > it
> > > yet. Once there is a patch available, it can go into the next release,
> so
> > > it’s certainly possible for it to be available in 1.1.2.
> > >
> > > -Taylor
> > >
> > >
> > > > On Jul 25, 2017, at 1:12 PM, Alexandre Vermeerbergen <
> > > avermeerber...@gmail.com> wrote:
> > > >
> > > > Hello,
> > > >
> > > > -1 (non binding)
> > > >
> > > > Maybe it's a bit selfish, but I really count on
> > > > https://issues.apache.org/jira/browse/STORM-2648 being part of Storm
> > > 1.1.1,
> > > > because we have a requirement to use Kafka 0.10 consumer in
> topologies
> > > > requiring at most once behavior.
> > > >
> > > > We understood that we could use storm-kafka-client with autocommit,
> but
> > > > then we're missing ack/fails and complete latency.
> > > >
> > > > We know that we could by-pass this limitation by implementing our 
own
> > > Kafka
> > > > 0.10 spout, but if possible it would be great to have Storm 1.1.1's
> > storm
> > > > kafka client cover the need of "at most once" requirements.
> > > >
> > > > Would it be possible to have this STORM-2648 to be part of 1.1.1 ?
> > > >
> > > > Best regards,
> > > > Alexandre Vermeerbergen
> > > >
> > > >
> > > > 2017-07-25 18:24 GMT+02:00 P. Taylor Goetz :
> > > >
> > > >> Yes. There were a number of important patches present in 1.x-branch
> > that
> > > >> were not in 1.1.x-branch.
> > > >>
> > > >> When I tried to merge 1.x-branch to 1.1.x-branch, I ran into
> > unexpected
> > > >> conflicts. I thought about deleting 1.1.x-branch and recreating it,
> > but
> > > >> decided against it, not wanting lose changes there that we might
> want
> > to
> > > >> keep in case we wanted to revisit the contents of that branch. In
> the
> > > end I
> > > >> decided to cut the release from 1.x-branch.
> > > >>
> > > >> Jungtaek, I believe you created 1.1.x-branch… Do you know why/how 
it
> > > >> diverged?
> > > >>
> > > >> -Taylor
> > > >>
> > > >>> On Jul 25, 2017, at 12:08 PM, Stig Rohde Døssing <
> > > stigdoess...@gmail.com>
> > > >> wrote:
> > > >>>
> > > >>> Is it on purpose that this is cut from 1.x-branch and not
> > 1.1.x-branch?
> > > >>>
> > > >>> 2017-07-25 17:09 GMT+02:00 P. Taylor Goetz :
> > > >>>
> > >  This is a call to vote on releasing Apache Storm 1.1.1 (rc1)
> > > 
> > >  Full list of changes in this release:
> > > 
> > >  https://git-wip-us.apache.org/repos/asf?p=storm.git;a=blob_

Re: Use of the KafkaConsumer.subscribe API in the storm-kafka-client spout

[Priyank Shah]

Hugo, I agree about the benefits of immutability and encapsulation. But are 
they so important for the case we are discussing? As far as code being super 
fragile, hard to debug and being thread unsafe, I don’t really think its 
applicable here. Can you elaborate on how does it make the code super fragile 
and hard to debug? As far as it being thread unsafe is concerned, 
KafkaSpoutConfig is anyway going to be serialized and reconstructed in worker. 
And it’s always going to be one instance of it per executor. Correct me if I am 
misunderstanding anything.

Stig, we could probably modify flux to do something like that. I need to put 
more thought to see if any better semantic can be applied for supporting 
builder pattern. For now, since you both prefer to keep the builder pattern we 
can do with the current work around of public constructor like you said. 

On 6/14/17, 2:21 PM, "Stig Døssing"  wrote:

It would obviously be ideal if Flux could be made to support object
creation via builders, but if that's not possible I think leaving the
KafkaSpoutConfig constructor public is a decent workaround. We are still
getting some of the benefits of the Builder pattern, even if Flux doesn't
use builder.build() to construct the KafkaSpoutConfig (immutability of
KafkaSpoutConfig, no telescoping constructors). I would prefer not to get
rid of it to work around a limitation in Flux.

My knowledge of Flux is very limited, but is there any reason it can't be
changed to support taking the result of a method call as a constructor
parameter? e.g.
components:
  - id: "SpoutConfigBuilder"
className: "org.apache.storm.kafka.spout.KafkaSpoutConfig.Builder"
configMethods:
  - name: "setProp"
args:
  - "max.poll.records"
  - 250

  - id: "Spout"
className: "org.apache.storm.kafka.spout.KafkaSpout"
constructorArgs:
  - ref: "SpoutConfigBuilder"
call: "build"

2017-06-14 22:09 GMT+02:00 Hugo Da Cruz Louro :

> Hi,
>
> Flux is simply a mechanism to enabling Java objects creation using a
> descriptor file. If Flux does not support creating classes that follow the
> Builder design pattern, that is a Flux limitation and has to be fixed. It
> is not reasonable to impose that no one can write a builder because Flux
> does not support it. My suggested approach was a simple solution to 
quickly
> get around it. Let’s identify the proper way to fix it.
>
> I do not think it is reasonable not to respect immutability and
> encapsulation unless there is a very strong reason to do so. It makes the
> code super fragile, hard to debug, and thread unsafe.
>
> > On Jun 14, 2017, at 12:22 PM, Priyank Shah 
> wrote:
> >
> > Hi Stig/Hugo,
> >
> > That constructor is indeed public. I actually made that change but
> forgot about it. https://github.com/apache/storm/commit/
> 5ff7865cf0b86f40e99b54e789fa60b8843191aa The reason for making that
> change is to make it work with flux.
> >
> > I think changing flux code to access private constructor is a hack and I
> prefer not doing that. On the other hand, having a public constructor
> defeats the purpose of Builder pattern since builder.build() is supposed 
to
> create the object. I personally don’t think immutability of
> KafkaSpoutConfig is that important here. I would rather get rid of the
> builder and have it with one or two constructors with some setXXX methods.
> Let me know what you guys think.
> >
> >
> > On 6/14/17, 9:46 AM, "Hugo Da Cruz Louro" 
> wrote:
> >
> >@Harsha @Stig, I agree with you. Let’s make the de facto
> implementation manual partition assignment. I have already adjusted the
> KafkaTridentSpout code to reflect @Stig’s changes and things seem to be
> working very well for Trident as well. I am tracking that on
> https://issues.apache.org/jira/browse/STORM-2554  and I will submit a PR
> soon. There were a couple minor fixes that I had to provide to
> https://github.com/apache/storm/pull/2150 to make it work; I will mention
> them as comment in the PR.
> >
> >@Priyank, the KafkaSpoutConfig class should be immutable, as it is a
> configuration class, which should not be possible to change once it is
> passed onto the KafkaSpout or KafkaTridentSpout. The builder that @Stig
> referenced should indeed be private or at most package protected if needed
> for unit tests, not public. If we have to leave it public for now

Re: Use of the KafkaConsumer.subscribe API in the storm-kafka-client spout

[Priyank Shah]

Hi Stig/Hugo,

That constructor is indeed public. I actually made that change but forgot about 
it. 
https://github.com/apache/storm/commit/5ff7865cf0b86f40e99b54e789fa60b8843191aa 
The reason for making that change is to make it work with flux.

I think changing flux code to access private constructor is a hack and I prefer 
not doing that. On the other hand, having a public constructor defeats the 
purpose of Builder pattern since builder.build() is supposed to create the 
object. I personally don’t think immutability of KafkaSpoutConfig is that 
important here. I would rather get rid of the builder and have it with one or 
two constructors with some setXXX methods. Let me know what you guys think.


On 6/14/17, 9:46 AM, "Hugo Da Cruz Louro"  wrote:

@Harsha @Stig, I agree with you. Let’s make the de facto implementation 
manual partition assignment. I have already adjusted the KafkaTridentSpout code 
to reflect @Stig’s changes and things seem to be working very well for Trident 
as well. I am tracking that on https://issues.apache.org/jira/browse/STORM-2554 
 and I will submit a PR soon. There were a couple minor fixes that I had to 
provide to https://github.com/apache/storm/pull/2150 to make it work; I will 
mention them as comment in the PR.

@Priyank, the KafkaSpoutConfig class should be immutable, as it is a 
configuration class, which should not be possible to change once it is passed 
onto the KafkaSpout or KafkaTridentSpout. The builder that @Stig referenced 
should indeed be private or at most package protected if needed for unit tests, 
not public. If we have to leave it public for now to make Flux work, so be it. 
However, the right fix for this would be to fix the Flux code to work with 
builders. Flux uses mostly Java reflection, so the fix may be as simple as 
allowing invocation of private constructors as described in 
here<https://stackoverflow.com/questions/11282265/how-to-call-a-private-method-from-outside-a-java-class>.

We should try to eliminate as many constructors possible. There should be 
one or two constructors that enforce the dependencies that are absolutely 
required and for which there are no reasonable defaults. Any other optional, or 
non default, configuration setting should go in a setter method. All the 
KafkaConsumer properties, as we seem to all agree, should be passed in a 
Map which is what KafkaConsumer needs in its constructor.

Hugo


On Jun 14, 2017, at 8:38 AM, Stig Døssing 
mailto:generalbas@gmail.com>> wrote:

It looks public to me?

https://github.com/apache/storm/blob/38e997ed96ce6627cabb4054224d7044fd2b40f9/external/storm-kafka-client/src/main/java/org/apache/storm/kafka/spout/KafkaSpoutConfig.java#L461

I think its good to be able to avoid telescoping constructors, while at the
same time not having a bunch of setters on the KafkaSpoutConfig. That's the
main purpose I think the builder has, allowing KafkaSpoutConfig to be
immutable.

I'd be happy to fiddle with it if you have an example to work from?
    
2017-06-14 1:11 GMT+02:00 Priyank Shah :

Hi Stig,

I think KafkaSpoutConfig constructor is private and it's throwing errors
while using the approach that you mentioned. Making it public defeats be
purpose of the builder. Can you give it a shot and confirm at your end if
it's possible?

Thanks
Priyank

Sent from my iPhone

On Jun 13, 2017, at 9:36 AM, Stig Døssing 
wrote:

Hi Priyank,

I changed my mind since those mails were sent. I don't think setKey/Value
are very useful. They couldn't be used with the default Kafka
deserializers, only for deserializers implemented by the user, and then
only if they were declared to implement SerializableDeserializer. I agree
that we should remove them, and I'm not going to undo anything currently
in
the PR (unless there are objections on the PR of course)

With regard to getting rid of the builder pattern, I think it is a pretty
nice pattern for Java. It looks to me like it should be possible to
declare
and configure the builder with "component:", and then pass it to the
KafkaSpoutConfig constructor with "ref:" after (which lets you avoid
calling build()). Doesn't this work?

2017-06-12 23:32 GMT+02:00 Priyank Shah :

Hi Stig,

I think PR https://github.com/apache/storm/pull/2155/files you created
gets rid of setKey and setValue. I am fine with it and in fact that’s
what
I was suggesting in first place. However, your last two email replies
suggest something else. Just making sure you are not going to undo
anything
in the PR and that we are same page about this change. i.e. no setKey or
setValue. Either for SerializableDeserializer implementations or Kafka
Deserialize

Re: Use of the KafkaConsumer.subscribe API in the storm-kafka-client spout

[Priyank Shah]

Hi Stig,

I think KafkaSpoutConfig constructor is private and it's throwing errors while 
using the approach that you mentioned. Making it public defeats be purpose of 
the builder. Can you give it a shot and confirm at your end if it's possible?

Thanks
Priyank

Sent from my iPhone

> On Jun 13, 2017, at 9:36 AM, Stig Døssing  wrote:
> 
> Hi Priyank,
> 
> I changed my mind since those mails were sent. I don't think setKey/Value
> are very useful. They couldn't be used with the default Kafka
> deserializers, only for deserializers implemented by the user, and then
> only if they were declared to implement SerializableDeserializer. I agree
> that we should remove them, and I'm not going to undo anything currently in
> the PR (unless there are objections on the PR of course)
> 
> With regard to getting rid of the builder pattern, I think it is a pretty
> nice pattern for Java. It looks to me like it should be possible to declare
> and configure the builder with "component:", and then pass it to the
> KafkaSpoutConfig constructor with "ref:" after (which lets you avoid
> calling build()). Doesn't this work?
> 
> 2017-06-12 23:32 GMT+02:00 Priyank Shah :
> 
>> Hi Stig,
>> 
>> I think PR https://github.com/apache/storm/pull/2155/files you created
>> gets rid of setKey and setValue. I am fine with it and in fact that’s what
>> I was suggesting in first place. However, your last two email replies
>> suggest something else. Just making sure you are not going to undo anything
>> in the PR and that we are same page about this change. i.e. no setKey or
>> setValue. Either for SerializableDeserializer implementations or Kafka
>> Deserializer interface. Only string in fqcn as a property.
>> 
>> The other thing I propose is to get rid of the builder class. Reason is
>> constructing an object with builder pattern requires builder.build and that
>> does work well with flux yaml. I think we should be careful about
>> implementing new connectors and make sure they work with yaml as well. I
>> have commented on the PR as well. Unless, someone else has a different
>> opinion can you address that as well?
>> 
>> On 6/10/17, 2:05 AM, "Stig Døssing"  wrote:
>> 
>>Priyank, I was a bit too hasty in the last response. The setKey/Value
>>functions are necessary when users want to set only the key or the
>> value
>>deserializer. I think we should keep those. It may be possible to
>>deduplicate the functionality on the API by removing the Builder
>>constructors that takes deserializers, and by getting rid of the
>>setKey/Value functions that take Class instances, since those seem
>> like a
>>duplication of the consumer config functionality. This should get rid
>> of a
>>lot of the overloads.
>> 
>>2017-06-10 10:20 GMT+02:00 Stig Døssing :
>> 
>>> Harsha,
>>> 
>>> +1 for simplifying away those methods that are just setting consumer
>>> config. The properties I think we should keep are all the spout
>>> configuration (timeouts, retry handling, tuple construction). Maybe
>> we
>>> deprecate the consumer config functions on 1.x and remove them on
>> master?
>>> 
>>> Priyank,
>>> 
>>> When the spout is declared, it takes type parameters to define the
>> key and
>>> value types of the consumer. We are able to check at compile time
>> that the
>>> deserializers match those expected types.
>>> e.g.
>>> SerializableDeserializer des = ...
>>> 
>>> KafkaSpoutConfig config = KafkaSpoutConfig.builder("
>> dummy",
>>> "dummy")
>>>.setKey(des)
>>>.build();
>>> 
>>> KafkaSpout wrongTypeSpout = new KafkaSpout<>(config);
>>> 
>>> will not compile, while
>>> 
>>> SerializableDeserializer des = ...
>>> 
>>> KafkaSpoutConfig config = KafkaSpoutConfig.builder("
>> dummy",
>>> "dummy")
>>>.setKey(des)
>>>.build();
>>> 
>>> KafkaSpout spout = new KafkaSpout<>(config);
>>> 
>>> will. If we want to simplify the API, maybe we should just mirror the
>>> KafkaConsumer API more closely and remove the Builder setKey/Value
>> methods.
>>> I can't think of a reason why a user should need to create a Builder
>> of one
>>> type, and then change the type later with setKey/Value. The
>> deserializers
>>> can just go in through the Bu

Re: Use of the KafkaConsumer.subscribe API in the storm-kafka-client spout

[Priyank Shah]

Hi Stig,

I think PR https://github.com/apache/storm/pull/2155/files you created gets rid 
of setKey and setValue. I am fine with it and in fact that’s what I was 
suggesting in first place. However, your last two email replies suggest 
something else. Just making sure you are not going to undo anything in the PR 
and that we are same page about this change. i.e. no setKey or setValue. Either 
for SerializableDeserializer implementations or Kafka Deserializer interface. 
Only string in fqcn as a property.

The other thing I propose is to get rid of the builder class. Reason is 
constructing an object with builder pattern requires builder.build and that 
does work well with flux yaml. I think we should be careful about implementing 
new connectors and make sure they work with yaml as well. I have commented on 
the PR as well. Unless, someone else has a different opinion can you address 
that as well?

On 6/10/17, 2:05 AM, "Stig Døssing"  wrote:

Priyank, I was a bit too hasty in the last response. The setKey/Value
functions are necessary when users want to set only the key or the value
deserializer. I think we should keep those. It may be possible to
deduplicate the functionality on the API by removing the Builder
constructors that takes deserializers, and by getting rid of the
setKey/Value functions that take Class instances, since those seem like a
duplication of the consumer config functionality. This should get rid of a
lot of the overloads.

2017-06-10 10:20 GMT+02:00 Stig Døssing :

> Harsha,
>
> +1 for simplifying away those methods that are just setting consumer
> config. The properties I think we should keep are all the spout
> configuration (timeouts, retry handling, tuple construction). Maybe we
> deprecate the consumer config functions on 1.x and remove them on master?
>
> Priyank,
>
> When the spout is declared, it takes type parameters to define the key and
> value types of the consumer. We are able to check at compile time that the
> deserializers match those expected types.
> e.g.
> SerializableDeserializer des = ...
>
> KafkaSpoutConfig config = 
KafkaSpoutConfig.builder("dummy",
> "dummy")
> .setKey(des)
> .build();
>
> KafkaSpout wrongTypeSpout = new KafkaSpout<>(config);
>
> will not compile, while
>
> SerializableDeserializer des = ...
>
> KafkaSpoutConfig config = 
KafkaSpoutConfig.builder("dummy",
> "dummy")
> .setKey(des)
> .build();
>
> KafkaSpout spout = new KafkaSpout<>(config);
>
> will. If we want to simplify the API, maybe we should just mirror the
> KafkaConsumer API more closely and remove the Builder setKey/Value 
methods.
> I can't think of a reason why a user should need to create a Builder of 
one
> type, and then change the type later with setKey/Value. The deserializers
> can just go in through the Builder constructor.
>
> About KafkaTuple, I think it was done this way originally since requiring
> users to subclass KafkaTuple would be a breaking change. If we want to do
> it it should go in 2.x only. I'm not necessarily opposed to doing it, but 
I
> don't really have a strong opinion on it.
>
> Hugo,
>
> I appreciate that the subscribe API is a major new feature of the 0.9
> consumer, but I can't come up with any reason to use it in Storm. I don't
> think we should support it just because it is there. As mentioned 
upthread,
> the features offered by that API are already covered by Storm, so I'm not
> seeing the value to having it. If we can't come up with a use case for it 
I
> don't see a reason to allow users to configure it, especially given the
> non-obvious problems users who choose it are likely to run into.
>
>
> 2017-06-10 <20%2017%2006%2010> 6:03 GMT+02:00 Harsha :
>
>> Dynamic assignment is what causing all the issues that we see now.
>> 1. Duplicates at the start of the KafkaSpout and upon any rebalance
>> 2. Trident Kafka Spout not holding the transactional batches.
>> Many corner cases can easily produce duplicates.
>>
>> There is no point in keeping dynamic assignment given all the issues
>> that are showing up.
>> Here is the excerpt from Kafka consumer docs
>> https://www-us.apache.org/dist/kafka/0.10.0.1/javadoc/org/
>> apache/kafka/clients/consumer/KafkaConsumer.html
>> "If the process itself is highly available and will be restarted if it
>> fails (perhaps using a cluster management framework like YARN, Mesos, or
>> AWS facilities, or as part of a stream processing framework). In this
>> case there is no need for Kafka to detect the failure and reassign the
>> partition since the consuming process will be restarted on another
>> machine."
>>
>> Manual assignment is 

Re: [Discussion]: Storm Improvemement Proposal (SIP) to discuss changes

[Priyank Shah]

+1 for SIPs including a new connector. The person writing SIP can provide 
details about the external system for which connector is being written to let 
others know why a certain design decision was made. This will make it easy for 
reviewers.

On 6/9/17, 5:24 PM, "Satish Duggana"  wrote:

+1 for SIPs. It is so useful as mentioned by others in earlier mails. It
would be very useful for new contributors and others who are looking out
for a feature design and decisions taken etc.

Whenever a minor feature is added to a connector it may not need a separate
SIP but the existing README can be updated with details for users. It can
be discussed and decided apropos whether a SIP is really required for any
enhancement which is not really big.


On Sat, Jun 10, 2017 at 5:13 AM, Roshan Naik  wrote:

> If I am looking at the Kafka site correctly, I see that Kafka has a total
> of 167 KIPs so far.
> So I assume that minor new features would not be parrt of the SIP ?
>
> Unlike Kafka, since Storm has a number of connectors (that keep growing),
> I am speculating the SIP process might get somewhat unwieldy if it were to
> track little changes in each of the connectors.
>
> Also thinking that a SIP may not be needed to justify a new connector, but
> useful if we are replacing an old connector with a new one.
>
> -roshan
>
>
>
> On 6/9/17, 3:19 PM, "Harsha"  wrote:
>
> Hi Bobby,
>In general, a KIP is required for adding New features,
> config
>changes or backward-incompatible changes. Don't require
>adding a KIP for bug-fixes.  Devs who wants to add any
>features will write up a wiki which has JIRA link, mailing
>list discussion link and outline the Motivation, Public
>interface changes and protocol changes etc ..a good example
>here is
>https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> 48+Delegation+token+support+for+Kafka.
> They can start the discussion thread once its ready and once everyone
> agrees its in a good shape, a Vote thread starts . Once there are
> required votes are in one can start the PR process and get it merged
> in.
>Each release we can collect what features/fixes especially
> to
>public interfaces that went in and roll it out in release
>notes. This will give a better idea for the users on what
>changed and added from previous version.
>  We can only enforce this to new feature/config/backward
>  incompatible change. Having this go through the discussion
>  phase will give us the early feedback and potentially caught
>  any issues before the implementation.
> Thanks,
> Harsha
>
> On Fri, Jun 9, 2017 at 2:24 PM Bobby Evans 
 >
> wrote:
>
> Can you please explain how KIP currently works and how you would
> like to see something similar in storm?
> If we make the process more formal we will probably have less
> people
> contributing, but we will probably have better overall patches.  
It
> is a balancing act and having never used KIP I would like to
> understand it better before going all in on it.
> - Bobby
>
>
> On Friday, June 9, 2017, 4:09:38 PM CDT, Stig Døssing
>  wrote:
>
> This sounds like a good idea. KIPs seem to work well for Kafka.
> It's
> easy
> for discussions to get lost or just not seen on the mailing list.
>
> 2017-06-09 21:36 GMT+02:00 Harsha :
>
> > Hi All,
> >  We’ve seen good adoption of KIP approach in Kafka
> community
> >  and would like to see we adopt the similar approach for
> storm
> >  as well.
> > Its hard to keep track of proposed changes and mailing list
> threads to
> > know what all changes that are coming into  and what
> design/backward
> > incompatible changes being approved.  It will be good to have
> this
> > documented and go through discussion then Vote phase to get them
> > approved before we merge the PRs. This will keep everyone
> informed of
> > what changes happened even if they are not following the mailing
> list
> > they can go to wiki to see the list of changes went into a
> release.
> > Community overall will be well informed of the changes as well.
> Would
> > like to hear your thoughts.
> >
> > Thanks,
> > Harsha
  

Re: Use of the KafkaConsumer.subscribe API in the storm-kafka-client spout

[Priyank Shah]

Just realized that from my email it was not clear what were my comments versus 
Stig’s. I have put mine in [] below.

On 6/9/17, 5:42 PM, "Priyank Shah"  wrote:

I want to add a few things other than the issues raised by Sriharsha and 
Hugo. I am pasting one of the other emails that I sent sometime back about 
cleaning up KafkaSpoutConfig. Stig responsed to that email. Trying to answer 
that in this email so that it is all in one place. Answers in line.



Hi Priyank,

For question 1 I can think of a couple of reasons (not sure how important
they are though): Using FQCN makes it impossible to check the generic type
of the deserializer, so you'd be able to pass the wrong type of
deserializer to the spout (e.g. a spout that otherwise expects String but
is passed an Integer deserializer). Passing class instances instead of
using FQCNs makes it possible to set up some configuration for the
serializer (e.g. call a constructor with a parameter). I would assume
that's why the KafkaConsumer API also supports passing instances instead of
FQCNs. We have SerializableDeserializer as a bit of a service to the user.
Kafka's Deserializer isn't inherently serializable, and if you configure
your spout to use a deserializer that isn't serializable, the topology
submission will fail when Nimbus tries to serialize the spout.

  Priyank [ Not sure where and how do we check the type of Deserializer to make 
sure that it’s a String and not Integer by error. Can you elaborate on that? It 
will be anyway throwing a RuntimeException in the worker. Developers are 
already aware that message needs to be deserialized with correct deserializer. 
And if that’s not the case then it should be fixed. As far as configuration 
state for deserializer object is concerned, api already has configure method 
here 
https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/common/serialization/Deserializer.java#L37
 Do you see any strong reason why one would use an instance of Deserializer 
with some constructor args and setter methods to create the state and not just 
use the string value for FQCN. The state can be passed through consumer 
properties in configure method as mentioned before. Getting rid of 4 instance 
variables keyDes, keyDesClass, valueDes, valueDesClazz and the related 
interface SerializableDeserializer and the different combinations of overloaded 
constructors because of that will be a big cleanup. Instead of doing a service 
to the user we are making the interface of KafkaSpoutConfig and hence 
KafkaSpout more complicated. On the same lines, I still don’t see a reason for 
creating a deserializer object on client side and serialize it to send it over 
to Nimbus. Let me know if you have a concrete example of where one would need 
this.]

For question 2, I think we didn't want to require people to subclass
KafkaTuple. If they aren't emitting to multiple streams, it's unnecessary
for the tuple to subclass KafkaTuple. I'm almost certain we don't want to
change it to List. Splitting a Kafka message into multiple
tuples can already be done by adding a splitter bolt that does that after
the KafkaSpout in the topology. I don't really see a good reason for
putting this functionality in the spout, especially since it complicates
ack/commit management a bit more if we have to keep track of multiple
tuples per Kafka message. Is there a reason you'd like the message split in
the spout, rather than in a downstream bolt?

Priyank [   I agree we don’t want return type to be List because of 
offset management you mentioned. I think making it return KafkaTuple still 
makes sense. I would rather prefer forcing user to subclass KafkaTuple. Any 
strong reasons for not requiring users to subclass it? The way it is currently, 
it can misguide the user. List can be anything, including 
List. I think having the instanceof check in KafkaSpout for object 
returned in apply method is unnecessary. We should change the apply method of 
the interface to return KafkaTuple and have DefaultRecordTranslator handle that 
i.e. return a KafkaTuple with getStream returning the default stream. The only 
contract user needs to bind to is that the stream in KafkaTuple has to be one 
of the streams returned by RecordTranslator that is used in declareOutputFields.

Talking about manual assignment, I remember already doing something like 
that in KinesisSpout. If everyone is okay with the above changes I want to take 
up the task of making the changes as discussed in this email thread(whatever 
conclusion we reach) and even switch the spout to manual assignments. I know 
it’s going to be backward incompatible but I prefer that since we will be 
cleaning up lot of stuff. We can decide which release to pick if at all we vote 
for these backward i

Re: Use of the KafkaConsumer.subscribe API in the storm-kafka-client spout

[Priyank Shah]

I want to add a few things other than the issues raised by Sriharsha and Hugo. 
I am pasting one of the other emails that I sent sometime back about cleaning 
up KafkaSpoutConfig. Stig responsed to that email. Trying to answer that in 
this email so that it is all in one place. Answers in line.



Hi Priyank,

For question 1 I can think of a couple of reasons (not sure how important
they are though): Using FQCN makes it impossible to check the generic type
of the deserializer, so you'd be able to pass the wrong type of
deserializer to the spout (e.g. a spout that otherwise expects String but
is passed an Integer deserializer). Passing class instances instead of
using FQCNs makes it possible to set up some configuration for the
serializer (e.g. call a constructor with a parameter). I would assume
that's why the KafkaConsumer API also supports passing instances instead of
FQCNs. We have SerializableDeserializer as a bit of a service to the user.
Kafka's Deserializer isn't inherently serializable, and if you configure
your spout to use a deserializer that isn't serializable, the topology
submission will fail when Nimbus tries to serialize the spout.

Not sure where and how do we check the type of Deserializer to make sure that 
it’s a String and not Integer by error. Can you elaborate on that? It will be 
anyway throwing a RuntimeException in the worker. Developers are already aware 
that message needs to be deserialized with correct deserializer. And if that’s 
not the case then it should be fixed. As far as configuration state for 
deserializer object is concerned, api already has configure method here 
https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/common/serialization/Deserializer.java#L37
 Do you see any strong reason why one would use an instance of Deserializer 
with some constructor args and setter methods to create the state and not just 
use the string value for FQCN. The state can be passed through consumer 
properties in configure method as mentioned before. Getting rid of 4 instance 
variables keyDes, keyDesClass, valueDes, valueDesClazz and the related 
interface SerializableDeserializer and the different combinations of overloaded 
constructors because of that will be a big cleanup. Instead of doing a service 
to the user we are making the interface of KafkaSpoutConfig and hence 
KafkaSpout more complicated. On the same lines, I still don’t see a reason for 
creating a deserializer object on client side and serialize it to send it over 
to Nimbus. Let me know if you have a concrete example of where one would need 
this.

For question 2, I think we didn't want to require people to subclass
KafkaTuple. If they aren't emitting to multiple streams, it's unnecessary
for the tuple to subclass KafkaTuple. I'm almost certain we don't want to
change it to List. Splitting a Kafka message into multiple
tuples can already be done by adding a splitter bolt that does that after
the KafkaSpout in the topology. I don't really see a good reason for
putting this functionality in the spout, especially since it complicates
ack/commit management a bit more if we have to keep track of multiple
tuples per Kafka message. Is there a reason you'd like the message split in
the spout, rather than in a downstream bolt?

I agree we don’t want return type to be List because of offset 
management you mentioned. I think making it return KafkaTuple still makes 
sense. I would rather prefer forcing user to subclass KafkaTuple. Any strong 
reasons for not requiring users to subclass it? The way it is currently, it can 
misguide the user. List can be anything, including List. I 
think having the instanceof check in KafkaSpout for object returned in apply 
method is unnecessary. We should change the apply method of the interface to 
return KafkaTuple and have DefaultRecordTranslator handle that i.e. return a 
KafkaTuple with getStream returning the default stream. The only contract user 
needs to bind to is that the stream in KafkaTuple has to be one of the streams 
returned by RecordTranslator that is used in declareOutputFields.

Talking about manual assignment, I remember already doing something like that 
in KinesisSpout. If everyone is okay with the above changes I want to take up 
the task of making the changes as discussed in this email thread(whatever 
conclusion we reach) and even switch the spout to manual assignments. I know 
it’s going to be backward incompatible but I prefer that since we will be 
cleaning up lot of stuff. We can decide which release to pick if at all we vote 
for these backward incompatible changes. 

2017-05-10 2:47 GMT+02:00 Priyank Shah :

I was going through new kafka spout code and had a couple of questions.


1.   https://github.com/apache/storm/blob/master/external/
storm-kafka-client/src/main/java/org/apache/storm/kafka/
spout/KafkaSpoutConfig.java#L98 The instance variable at that line and
fol

Some questions on storm-kaka-client KafkaSpout

[Priyank Shah]

I was going through new kafka spout code and had a couple of questions.


1.   
https://github.com/apache/storm/blob/master/external/storm-kafka-client/src/main/java/org/apache/storm/kafka/spout/KafkaSpoutConfig.java#L98
 The instance variable at that line and following 3 lines. Why do we need them? 
Because of that we have Builder constructors with different parameters for key 
and value deserializers. We even have 
https://github.com/apache/storm/blob/master/external/storm-kafka-client/src/main/java/org/apache/storm/kafka/spout/SerializableDeserializer.java
 Not sure if we really need it. 
https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/clients/consumer/KafkaConsumer.java#L555
  already has a constructor that takes in Properties or a Map and if 
key.deserializer and value.deserialzer keys are set to fqcns then it will 
instantiate them and take care of them at 
https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/clients/consumer/KafkaConsumer.java#L642
 . And we already have setProp method on builder to set different kafka configs 
that will be passed to KafkaConsumer constructor. We can get rid of the 
SerializableDeserializer interface and a bunch of constructors and instance 
variables related to Key and Value Deserializable.

2.   We have a RecordTranslator interface that is used to 
declareOutputFields at 
https://github.com/apache/storm/blob/master/external/storm-kafka-client/src/main/java/org/apache/storm/kafka/spout/KafkaSpout.java#L486
 and then we have this special instanceof check here 
https://github.com/apache/storm/blob/master/external/storm-kafka-client/src/main/java/org/apache/storm/kafka/spout/KafkaSpout.java#L333
 Why is return type of apply a List ? Should we change it to 
List? That way we can get rid of instanceof check and support 
multiple tuples emitted for one kafka message.


Fixes for above two might not be backward compatible but if everyone is okay 
with above changes then I can create a patch.

Re: [VOTE] Java Code Style Standard for Apache Storm.

[Priyank Shah]

[1] Google Java Style Guide

[2] Sun Java Code 
Conventions

[3] HBase 
style
 (sorry all I could find was an eclipse XML config)

[4] Hadoop style which is 
described as java but with 2 spaces instead of 4.

On 4/26/17, 12:06 PM, "Hugo Da Cruz Louro"  wrote:

My vote is in the following order:

[1] Google Java Style Guide
[2] Sun Java Code Conventions
[3] HBase style
[4] Hadoop style, which is described as java but with 2 spaces instead of 4

Thanks,
Hugo

> On Apr 26, 2017, at 12:03 PM, P. Taylor Goetz  wrote:
> 
> [1 ] Google Java Style 
Guide>
> 
> [ 2] Sun Java Code 
Conventions>
> 
> [ 3] HBase 
style>
 (sorry all I could find was an eclipse XML config)
> 
> [4 ] Hadoop style> which is described as 
java but with 2 spaces instead of 4.
> 
> [ ] Other (Please specify)
> 
> -Taylor
> 
>> On Apr 26, 2017, at 2:50 PM, Bobby Evans  
wrote:
>> 
>> We would like to adopt a code style standard for Apache Storm.  Please 
rank the following with 1 being the most desired and 4 being the least desired 
(5 if you have a write in choice).  This is not an official vote as per the 
ByLaws, but we will probably go with whichever wins the most 1 votes (unless it 
is really close and then we may go with some STeVe like ranking, but I want to 
avoid it because it is hard and I might get the math wrong) This is open to 
everyone so please vote.
>> 
>> 
>> [ ] Google Java Style 
Guide
>> 
>> [ ] Sun Java Code 
Conventions
>> 
>> [ ] HBase 
style
 (sorry all I could find was an eclipse XML config)
>> 
>> [ ] Hadoop style 
which is described as java but with 2 spaces instead of 4.
>> 
>> [ ] Other (Please specify)
>> 
>> 
>> I apologize if the formatting is bad I am forced to use a corporate 
sponsored mail client that somehow messed up plain text formatting.  I don't 
know how.
>> 
>> 
>> - Bobby
> 

Re: [DISCUSS] Move non-connectors modules to out of external

[Priyank Shah]

+1 to moving non-conncectors to top level. I think we should keep 
stom-kafka-monitor under external or connectors(after renaming).

Jungtaek, just to clarify on what you said regarding storm core referencing 
storm-kafka-monitor. Like you said its just calling the script from ui jvm. 
There is no dependency in terms of class files needed to run the script from 
ui. The script itself adds a –cp argument and all it needs is 
storm-kafka-monitor jar in classpath. As far as packaging the script is 
concerned we can do what Satish suggested. i.e. move it to storm-kafka-monitor 
in source and while packaging put it under bin. Reiterating to make sure I am 
not mis-understanding anything.

On 3/24/17, 9:14 AM, "Harsha Chintalapani"  wrote:

+1 on moving non-connectors to top-level like sql and storm-perf.
Regarding storm-kafka-monitor we can move this into "util" folder or keep
in the external.
-Harsha

On Fri, Mar 24, 2017 at 2:23 AM Satish Duggana 
wrote:

> storm-kafka-monitor is not a connector by itself but it is related to 
kafka
> connectors. So, any utility related to that connector should be part of
> that connector module(can be a submodule) instead of a top level module.
> core/ui uses this utility referring directly in a hacky way, which we may
> want to fix later. storm-kafka-monitor script exists in bin directory 
which
> can be moved to storm-kafka-monitor module and the same script can be
> packaged as part of storm/bin directory while packaging the distribution.
>
> Thanks,
> ~Satish.
>
> On Fri, Mar 24, 2017 at 1:07 PM, Jungtaek Lim  wrote:
>
> > storm-kafka-monitor is referred by storm-core, though it's referenced 
via
> > executing command. Yes it's a bit odd to place it as top directory, but
> > it's not a connector for that reason too. Neither is ideal for me, so
> > ironically, either is fine.
> >
> > - Jungtaek Lim (HeartSaVioR)
> >
> > 2017년 3월 24일 (금) 오후 4:19, Satish Duggana 님이
> 작성:
> >
> > > +1 except for storm-kafka-monitor module as this utility is more about
> > > querying topic/partition offsets of kafka spouts in a topology. Do not
> we
> > > want to push this module into connectors/kafka as a submodule along
> with
> > > other submodules including old/new kafka spout modules?
> > >
> > > Thanks,
> > > Satish.
> > >
> > >
> > > On Fri, Mar 24, 2017 at 12:10 PM, Arun Iyer 
> > wrote:
> > >
> > > > +1
> > > >
> > > > Makes sense to move the non-connectors to top level and keep only 
the
> > > > connectors under “connectors” folder.
> > > >
> > > >
> > > > On 3/24/17, 12:00 PM, "Jungtaek Lim"  wrote:
> > > >
> > > > >(Sent this yesterday but can't find this from storm-dev mbox...
> > sending
> > > it
> > > > >again)
> > > > >
> > > > >Hi dev,
> > > > >
> > > > >I'd like to start discussion regarding moving non-connectors 
modules
> > out
> > > > of
> > > > >external, maybe top directory.
> > > > >
> > > > >"external" directory has non-connectors (SQL, Flux,
> > storm-kafka-monitor,
> > > > >storm-submit-tools), and except Flux, others should be placed to 
the
> > > > binary
> > > > >dist. since Storm itself (not from user topology) needs to refer
> them.
> > > > >
> > > > >They're actually tied to the core of Storm, so I feel that it would
> be
> > > > >better to treat them (including Flux) as non-external, maybe same
> > level
> > > as
> > > > >storm-core.
> > > > >(I'm not sure what "external" actually means for Storm project 
btw.)
> > > > >
> > > > >In addition, after doing that I'd like to change the directory name
> > > > >"external" to "connector" or so, so that the name could be
> > > self-describing
> > > > >and we can only place connectors to that directory.
> > > > >(I know it would be painful for already opened pull requests, so no
> > > strong
> > > > >opinion regarding this.)
> > > > >
> > > > >Looking forward to your opinion!
> > > > >
> > > > >Thanks,
> > > > >Jungtaek Lim (HeartSaVioR)
> > > >
> > > >
> > >
> >
>

Re: [VOTE] Release Apache Storm 1.1.0 (RC2)

[Priyank Shah]

+1(non-binding)

- Verified signatures, md5s and shas for all files
- Mvn clean install on source successsully
- Started daemons and killed them successfully for both source and binary
- Submitted WodCountTopology and FastWordCountTopology for both as well
- Tried out different commands from UI like Deactivate, Activate, Rebalance and 
Kill
- All of test runs were done on single node cluster on my local machine with 
each daemon running in its own jvm


On 2/27/17, 12:26 PM, "Alexandre Vermeerbergen"  
wrote:

Hello,

+1 (non-binding)

My first attempt to deploy our Supervision app (made of 10 topologies with
a Redis server for sharing memory) intto a small Storm 1.1.0rc2 cluster (1
nimbus node, 2 supervisors, 3 zookeeper VMs) failed because operations on
Nimbus VM were taking too long.

I realized that this was because my Nimbus VM was a "m1.small" (provided by
OUTSCALE Europe 2 datacenter) and it was too slow. I have the impression
that this setup was OK with Storm 1.0.2, but I have no proofs. I re-did my
1.1.0rc2 with a m1.medium VM for Nimbus and everything works like a charm.

One detail: on our Kafka spouts, I got the following message instead of a
lag information:


Offset lags for kafka not supported for older versions. Please update kafka
spout to latest version.
Is there some documentation which provides the clear requirements to get
Kafka monitoring from Nimbus UI?
PS: I'm using Kafka Spout (not yet migrated to Kafka Client Spout with
Kafka 0.9.01 libraries.

Best regards,
Alexandre



2017-02-27 14:13 GMT+01:00 Alexandre Vermeerbergen :

> Hello,
>
> Tried with our supervision topologies on a small cluster (1 Nimbus VM, 3
> Zookeeper VMs, 2 Supervisor VMs).
>
> For the moment I have trouble with Nimbus & Nimbus UI startup time, the
> "bin/storm nimbus" starts the following process with runs for quite a long
> time (5 to 10 minutes) before the actual Nimbus process (which has a "java
> -server" pattern instead of "java -client" in ps aux list) is actually
> started:
>
> root 29178 81.0  0.9 1965584 18404 pts/0   Sl   13:04   0:00
> /usr/local/jdk/bin/java -client -Dstorm.options= -Dstorm.conf.file= -cp
> /usr/local/Storm/storm-stable/lib/servlet-api-2.5.jar:/usr/l
> ocal/Storm/storm-stable/lib/log4j-slf4j-impl-2.8.jar:/usr/lo
> cal/Storm/storm-stable/lib/minlog-1.3.0.jar:/usr/local/Storm
> /storm-stable/lib/storm-rename-hack-1.1.0.jar:/usr/local/
> Storm/storm-stable/lib/slf4j-api-1.7.21.jar:/usr/local/
> Storm/storm-stable/lib/disruptor-3.3.2.jar:/usr/local/Storm/
> storm-stable/lib/log4j-core-2.8.jar:/usr/local/Storm/storm-
> stable/lib/log4j-over-slf4j-1.6.6.jar:/usr/local/Storm/
> storm-stable/lib/reflectasm-1.10.1.jar:/usr/local/Storm/
> storm-stable/lib/ring-cors-0.1.5.jar:/usr/local/Storm/
> storm-stable/lib/clojure-1.7.0.jar:/usr/local/Storm/storm-
> stable/lib/kryo-3.0.3.jar:/usr/local/Storm/storm-stable/
> lib/asm-5.0.3.jar:/usr/local/Storm/storm-stable/lib/
> objenesis-2.1.jar:/usr/local/Storm/storm-stable/lib/log4j-
> api-2.8.jar:/usr/local/Storm/storm-stable/lib/storm-core-1.
> 1.0.jar:/usr/local/Storm/storm-stable/extlib/flux-
> wrappers-1.1.0.jar:/usr/local/Storm/storm-stable/extlib/
> flux-core-1.1.0.jar:/usr/local/Storm/storm-stable/conf
> org.apache.storm.command.config_value java.library.path
>
> I have no logs generated by this first process.
> Is there a way to turn on some logs to understand what takes so long?
>
> Best regards,
> Alexandre
>
>
> 2017-02-24 21:40 GMT+01:00 P. Taylor Goetz :
>
>> This is a call to vote on releasing Apache Storm 1.1.0 (rc2)
>>
>> Full list of changes in this release:
>>
>> https://git-wip-us.apache.org/repos/asf?p=storm.git;a=blob_p
>> lain;f=CHANGELOG.md;hb=17a2017fb644e353fb2a0f5bf50d400ee28036ba
>>
>> The tag/commit to be voted upon is v1.1.0:
>>
>> https://git-wip-us.apache.org/repos/asf?p=storm.git;a=blob;f
>> =CHANGELOG.md;h=0352f6d27dad1c88c24bb99fd894a211ea6c1e71;hb=
>> 7c8ea09e09160d36bba7e363ed53781816b46597
>>
>> The source archive being voted upon can be found here:
>>
>> https://dist.apache.org/repos/dist/dev/storm/apache-storm-1.
>> 1.0-rc2/apache-storm-1.1.0-src.tar.gz
>>
>> Other release files, signatures and digests can be found here:
>>
>> https://dist.apache.org/repos/dist/dev/storm/apache-storm-1.1.0-rc2/
>>
>> The release artifacts are signed with the following key:
>>
>> https://git-wip-us.apache.org/repos/asf?p=storm.git;a=blob_p
>> lain;f=KEYS;hb=22b832708295fa2c15c4f3c70ac0d2bc6fded4bd
>>
>> The Nexus staging repository for this release is:
>>
>> https://repository.apache.org/content/repositories/orgapache

[jira] [Commented] (STORM-2021) storm-kinesis missing licenses

[Priyank Shah (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-2021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15408143#comment-15408143
 ] 

Priyank Shah commented on STORM-2021:
-

[~sriharsha] [~revans2] Sorry I missed the license in the test files. Just 
created a PR.

> storm-kinesis missing licenses
> --
>
> Key: STORM-2021
> URL: https://issues.apache.org/jira/browse/STORM-2021
> Project: Apache Storm
>  Issue Type: Bug
>  Components: storm-kinesis
>Affects Versions: 2.0.0
>Reporter: Robert Joseph Evans
>Assignee: Priyank Shah
>Priority: Blocker
>
> {code}
> Unapproved licenses:
>   
> external/storm-kinesis/src/test/java/org/apache/storm/kinesis/spout/test/KinesisBoltTest.java
>   
> external/storm-kinesis/src/test/java/org/apache/storm/kinesis/spout/test/KinesisSpoutTopology.java
>   
> external/storm-kinesis/src/test/java/org/apache/storm/kinesis/spout/test/TestRecordToTupleMapper.java
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [PROPOSAL] submitting topology with adding jars and maven artifacts

[Priyank Shah]

Hi Jungtaek,

For adding jars and maven at submission, you have used the word Submitter. Is 
Submitter the person running storm jar command or is Submitter the java code 
that actually submits it to Nimbus?
Also, I did not quite understand the --jars option. If you could please 
elaborate a little on that, that will be great

Thanks
Priyank






On 8/2/16, 7:05 AM, "Jungtaek Lim"  wrote:

>Ah, Satish you got the point. I meant copied version of files in
>supervisor, but itself can be isolated.
>I didn't think about removing blobs, and it seems not easy to do.
>
>Jungtaek Lim (HeartSaVIoR)
>
>
>2016년 8월 2일 (화) 오후 7:35, Satish Duggana 님이 작성:
>
>> Hi Jungtaek,
>> With the current proposal, are we removing blob store files referred by a
>> topology when it is killed?
>>
>> Thanks,
>> Satish.
>>
>> On Tue, Aug 2, 2016 at 3:50 PM, Jungtaek Lim  wrote:
>>
>> > Hi Satish,
>> >
>> > Thanks for reviewing and share your idea.
>> >
>> > Yes this is shared dependencies vs isolated dependencies.
>> > If we name file of dependency to contain group name, artifact name, and
>> > version, that can be shared.
>> > One downside of this approach is storage space since we don't know when
>> > it's safe to delete without additional care, but I'm curious that disk
>> > fills up due to dependency blob jar files in normal situation.
>> > So I think we're OK to do this but I would like to see others opinions.
>> >
>> > Btw, I'm designing details based on proposal. Will update to this thread
>> if
>> > there're not covered things with initial design.
>> >
>> > Thanks,
>> > Jungtaek Lim (HeartSaVioR)
>> >
>> > 2016년 8월 2일 (화) 오후 6:58, Satish Duggana 님이 작성:
>> >
>> > > Hi Jungtaek,
>> > > Proposal looks good to me. Good that we are not going with other
>> > > alternative using mutable classloader etc.
>> > >
>> > > Good to have the mentioned config in proposal to add those jars before
>> or
>> > > after storm core/libs. There is a property Config.
>> > > TOPOLOGY_CLASSPATH_BEGINNING which is to have that value as initial
>> > > classpath and that should continue to be working as expected even with
>> > the
>> > > new configuration.
>> > >
>> > > One enhancement which we may want to add to the existing proposal.
>> > > When --packages are used, storm submitter can upload those dependencies
>> > in
>> > > blob store with a defined naming convention so that same set of
>> packages
>> > > are not uploaded again and they can be used again for other topologies
>> if
>> > > they use same package.
>> > >
>> > > Thanks,
>> > > Satish.
>> > >
>> > >
>> > > On Tue, Aug 2, 2016 at 7:25 AM, Jungtaek Lim 
>> wrote:
>> > >
>> > > > Hi dev,
>> > > >
>> > > > This is proposal review thread for submitting topology with adding
>> jars
>> > > and
>> > > > maven artifacts. This is also following up discussion thread for
>> > > > [DISCUSSION]
>> > > > Policy of resolving dependencies for non storm-core modules.[1]
>> > > >
>> > > > I've written design doc which also describes motivation on this.
>> > > >
>> > > >
>> > >
>> >
>> https://cwiki.apache.org/confluence/display/STORM/A.+Design+doc%3A+adding+jars+and+maven+artifacts+at+submission
>> > > >
>> > > > Please review this and comment to "this thread" instead of wiki page
>> so
>> > > > that all devs can be notified for the update.
>> > > >
>> > > > Thanks,
>> > > > Jungtaek Lim (HeartSaVioR)
>> > > >
>> > > > [1]
>> > > >
>> > > >
>> > >
>> >
>> http://mail-archives.apache.org/mod_mbox/storm-dev/201607.mbox/%3CCAF5108jByyJLTKrV_P4fS=dj8rsr_o5oubzqbviscggsc1c...@mail.gmail.com%3E
>> > > >
>> > >
>> >
>>

[jira] [Updated] (STORM-1891) Incorporate trident kafka spout for the kafka offset lag tool

[Priyank Shah (JIRA)]

 [ 
https://issues.apache.org/jira/browse/STORM-1891?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Priyank Shah updated STORM-1891:

Description: This is a follow up JIRA to support trident kafka spouts for 
tool introduced in https://issues.apache.org/jira/browse/STORM-1136  (was: We 
store kafkaspout offsets in zkroot + id path in zookeeper. Kafka provides a 
utility and a protocol request to fetch latest offsets into topic
{code}
example:
bin/kafka-run-classh.sh kafka.tools.GetOffsetTool 
{code}

we should provide a way for the user to check how far the kafka spout read into 
topic and whats the lag. If we can expose this via UI even better.)

> Incorporate trident kafka spout for the kafka offset lag tool
> -
>
> Key: STORM-1891
> URL: https://issues.apache.org/jira/browse/STORM-1891
> Project: Apache Storm
>  Issue Type: Improvement
>  Components: storm-kafka
>    Reporter: Priyank Shah
>Assignee: Priyank Shah
>
> This is a follow up JIRA to support trident kafka spouts for tool introduced 
> in https://issues.apache.org/jira/browse/STORM-1136



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (STORM-1891) Incorporate trident kafka spout for the kafka offset lag tool

[Priyank Shah (JIRA)]

Priyank Shah created STORM-1891:
---

 Summary: Incorporate trident kafka spout for the kafka offset lag 
tool
 Key: STORM-1891
 URL: https://issues.apache.org/jira/browse/STORM-1891
 Project: Apache Storm
  Issue Type: Improvement
  Components: storm-kafka
Reporter: Priyank Shah
Assignee: Priyank Shah


We store kafkaspout offsets in zkroot + id path in zookeeper. Kafka provides a 
utility and a protocol request to fetch latest offsets into topic
{code}
example:
bin/kafka-run-classh.sh kafka.tools.GetOffsetTool 
{code}

we should provide a way for the user to check how far the kafka spout read into 
topic and whats the lag. If we can expose this via UI even better.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (STORM-1136) Provide a bin script to check consumer lag from KafkaSpout to Kafka topic offsets and integrate with storm ui

[Priyank Shah (JIRA)]

 [ 
https://issues.apache.org/jira/browse/STORM-1136?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Priyank Shah updated STORM-1136:

Summary: Provide a bin script to check consumer lag from KafkaSpout to 
Kafka topic offsets and integrate with storm ui  (was: Provide a bin script to 
check consumer lag from KafkaSpout to Kafka topic offsets)

> Provide a bin script to check consumer lag from KafkaSpout to Kafka topic 
> offsets and integrate with storm ui
> -
>
> Key: STORM-1136
> URL: https://issues.apache.org/jira/browse/STORM-1136
> Project: Apache Storm
>  Issue Type: Improvement
>  Components: storm-kafka
>Reporter: Sriharsha Chintalapani
>    Assignee: Priyank Shah
>
> We store kafkaspout offsets in zkroot + id path in zookeeper. Kafka provides 
> a utility and a protocol request to fetch latest offsets into topic
> {code}
> example:
> bin/kafka-run-classh.sh kafka.tools.GetOffsetTool 
> {code}
> we should provide a way for the user to check how far the kafka spout read 
> into topic and whats the lag. If we can expose this via UI even better.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (STORM-1839) Kinesis Spout

[Priyank Shah (JIRA)]

 [ 
https://issues.apache.org/jira/browse/STORM-1839?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Priyank Shah reassigned STORM-1839:
---

Assignee: Priyank Shah  (was: Sriharsha Chintalapani)

> Kinesis Spout
> -
>
> Key: STORM-1839
> URL: https://issues.apache.org/jira/browse/STORM-1839
> Project: Apache Storm
>  Issue Type: Improvement
>Reporter: Sriharsha Chintalapani
>    Assignee: Priyank Shah
>
> As Storm is increasingly used in Cloud environments. It will great to have a 
> Kinesis Spout integration in Apache Storm.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Storm Topology - Paralellism Help

[Priyank Shah]

I ran in to a similar issue about inability to create native threads. Can you 
check nproc and nofile values for your user? I tried 65536 for hard and 32768 
for soft in /etc/security/limits.conf
Also, make sure to check the same values in 
/etc/security/limits.d/90-nproc.conf as well. Those value override the previous 
ones and they solved the problem for me.




On 5/12/16, 8:40 AM, "numan goceri"  wrote:

>I've set in the config 2 Workers (each one is related to a port - 
>supervisor.slots.ports: [6700, 6701]). I also tried to involve 2 more 
>additional ports but in all cases I've noticed that somehow only 1 port 
>(worker) has been used according to Storm UI. Why is that is also a question.
>
>For my spout and 2 bolts, I set different number of parallelism_hints to see 
>the reaction of the topology. (started from 2 until 100).. I'm either having 
>"unable to create new native thread", or the hbase connection error.
>As you said, the best would be to find out why so many threads are created and 
>how to close these threads.It should be independent from the amount of tuples 
>that I'm sending via kafka producer, shoudn't it?
> --- Numan Göceri 
>
>On Thursday, May 12, 2016 4:26 PM, Bobby Evans  wrote:
> 
>
> max user processes  (-u) 47555 is the limit that is probably 
> being hit.  So you probably have 47555+ threads in execution on that host.
>I honestly don't know where all of the threads are coming from.  Storm will 
>create two threads for each executor, and a small fixed number more for the 
>system in each worker.  By increasing the parallelism hint of your 
>bolts/spouts and increasing the number of workers the number of threads will 
>go up and linux treats threads against your max user processes limit.
>Most of the time when I run into a situation like this I try to get a jstack 
>or a heap dump of the workers to try and understand what all of the threads 
>are associated with.  My guess is that something is happening with HBase, like 
>you said, where the client is not shutting down all of it's threads when you 
>close the table. 
> - Bobby 
>
>On Thursday, May 12, 2016 9:00 AM, numan goceri  
> wrote:
> 
>
> Hi Bobby,
>Thanks for your reply. I've searched a lot on the net, what OutOfMemoryError 
>actually means. And it seems like an OS resource problem.Link#1: 
>http://stackoverflow.com/questions/16789288/java-lang-outofmemoryerror-unable-to-create-new-native-threadLink#2:
> https://plumbr.eu/outofmemoryerror/unable-to-create-new-native-thread
>I'm running this on my VM-Player and so is the ulimit 
>configurations:[root@my_Project]# ulimit -a
>core file size  (blocks, -c) 0
>data seg size   (kbytes, -d) unlimited
>scheduling priority (-e) 0
>file size   (blocks, -f) unlimited
>pending signals (-i) 47555
>max locked memory   (kbytes, -l) 64
>max memory size (kbytes, -m) unlimited
>open files  (-n) 1024
>pipe size(512 bytes, -p) 8
>POSIX message queues (bytes, -q) 819200
>real-time priority  (-r) 0
>stack size  (kbytes, -s) 10240
>cpu time   (seconds, -t) unlimited
>max user processes  (-u) 47555
>virtual memory  (kbytes, -v) unlimited
>file locks  (-x) unlimited
>[root@my_Project]#
>
>I'm guessing Storm is not killing any threads, which have been opened so far.
>if it's opening a new thread, each time when I want to "put" a result into the 
>hbase, then it should kill, since I call "close" afterwards.
>My code is basically like followings:
>prepare(){this.connection = 
>ConnectionFactory.createConnection(constructConfiguration());
>this.allTuplesTable = connection.getTable("allTuples");
>this.resultsTable = connection.getTable("results");
>}
>execute(){readFromAllTuples();
>... 
>this.resultsTable.put(put);this.resultsTable.close();}cleanup(){
> this.allTuplesTable.close();this.resultsTable.close();}
>
>I'd appreciate if you could show me a way to continue.. the data, I'm dealing 
>with, should be not soo "big data" for Storm.
>
>
> --- Numan Göceri 
>
>On Thursday, May 12, 2016 3:31 PM, Bobby Evans  wrote:
> 
>
> Numan,
>Please read up a bit on what the OutOfMemeroyError actually means.  
>
>https://duckduckgo.com/?q=java.lang.OutOfMemoryError%3A+unable+to+create+new+native+thread&t=ffab
>
>Most of the time on Linux it means that you have hit a ulimit on the number of 
>processes/threads that your headless user can handle.  By increasing the 
>parallelism you did not make the problem go away, you just created a new 
>problem, and when you fixed that problem the original problem came back.  You 
>either need to increase the ulimit (possibly both hard and soft) for the user 
>your worker processes are running as, or you need to find a way to reduce the 
>number of threads that you are using.  
> - Bobby 
>
>On Thursday, May 12, 2016 

[jira] [Assigned] (STORM-1535) Storm-HDFS - When using keytab Kerberos ticket is not renewed with multiple bolts/executors

[Priyank Shah (JIRA)]

 [ 
https://issues.apache.org/jira/browse/STORM-1535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Priyank Shah reassigned STORM-1535:
---

Assignee: Priyank Shah  (was: Dan Bahir)

> Storm-HDFS - When using keytab Kerberos ticket is not renewed with multiple 
> bolts/executors
> ---
>
> Key: STORM-1535
> URL: https://issues.apache.org/jira/browse/STORM-1535
> Project: Apache Storm
>  Issue Type: Bug
>  Components: storm-hdfs
>Affects Versions: 0.10.0, 0.9.5
>Reporter: Dan Bahir
>Assignee: Priyank Shah
>
> When logging in with a keytab, if the topology has more than one instance of 
> an HDFS bolt then the ticket might not be automatically renewed.
> Bug has been fixed for HBase bolt, code is slightly different for HDFS bolt 
> and therefore requires further investigation



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (STORM-1668) Flux silently fails while setting a non-existent property.

[Priyank Shah (JIRA)]

Priyank Shah created STORM-1668:
---

 Summary: Flux silently fails while setting a non-existent property.
 Key: STORM-1668
 URL: https://issues.apache.org/jira/browse/STORM-1668
 Project: Apache Storm
  Issue Type: Bug
  Components: Flux
Affects Versions: 1.0.0, 2.0.0
Reporter: Priyank Shah
Assignee: Priyank Shah
 Fix For: 1.0.0


Currently, if a yaml file has a property with a name that does not exist on the 
java topology component object then flux silently fails by logging a message 
and does not throw an exception. This needs to be changed so that flux throws 
the exception failing topology submission so that user can take corrective 
action.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (STORM-1663) Clicking on an active topology from storm ui home page and then refreshing the page throws exception

[Priyank Shah (JIRA)]

Priyank Shah created STORM-1663:
---

 Summary: Clicking on an active topology from storm ui home page 
and then refreshing the page throws exception
 Key: STORM-1663
 URL: https://issues.apache.org/jira/browse/STORM-1663
 Project: Apache Storm
  Issue Type: Bug
Affects Versions: 2.0.0
Reporter: Priyank Shah


The exception thrown is:

org.apache.storm.thrift.transport.TTransportException
at 
org.apache.storm.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)
at 
org.apache.storm.thrift.transport.TTransport.readAll(TTransport.java:86)
at 
org.apache.storm.thrift.transport.TFramedTransport.readFrame(TFramedTransport.java:129)
at 
org.apache.storm.thrift.transport.TFramedTransport.read(TFramedTransport.java:101)
at 
org.apache.storm.thrift.transport.TTransport.readAll(TTransport.java:86)
at 
org.apache.storm.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429)
at 
org.apache.storm.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318)
at 
org.apache.storm.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219)
at 
org.apache.storm.thrift.TServiceClient.receiveBase(TServiceClient.java:77)
at 
org.apache.storm.generated.Nimbus$Client.recv_getTopologyPageInfo(Nimbus.java:1243)
at 
org.apache.storm.generated.Nimbus$Client.getTopologyPageInfo(Nimbus.java:1228)
at org.apache.storm.ui.core$topology_page.invoke(core.clj:638)
at org.apache.storm.ui.core$fn__3662.invoke(core.clj:987)
at 
org.apache.storm.shade.compojure.core$make_route$fn__302.invoke(core.clj:93)
at 
org.apache.storm.shade.compojure.core$if_route$fn__290.invoke(core.clj:39)
at 
org.apache.storm.shade.compojure.core$if_method$fn__283.invoke(core.clj:24)
at 
org.apache.storm.shade.compojure.core$routing$fn__308.invoke(core.clj:106)
at clojure.core$some.invoke(core.clj:2570)
at org.apache.storm.shade.compojure.core$routing.doInvoke(core.clj:106)
at clojure.lang.RestFn.applyTo(RestFn.java:139)
at clojure.core$apply.invoke(core.clj:632)
at 
org.apache.storm.shade.compojure.core$routes$fn__312.invoke(core.clj:111)
at 
org.apache.storm.shade.ring.middleware.json$wrap_json_params$fn__1204.invoke(json.clj:56)
at 
org.apache.storm.shade.ring.middleware.multipart_params$wrap_multipart_params$fn__765.invoke(multipart_params.clj:103)
at 
org.apache.storm.shade.ring.middleware.reload$wrap_reload$fn__724.invoke(reload.clj:22)
at 
org.apache.storm.ui.helpers$requests_middleware$fn__3091.invoke(helpers.clj:50)
at org.apache.storm.ui.core$catch_errors$fn__3837.invoke(core.clj:1250)
at 
org.apache.storm.shade.ring.middleware.keyword_params$wrap_keyword_params$fn__2852.invoke(keyword_params.clj:27)
at 
org.apache.storm.shade.ring.middleware.nested_params$wrap_nested_params$fn__2892.invoke(nested_params.clj:65)
at 
org.apache.storm.shade.ring.middleware.params$wrap_params$fn__2823.invoke(params.clj:55)
at 
org.apache.storm.shade.ring.middleware.multipart_params$wrap_multipart_params$fn__765.invoke(multipart_params.clj:103)
at 
org.apache.storm.shade.ring.middleware.flash$wrap_flash$fn__3075.invoke(flash.clj:14)
at 
org.apache.storm.shade.ring.middleware.session$wrap_session$fn__3063.invoke(session.clj:43)
at 
org.apache.storm.shade.ring.middleware.cookies$wrap_cookies$fn__2991.invoke(cookies.clj:160)
at 
org.apache.storm.shade.ring.util.servlet$make_service_method$fn__2729.invoke(servlet.clj:127)
at 
org.apache.storm.shade.ring.util.servlet$servlet$fn__2733.invoke(servlet.clj:136)
at 
org.apache.storm.shade.ring.util.servlet.proxy$javax.servlet.http.HttpServlet$ff19274a.service(Unknown
 Source)
at 
org.apache.storm.shade.org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:654)
at 
org.apache.storm.shade.org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1320)
at 
org.apache.storm.logging.filters.AccessLoggingFilter.handle(AccessLoggingFilter.java:47)
at 
org.apache.storm.logging.filters.AccessLoggingFilter.doFilter(AccessLoggingFilter.java:39)
at 
org.apache.storm.shade.org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1291)
at 
org.apache.storm.shade.org.eclipse.jetty.servlets.CrossOriginFilter.handle(CrossOriginFilter.java:247)
at 
org.apache.storm.shade.org.eclipse.jetty.servlets.CrossOriginFilter.doFilter(CrossOriginFilter.java:210)
at 
org.apache.storm.shade.org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1291)
at 
org.apache.storm.shade.org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:443)
at

[jira] [Closed] (STORM-1626) HBaseBolt tuple counts too high in storm ui.

[Priyank Shah (JIRA)]

 [ 
https://issues.apache.org/jira/browse/STORM-1626?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Priyank Shah closed STORM-1626.
---
Resolution: Duplicate

Duplicate of https://issues.apache.org/jira/browse/STORM-1654

> HBaseBolt tuple counts too high in storm ui.
> 
>
> Key: STORM-1626
> URL: https://issues.apache.org/jira/browse/STORM-1626
> Project: Apache Storm
>  Issue Type: Bug
>  Components: storm-hbase
>Affects Versions: 1.0.0
>Reporter: Priyank Shah
>Priority: Blocker
>
> When a storm topology has an HbaseBolt the storm ui numbers for executed and 
> acked tuples seem to be too high for HbaseBolt component. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (STORM-1626) HBaseBolt tuple counts too high in storm ui.

[Priyank Shah (JIRA)]

Priyank Shah created STORM-1626:
---

 Summary: HBaseBolt tuple counts too high in storm ui.
 Key: STORM-1626
 URL: https://issues.apache.org/jira/browse/STORM-1626
 Project: Apache Storm
  Issue Type: Bug
  Components: storm-hbase
Affects Versions: 1.0.0
Reporter: Priyank Shah
Priority: Blocker


When a storm topology has an HbaseBolt the storm ui numbers for executed and 
acked tuples seem to be too high for HbaseBolt component. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (STORM-1160) add hadoop-auth dependency back to storm-core

[Priyank Shah (JIRA)]

 [ 
https://issues.apache.org/jira/browse/STORM-1160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Priyank Shah reassigned STORM-1160:
---

Assignee: Priyank Shah  (was: Sriharsha Chintalapani)

> add hadoop-auth dependency back to storm-core
> -
>
> Key: STORM-1160
> URL: https://issues.apache.org/jira/browse/STORM-1160
> Project: Apache Storm
>  Issue Type: Bug
>Reporter: Sriharsha Chintalapani
>    Assignee: Priyank Shah
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (STORM-1458) Add check to see if nimbus is already running.

[Priyank Shah (JIRA)]

Priyank Shah created STORM-1458:
---

 Summary: Add check to see if nimbus is already running.
 Key: STORM-1458
 URL: https://issues.apache.org/jira/browse/STORM-1458
 Project: Apache Storm
  Issue Type: Bug
Reporter: Priyank Shah


Adding a validation to ensure if a nimbus is already running another nimbus 
fails on startup before modifying any zookeeper state.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (STORM-1451) Storm topology submission can take upto 5 minutes in HA mode when zookeeper reconnects. Nimbus discovery can fail when zookeeper reconnect happens.

[Priyank Shah (JIRA)]

Priyank Shah created STORM-1451:
---

 Summary: Storm topology submission can take upto 5 minutes in HA 
mode when zookeeper reconnects. Nimbus discovery can fail when zookeeper 
reconnect happens.
 Key: STORM-1451
 URL: https://issues.apache.org/jira/browse/STORM-1451
 Project: Apache Storm
  Issue Type: Bug
  Components: storm-core
Reporter: Priyank Shah


We discovered couple of issues when testing storm under vagrant clusters.
1. When a nimbus zookeeper connection is dropped and reconnected the ephemeral 
entry for that host under /nimbuses gets deleted and is not auto recreated when 
reconnection happens. This means even though nimbus is up no client will be 
able to actually discover it. To address this issue we now have a listener that 
listens for RECONNECT events and recreates the entry.
2. Zookeeper is eventual consistent when multiple clients are involved. In 
practice we did not notice this issue but in the vagrant cluster due to 
resource constrained it was pretty evident that updates created by leader 
nimbuses were not observed by other nimbus host unless they waited for a few 
second. Due to this topology submission can take upto 5 minutes which is super 
bad user experience.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Welcome new Apache Storm Committers/PMC Members

[Priyank Shah]

Congratulation everyone!




On 12/8/15, 3:17 PM, "Hugo Da Cruz Louro"  wrote:

>Congrats everyone. Looking forward to working with you!
>
>> On Dec 8, 2015, at 2:54 PM, Aaron.Dossett  wrote:
>> 
>> Thanks, everyone, and congratulations to the other new committers as well!
>> 
>> On 12/8/15, 4:12 PM, "임정택"  wrote:
>> 
>>> Congratulation! Looking forward to work with you all.
>>> 
>>> Best,
>>> Jungtaek Lim (HeartSaVioR)
>>> 
>>> On 2015년 12월 9일 (수) at 오전 6:55 P. Taylor Goetz  wrote:
>>> 
 I’m pleased to announce that the following individuals have joined as
 Apache Storm Committers/PMC Members:
 
 - Longda Feng
 - Arun Mahadevan
 - Boyang Jerry Peng
 - Matthias J. Sax
 - Aaron Dossett
 
 Longda, Arun, Jerry, Matthias, and Aaron have all demonstrated technical
 merit and dedication to Apache Storm and its community, and as PMC
 members
 they will help drive innovation and community development.
 
 Please join me in welcoming and congratulating Londa, Arun, Jerry,
 Matthias, and Aaron. We look forward to your continued dedication to the
 Storm community.
 
 -Taylor
 
>> 
>

[jira] [Commented] (STORM-904) move storm bin commands to java and provide appropriate bindings for windows and linux

[Priyank Shah (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15009320#comment-15009320
 ] 

Priyank Shah commented on STORM-904:


I have not worked on this in a while. The windows implementation is not done 
yet. Besides, the Unix based implementation will also need to be updated based 
on any additions to the command line. It needs an upmerge and additional review 
as well. I have updated the JIRA to Stop Progress for now. Sorry for not doing 
it sooner. 

> move storm bin commands to java and provide appropriate bindings for windows 
> and linux
> --
>
> Key: STORM-904
> URL: https://issues.apache.org/jira/browse/STORM-904
> Project: Apache Storm
>  Issue Type: Bug
>  Components: storm-core
>Reporter: Sriharsha Chintalapani
>Assignee: Priyank Shah
>
> Currently we have python and .cmd implementation for windows. This is 
> becoming increasing difficult upkeep both versions. Lets make all the main 
> code of starting daemons etc. to java and provider wrapper scripts in shell 
> and batch for linux and windows respectively. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Welcome JStorm Community!

[Priyank Shah]

Welcome everyone from JStorm community. Look forward to working with you all.




On 11/4/15, 8:49 AM, "Hugo Da Cruz Louro"  wrote:

>Welcome to the Community! It’s exciting to have JStorm as part of the Apache 
>family. Looking forward to working and getting to meet you all sometime during 
>our Apache contributors journey.
>
>Please feel free to reach out in case you need anything.
>
>Best,
>Hugo
>
>On Nov 3, 2015, at 2:05 PM, P. Taylor Goetz 
>mailto:ptgo...@apache.org>> wrote:
>
>For those who may not be aware, Alibaba has donated JStorm to Apache Storm, 
>and we have completed the IP Clearance process. JStorm is a fork of Storm with 
>the clojure components reimplemented in Java, and includes a number of 
>enhancements in terms of performance and features. JStorm is used widely in 
>production at Alibaba across many different areas.
>
>Of equal importance is the JStorm community, which has expressed a commitment 
>to join forces with the Apache Storm community to bring JStorm’s improvements 
>to Apache Storm and collaborate on new features and improvements.
>
>Over the coming days and weeks we will have a number of discussions on the 
>dev@ list regarding how to best approach merging both the incoming code and 
>associated community.
>
>Please join me in welcoming the JStorm contributors to the Apache Storm 
>community.
>
>
>JStorm community:
>
>Welcome abord! We’re glad you’re here.
>
>As you’ve probably noticed, all discussions and decision making at Apache 
>takes place on the mailing list in English. I understand that might be a 
>burden for non-English speakers, but there a number of Apache Storm 
>contributors that speak both English and Chinese that would likely be willing 
>to help translate. Google translate, while not perfect, is also helpful in 
>some cases. We don’t expect perfect translation — just enough to get your 
>point across is fine.
>
>When signing your name on an email, etc., please use the western name of 
>yourself and others. This makes it easier for those of us who can't read Asian 
>characters to identify individuals.
>
>If you’re new to Apache and the way it works, here is a good starting point 
>for learning “The Apache Way”:
>
>https://community.apache.org/contributors/
>
>Feel free to introduce yourself to the Storm community by replying to this 
>thread and letting us know a little more about yourself.
>
>Again, thank you and welcome!
>
>-Taylor
>

Re: [VOTE] Accept Alibaba JStorm Code Donation

[Priyank Shah]

+1

From: "P. Taylor Goetz"
Reply-To: mailto:dev@storm.apache.org>>
Date: Tuesday, October 27, 2015 at 11:05 AM
To: mailto:dev@storm.apache.org>>
Subject: Re: [VOTE] Accept Alibaba JStorm Code Donation

+1 (binding)

On Oct 27, 2015, at 1:48 PM, P. Taylor Goetz 
mailto:ptgo...@apache.org>> wrote:

All,

The IP Clearance process for the Alibaba JStorm code donation has completed.

The IP Clearance Status document can be found here:

http://incubator.apache.org/ip-clearance/storm-jstorm.html

The source code can be found at https://github.com/alibaba/jstorm with the 
following git commit SHA: e935da91a897797dad56e24c4ffa57860ac91878

This is a VOTE to accept the code donation, and import the donated code into 
the Apache Storm git repository. Discussion regarding how to proceed with 
merging the codebases can take place in separate thread.

[ ] +1 Accept the Alibaba JStorm code donation.
[ ] +0 Indifferent
[ ] -1 Do not accept the code donation because…

This VOTE will be open for at least 72 hours.

-Taylor

[jira] [Created] (STORM-1129) Storm should use topology name instead of ids for url in storm UI.

[Priyank Shah (JIRA)]

Priyank Shah created STORM-1129:
---

 Summary: Storm should use topology name instead of ids for url in 
storm UI.
 Key: STORM-1129
 URL: https://issues.apache.org/jira/browse/STORM-1129
 Project: Apache Storm
  Issue Type: Bug
Reporter: Priyank Shah
Assignee: Priyank Shah


Currently, in storm UI details about a topology can be viewed at a URL which 
has a topology id as a query parameter. When a topology is updated and  
redeployed a new id is assigned by storm and existing URL(and any bookmarks 
relying on it) for the topology do not work since the id has changed. We should 
change it so that topology name is used instead of id.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (STORM-1017) If ignoreZkOffsets set true,KafkaSpout will reset zk offset when recover from failure.

[Priyank Shah (JIRA)]

 [ 
https://issues.apache.org/jira/browse/STORM-1017?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Priyank Shah resolved STORM-1017.
-
Resolution: Fixed

> If ignoreZkOffsets set true,KafkaSpout will reset zk offset when recover from 
> failure.
> --
>
> Key: STORM-1017
> URL: https://issues.apache.org/jira/browse/STORM-1017
> Project: Apache Storm
>  Issue Type: Bug
>  Components: storm-kafka
>Reporter: Renkai Ge
>    Assignee: Priyank Shah
>
> when ignoreZkOffsets set true and startOffsetTime = 
> kafka.api.OffsetRequest.EarliestTime().
> workers running -> topology shutdown by user and restart -> workers will read 
> from earliest time again
> workers running -> one of workers shutdown by accident and supervisor restart 
> the worker -> what offset will the restarted worker read from?
> More details on 
> https://github.com/apache/storm/pull/493#issuecomment-135783234
> It will cause a lot of unwanted duplicated messages in some scenes.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (STORM-1061) Create integration tests around ignoreZkOffsets config for kafka spout - core and trident

[Priyank Shah (JIRA)]

 [ 
https://issues.apache.org/jira/browse/STORM-1061?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Priyank Shah updated STORM-1061:

Description: Related to STORM-1017 changes. Add tests to make sure that 
ignoreZkOffsets config is used only user kills and resubmits the topology and 
not when a worker is crashed and restarted.  (was: when ignoreZkOffsets set 
true and startOffsetTime = kafka.api.OffsetRequest.EarliestTime().
workers running -> topology shutdown by user and restart -> workers will read 
from earliest time again
workers running -> one of workers shutdown by accident and supervisor restart 
the worker -> what offset will the restarted worker read from?

More details on https://github.com/apache/storm/pull/493#issuecomment-135783234
It will cause a lot of unwanted duplicated messages in some scenes.)
 Issue Type: Improvement  (was: Bug)

> Create integration tests around ignoreZkOffsets config for kafka spout - core 
> and trident
> -
>
> Key: STORM-1061
> URL: https://issues.apache.org/jira/browse/STORM-1061
> Project: Apache Storm
>  Issue Type: Improvement
>  Components: storm-kafka
>Reporter: Priyank Shah
>Assignee: Priyank Shah
>
> Related to STORM-1017 changes. Add tests to make sure that ignoreZkOffsets 
> config is used only user kills and resubmits the topology and not when a 
> worker is crashed and restarted.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (STORM-1061) Create integration tests around ignoreZkOffsets config for kafka spout - core and trident

[Priyank Shah (JIRA)]

Priyank Shah created STORM-1061:
---

 Summary: Create integration tests around ignoreZkOffsets config 
for kafka spout - core and trident
 Key: STORM-1061
 URL: https://issues.apache.org/jira/browse/STORM-1061
 Project: Apache Storm
  Issue Type: Bug
  Components: storm-kafka
Reporter: Priyank Shah
Assignee: Priyank Shah


when ignoreZkOffsets set true and startOffsetTime = 
kafka.api.OffsetRequest.EarliestTime().
workers running -> topology shutdown by user and restart -> workers will read 
from earliest time again
workers running -> one of workers shutdown by accident and supervisor restart 
the worker -> what offset will the restarted worker read from?

More details on https://github.com/apache/storm/pull/493#issuecomment-135783234
It will cause a lot of unwanted duplicated messages in some scenes.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-1017) If ignoreZkOffsets set true,KafkaSpout will reset zk offset when recover from failure.

[Priyank Shah (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1017?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14902893#comment-14902893
 ] 

Priyank Shah commented on STORM-1017:
-

https://github.com/apache/storm/pull/754/files

> If ignoreZkOffsets set true,KafkaSpout will reset zk offset when recover from 
> failure.
> --
>
> Key: STORM-1017
> URL: https://issues.apache.org/jira/browse/STORM-1017
> Project: Apache Storm
>  Issue Type: Bug
>  Components: storm-kafka
>Reporter: Renkai Ge
>Assignee: Priyank Shah
>
> when ignoreZkOffsets set true and startOffsetTime = 
> kafka.api.OffsetRequest.EarliestTime().
> workers running -> topology shutdown by user and restart -> workers will read 
> from earliest time again
> workers running -> one of workers shutdown by accident and supervisor restart 
> the worker -> what offset will the restarted worker read from?
> More details on 
> https://github.com/apache/storm/pull/493#issuecomment-135783234
> It will cause a lot of unwanted duplicated messages in some scenes.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (STORM-965) excessive logging in storm when non-kerberos client tries to connect.

[Priyank Shah (JIRA)]

 [ 
https://issues.apache.org/jira/browse/STORM-965?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Priyank Shah resolved STORM-965.

Resolution: Fixed

> excessive logging in storm when non-kerberos client tries to connect.
> -
>
> Key: STORM-965
> URL: https://issues.apache.org/jira/browse/STORM-965
> Project: Apache Storm
>  Issue Type: Bug
>Reporter: Sriharsha Chintalapani
>    Assignee: Priyank Shah
>
> Ambari alert service pings nimbus thrift port to check if the process is 
> alive or not. This causes unnecessary logging 
> b.s.s.a.k.KerberosSaslTransportPlugin [ERROR] Storm server failed to open 
> transport to interact with a client during session initiation: 
> java.lang.NullPointerException
> java.lang.NullPointerException: null
>   at org.apache.thrift7.transport.TSocket.open(TSocket.java:170) 
> ~[storm-core-0.10.0.2.3.0.0-2557.jar:0.10.0.2.3.0.0-2557]
>   at 
> org.apache.thrift7.transport.TSaslTransport.open(TSaslTransport.java:266) 
> ~[storm-core-0.10.0.2.3.0.0-2557.jar:0.10.0.2.3.0.0-2557]
>   at 
> org.apache.thrift7.transport.TSaslServerTransport.open(TSaslServerTransport.java:41)
>  ~[storm-core-0.10.0.2.3.0.0-2557.jar:0.10.0.2.3.0.0-2557]
>   at 
> org.apache.thrift7.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:216)
>  ~[storm-core-0.10.0.2.3.0.0-2557.jar:0.10.0.2.3.0.0-2557]
>   at 
> backtype.storm.security.auth.kerberos.KerberosSaslTransportPlugin$TUGIAssumingTransportFactory$1.run(KerberosSaslTransportPlugin.java:195)
>  [storm-core-0.10.0.2.3.0.0-2557.jar:0.10.0.2.3.0.0-2557]
>   at 
> backtype.storm.security.auth.kerberos.KerberosSaslTransportPlugin$TUGIAssumingTransportFactory$1.run(KerberosSaslTransportPlugin.java:192)
>  [storm-core-0.10.0.2.3.0.0-2557.jar:0.10.0.2.3.0.0-2557]
>   at java.security.AccessController.doPrivileged(Native Method) 
> [na:1.7.0_67]
>   at javax.security.auth.Subject.doAs(Subject.java:415) [na:1.7.0_67]
>   at 
> backtype.storm.security.auth.kerberos.KerberosSaslTransportPlugin$TUGIAssumingTransportFactory.getTransport(KerberosSaslTransportPlugin.java:191)
>  [storm-core-0.10.0.2.3.0.0-2557.jar:0.10.0.2.3.0.0-2557]
>   at 
> org.apache.thrift7.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:269)
>  [storm-core-0.10.0.2.3.0.0-2557.jar:0.10.0.2.3.0.0-2557]
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>  [na:1.7.0_67]
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>  [na:1.7.0_67]
>   at java.lang.Thread.run(Thread.java:745) [na:1.7.0_67]
> 2015-07-17T10:35:52.723+ o.a.t.s.TThreadPoolServer [ERROR] Error occurred 
> during processing of message.
> java.lang.NullPointerException: null
>   at 
> backtype.storm.security.auth.SaslTransportPlugin$TUGIWrapProcessor.process(SaslTransportPlugin.java:120)
>  ~[storm-core-0.10.0.2.3.0.0-2557.jar:0.10.0.2.3.0.0-2557]
>   at 
> org.apache.thrift7.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:285)
>  ~[storm-core-0.10.0.2.3.0.0-2557.jar:0.10.0.2.3.0.0-2557]
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>  [na:1.7.0_67]
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>  [na:1.7.0_67]
>   at java.lang.Thread.run(Thread.java:745) [na:1.7.0_67]



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (STORM-1039) Storm Web UI gives 500 error: Remove commons-codec shading, commons-codec is used by hadoop authentication library which is used during ui authentication in secured envir

[Priyank Shah (JIRA)]

Priyank Shah created STORM-1039:
---

 Summary: Storm Web UI gives 500 error: Remove commons-codec 
shading, commons-codec is used by hadoop authentication library which is used 
during ui authentication in secured environment.
 Key: STORM-1039
 URL: https://issues.apache.org/jira/browse/STORM-1039
 Project: Apache Storm
  Issue Type: Bug
Reporter: Priyank Shah
Assignee: Priyank Shah






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (STORM-999) Add support for user specified UGI - (UserGroupInformation) for storm hive connector

[Priyank Shah (JIRA)]

 [ 
https://issues.apache.org/jira/browse/STORM-999?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Priyank Shah updated STORM-999:
---
Description: In a non-secure environment, Storm Hive component that 
provides interaction with Hive from storm currently does that as the user storm 
with which the worker process had been started. We want to allow the component 
to interact with Hive as the user provided instead of user running the worker 
process

> Add support for user specified UGI - (UserGroupInformation) for storm hive 
> connector
> 
>
> Key: STORM-999
> URL: https://issues.apache.org/jira/browse/STORM-999
> Project: Apache Storm
>  Issue Type: Sub-task
>    Reporter: Priyank Shah
>Assignee: Priyank Shah
>
> In a non-secure environment, Storm Hive component that provides interaction 
> with Hive from storm currently does that as the user storm with which the 
> worker process had been started. We want to allow the component to interact 
> with Hive as the user provided instead of user running the worker process



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (STORM-998) Add support for user specified UGI - (UserGroupInformation) for storm hbase connector

[Priyank Shah (JIRA)]

 [ 
https://issues.apache.org/jira/browse/STORM-998?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Priyank Shah updated STORM-998:
---
Description: In a non-secure environment, Storm HBase component that 
provides interaction with HBase from storm currently does that as the user 
storm with which the worker process had been started. We want to allow the 
component to interact with HBase as the user provided instead of user running 
the worker process  (was: In a non-secure environment, Storm HBase component 
that provides interaction with HBase from storm currently does that as the user 
storm with which the worker process had been started. We want to allow the 
component to interact with hdfs as the user provided instead of user running 
the worker process)

> Add support for user specified UGI - (UserGroupInformation) for storm hbase 
> connector
> -
>
> Key: STORM-998
> URL: https://issues.apache.org/jira/browse/STORM-998
> Project: Apache Storm
>  Issue Type: Sub-task
>    Reporter: Priyank Shah
>Assignee: Priyank Shah
>
> In a non-secure environment, Storm HBase component that provides interaction 
> with HBase from storm currently does that as the user storm with which the 
> worker process had been started. We want to allow the component to interact 
> with HBase as the user provided instead of user running the worker process



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (STORM-998) Add support for user specified UGI - (UserGroupInformation) for storm hbase connector

[Priyank Shah (JIRA)]

 [ 
https://issues.apache.org/jira/browse/STORM-998?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Priyank Shah updated STORM-998:
---
Description: In a non-secure environment, Storm HBase component that 
provides interaction with HBase from storm currently does that as the user 
storm with which the worker process had been started. We want to allow the 
component to interact with hdfs as the user provided instead of user running 
the worker process

> Add support for user specified UGI - (UserGroupInformation) for storm hbase 
> connector
> -
>
> Key: STORM-998
> URL: https://issues.apache.org/jira/browse/STORM-998
> Project: Apache Storm
>  Issue Type: Sub-task
>    Reporter: Priyank Shah
>Assignee: Priyank Shah
>
> In a non-secure environment, Storm HBase component that provides interaction 
> with HBase from storm currently does that as the user storm with which the 
> worker process had been started. We want to allow the component to interact 
> with hdfs as the user provided instead of user running the worker process



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (STORM-997) Add support for user specified UGI - (UserGroupInformation) for storm hdfs connector

[Priyank Shah (JIRA)]

 [ 
https://issues.apache.org/jira/browse/STORM-997?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Priyank Shah updated STORM-997:
---
Description: In a non-secure environment, Storm HDFS component that 
provides interaction with HDFS from storm currently does that as the user storm 
with which the worker process had been started. We want to allow the component 
to interact with hdfs as the user provided instead of user running the worker 
process

> Add support for user specified UGI - (UserGroupInformation) for storm hdfs 
> connector
> 
>
> Key: STORM-997
> URL: https://issues.apache.org/jira/browse/STORM-997
> Project: Apache Storm
>  Issue Type: Sub-task
>    Reporter: Priyank Shah
>Assignee: Priyank Shah
>
> In a non-secure environment, Storm HDFS component that provides interaction 
> with HDFS from storm currently does that as the user storm with which the 
> worker process had been started. We want to allow the component to interact 
> with hdfs as the user provided instead of user running the worker process



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (STORM-997) Add support for user specified UGI - (UserGroupInformation) for storm hdfs connector

[Priyank Shah (JIRA)]

Priyank Shah created STORM-997:
--

 Summary: Add support for user specified UGI - 
(UserGroupInformation) for storm hdfs connector
 Key: STORM-997
 URL: https://issues.apache.org/jira/browse/STORM-997
 Project: Apache Storm
  Issue Type: Sub-task
Reporter: Priyank Shah
Assignee: Priyank Shah






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Issue Comment Deleted] (STORM-997) Add support for user specified UGI - (UserGroupInformation) for storm hdfs connector

[Priyank Shah (JIRA)]

 [ 
https://issues.apache.org/jira/browse/STORM-997?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Priyank Shah updated STORM-997:
---
Comment: was deleted

(was: In a non-secure environment, Storm HDFS component that provides 
interaction with HDFS from storm currently does that as the user storm with 
which the worker process had been started. We want to allow the component to 
interact with hdfs as the user provided instead of user running the worker 
process.)

> Add support for user specified UGI - (UserGroupInformation) for storm hdfs 
> connector
> 
>
> Key: STORM-997
> URL: https://issues.apache.org/jira/browse/STORM-997
> Project: Apache Storm
>  Issue Type: Sub-task
>    Reporter: Priyank Shah
>Assignee: Priyank Shah
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (STORM-999) Add support for user specified UGI - (UserGroupInformation) for storm hive connector

[Priyank Shah (JIRA)]

Priyank Shah created STORM-999:
--

 Summary: Add support for user specified UGI - 
(UserGroupInformation) for storm hive connector
 Key: STORM-999
 URL: https://issues.apache.org/jira/browse/STORM-999
 Project: Apache Storm
  Issue Type: Sub-task
Reporter: Priyank Shah
Assignee: Priyank Shah






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (STORM-998) Add support for user specified UGI - (UserGroupInformation) for storm hbase connector

[Priyank Shah (JIRA)]

Priyank Shah created STORM-998:
--

 Summary: Add support for user specified UGI - 
(UserGroupInformation) for storm hbase connector
 Key: STORM-998
 URL: https://issues.apache.org/jira/browse/STORM-998
 Project: Apache Storm
  Issue Type: Sub-task
Reporter: Priyank Shah
Assignee: Priyank Shah






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-997) Add support for user specified UGI - (UserGroupInformation) for storm hdfs connector

[Priyank Shah (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-997?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14701690#comment-14701690
 ] 

Priyank Shah commented on STORM-997:


In a non-secure environment, Storm HDFS component that provides interaction 
with HDFS from storm currently does that as the user storm with which the 
worker process had been started. We want to allow the component to interact 
with hdfs as the user provided instead of user running the worker process.

> Add support for user specified UGI - (UserGroupInformation) for storm hdfs 
> connector
> 
>
> Key: STORM-997
> URL: https://issues.apache.org/jira/browse/STORM-997
> Project: Apache Storm
>  Issue Type: Sub-task
>    Reporter: Priyank Shah
>Assignee: Priyank Shah
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-904) move storm bin commands to java and provide appropriate bindings for windows and linux

[Priyank Shah (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14621252#comment-14621252
 ] 

Priyank Shah commented on STORM-904:


[~harsha_ch] please ignore the previous comment. You already mentioned it in 
the subject and i found it. storm.py and storm.cmd in bin folder of storm git 
repo source code.

> move storm bin commands to java and provide appropriate bindings for windows 
> and linux
> --
>
> Key: STORM-904
> URL: https://issues.apache.org/jira/browse/STORM-904
> Project: Apache Storm
>  Issue Type: Bug
>Reporter: Sriharsha Chintalapani
>Assignee: Priyank Shah
>
> Currently we have python and .cmd implementation for windows. This is 
> becoming increasing difficult upkeep both versions. Lets make all the main 
> code of starting daemons etc. to java and provider wrapper scripts in shell 
> and batch for linux and windows respectively. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-904) move storm bin commands to java and provide appropriate bindings for windows and linux

[Priyank Shah (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14621248#comment-14621248
 ] 

Priyank Shah commented on STORM-904:


[~harsha_ch] what is the directory where i can find all the python and cmd 
scripts?

> move storm bin commands to java and provide appropriate bindings for windows 
> and linux
> --
>
> Key: STORM-904
> URL: https://issues.apache.org/jira/browse/STORM-904
> Project: Apache Storm
>  Issue Type: Bug
>Reporter: Sriharsha Chintalapani
>Assignee: Priyank Shah
>
> Currently we have python and .cmd implementation for windows. This is 
> becoming increasing difficult upkeep both versions. Lets make all the main 
> code of starting daemons etc. to java and provider wrapper scripts in shell 
> and batch for linux and windows respectively. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (STORM-904) move storm bin commands to java and provide appropriate bindings for windows and linux

[Priyank Shah (JIRA)]

 [ 
https://issues.apache.org/jira/browse/STORM-904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Priyank Shah reassigned STORM-904:
--

Assignee: Priyank Shah  (was: Hugo Louro)

> move storm bin commands to java and provide appropriate bindings for windows 
> and linux
> --
>
> Key: STORM-904
> URL: https://issues.apache.org/jira/browse/STORM-904
> Project: Apache Storm
>  Issue Type: Bug
>Reporter: Sriharsha Chintalapani
>Assignee: Priyank Shah
>
> Currently we have python and .cmd implementation for windows. This is 
> becoming increasing difficult upkeep both versions. Lets make all the main 
> code of starting daemons etc. to java and provider wrapper scripts in shell 
> and batch for linux and windows respectively. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)