[GitHub] storm issue #1655: STORM-2067 Fix "array element type mismatch" from compute...

[HeartSaVioR]

Github user HeartSaVioR commented on the issue:

https://github.com/apache/storm/pull/1655
  
+1 from STORM-2057 #1656 (by @arunmahadevan )


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request #1656: STORM-2057 Support JOIN statement in Storm SQL

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/storm/pull/1656


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request #1655: STORM-2067 Fix "array element type mismatch" from ...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/storm/pull/1655


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm issue #1656: STORM-2057 Support JOIN statement in Storm SQL

[HeartSaVioR]

Github user HeartSaVioR commented on the issue:

https://github.com/apache/storm/pull/1656
  
Thanks @arunmahadevan for reviewing. Since I've also addressed style review 
comments I'll just go on merging this.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request #1682: STORM-2089 Replace Consumer of ISqlTridentDataSour...

[HeartSaVioR]

GitHub user HeartSaVioR opened a pull request:

https://github.com/apache/storm/pull/1682

STORM-2089 Replace Consumer of ISqlTridentDataSource with SqlTridentConsumer

* issue link: https://issues.apache.org/jira/browse/STORM-2089

This is for handling INSERT for tuples as batch if possible.

I also changed storm-sql-kafka to expose State directly, but this is 
effectively same as current since TridentKafkaState.updateState() does update 
via fully synchronous way.
(by looping each tuple and calling Future.get() immediately)

```
public void updateState(List tuples, TridentCollector 
collector) {
String topic = null;
for (TridentTuple tuple : tuples) {
try {
topic = topicSelector.getTopic(tuple);

if(topic != null) {
Future result = producer.send(new 
ProducerRecord(topic,
mapper.getKeyFromTuple(tuple), 
mapper.getMessageFromTuple(tuple)));
try {
result.get();
} catch (ExecutionException e) {
String errorMsg = "Could not retrieve result for 
message with key = "
+ mapper.getKeyFromTuple(tuple) + " from 
topic = " + topic;
LOG.error(errorMsg, e);
throw new FailedException(errorMsg, e);
}
} else {
LOG.warn("skipping key = " + 
mapper.getKeyFromTuple(tuple) + ", topic selector returned null.");
}
} catch (Exception ex) {
String errorMsg = "Could not send message with key = " + 
mapper.getKeyFromTuple(tuple)
+ " to topic = " + topic;
LOG.warn(errorMsg, ex);
throw new FailedException(errorMsg, ex);
}
}
}
```

We could improve it, but it would be better to handle it as another issue 
if needed.

Please review and comment. Thanks!

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/HeartSaVioR/storm STORM-2089

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/storm/pull/1682.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1682


commit cc30363c7550ac34f9f8fed6abc4713a7680ee36
Author: Jungtaek Lim 
Date:   2016-09-13T02:40:13Z

STORM-2089 Replace Consumer of ISqlTridentDataSource with SqlTridentConsumer

* SqlTridentConsumer contains StateFactory and StateUpdater which is needed 
to store tuples to State via batch
* Apply the change to storm-sql-kafka




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm issue #1679: STORM-2087: storm-kafka-client - tuples not always being ...

[jfenc91]

Github user jfenc91 commented on the issue:

https://github.com/apache/storm/pull/1679
  
A larger refactor here is probably needed to make this more performant. 
These changes understandably seem to make the spout struggle to get the 
processing tuple count anywhere near the max spout pending. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: Flux exception while upgrading to Storm 1.0.2

[S G]

Hi Manu,

I do not see any 0.10.0 versions being downloaded in my ~/.m2 repo:

ls ~/.m2/repository/org/apache/storm/*

~/.m2/repository/org/apache/storm/flux:
1.0.2

~/.m2/repository/org/apache/storm/flux-core:
1.0.2

~/.m2/repository/org/apache/storm/storm:
1.0.2

~/.m2/repository/org/apache/storm/storm-kafka:
1.0.2

And yet surprisingly, when I do `mvn clean install` and explore the jar
file, I see two versions of KeyValueSchemeAsMultiScheme.class

jar -xvf my_shaded_fat_jar.jar

find . -name KeyValueSchemeAsMultiScheme.class
./org/apache/storm/kafka/KeyValueSchemeAsMultiScheme.class
./storm/kafka/KeyValueSchemeAsMultiScheme.class




On Mon, Sep 12, 2016 at 5:56 PM, Manu Zhang  wrote:

> Hi,
>
> Could you share your maven file ? When running `mvn package`, you should
> see what and which version of jar files are included ?
> That may give you a hint.
>
> Thanks,
> Manu
>
>
> On Tue, Sep 13, 2016 at 6:50 AM S G  wrote:
>
> > From the
> > https://github.com/apache/storm/tree/1.0.x-branch/external/storm-kafka
> > docs,
> >
> > "This means that pre 1.0 kafka spouts will not work with Storm versions
> 1.0
> > and higher. While running topologies in Storm version 1.0 and higher, it
> > must be ensured that the storm-kafka version is at least 1.0. Pre 1.0
> > shaded topology jars that bundles storm-kafka classes must be rebuilt
> with
> > storm-kafka version 1.0 for running in clusters with storm 1.0 and
> higher."
> >
> > Any idea how do I ensure that?
> > I have done a thorough checking of my pom.xmls and also ran grep
> > expressions.
> >
> > On Mon, Sep 12, 2016 at 3:44 PM, S G  wrote:
> >
> > > Can anyone help with this?
> > >
> > > I tried relocating classes too during shading as:
> > > 
> > >   
> > > backtype.storm
> > > org.apache.storm
> > >   
> > >   
> > > storm.trident
> > > org.apache.storm.trident
> > >   
> > >   
> > > org.apache.thrift7
> > > org.apache.storm.thrift
> > >   
> > > 
> > >
> > > But then the deployed topology began to complain about:
> > > java.lang.NoSuchMethodError: org.apache.storm.spout.
> > > MultiScheme.deserialize([B)Ljava/lang/Iterable; at
> > storm.kafka.KafkaUtils.
> > > generateTuples(KafkaUtils.java:206)
> > >
> > > So it seems that I really need to get the 1.0.2 storm-kafka into my
> > > project to make it work.
> > >
> > > I even tried to exclude storm-kafka everywhere from all the
> dependencies
> > > my project had and then including it as an explicit dependency, but it
> > > still did not work.
> > >
> > > mvn dependency:tree shows that storm-kafka is nowhere being included
> as a
> > > transitive dependency in my project.
> > >
> > > What to do now?
> > >
> > > Thx,
> > > SG
> > >
> > > On Mon, Sep 12, 2016 at 12:24 PM, S G 
> wrote:
> > >
> > >> Hi,
> > >>
> > >> We are upgrading to Storm 1.0.2 and have done all the package name
> > >> changes.
> > >>
> > >> grep -ri backtype in our code repo returns nothing.
> > >>
> > >> Yet on deploying the jar to Storm, following error is seen:
> > >>
> > >>
> > >> storm/storm_installed/apache-storm-1.0.2/bin/storm \
> > >>  jar \
> > >> $jarFile \
> > >> org.apache.storm.flux.Flux \
> > >> $yamlFileTarget \
> > >> --remote
> > >>
> > >>
> > >>
> > >> +- Apache Storm-+
> > >> +-  data FLow User eXperience  -+
> > >> Version: 1.0.2
> > >> Parsing file: topology-config.yaml
> > >> 333  [main] INFO  o.a.s.f.p.FluxParser - loading YAML from input
> > stream...
> > >> 335  [main] INFO  o.a.s.f.p.FluxParser - Not performing property
> > >> substitution.
> > >> 335  [main] INFO  o.a.s.f.p.FluxParser - Not performing environment
> > >> variable substitution.
> > >> Exception in thread "main" java.lang.NoClassDefFoundError:
> > >> backtype/storm/spout/MultiScheme
> > >> at java.lang.Class.forName0(Native Method)
> > >> at java.lang.Class.forName(Class.java:264)
> > >> at org.apache.storm.flux.FluxBuilder.buildObject(
> FluxBuilder.java:291)
> > >> at org.apache.storm.flux.FluxBuilder.buildComponents(FluxBuilde
> > >> r.java:350)
> > >> at org.apache.storm.flux.FluxBuilder.buildTopology(
> FluxBuilder.java:75)
> > >> at org.apache.storm.flux.Flux.runCli(Flux.java:153)
> > >> at org.apache.storm.flux.Flux.main(Flux.java:98)
> > >> Caused by: java.lang.ClassNotFoundException:
> > >> backtype.storm.spout.MultiScheme
> > >> at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
> > >> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> > >> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
> > >> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> > >> ... 7 more
> > >>
> > >>
> > >>
> > >> This used to work fine with 0.10.0
> > >>
> > >> I have checked all the places in my code including mvn
> dependency:tree's
> > >> output
> > >> All versions 

Re: Flux exception while upgrading to Storm 1.0.2

[Manu Zhang]

Hi,

Could you share your maven file ? When running `mvn package`, you should
see what and which version of jar files are included ?
That may give you a hint.

Thanks,
Manu


On Tue, Sep 13, 2016 at 6:50 AM S G  wrote:

> From the
> https://github.com/apache/storm/tree/1.0.x-branch/external/storm-kafka
> docs,
>
> "This means that pre 1.0 kafka spouts will not work with Storm versions 1.0
> and higher. While running topologies in Storm version 1.0 and higher, it
> must be ensured that the storm-kafka version is at least 1.0. Pre 1.0
> shaded topology jars that bundles storm-kafka classes must be rebuilt with
> storm-kafka version 1.0 for running in clusters with storm 1.0 and higher."
>
> Any idea how do I ensure that?
> I have done a thorough checking of my pom.xmls and also ran grep
> expressions.
>
> On Mon, Sep 12, 2016 at 3:44 PM, S G  wrote:
>
> > Can anyone help with this?
> >
> > I tried relocating classes too during shading as:
> > 
> >   
> > backtype.storm
> > org.apache.storm
> >   
> >   
> > storm.trident
> > org.apache.storm.trident
> >   
> >   
> > org.apache.thrift7
> > org.apache.storm.thrift
> >   
> > 
> >
> > But then the deployed topology began to complain about:
> > java.lang.NoSuchMethodError: org.apache.storm.spout.
> > MultiScheme.deserialize([B)Ljava/lang/Iterable; at
> storm.kafka.KafkaUtils.
> > generateTuples(KafkaUtils.java:206)
> >
> > So it seems that I really need to get the 1.0.2 storm-kafka into my
> > project to make it work.
> >
> > I even tried to exclude storm-kafka everywhere from all the dependencies
> > my project had and then including it as an explicit dependency, but it
> > still did not work.
> >
> > mvn dependency:tree shows that storm-kafka is nowhere being included as a
> > transitive dependency in my project.
> >
> > What to do now?
> >
> > Thx,
> > SG
> >
> > On Mon, Sep 12, 2016 at 12:24 PM, S G  wrote:
> >
> >> Hi,
> >>
> >> We are upgrading to Storm 1.0.2 and have done all the package name
> >> changes.
> >>
> >> grep -ri backtype in our code repo returns nothing.
> >>
> >> Yet on deploying the jar to Storm, following error is seen:
> >>
> >>
> >> storm/storm_installed/apache-storm-1.0.2/bin/storm \
> >>  jar \
> >> $jarFile \
> >> org.apache.storm.flux.Flux \
> >> $yamlFileTarget \
> >> --remote
> >>
> >>
> >>
> >> +- Apache Storm-+
> >> +-  data FLow User eXperience  -+
> >> Version: 1.0.2
> >> Parsing file: topology-config.yaml
> >> 333  [main] INFO  o.a.s.f.p.FluxParser - loading YAML from input
> stream...
> >> 335  [main] INFO  o.a.s.f.p.FluxParser - Not performing property
> >> substitution.
> >> 335  [main] INFO  o.a.s.f.p.FluxParser - Not performing environment
> >> variable substitution.
> >> Exception in thread "main" java.lang.NoClassDefFoundError:
> >> backtype/storm/spout/MultiScheme
> >> at java.lang.Class.forName0(Native Method)
> >> at java.lang.Class.forName(Class.java:264)
> >> at org.apache.storm.flux.FluxBuilder.buildObject(FluxBuilder.java:291)
> >> at org.apache.storm.flux.FluxBuilder.buildComponents(FluxBuilde
> >> r.java:350)
> >> at org.apache.storm.flux.FluxBuilder.buildTopology(FluxBuilder.java:75)
> >> at org.apache.storm.flux.Flux.runCli(Flux.java:153)
> >> at org.apache.storm.flux.Flux.main(Flux.java:98)
> >> Caused by: java.lang.ClassNotFoundException:
> >> backtype.storm.spout.MultiScheme
> >> at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
> >> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> >> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
> >> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> >> ... 7 more
> >>
> >>
> >>
> >> This used to work fine with 0.10.0
> >>
> >> I have checked all the places in my code including mvn dependency:tree's
> >> output
> >> All versions are 1.0.2 (storm and flux) and there is no mention of
> >> backtype anywhere in the code, flux-file or the jar we are deploying.
> >>
> >> What is wrong here?
> >>
> >> Thanks
> >> SG
> >>
> >
> >
>

Re: Flux exception while upgrading to Storm 1.0.2

[S G]

>From the
https://github.com/apache/storm/tree/1.0.x-branch/external/storm-kafka docs,

"This means that pre 1.0 kafka spouts will not work with Storm versions 1.0
and higher. While running topologies in Storm version 1.0 and higher, it
must be ensured that the storm-kafka version is at least 1.0. Pre 1.0
shaded topology jars that bundles storm-kafka classes must be rebuilt with
storm-kafka version 1.0 for running in clusters with storm 1.0 and higher."

Any idea how do I ensure that?
I have done a thorough checking of my pom.xmls and also ran grep
expressions.

On Mon, Sep 12, 2016 at 3:44 PM, S G  wrote:

> Can anyone help with this?
>
> I tried relocating classes too during shading as:
> 
>   
> backtype.storm
> org.apache.storm
>   
>   
> storm.trident
> org.apache.storm.trident
>   
>   
> org.apache.thrift7
> org.apache.storm.thrift
>   
> 
>
> But then the deployed topology began to complain about:
> java.lang.NoSuchMethodError: org.apache.storm.spout.
> MultiScheme.deserialize([B)Ljava/lang/Iterable; at storm.kafka.KafkaUtils.
> generateTuples(KafkaUtils.java:206)
>
> So it seems that I really need to get the 1.0.2 storm-kafka into my
> project to make it work.
>
> I even tried to exclude storm-kafka everywhere from all the dependencies
> my project had and then including it as an explicit dependency, but it
> still did not work.
>
> mvn dependency:tree shows that storm-kafka is nowhere being included as a
> transitive dependency in my project.
>
> What to do now?
>
> Thx,
> SG
>
> On Mon, Sep 12, 2016 at 12:24 PM, S G  wrote:
>
>> Hi,
>>
>> We are upgrading to Storm 1.0.2 and have done all the package name
>> changes.
>>
>> grep -ri backtype in our code repo returns nothing.
>>
>> Yet on deploying the jar to Storm, following error is seen:
>>
>>
>> storm/storm_installed/apache-storm-1.0.2/bin/storm \
>>  jar \
>> $jarFile \
>> org.apache.storm.flux.Flux \
>> $yamlFileTarget \
>> --remote
>>
>>
>>
>> +- Apache Storm-+
>> +-  data FLow User eXperience  -+
>> Version: 1.0.2
>> Parsing file: topology-config.yaml
>> 333  [main] INFO  o.a.s.f.p.FluxParser - loading YAML from input stream...
>> 335  [main] INFO  o.a.s.f.p.FluxParser - Not performing property
>> substitution.
>> 335  [main] INFO  o.a.s.f.p.FluxParser - Not performing environment
>> variable substitution.
>> Exception in thread "main" java.lang.NoClassDefFoundError:
>> backtype/storm/spout/MultiScheme
>> at java.lang.Class.forName0(Native Method)
>> at java.lang.Class.forName(Class.java:264)
>> at org.apache.storm.flux.FluxBuilder.buildObject(FluxBuilder.java:291)
>> at org.apache.storm.flux.FluxBuilder.buildComponents(FluxBuilde
>> r.java:350)
>> at org.apache.storm.flux.FluxBuilder.buildTopology(FluxBuilder.java:75)
>> at org.apache.storm.flux.Flux.runCli(Flux.java:153)
>> at org.apache.storm.flux.Flux.main(Flux.java:98)
>> Caused by: java.lang.ClassNotFoundException:
>> backtype.storm.spout.MultiScheme
>> at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
>> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
>> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>> ... 7 more
>>
>>
>>
>> This used to work fine with 0.10.0
>>
>> I have checked all the places in my code including mvn dependency:tree's
>> output
>> All versions are 1.0.2 (storm and flux) and there is no mention of
>> backtype anywhere in the code, flux-file or the jar we are deploying.
>>
>> What is wrong here?
>>
>> Thanks
>> SG
>>
>
>

Re: Flux exception while upgrading to Storm 1.0.2

[S G]

Can anyone help with this?

I tried relocating classes too during shading as:

  
backtype.storm
org.apache.storm
  
  
storm.trident
org.apache.storm.trident
  
  
org.apache.thrift7
org.apache.storm.thrift
  


But then the deployed topology began to complain about:
java.lang.NoSuchMethodError:
org.apache.storm.spout.MultiScheme.deserialize([B)Ljava/lang/Iterable; at
storm.kafka.KafkaUtils.generateTuples(KafkaUtils.java:206)

So it seems that I really need to get the 1.0.2 storm-kafka into my project
to make it work.

I even tried to exclude storm-kafka everywhere from all the dependencies my
project had and then including it as an explicit dependency, but it still
did not work.

mvn dependency:tree shows that storm-kafka is nowhere being included as a
transitive dependency in my project.

What to do now?

Thx,
SG

On Mon, Sep 12, 2016 at 12:24 PM, S G  wrote:

> Hi,
>
> We are upgrading to Storm 1.0.2 and have done all the package name changes.
>
> grep -ri backtype in our code repo returns nothing.
>
> Yet on deploying the jar to Storm, following error is seen:
>
>
> storm/storm_installed/apache-storm-1.0.2/bin/storm \
>  jar \
> $jarFile \
> org.apache.storm.flux.Flux \
> $yamlFileTarget \
> --remote
>
>
>
> +- Apache Storm-+
> +-  data FLow User eXperience  -+
> Version: 1.0.2
> Parsing file: topology-config.yaml
> 333  [main] INFO  o.a.s.f.p.FluxParser - loading YAML from input stream...
> 335  [main] INFO  o.a.s.f.p.FluxParser - Not performing property
> substitution.
> 335  [main] INFO  o.a.s.f.p.FluxParser - Not performing environment
> variable substitution.
> Exception in thread "main" java.lang.NoClassDefFoundError:
> backtype/storm/spout/MultiScheme
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:264)
> at org.apache.storm.flux.FluxBuilder.buildObject(FluxBuilder.java:291)
> at org.apache.storm.flux.FluxBuilder.buildComponents(FluxBuilder.java:350)
> at org.apache.storm.flux.FluxBuilder.buildTopology(FluxBuilder.java:75)
> at org.apache.storm.flux.Flux.runCli(Flux.java:153)
> at org.apache.storm.flux.Flux.main(Flux.java:98)
> Caused by: java.lang.ClassNotFoundException: backtype.storm.spout.
> MultiScheme
> at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> ... 7 more
>
>
>
> This used to work fine with 0.10.0
>
> I have checked all the places in my code including mvn dependency:tree's
> output
> All versions are 1.0.2 (storm and flux) and there is no mention of
> backtype anywhere in the code, flux-file or the jar we are deploying.
>
> What is wrong here?
>
> Thanks
> SG
>

Re: JIRI Notifications Moved to iss...@storm.apache.org

[Jungtaek Lim]

Nice work.

Will Github notification (posted to dev@) be also moved to issues@? Or will
keep posted to dev@?
On 2016년 9월 13일 (화) at 오전 6:00 P. Taylor Goetz  wrote:

> Infra has now moved JIRA notifications from dev@storm.apache.org to
> issues.storm.apache.org.
>
> If you would like to continue to receive JIRA notifications, you should
> subscribe to that list by sending an email to:
>
> issues-subscr...@storm.apache.org
>
> -Taylor
>

JIRI Notifications Moved to iss...@storm.apache.org

[P. Taylor Goetz]

Infra has now moved JIRA notifications from dev@storm.apache.org 
 to issues.storm.apache.org 
.

If you would like to continue to receive JIRA notifications, you should 
subscribe to that list by sending an email to:

issues-subscr...@storm.apache.org 

-Taylor


signature.asc
Description: Message signed with OpenPGP using GPGMail

[GitHub] storm issue #1680: Minor typos in documentation

[redsanket]

Github user redsanket commented on the issue:

https://github.com/apache/storm/pull/1680
  
+1


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: Clojure to Java Migration (was: How does Storm's OutputCollector work?)

[P. Taylor Goetz]

> On Sep 12, 2016, at 3:26 PM, S G  wrote:
> 
> Just curious, how much of the code-base is planned to be moved away from
> clojure to Java?

Ultimately all the Clojure code will move to Java. The initial focus is to get 
the main (i.e. non-test) code converted before migrating the unit test code.

> When do we plan to make a release for the Java-based code?

There is no specific date set, so the release will happen once the conversion 
is complete, blocker bugs resolved, etc. I imagine we could release before the 
test code is migrated.

-Taylor


> 
> On Mon, Sep 12, 2016 at 6:21 AM, Bobby Evans 
> wrote:
> 
>> Manu Zhang is correct in his answer.  The main reason why you cannot find
>> a class that implements that interface is because in all released versions
>> of storm the code was written in clojure, so it is hidden away for people
>> who are used to just java. - Bobby
>> 
>>On Monday, September 12, 2016 12:09 AM, Manu Zhang <
>> owenzhang1...@gmail.com> wrote:
>> 
>> 
>> Hi T.I.,
>> 
>> Answered your question on StackOverflow. Hope that helps.
>> 
>> Thanks,
>> Manu
>> 
>> On Mon, Sep 12, 2016 at 11:36 AM Tech Id  wrote:
>> 
>>> Hi,
>>> 
>>> Can someone help provide a better understanding of Storm's
>> OutputCollector?
>>> A question for the same can be found at
>>> 
>>> http://stackoverflow.com/questions/39439336/how-does-
>> storms-outputcollector-work
>>> 
>>> Thanks,
>>> T.I.
>>> 
>> 
>> 
>> 
>> 



signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: How does Storm's OutputCollector work?

[Tech Id]

Thanks a lot Manu, Those pointers are of great help.

Can you also help with
http://stackoverflow.com/questions/39456590/is-nimbus-host-a-valid-option-in-storm-1-0-2-version
?



On Mon, Sep 12, 2016 at 12:26 PM, S G  wrote:

> Thx Bobby,
> Just curious, how much of the code-base is planned to be moved away from
> clojure to Java?
> When do we plan to make a release for the Java-based code?
>
> On Mon, Sep 12, 2016 at 6:21 AM, Bobby Evans 
> wrote:
>
> > Manu Zhang is correct in his answer.  The main reason why you cannot find
> > a class that implements that interface is because in all released
> versions
> > of storm the code was written in clojure, so it is hidden away for people
> > who are used to just java. - Bobby
> >
> > On Monday, September 12, 2016 12:09 AM, Manu Zhang <
> > owenzhang1...@gmail.com> wrote:
> >
> >
> >  Hi T.I.,
> >
> > Answered your question on StackOverflow. Hope that helps.
> >
> > Thanks,
> > Manu
> >
> > On Mon, Sep 12, 2016 at 11:36 AM Tech Id 
> wrote:
> >
> > > Hi,
> > >
> > > Can someone help provide a better understanding of Storm's
> > OutputCollector?
> > > A question for the same can be found at
> > >
> > > http://stackoverflow.com/questions/39439336/how-does-
> > storms-outputcollector-work
> > >
> > > Thanks,
> > > T.I.
> > >
> >
> >
> >
> >
>

Re: How does Storm's OutputCollector work?

[S G]

Thx Bobby,
Just curious, how much of the code-base is planned to be moved away from
clojure to Java?
When do we plan to make a release for the Java-based code?

On Mon, Sep 12, 2016 at 6:21 AM, Bobby Evans 
wrote:

> Manu Zhang is correct in his answer.  The main reason why you cannot find
> a class that implements that interface is because in all released versions
> of storm the code was written in clojure, so it is hidden away for people
> who are used to just java. - Bobby
>
> On Monday, September 12, 2016 12:09 AM, Manu Zhang <
> owenzhang1...@gmail.com> wrote:
>
>
>  Hi T.I.,
>
> Answered your question on StackOverflow. Hope that helps.
>
> Thanks,
> Manu
>
> On Mon, Sep 12, 2016 at 11:36 AM Tech Id  wrote:
>
> > Hi,
> >
> > Can someone help provide a better understanding of Storm's
> OutputCollector?
> > A question for the same can be found at
> >
> > http://stackoverflow.com/questions/39439336/how-does-
> storms-outputcollector-work
> >
> > Thanks,
> > T.I.
> >
>
>
>
>

Flux exception while upgrading to Storm 1.0.2

[S G]

Hi,

We are upgrading to Storm 1.0.2 and have done all the package name changes.

grep -ri backtype in our code repo returns nothing.

Yet on deploying the jar to Storm, following error is seen:


storm/storm_installed/apache-storm-1.0.2/bin/storm \
 jar \
$jarFile \
org.apache.storm.flux.Flux \
$yamlFileTarget \
--remote



+- Apache Storm-+
+-  data FLow User eXperience  -+
Version: 1.0.2
Parsing file: topology-config.yaml
333  [main] INFO  o.a.s.f.p.FluxParser - loading YAML from input stream...
335  [main] INFO  o.a.s.f.p.FluxParser - Not performing property
substitution.
335  [main] INFO  o.a.s.f.p.FluxParser - Not performing environment
variable substitution.
Exception in thread "main" java.lang.NoClassDefFoundError:
backtype/storm/spout/MultiScheme
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:264)
at org.apache.storm.flux.FluxBuilder.buildObject(FluxBuilder.java:291)
at org.apache.storm.flux.FluxBuilder.buildComponents(FluxBuilder.java:350)
at org.apache.storm.flux.FluxBuilder.buildTopology(FluxBuilder.java:75)
at org.apache.storm.flux.Flux.runCli(Flux.java:153)
at org.apache.storm.flux.Flux.main(Flux.java:98)
Caused by: java.lang.ClassNotFoundException:
backtype.storm.spout.MultiScheme
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 7 more



This used to work fine with 0.10.0

I have checked all the places in my code including mvn dependency:tree's
output
All versions are 1.0.2 (storm and flux) and there is no mention of backtype
anywhere in the code, flux-file or the jar we are deploying.

What is wrong here?

Thanks
SG

[jira] [Created] (STORM-2091) storm-kafka-client, NoSuchMethod on KafkaConsumer.subscribe

[Pierre Padrixe (JIRA)]

Pierre Padrixe created STORM-2091:
-

 Summary: storm-kafka-client, NoSuchMethod on 
KafkaConsumer.subscribe 
 Key: STORM-2091
 URL: https://issues.apache.org/jira/browse/STORM-2091
 Project: Apache Storm
  Issue Type: Bug
Reporter: Pierre Padrixe


Hello,

I am currently using storm-kafka-client "1.0.2" with kafka-clients "0.10.0.1" 
and I am encountering the following stacktrace at runtime of my topology:

java.lang.NoSuchMethodError: 
org.apache.kafka.clients.consumer.KafkaConsumer.subscribe(Ljava/util/List;Lorg/apache/kafka/clients/consumer/ConsumerRebalanceListener;)V
 at 
org.apache.storm.kafka.spout.KafkaSpout.subscribeKafkaConsumer(KafkaSpout.java:350)
 at org.apache.storm.kafka.spout.KafkaSpout.activate(KafkaSpout.java:341) at 
org.apache.storm.daemon.executor$fn__7885$fn__7900$fn__7931.invoke(executor.clj:640)
 at org.apache.storm.util$async_loop$fn__625.invoke(util.clj:484) at 
clojure.lang.AFn.run(AFn.java:22) at java.lang.Thread.run(Thread.java:745)

So I checked the source code and Indeed the subscribe method takes a collection 
of topics in addition of the listener: 
https://github.com/apache/kafka/blob/0.10.0.1/clients/src/main/java/org/apache/kafka/clients/consumer/KafkaConsumer.java#L795

I though version 1.0.2 of storm-kafka-clients should work with kafka 0.10.0.1, 
do you have any idea ? Thanks.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-2090) Add integration test for storm windowing

[Raghav Kumar Gautam (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-2090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15484786#comment-15484786
 ] 

Raghav Kumar Gautam commented on STORM-2090:


For now I am using vagrant setup for debugging and we can use it for problems 
that reproduce consistently.

> Add integration test for storm windowing
> 
>
> Key: STORM-2090
> URL: https://issues.apache.org/jira/browse/STORM-2090
> Project: Apache Storm
>  Issue Type: Test
>  Components: build
>Affects Versions: 1.0.2
>Reporter: Raghav Kumar Gautam
>Assignee: Raghav Kumar Gautam
> Fix For: 1.0.3
>
>
> We want to add integration test for storm windowing feature.
> It will be nice to have this running with existing travis-ci setup.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-2090) Add integration test for storm windowing

[Raghav Kumar Gautam (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-2090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15484781#comment-15484781
 ] 

Raghav Kumar Gautam commented on STORM-2090:


It seems to get logs from travis ci we have following options:
1. print logs to stdout/stderr - can be done but since there will be multiple 
logs from nimbus, supervisor, tests etc this will exceed 10 mb travis-ci limit, 
(thanks [~kabhwan])
2. copy logs to s3 - we can do this if we can get an account for doing this


> Add integration test for storm windowing
> 
>
> Key: STORM-2090
> URL: https://issues.apache.org/jira/browse/STORM-2090
> Project: Apache Storm
>  Issue Type: Test
>  Components: build
>Affects Versions: 1.0.2
>Reporter: Raghav Kumar Gautam
>Assignee: Raghav Kumar Gautam
> Fix For: 1.0.3
>
>
> We want to add integration test for storm windowing feature.
> It will be nice to have this running with existing travis-ci setup.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (STORM-2090) Add integration test for storm windowing

[Raghav Kumar Gautam (JIRA)]

Raghav Kumar Gautam created STORM-2090:
--

 Summary: Add integration test for storm windowing
 Key: STORM-2090
 URL: https://issues.apache.org/jira/browse/STORM-2090
 Project: Apache Storm
  Issue Type: Test
  Components: build
Affects Versions: 1.0.2
Reporter: Raghav Kumar Gautam
Assignee: Raghav Kumar Gautam
 Fix For: 1.0.3


We want to add integration test for storm windowing feature.
It will be nice to have this running with existing travis-ci setup.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [VOTE] Release Apache Storm 0.10.2 (rc2)

[P. Taylor Goetz]

This vote is now closed, and passes with 4 binding +1 votes and no 0 or -1 
votes.

Vote tally:

+1:
P. Taylor Goetz
Bobby Evans
Jungtaek Lim
Satish Duggana


Thanks to everyone who took the time to vote.

-Taylor


> On Sep 2, 2016, at 3:40 PM, P. Taylor Goetz  wrote:
> 
> This is a call to vote on releasing Apache Storm 0.10.2 (rc2)
> 
> Full list of changes in this release:
> 
> https://git-wip-us.apache.org/repos/asf?p=storm.git;a=blob_plain;f=CHANGELOG.md;hb=d8e4a3b59e6a97165ff769f11c9d4d2d3ecbb9dc
> 
> The tag/commit to be voted upon is v0.10.2:
> 
> https://git-wip-us.apache.org/repos/asf?p=storm.git;a=tree;h=d51a27fae2a9549c3ccbbf23b4ad81a0c50a3a65;hb=d8e4a3b59e6a97165ff769f11c9d4d2d3ecbb9dc
> 
> The source archive being voted upon can be found here:
> 
> https://dist.apache.org/repos/dist/dev/storm/apache-storm-0.10.2-rc2/apache-storm-0.10.2-src.tar.gz
> 
> Other release files, signatures and digests can be found here:
> 
> https://dist.apache.org/repos/dist/dev/storm/apache-storm-0.10.2-rc2/
> 
> The release artifacts are signed with the following key:
> 
> https://git-wip-us.apache.org/repos/asf?p=storm.git;a=blob_plain;f=KEYS;hb=22b832708295fa2c15c4f3c70ac0d2bc6fded4bd
> 
> The Nexus staging repository for this release is:
> 
> https://repository.apache.org/content/repositories/orgapachestorm-1042
> 
> Please vote on releasing this package as Apache Storm 0.10.2.
> 
> When voting, please list the actions taken to verify the release.
> 
> This vote will be open for at least 72 hours.
> 
> [ ] +1 Release this package as Apache Storm 0.10.2
> [ ]  0 No opinion
> [ ] -1 Do not release this package because...
> 
> Thanks to everyone who contributed to this release.
> 
> -Taylor



signature.asc
Description: Message signed with OpenPGP using GPGMail

[GitHub] storm issue #1661: [STORM-2071] Add in retry after rebalance in unit test

[ppoulosk]

Github user ppoulosk commented on the issue:

https://github.com/apache/storm/pull/1661
  
@HeartSaVioR @revans2 I have no preference which solution gets pulled in 
first.  

I fixed the comment, retested, squashed the three commits into one and 
pushed it to my branch.

Thanks!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (STORM-2089) Replace Consumer of ISqlTridentDataSource with StateFactory and StateUpdater

[Jungtaek Lim (JIRA)]

Jungtaek Lim created STORM-2089:
---

 Summary: Replace Consumer of ISqlTridentDataSource with 
StateFactory and StateUpdater
 Key: STORM-2089
 URL: https://issues.apache.org/jira/browse/STORM-2089
 Project: Apache Storm
  Issue Type: Improvement
  Components: storm-sql
Reporter: Jungtaek Lim


Currently ISqlTridentDataSource exposes Function as Consumer which provides 
only single row update. To maximize the performance, it should be changed to 
StateFactory (or StateSpec) and StateUpdater.

This also includes change of storm-sql-kafka.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: How does Storm's OutputCollector work?

[Bobby Evans]

Manu Zhang is correct in his answer.  The main reason why you cannot find a 
class that implements that interface is because in all released versions of 
storm the code was written in clojure, so it is hidden away for people who are 
used to just java. - Bobby 

On Monday, September 12, 2016 12:09 AM, Manu Zhang 
 wrote:
 

 Hi T.I.,

Answered your question on StackOverflow. Hope that helps.

Thanks,
Manu

On Mon, Sep 12, 2016 at 11:36 AM Tech Id  wrote:

> Hi,
>
> Can someone help provide a better understanding of Storm's OutputCollector?
> A question for the same can be found at
>
> http://stackoverflow.com/questions/39439336/how-does-storms-outputcollector-work
>
> Thanks,
> T.I.
>


   

[jira] [Commented] (STORM-2075) Storm SQL Phase III

[Jungtaek Lim (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-2075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15484071#comment-15484071
 ] 

Jungtaek Lim commented on STORM-2075:
-

Below is Trident support for external modules.

|| module || producer || consumer ||
| Cassandra | N/A | State |
| Druid | N/A | State |
| ElasticSearch | N/A | State |
| EventHubs | Opaque/Transactional Spout | N/A |
| HBase | N/A | State |
| HDFS | N/A | State |
| Hive | N/A | State |
| JDBC | N/A | State |
| Kafka | Opaque/Transactional Spout | State |
| Kafka-Client (New) | N/A | N/A |
| Kinesis | N/A | N/A |
| Mongo | N/A | State |
| MQTT | N/A | BaseFunction |
| OpenTSDB | N/A | State |
| Redis | N/A | State |
| Solr | N/A | State |

Btw, I found that storm-sql-kafka provides BaseFunction via State. To maximize 
performance we should support State directly.

> Storm SQL Phase III
> ---
>
> Key: STORM-2075
> URL: https://issues.apache.org/jira/browse/STORM-2075
> Project: Apache Storm
>  Issue Type: Epic
>  Components: storm-sql
>Reporter: Jungtaek Lim
>
> This epic tracks the effort of the phase III development of StormSQL.
> For now Storm SQL only supports Kafka as a data source, which is limited for 
> normal use cases. We would need to support others as well. Candidates are 
> external modules.
> And also consider supporting State since Trident provides a way to set / get 
> in batch manner to only State. Current way (each) does insert a row 1 by 1.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] storm pull request #1681: STORM-1444 Support EXPLAIN statement in StormSQL

[HeartSaVioR]

GitHub user HeartSaVioR opened a pull request:

https://github.com/apache/storm/pull/1681

STORM-1444 Support EXPLAIN statement in StormSQL

* when '--explain' is specified to topology name, activate explain mode
* when explain mode is activated, it analyzes each query statement, and 
shows query plan
  * it doesn't submit topologies
* Showing plan is provided by Calcite: it just uses that feature

As I also described to the doc, query plan for below query 
```
INSERT INTO LARGE_ORDERS SELECT ID, UNIT_PRICE * QUANTITY AS TOTAL FROM 
ORDERS 
WHERE UNIT_PRICE * QUANTITY > 50
```

is shown as: 

```
LogicalTableModify(table=[[LARGE_ORDERS]], operation=[INSERT], 
updateColumnList=[[]], flattened=[true]), id = 8
  LogicalProject(ID=[$0], TOTAL=[*($1, $2)]), id = 7
LogicalFilter(condition=[>(*($1, $2), 50)]), id = 6
  EnumerableTableScan(table=[[ORDERS]]), id = 5
```

FYI, $0 is column index 0 (starting from 0), and so on. Expression is 
printed as prefix notation.
Calcite just provides this functionality, though I'm not clear whether this 
is friendly (acceptable by end-user) or not.
There's staled issue from Calcite 
([CALCITE-155](https://issues.apache.org/jira/browse/CALCITE-155)) for 
displaying actual column names/aliases in explain plan.

Please have a look and share your opinion about this.
Thanks in advance!

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/HeartSaVioR/storm STORM-1444

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/storm/pull/1681.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1681






---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---