On Thu, May 22, 2014 at 11:46 AM, Steve Blackmon <[email protected]>wrote:
> Renato, > > Yes there's no question we need to update the website - not least to > help newcomers locate and get started with the examples. > > I would like to see us creating more integration tests like > Top100FeedsTest which source and pass current-time real-world data > through provider modules to confirm they work. Social APIs are > continuously evolving and this could help us catch any changes that we > weren't aware of and prepared for prior to the change. > > Persistance module testing should also be done and would probably look > a lot like gora's testing. Let's follow up on ideas for aligning > efforts along those lines. A 10-20 minute build when integration > testing is enabled isn't unreasonable. > > I'm not sure you meant to suggest it explicitly, but this note > triggered a thought that the contrib modules perhaps could be split > out into another repo as well. Like examples, a failing test within a > specific datasource or database integration shouldn't necessarily > break the whole build. APIs aren't perfect and may have transient > issues, and given that our poms lean toward the bleeding edge > occasionally database or database driver bugs are going to cause test > failures. > > I like the way storm has handled this - they've use git submodules to > associate useful integrations that make storm easier to build on with > the top-level project, while essentially delegating maintenance of > those modules to other projects/companies. A similar model could work > well for streams. > IMO, we should avoid git submodules for this. I also don't think it needs to be another entire repo, it could just be moved to a separate top-level directory > > Thanks for the suggestions! > > Steve Blackmon > [email protected] > > On Thu, May 22, 2014 at 7:22 AM, Renato Marroquín Mogrovejo > <[email protected]> wrote: > > Maybe even something like this should be done but for the contrib > modules. > > > > > https://github.com/apache/incubator-streams/blob/master/streams-contrib/streams-provider-rss/src/test/java/org/apache/streams/rss/test/Top100FeedsTest.java > > > > That would be the something like examples. Wdyt? > > > > > > Renato M. > > > > > > 2014-05-22 12:05 GMT+02:00 Renato Marroquín Mogrovejo < > > [email protected]>: > > > >> Hi Steve, > >> > >> Comments inline. > >> > >> 2014-05-21 20:59 GMT+02:00 Steve Blackmon <[email protected]>: > >> > >> I will happily accept pull requests to streams-examples, although I > >>> don't think it would need to change for an update to the provider to > >>> be applied (except perhaps to add that verb filtering step) > >>> > >> > >> This is what I meant. > >> > >> > >>> I am willing to donate all of the examples, but I don't think they > >>> belong in the main repo. What if there were 50 examples many > >>> containing integration tests? Build would take forever - plus I don't > >>> think that a single example failing a test should break the build. > >>> Mahout suffered from really long-builds and handled it by separating > >>> the examples out from core. > >>> > >> > >> Well, over in Apache Giraph they have over 20 graph example algorithms > and > >> the way they test them is just by doing some simple computations, I > don't > >> know how long it takes, but I think it makes sense to at least try it > and > >> see how long it takes. > >> Over in Gora we have a test suite with the different operations > supported > >> that is run over the different data stores, and depending on your > machine, > >> it might take from 10 - 20 mins. There was a time where the HBase tests > >> were taking around 30 mins by themselves but we fixed them and are now > run > >> in less than 5 mins. > >> Maybe your examples could be converted into a test suite? I mean reading > >> from twitter(or a file) and persisting into different places. But > anyhow, > >> your examples should be referenced somewhere as it is what it makes > sense > >> for a newcomer :D > >> > >> When the project was hosting in SVN, we had the ability to create and > >>> checkout different roots (the master pom was in a different root for > >>> example). If we can do the same using Git I propose we put the > >>> examples in as a flat multi-module project that way. > >>> > >> > >> +1 > >> > >> > >> Renato M. > >> > >> Any thoughts? > >>> > >>> Steve Blackmon > >>> [email protected] > >>> > >>> On Wed, May 21, 2014 at 11:59 AM, Renato Marroquín Mogrovejo > >>> <[email protected]> wrote: > >>> > I tried updating the site through ApacheCMS but I guess I don't have > the > >>> > rights to do it ;) > >>> > This is what I got: > >>> > > >>> > The operation is forbidden by the server: Commit failed (details > >>> follow):: > >>> > Changing file > >>> > > >>> > '/usr/local/cms/wc/streams/rmarroquin-NlOO6g/trunk/content/source-repository.mdtext' > >>> > is forbidden by the server: Access to > >>> > > >>> > '/repos/asf/!svn/txr/1596615-yz8u/incubator/streams/site/trunk/content/source-repository.mdtext' > >>> > forbidden at /usr/local/cms/webgui/lib/ASF/CMS/WC/Commit.pm line 64 > >>> > > >>> > So what is the best/easy way to update the site? > >>> > > >>> > @Steve > >>> > Yeah, I finally got the time to try this out :) and looks very > promising > >>> > for sure! > >>> > Is anybody working on STREAMS-57? Could I pick it up? I think this > >>> should > >>> > be pretty straight forward. > >>> > And are there any plans to put your examples into the code base? It's > >>> quite > >>> > puzzling going from repo to another (plus the one from the site) to > >>> > accomplish this. And then if I could work on STREAMS-57, would I have > >>> to do > >>> > the pull request into W2OGroup/streams-examples repo? I don't think > so > >>> > right? how would this work? > >>> > Thanks! > >>> > > >>> > > >>> > Renato M. > >>> > > >>> > > >>> > 2014-05-21 17:42 GMT+02:00 Steve Blackmon <[email protected]>: > >>> > > >>> >> Cool I'm glad to hear you got it running. That exception occurs > because > >>> >> twitter sends a preamble listing all of your friends when you open a > >>> >> stream. The provider doesn't have logic to handle it yet, so it > tries > >>> to > >>> >> build an activity and throws that exception. > >>> >> > >>> >> STREAMS-57 would suppress this warning - although once non-content > >>> >> activities with new verbs like follow are generated we will want a > >>> >> filtering mechanism in place so it remains easy to build indexes of > >>> just > >>> >> shares and posts. > >>> >> > >>> >> On Wed, May 21, 2014 at 9:23 AM, Renato Marroquín Mogrovejo < > >>> >> [email protected]> wrote: > >>> >> > Hey Matt, > >>> >> > > >>> >> > So the examples worked :D at least the one retrieving the twitter > >>> user's > >>> >> > history. The one trying to track users activity keeps on getting > this > >>> >> > exception: > >>> >> > > >>> >> > 16:09:11.151 [pool-2-thread-2] DEBUG > o.a.s.t.p.TwitterTypeConverter - > >>> >> > ACTIVITY > >>> >> > org.apache.streams.exceptions.ActivitySerializerException: > >>> unrecognized > >>> >> type > >>> >> > at > >>> >> > > >>> >> > >>> >> > >>> > org.apache.streams.twitter.serializer.TwitterJsonActivitySerializer.deserialize(TwitterJsonActivitySerializer.java:74) > >>> >> > at > >>> >> > > >>> >> > >>> >> > >>> > org.apache.streams.twitter.processor.TwitterTypeConverter.convert(TwitterTypeConverter.java:65) > >>> >> > at > >>> >> > > >>> >> > >>> >> > >>> > org.apache.streams.twitter.processor.TwitterTypeConverter.process(TwitterTypeConverter.java:146) > >>> >> > at > >>> >> > > >>> >> > >>> >> > >>> > org.apache.streams.local.tasks.StreamsProcessorTask.run(StreamsProcessorTask.java:72) > >>> >> > at > >>> >> > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > >>> >> > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > >>> >> > at > >>> >> > > >>> >> > >>> >> > >>> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > >>> >> > at > >>> >> > > >>> >> > >>> >> > >>> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > >>> >> > at java.lang.Thread.run(Thread.java:744) > >>> >> > > >>> >> > But everything seems to work in spite of that exception :) Cool > >>> stuff! So > >>> >> > can I open a JIRA issue for updating the site and putting the > patch > >>> there > >>> >> > then? > >>> >> > I will write another email with some other issues. > >>> >> > Thanks Matt! > >>> >> > > >>> >> > > >>> >> > Renato M. > >>> >> > > >>> >> > > >>> >> > 2014-05-21 15:59 GMT+02:00 Matt Franklin < > [email protected]>: > >>> >> > > >>> >> >> On Wed, May 21, 2014 at 9:00 AM, Renato Marroquín Mogrovejo < > >>> >> >> [email protected]> wrote: > >>> >> >> > >>> >> >> > Hi Matt, > >>> >> >> > > >>> >> >> > Thank for pointing me to these examples. I will try them out > and > >>> >> report > >>> >> >> > back. > >>> >> >> > But I have more questions now. I have found: > >>> >> >> > > >>> >> >> > https://github.com/apache/incubator-streams --> referenced by > the > >>> >> >> examples > >>> >> >> > you sent me and more active > >>> >> >> > https://github.com/apache/streams --> referenced by the > website > >>> >> >> > > >>> >> >> > >>> >> >> Oops. Yes it is the first one. The second one was a mirror of > SVN. > >>> >> >> > >>> >> >> > >>> >> >> > > >>> >> >> > Which one is the official one? I guess the first one right. > >>> Shouldn't > >>> >> we > >>> >> >> > change this on the website? does that mean that Apache Streams > >>> have > >>> >> two > >>> >> >> git > >>> >> >> > repos? I can fix that on the website if you point me to the > right > >>> >> place. > >>> >> >> > > >>> >> >> > >>> >> >> Website is in SVN using Apache CMS. > >>> >> >> > >>> >> >> > >>> >> >> > > >>> >> >> > > >>> >> >> > Renato M. > >>> >> >> > > >>> >> >> > > >>> >> >> > > >>> >> >> > 2014-05-21 2:14 GMT+02:00 Matt Franklin < > [email protected] > >>> >: > >>> >> >> > > >>> >> >> > > On Tue, May 20, 2014 at 4:28 PM, Renato Marroquín Mogrovejo < > >>> >> >> > > [email protected]> wrote: > >>> >> >> > > > >>> >> >> > > > Sorry I pressed the sent button before I finished writing > the > >>> >> email. > >>> >> >> > > > Anyways . . . Tomcat 7.0.39, ServiceMix 5.0.1 and Cassandra > >>> 2.0.4. > >>> >> >> > > > But I am getting this exception: > >>> >> >> > > > > >>> >> >> > > > >>> >> >> > > We are unfortunately in the middle of converging the new > model > >>> and > >>> >> the > >>> >> >> > old > >>> >> >> > > model. This runtime should be working, but it needs some > work to > >>> >> >> ensure > >>> >> >> > > that the new components in streams-contrib work. If you are > >>> looking > >>> >> to > >>> >> >> > get > >>> >> >> > > started with a newer runtime, streams-runtime-local is a good > >>> place. > >>> >> >> > > > >>> >> >> > > Steve posted a link to some examples he wrote for how to get > >>> >> started: > >>> >> >> > > > >>> >> >> > > > >>> >> >> > > > >>> >> >> > > >>> >> >> > >>> >> > >>> >> > >>> > https://github.com/w2ogroup/streams-examples/tree/master/twitter-history-elasticsearch > >>> >> >> > > > >>> >> >> > > We need to create some of our own examples (or ask him > kindly to > >>> >> donate > >>> >> >> > > some to us), but until the site is fixed, this is a good > start. > >>> I > >>> >> will > >>> >> >> > > create an issue for the Caml runtime issue you found here. > >>> >> >> > > > >>> >> >> > > > >>> >> >> > > > > >>> >> >> > > > May 20, 2014 10:15:29 PM > >>> org.apache.catalina.startup.ContextConfig > >>> >> >> > > > getServletContainerInitializer > >>> >> >> > > > SEVERE: The ServletContentInitializer > >>> >> >> > > > [org.springframework.web.SpringServletContainerInitializer] > >>> could > >>> >> not > >>> >> >> > be > >>> >> >> > > > created > >>> >> >> > > > java.lang.ClassNotFoundException: > >>> >> >> > > > org.springframework.web.SpringServletContainerInitializer > >>> >> >> > > > at > >>> >> >> > > > > >>> >> >> > > > > >>> >> >> > > > >>> >> >> > > >>> >> >> > >>> >> > >>> >> > >>> > org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1713) > >>> >> >> > > > at > >>> >> >> > > > > >>> >> >> > > > > >>> >> >> > > > >>> >> >> > > >>> >> >> > >>> >> > >>> >> > >>> > org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1558) > >>> >> >> > > > at java.lang.Class.forName0(Native Method) > >>> >> >> > > > at java.lang.Class.forName(Class.java:270) > >>> >> >> > > > at > >>> >> >> > > > > >>> >> >> > > > > >>> >> >> > > > >>> >> >> > > >>> >> >> > >>> >> > >>> >> > >>> > org.apache.catalina.startup.ContextConfig.getServletContainerInitializer(ContextConfig.java:1659) > >>> >> >> > > > at > >>> >> >> > > > > >>> >> >> > > > > >>> >> >> > > > >>> >> >> > > >>> >> >> > >>> >> > >>> >> > >>> > org.apache.catalina.startup.ContextConfig.processServletContainerInitializers(ContextConfig.java:1569) > >>> >> >> > > > at > >>> >> >> > > > > >>> >> >> > > > > >>> >> >> > > > >>> >> >> > > >>> >> >> > >>> >> > >>> >> > >>> > org.apache.catalina.startup.ContextConfig.webConfig(ContextConfig.java:1277) > >>> >> >> > > > at > >>> >> >> > > > > >>> >> >> > > > > >>> >> >> > > > >>> >> >> > > >>> >> >> > >>> >> > >>> >> > >>> > org.apache.catalina.startup.ContextConfig.configureStart(ContextConfig.java:878) > >>> >> >> > > > at > >>> >> >> > > > > >>> >> >> > > > > >>> >> >> > > > >>> >> >> > > >>> >> >> > >>> >> > >>> >> > >>> > org.apache.catalina.startup.ContextConfig.lifecycleEvent(ContextConfig.java:369) > >>> >> >> > > > at > >>> >> >> > > > > >>> >> >> > > > > >>> >> >> > > > >>> >> >> > > >>> >> >> > >>> >> > >>> >> > >>> > org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:119) > >>> >> >> > > > at > >>> >> >> > > > > >>> >> >> > > > > >>> >> >> > > > >>> >> >> > > >>> >> >> > >>> >> > >>> >> > >>> > org.apache.catalina.util.LifecycleBase.fireLifecycleEvent(LifecycleBase.java:90) > >>> >> >> > > > at > >>> >> >> > > > > >>> >> >> > > > > >>> >> >> > > > >>> >> >> > > >>> >> >> > >>> >> > >>> >> > >>> > org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5269) > >>> >> >> > > > at > >>> >> >> > > > > >>> >> org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150) > >>> >> >> > > > > >>> >> >> > > > That let me to [2]. Is this related to this? or is it just > me > >>> >> doing > >>> >> >> > > things > >>> >> >> > > > wrong here? > >>> >> >> > > > Thanks for any pointers! > >>> >> >> > > > > >>> >> >> > > > > >>> >> >> > > > Renato M. > >>> >> >> > > > > >>> >> >> > > > 2014-05-20 22:25 GMT+02:00 Renato Marroquín Mogrovejo < > >>> >> >> > > > [email protected]>: > >>> >> >> > > > > >>> >> >> > > > > Hey guys, > >>> >> >> > > > > > >>> >> >> > > > > This is a long overdue email but with little steps I will > >>> start > >>> >> >> going > >>> >> >> > > :) > >>> >> >> > > > > So I have been trying to run the whole project based > on[1]. > >>> As > >>> >> >> there > >>> >> >> > is > >>> >> >> > > > no > >>> >> >> > > > > version for any of the pieces involved in running this > (if > >>> there > >>> >> is > >>> >> >> > > > please > >>> >> >> > > > > point me to them, if not, I am happy to put them on the > site > >>> >> >> > > somewhere). > >>> >> >> > > > I > >>> >> >> > > > > chose > >>> >> >> > > > > > >>> >> >> > > > > > >>> >> >> > > > > > >>> >> >> > > > > [1] > http://streams.incubator.apache.org/streams-usage.html > >>> >> >> > > > > [2] > >>> >> >> > > > > > >>> >> >> > > > > >>> >> >> > > > >>> >> >> > > >>> >> >> > >>> >> > >>> >> > >>> > http://stackoverflow.com/questions/7692497/tomcat-wont-load-my-meta-inf-services-javax-servlet-servletcontainerinitializ > >>> >> >> > > > > > >>> >> >> > > > > >>> >> >> > > > >>> >> >> > > >>> >> >> > >>> >> > >>> > >> > >> >
