Just to finish this thread. @Matt +1
@Steve I found that Apache Giraph tests their algorithms with Mockito[1], but we could other frameworks like Easymock , or JMock if we decide to go down that road. Renato M. [1] https://code.google.com/p/mockito/ 2014-05-22 22:46 GMT+02:00 Matt Franklin <[email protected]>: > On Thu, May 22, 2014 at 11:46 AM, Steve Blackmon <[email protected] > >wrote: > > > Renato, > > > > Yes there's no question we need to update the website - not least to > > help newcomers locate and get started with the examples. > > > > I would like to see us creating more integration tests like > > Top100FeedsTest which source and pass current-time real-world data > > through provider modules to confirm they work. Social APIs are > > continuously evolving and this could help us catch any changes that we > > weren't aware of and prepared for prior to the change. > > > > Persistance module testing should also be done and would probably look > > a lot like gora's testing. Let's follow up on ideas for aligning > > efforts along those lines. A 10-20 minute build when integration > > testing is enabled isn't unreasonable. > > > > I'm not sure you meant to suggest it explicitly, but this note > > triggered a thought that the contrib modules perhaps could be split > > out into another repo as well. Like examples, a failing test within a > > specific datasource or database integration shouldn't necessarily > > break the whole build. APIs aren't perfect and may have transient > > issues, and given that our poms lean toward the bleeding edge > > occasionally database or database driver bugs are going to cause test > > failures. > > > > I like the way storm has handled this - they've use git submodules to > > associate useful integrations that make storm easier to build on with > > the top-level project, while essentially delegating maintenance of > > those modules to other projects/companies. A similar model could work > > well for streams. > > > > IMO, we should avoid git submodules for this. I also don't think it needs > to be another entire repo, it could just be moved to a separate top-level > directory > > > > > > Thanks for the suggestions! > > > > Steve Blackmon > > [email protected] > > > > On Thu, May 22, 2014 at 7:22 AM, Renato Marroquín Mogrovejo > > <[email protected]> wrote: > > > Maybe even something like this should be done but for the contrib > > modules. > > > > > > > > > https://github.com/apache/incubator-streams/blob/master/streams-contrib/streams-provider-rss/src/test/java/org/apache/streams/rss/test/Top100FeedsTest.java > > > > > > That would be the something like examples. Wdyt? > > > > > > > > > Renato M. > > > > > > > > > 2014-05-22 12:05 GMT+02:00 Renato Marroquín Mogrovejo < > > > [email protected]>: > > > > > >> Hi Steve, > > >> > > >> Comments inline. > > >> > > >> 2014-05-21 20:59 GMT+02:00 Steve Blackmon <[email protected]>: > > >> > > >> I will happily accept pull requests to streams-examples, although I > > >>> don't think it would need to change for an update to the provider to > > >>> be applied (except perhaps to add that verb filtering step) > > >>> > > >> > > >> This is what I meant. > > >> > > >> > > >>> I am willing to donate all of the examples, but I don't think they > > >>> belong in the main repo. What if there were 50 examples many > > >>> containing integration tests? Build would take forever - plus I > don't > > >>> think that a single example failing a test should break the build. > > >>> Mahout suffered from really long-builds and handled it by separating > > >>> the examples out from core. > > >>> > > >> > > >> Well, over in Apache Giraph they have over 20 graph example algorithms > > and > > >> the way they test them is just by doing some simple computations, I > > don't > > >> know how long it takes, but I think it makes sense to at least try it > > and > > >> see how long it takes. > > >> Over in Gora we have a test suite with the different operations > > supported > > >> that is run over the different data stores, and depending on your > > machine, > > >> it might take from 10 - 20 mins. There was a time where the HBase > tests > > >> were taking around 30 mins by themselves but we fixed them and are now > > run > > >> in less than 5 mins. > > >> Maybe your examples could be converted into a test suite? I mean > reading > > >> from twitter(or a file) and persisting into different places. But > > anyhow, > > >> your examples should be referenced somewhere as it is what it makes > > sense > > >> for a newcomer :D > > >> > > >> When the project was hosting in SVN, we had the ability to create and > > >>> checkout different roots (the master pom was in a different root for > > >>> example). If we can do the same using Git I propose we put the > > >>> examples in as a flat multi-module project that way. > > >>> > > >> > > >> +1 > > >> > > >> > > >> Renato M. > > >> > > >> Any thoughts? > > >>> > > >>> Steve Blackmon > > >>> [email protected] > > >>> > > >>> On Wed, May 21, 2014 at 11:59 AM, Renato Marroquín Mogrovejo > > >>> <[email protected]> wrote: > > >>> > I tried updating the site through ApacheCMS but I guess I don't > have > > the > > >>> > rights to do it ;) > > >>> > This is what I got: > > >>> > > > >>> > The operation is forbidden by the server: Commit failed (details > > >>> follow):: > > >>> > Changing file > > >>> > > > >>> > > > '/usr/local/cms/wc/streams/rmarroquin-NlOO6g/trunk/content/source-repository.mdtext' > > >>> > is forbidden by the server: Access to > > >>> > > > >>> > > > '/repos/asf/!svn/txr/1596615-yz8u/incubator/streams/site/trunk/content/source-repository.mdtext' > > >>> > forbidden at /usr/local/cms/webgui/lib/ASF/CMS/WC/Commit.pm line 64 > > >>> > > > >>> > So what is the best/easy way to update the site? > > >>> > > > >>> > @Steve > > >>> > Yeah, I finally got the time to try this out :) and looks very > > promising > > >>> > for sure! > > >>> > Is anybody working on STREAMS-57? Could I pick it up? I think this > > >>> should > > >>> > be pretty straight forward. > > >>> > And are there any plans to put your examples into the code base? > It's > > >>> quite > > >>> > puzzling going from repo to another (plus the one from the site) to > > >>> > accomplish this. And then if I could work on STREAMS-57, would I > have > > >>> to do > > >>> > the pull request into W2OGroup/streams-examples repo? I don't think > > so > > >>> > right? how would this work? > > >>> > Thanks! > > >>> > > > >>> > > > >>> > Renato M. > > >>> > > > >>> > > > >>> > 2014-05-21 17:42 GMT+02:00 Steve Blackmon <[email protected]>: > > >>> > > > >>> >> Cool I'm glad to hear you got it running. That exception occurs > > because > > >>> >> twitter sends a preamble listing all of your friends when you > open a > > >>> >> stream. The provider doesn't have logic to handle it yet, so it > > tries > > >>> to > > >>> >> build an activity and throws that exception. > > >>> >> > > >>> >> STREAMS-57 would suppress this warning - although once non-content > > >>> >> activities with new verbs like follow are generated we will want a > > >>> >> filtering mechanism in place so it remains easy to build indexes > of > > >>> just > > >>> >> shares and posts. > > >>> >> > > >>> >> On Wed, May 21, 2014 at 9:23 AM, Renato Marroquín Mogrovejo < > > >>> >> [email protected]> wrote: > > >>> >> > Hey Matt, > > >>> >> > > > >>> >> > So the examples worked :D at least the one retrieving the > twitter > > >>> user's > > >>> >> > history. The one trying to track users activity keeps on getting > > this > > >>> >> > exception: > > >>> >> > > > >>> >> > 16:09:11.151 [pool-2-thread-2] DEBUG > > o.a.s.t.p.TwitterTypeConverter - > > >>> >> > ACTIVITY > > >>> >> > org.apache.streams.exceptions.ActivitySerializerException: > > >>> unrecognized > > >>> >> type > > >>> >> > at > > >>> >> > > > >>> >> > > >>> >> > > >>> > > > org.apache.streams.twitter.serializer.TwitterJsonActivitySerializer.deserialize(TwitterJsonActivitySerializer.java:74) > > >>> >> > at > > >>> >> > > > >>> >> > > >>> >> > > >>> > > > org.apache.streams.twitter.processor.TwitterTypeConverter.convert(TwitterTypeConverter.java:65) > > >>> >> > at > > >>> >> > > > >>> >> > > >>> >> > > >>> > > > org.apache.streams.twitter.processor.TwitterTypeConverter.process(TwitterTypeConverter.java:146) > > >>> >> > at > > >>> >> > > > >>> >> > > >>> >> > > >>> > > > org.apache.streams.local.tasks.StreamsProcessorTask.run(StreamsProcessorTask.java:72) > > >>> >> > at > > >>> >> > > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > > >>> >> > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > > >>> >> > at > > >>> >> > > > >>> >> > > >>> >> > > >>> > > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > > >>> >> > at > > >>> >> > > > >>> >> > > >>> >> > > >>> > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > > >>> >> > at java.lang.Thread.run(Thread.java:744) > > >>> >> > > > >>> >> > But everything seems to work in spite of that exception :) Cool > > >>> stuff! So > > >>> >> > can I open a JIRA issue for updating the site and putting the > > patch > > >>> there > > >>> >> > then? > > >>> >> > I will write another email with some other issues. > > >>> >> > Thanks Matt! > > >>> >> > > > >>> >> > > > >>> >> > Renato M. > > >>> >> > > > >>> >> > > > >>> >> > 2014-05-21 15:59 GMT+02:00 Matt Franklin < > > [email protected]>: > > >>> >> > > > >>> >> >> On Wed, May 21, 2014 at 9:00 AM, Renato Marroquín Mogrovejo < > > >>> >> >> [email protected]> wrote: > > >>> >> >> > > >>> >> >> > Hi Matt, > > >>> >> >> > > > >>> >> >> > Thank for pointing me to these examples. I will try them out > > and > > >>> >> report > > >>> >> >> > back. > > >>> >> >> > But I have more questions now. I have found: > > >>> >> >> > > > >>> >> >> > https://github.com/apache/incubator-streams --> referenced > by > > the > > >>> >> >> examples > > >>> >> >> > you sent me and more active > > >>> >> >> > https://github.com/apache/streams --> referenced by the > > website > > >>> >> >> > > > >>> >> >> > > >>> >> >> Oops. Yes it is the first one. The second one was a mirror of > > SVN. > > >>> >> >> > > >>> >> >> > > >>> >> >> > > > >>> >> >> > Which one is the official one? I guess the first one right. > > >>> Shouldn't > > >>> >> we > > >>> >> >> > change this on the website? does that mean that Apache > Streams > > >>> have > > >>> >> two > > >>> >> >> git > > >>> >> >> > repos? I can fix that on the website if you point me to the > > right > > >>> >> place. > > >>> >> >> > > > >>> >> >> > > >>> >> >> Website is in SVN using Apache CMS. > > >>> >> >> > > >>> >> >> > > >>> >> >> > > > >>> >> >> > > > >>> >> >> > Renato M. > > >>> >> >> > > > >>> >> >> > > > >>> >> >> > > > >>> >> >> > 2014-05-21 2:14 GMT+02:00 Matt Franklin < > > [email protected] > > >>> >: > > >>> >> >> > > > >>> >> >> > > On Tue, May 20, 2014 at 4:28 PM, Renato Marroquín > Mogrovejo < > > >>> >> >> > > [email protected]> wrote: > > >>> >> >> > > > > >>> >> >> > > > Sorry I pressed the sent button before I finished writing > > the > > >>> >> email. > > >>> >> >> > > > Anyways . . . Tomcat 7.0.39, ServiceMix 5.0.1 and > Cassandra > > >>> 2.0.4. > > >>> >> >> > > > But I am getting this exception: > > >>> >> >> > > > > > >>> >> >> > > > > >>> >> >> > > We are unfortunately in the middle of converging the new > > model > > >>> and > > >>> >> the > > >>> >> >> > old > > >>> >> >> > > model. This runtime should be working, but it needs some > > work to > > >>> >> >> ensure > > >>> >> >> > > that the new components in streams-contrib work. If you are > > >>> looking > > >>> >> to > > >>> >> >> > get > > >>> >> >> > > started with a newer runtime, streams-runtime-local is a > good > > >>> place. > > >>> >> >> > > > > >>> >> >> > > Steve posted a link to some examples he wrote for how to > get > > >>> >> started: > > >>> >> >> > > > > >>> >> >> > > > > >>> >> >> > > > > >>> >> >> > > > >>> >> >> > > >>> >> > > >>> >> > > >>> > > > https://github.com/w2ogroup/streams-examples/tree/master/twitter-history-elasticsearch > > >>> >> >> > > > > >>> >> >> > > We need to create some of our own examples (or ask him > > kindly to > > >>> >> donate > > >>> >> >> > > some to us), but until the site is fixed, this is a good > > start. > > >>> I > > >>> >> will > > >>> >> >> > > create an issue for the Caml runtime issue you found here. > > >>> >> >> > > > > >>> >> >> > > > > >>> >> >> > > > > > >>> >> >> > > > May 20, 2014 10:15:29 PM > > >>> org.apache.catalina.startup.ContextConfig > > >>> >> >> > > > getServletContainerInitializer > > >>> >> >> > > > SEVERE: The ServletContentInitializer > > >>> >> >> > > > > [org.springframework.web.SpringServletContainerInitializer] > > >>> could > > >>> >> not > > >>> >> >> > be > > >>> >> >> > > > created > > >>> >> >> > > > java.lang.ClassNotFoundException: > > >>> >> >> > > > org.springframework.web.SpringServletContainerInitializer > > >>> >> >> > > > at > > >>> >> >> > > > > > >>> >> >> > > > > > >>> >> >> > > > > >>> >> >> > > > >>> >> >> > > >>> >> > > >>> >> > > >>> > > > org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1713) > > >>> >> >> > > > at > > >>> >> >> > > > > > >>> >> >> > > > > > >>> >> >> > > > > >>> >> >> > > > >>> >> >> > > >>> >> > > >>> >> > > >>> > > > org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1558) > > >>> >> >> > > > at java.lang.Class.forName0(Native Method) > > >>> >> >> > > > at java.lang.Class.forName(Class.java:270) > > >>> >> >> > > > at > > >>> >> >> > > > > > >>> >> >> > > > > > >>> >> >> > > > > >>> >> >> > > > >>> >> >> > > >>> >> > > >>> >> > > >>> > > > org.apache.catalina.startup.ContextConfig.getServletContainerInitializer(ContextConfig.java:1659) > > >>> >> >> > > > at > > >>> >> >> > > > > > >>> >> >> > > > > > >>> >> >> > > > > >>> >> >> > > > >>> >> >> > > >>> >> > > >>> >> > > >>> > > > org.apache.catalina.startup.ContextConfig.processServletContainerInitializers(ContextConfig.java:1569) > > >>> >> >> > > > at > > >>> >> >> > > > > > >>> >> >> > > > > > >>> >> >> > > > > >>> >> >> > > > >>> >> >> > > >>> >> > > >>> >> > > >>> > > > org.apache.catalina.startup.ContextConfig.webConfig(ContextConfig.java:1277) > > >>> >> >> > > > at > > >>> >> >> > > > > > >>> >> >> > > > > > >>> >> >> > > > > >>> >> >> > > > >>> >> >> > > >>> >> > > >>> >> > > >>> > > > org.apache.catalina.startup.ContextConfig.configureStart(ContextConfig.java:878) > > >>> >> >> > > > at > > >>> >> >> > > > > > >>> >> >> > > > > > >>> >> >> > > > > >>> >> >> > > > >>> >> >> > > >>> >> > > >>> >> > > >>> > > > org.apache.catalina.startup.ContextConfig.lifecycleEvent(ContextConfig.java:369) > > >>> >> >> > > > at > > >>> >> >> > > > > > >>> >> >> > > > > > >>> >> >> > > > > >>> >> >> > > > >>> >> >> > > >>> >> > > >>> >> > > >>> > > > org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:119) > > >>> >> >> > > > at > > >>> >> >> > > > > > >>> >> >> > > > > > >>> >> >> > > > > >>> >> >> > > > >>> >> >> > > >>> >> > > >>> >> > > >>> > > > org.apache.catalina.util.LifecycleBase.fireLifecycleEvent(LifecycleBase.java:90) > > >>> >> >> > > > at > > >>> >> >> > > > > > >>> >> >> > > > > > >>> >> >> > > > > >>> >> >> > > > >>> >> >> > > >>> >> > > >>> >> > > >>> > > > org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5269) > > >>> >> >> > > > at > > >>> >> >> > > > > > >>> >> > org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150) > > >>> >> >> > > > > > >>> >> >> > > > That let me to [2]. Is this related to this? or is it > just > > me > > >>> >> doing > > >>> >> >> > > things > > >>> >> >> > > > wrong here? > > >>> >> >> > > > Thanks for any pointers! > > >>> >> >> > > > > > >>> >> >> > > > > > >>> >> >> > > > Renato M. > > >>> >> >> > > > > > >>> >> >> > > > 2014-05-20 22:25 GMT+02:00 Renato Marroquín Mogrovejo < > > >>> >> >> > > > [email protected]>: > > >>> >> >> > > > > > >>> >> >> > > > > Hey guys, > > >>> >> >> > > > > > > >>> >> >> > > > > This is a long overdue email but with little steps I > will > > >>> start > > >>> >> >> going > > >>> >> >> > > :) > > >>> >> >> > > > > So I have been trying to run the whole project based > > on[1]. > > >>> As > > >>> >> >> there > > >>> >> >> > is > > >>> >> >> > > > no > > >>> >> >> > > > > version for any of the pieces involved in running this > > (if > > >>> there > > >>> >> is > > >>> >> >> > > > please > > >>> >> >> > > > > point me to them, if not, I am happy to put them on the > > site > > >>> >> >> > > somewhere). > > >>> >> >> > > > I > > >>> >> >> > > > > chose > > >>> >> >> > > > > > > >>> >> >> > > > > > > >>> >> >> > > > > > > >>> >> >> > > > > [1] > > http://streams.incubator.apache.org/streams-usage.html > > >>> >> >> > > > > [2] > > >>> >> >> > > > > > > >>> >> >> > > > > > >>> >> >> > > > > >>> >> >> > > > >>> >> >> > > >>> >> > > >>> >> > > >>> > > > http://stackoverflow.com/questions/7692497/tomcat-wont-load-my-meta-inf-services-javax-servlet-servletcontainerinitializ > > >>> >> >> > > > > > > >>> >> >> > > > > > >>> >> >> > > > > >>> >> >> > > > >>> >> >> > > >>> >> > > >>> > > >> > > >> > > >
