STREAMS-403 : run embedded, outside of a streams runtime STREAMS-411 : run from console via main method
On Tue, Oct 11, 2016 at 1:43 PM, Matt Franklin <m.ben.frank...@gmail.com> wrote: > On Thu, Oct 6, 2016 at 10:32 AM Steve Blackmon <st...@blackmon.org> wrote: > >> This issue order makes sense to me, based on simplicity and complexity >> >> of improvement: >> >> >> >> > > STREAMS-403(https://issues.apache.org/jira/browse/STREAMS-403) >> >> > > Ensure all providers function stand-alone >> > > Just so I am following, what is the purpose of this? Run standalone from a > console or run standalone outside of a streams runtime. > > >> >> > > STREAMS-411(https://issues.apache.org/jira/browse/STREAMS-411) >> >> > > ability (and instructions on how) to run providers directly from >> console >> >> > > STREAMS-422(https://issues.apache.org/jira/browse/STREAMS-422) >> >> > > SNS providers should emit appropriately typed documents by default >> >> > > STREAMS-415(https://issues.apache.org/jira/browse/STREAMS-415) >> >> > > Proof of concept integration test that pulls actual data from generator >> >> >> >> In terms of which providers to target in what order, I’d suggest: >> >> >> >> twitter >> >> instagram >> >> rss >> >> youtube >> >> facebook >> >> gplus >> >> sysomos >> >> moreover >> >> >> >> Further down the list: more uncertainly about what exactly to do, how >> >> to do it, how to live test it, higher likelihood of encountering >> >> technical debt. >> >> >> >> If anyone wants to take a shot at any piece of this, go for it. No >> >> need for omni-bus PRs covering a full story or a full provider, every >> >> incremental improvement helps. I’ll happily code review any PR that’s >> >> opened and help get it finished. >> >> >> >> Steve >> >> >> >> >> >> On October 6, 2016 at 7:52:42 AM, Suneel Marthi >> >> (smar...@apache.org(mailto:smar...@apache.org)) wrote: >> >> >> >> > On Tue, Oct 4, 2016 at 9:58 PM, sblackmon wrote: >> >> > >> >> > > >> >> > > All, >> >> > > >> >> > > I’ve pushed a branch that depicts the sort of provider upgrades that I >> >> > > envision making across all providers this quarter. >> >> > > >> >> > > https://github.com/apache/incubator-streams/compare/ >> >> > > master...steveblackmon:STREAMS-415?expand=1 >> >> > > >> >> > > It solves for all of the following clean-up and improvement issues, >> >> > > specifically for the TwitterTimelineProvider >> >> > > >> >> > > STREAMS-403(https://issues.apache.org/jira/browse/STREAMS-403) >> >> > > Ensure all providers function stand-alone >> >> > > >> >> > > STREAMS-422(https://issues.apache.org/jira/browse/STREAMS-422) >> >> > > SNS providers should emit appropriately typed documents by default >> >> > > >> >> > > STREAMS-411(https://issues.apache.org/jira/browse/STREAMS-411) >> >> > > ability (and instructions on how) to run providers directly from >> console >> >> > > >> >> > > STREAMS-415(https://issues.apache.org/jira/browse/STREAMS-415) >> >> > > Proof of concept integration test that pulls actual data from generator >> >> > > >> >> > > Rather than tackle these stories one at a time laterally across all >> >> > > providers, we could consider divide-and-conquer by module or by class >> to >> >> > > reduce the number of commits and committers hitting each provider. >> >> > > >> >> > > Curious what others want to try. Would you rather take on one provider >> at >> >> > > a time and improve it’s quality across several dimensions, or improve >> many >> >> > > at once in a single dimension? >> >> > > >> >> > >> >> > I think we should do one at a time given the available and committed >> >> > resources. Which one would u like to tackle fist from the list above? >> >> > >> >> > > >> >> > > Steve >> >> > > >> >> > > On October 4, 2016 at 11:56:18 AM, sblackmon (sblack...@apache.org >> (mailto: >> >> > > sblack...@apache.org)) wrote: >> >> > > >> >> > > > With 0.3-incubating in the books, it’s time to take a look at what is >> >> > > most important for the next release. >> >> > > > >> >> > > > Here are a few concrete things I propose we tackle in October. They >> are >> >> > > tagged in JIRA with fixVersion=0.4 >> >> > > > >> >> > > > https://issues.apache.org/jira/browse/STREAMS-418?filter=12338530 >> >> > > > >> >> > > > I saw some new issues being created the last few days. Please keep it >> >> > > up, no matter how small! >> >> > > > >> >> > > > Major themes: >> >> > > > * Reboot and Clean-up >> >> > > > * Provider and Configuration consistency >> >> > > > * Prep for the AS 2.0 switch >> >> > > > Also: >> >> > > > * POC persist integration tests with docker >> >> > > > * POC provider integration tests that perform collection >> >> > > > * Fix lingering and simple bugs / documentation >> >> > > > * Flink example code >> >> > > > >> >> > > > If we target early week of October 24 for code freeze and cut rc1 >> that >> >> > > should give us time to get the release passed (hopefully with unanimous >> >> > > support from three active IPMC members on PPMC) by our November report. >> >> > > > >> >> > > > Feedback welcome! And feel free to assign yourself or let us someone >> >> > > with permissions know if you want to have something assigned to you. >> >> > > > >> >> > > > Steve >> >> > > > >> >> > > > On September 28, 2016 at 3:00:01 PM, sblackmon (sblack...@apache.org >> >> > > (mailto:sblack...@apache.org)) wrote: >> >> > > > >> >> > > > > All, >> >> > > > > >> >> > > > > Joey brought this up over the weekend and I think a discussion is >> >> > > overdue on the topic. >> >> > > > > >> >> > > > > Encouraging community growth and performing regular releases are on >> >> > > our list of graduation criteria. >> >> > > > > >> >> > > > > A few easy behaviors we can adopt to take to make progress on these >> >> > > goals: >> >> > > > > - planning release versions around one or two significant >> improvements >> >> > > > > - setting target dates to kick off upcoming releases >> >> > > > > - prioritizing our backlog after each release >> >> > > > > - discussing project and community milestones openly on the list >> >> > > > > - organizing JIRA so that all contributors (especially new) can >> decide >> >> > > where it’s most important to focus their efforts >> >> > > > > >> >> > > > > I think to get things moving again and demonstrate we are capable >> of >> >> > > consistent progress, we should aim to perform a release once per month >> >> > > around the end of the month. >> >> > > > > >> >> > > > > As for what to focus on, I think it’s time to discuss adopting >> >> > > Activity Streams 2.0, figure out what form that transition would take, >> and >> >> > > get started down that path. Working implementations demonstrate the >> >> > > suitability of the standard and drive it’s adoption, and the prospects >> of >> >> > > this project are closely tied to those of the standard. Separate >> DISCUSS >> >> > > coming on this topic. >> >> > > > > >> >> > > > > Also important for the ‘reboot’ theme, we should delete any >> modules we >> >> > > aren’t going to maintain, and bring all modules we are going to >> maintain up >> >> > > to acceptable standards - exactly what that means is an open question >> but >> >> > > broadly they should have documentation, code comments, and tests at the >> >> > > level of a typical module in a typical TLP. >> >> > > > > >> >> > > > > Expanding the examples to demonstrate how to use streams providers >> and >> >> > > processors within various execution engines and fixing any bugs that >> have >> >> > > been reported is desirable as well. Adding at least one new example per >> >> > > release is a good target for now. >> >> > > > > >> >> > > > > I have created some future versions with target release dates in >> JIRA >> >> > > and invite all committers to associate existing or new issues with >> those >> >> > > releases, or anyone who can’t modify JIRA to summarize their thoughts >> and >> >> > > share with the list and I will incorporate those ideas into JIRA. This >> >> > > should be the default reference for anyone looking for a way to help - >> look >> >> > > at issues associated with the next few releases and the top of the >> backlog >> >> > > and pick something that appeals and is in line with your experience. >> >> > > > > >> >> > > > > Anything else that should be a top priority for the rest of the >> year? >> >> > > Or other ideas on improving planning and coordination? >> >> > > > > >> >> > > > > Steve >> >> > > > > >> >> > > > > On September 24, 2016 at 1:01:02 PM, apache (sblack...@apache.org >> >> > > (mailto:sblack...@apache.org)) wrote: >> >> > > > > > - This has already come up, but maybe ActivityStreams 2.0 support >> >> > > would broaden the community and motivate more work. It's also a >> concrete >> >> > > > > >> >> > > > > >> >> > > > > > goal to work toward so people would know where they can start. >> >> > > > > >> >> > > > > >> >> > > > > >> >> > > > > >> >> > > > > > - Steve and I did a little work here a few months ago, but the >> JIRA >> >> > > could reflect the priorities better and I think keep the community >> working >> >> > > in a common direction. >> >> > > > > >> >> > > > > >> >> > > > > >> >> > > > > >> >> > > > > >> >> > > > >> >> > > > >> >> > > > >> >> > > > >> >> > > >> >> > > >> >> > > >> >> > > >> >> > > >> >>
