I think this would be great. I have written a lot of code that binds Streams
provider and processor modules in spark and flink in Scala nd it would be great
to be able to do the same with pyspark and pulling code.
On Sep 22, 2021, 11:21 AM -0500, Trevor Grant <trevor.d.gr...@gmail.com>, wrote:
> Heya,
>
> I've been working on some Python bindings for Mahout- I could probably use
> that knowledge and do the same for Streams.
>
> On Fri, Sep 3, 2021 at 6:44 PM sblack...@apache.org <sblack...@apache.org>
> wrote:
>
> > Joey, thanks for posting to start discussion.
> >
> > I would love to initiate a push to a 1.0 release!
> >
> > As important as 1.0 I think, is being able to do a binary + docker release
> > where everything is ready to run instantly from anywhere.
> >
> > The last remaining obstacle to that, is to build a NOTICE file containing
> > all transitive dependencies NOTICEs. And because we have hundreds of deps,
> > that is a daunting task.
> >
> > So personally I’m on board with removing a bunch of modules in the next
> > release, to make that problem more manageable. I’d also start with the
> > runtimes and the persist modules.
> >
> > (I think the most valuable modules are the providers and processors,
> > together with some core modules like streams-config and streams-pojo.)
> >
> > I’ve also gotten some good utility across multiple projects running the
> > classes in streams-examples-flink/flink-twitter-collection as is. They are
> > simple pipelines, but they 'just work'.
> >
> > I’ll be presenting one of those projects in the Federated Data track at
> > ACNA in session ‘Open Source Social Applications with Apache Streams’
> > https://www.apachecon.com/acah2021/tracks/feddata.html
> >
> >
> > Which bring me to the main obstacle I think Streams has to wide adoption -
> > no user interface.
> >
> > The app I’ll demo at ACNA uses zeppelin+flink to collect data, puts it in
> > Postgres, and serves it up for discovery and analysis with an open-source
> > node.js app.
> >
> > It would be neat I think to package all of the valuable connectors and
> > utilities in the project up into docker image(s), along with a new JSON-LD
> > ActivityPub-style API, that anyone could run and build their own social
> > apps on top of. A lot of work, but worthwhile.
> >
> > sblack...@apache.org
> > On Aug 27, 2021, 4:15 PM -0500, Joey Frazee
> > <joey.fra...@icloud.com.invalid>,
> > wrote:
> > > Hey, I don’t think it’s any secret that we’ve been a little dormant
> > lately. I thought I’d drop a note to say hi and mention a few things I’ve
> > been thinking about.
> > >
> > > First off, I firmly believe that not every ASF project needs to have
> > high levels of activity, community size, etc. to do some good in the world,
> > so while I think about the lull we’ve had, I don’t think it’s bad
> > (especially how the last year has gone). A good JVM-based ActivityStreams
> > implementation and related tooling is something that’s needed so the goal
> > is still good IMO.
> > >
> > > A lot of stuff has changed though since the project started. Working
> > with social data has become a lot more complicated for both business and
> > ethical reasons, there are tons of frameworks (ASF and otherwise) that do
> > ingest and persistence, the Java ecosystem has evolved, ActivityStreams 2.0
> > exists.
> > >
> > > So on those days where I find myself poking around on Streams, I think
> > about what could a 1.0 or breaking release would be like if we were able to
> > pick up the pace again:
> > >
> > > - A JSON-LD processor and ActivityStreams 2.0 support would be huge.
> > >
> > > - We need to think through what to do about the runtimes and
> > persistence. I don’t think there’s a big need for them anymore; or at least
> > as they are. However, I think the database schema generation stuff Steve
> > previously started very much is and would be cool if it had wide coverage.
> > >
> > > - We’re still on Java 1.8 (minus some baby steps for 11), as is a lot of
> > the world. 17-LTS is due out next month.
> > >
> > > - There’s probably community building stuff we could do.
> > >
> > > Anywho, I was just thinking about what might be fun to work on and
> > wondering whether anyone else has ideas you’ve been thinking about.
> > >
> > > Best,
> > >
> > > -joey
> > >
> >
