And pyflink (auto-correct)
On Oct 10, 2021, 2:38 PM -0500, sblack...@apache.org <sblack...@apache.org>,
wrote:
> I think this would be great. I have written a lot of code that binds Streams
> provider and processor modules in spark and flink in Scala nd it would be
> great to be able to do the same with pyspark and pulling code.
> On Sep 22, 2021, 11:21 AM -0500, Trevor Grant <trevor.d.gr...@gmail.com>,
> wrote:
> > Heya,
> >
> > I've been working on some Python bindings for Mahout- I could probably use
> > that knowledge and do the same for Streams.
> >
> > On Fri, Sep 3, 2021 at 6:44 PM sblack...@apache.org <sblack...@apache.org>
> > wrote:
> >
> > > Joey, thanks for posting to start discussion.
> > >
> > > I would love to initiate a push to a 1.0 release!
> > >
> > > As important as 1.0 I think, is being able to do a binary + docker release
> > > where everything is ready to run instantly from anywhere.
> > >
> > > The last remaining obstacle to that, is to build a NOTICE file containing
> > > all transitive dependencies NOTICEs. And because we have hundreds of deps,
> > > that is a daunting task.
> > >
> > > So personally I’m on board with removing a bunch of modules in the next
> > > release, to make that problem more manageable. I’d also start with the
> > > runtimes and the persist modules.
> > >
> > > (I think the most valuable modules are the providers and processors,
> > > together with some core modules like streams-config and streams-pojo.)
> > >
> > > I’ve also gotten some good utility across multiple projects running the
> > > classes in streams-examples-flink/flink-twitter-collection as is. They are
> > > simple pipelines, but they 'just work'.
> > >
> > > I’ll be presenting one of those projects in the Federated Data track at
> > > ACNA in session ‘Open Source Social Applications with Apache Streams’
> > > https://www.apachecon.com/acah2021/tracks/feddata.html
> > >
> > >
> > > Which bring me to the main obstacle I think Streams has to wide adoption -
> > > no user interface.
> > >
> > > The app I’ll demo at ACNA uses zeppelin+flink to collect data, puts it in
> > > Postgres, and serves it up for discovery and analysis with an open-source
> > > node.js app.
> > >
> > > It would be neat I think to package all of the valuable connectors and
> > > utilities in the project up into docker image(s), along with a new JSON-LD
> > > ActivityPub-style API, that anyone could run and build their own social
> > > apps on top of. A lot of work, but worthwhile.
> > >
> > > sblack...@apache.org
> > > On Aug 27, 2021, 4:15 PM -0500, Joey Frazee
> > > <joey.fra...@icloud.com.invalid>,
> > > wrote:
> > > > Hey, I don’t think it’s any secret that we’ve been a little dormant
> > > lately. I thought I’d drop a note to say hi and mention a few things I’ve
> > > been thinking about.
> > > >
> > > > First off, I firmly believe that not every ASF project needs to have
> > > high levels of activity, community size, etc. to do some good in the
> > > world,
> > > so while I think about the lull we’ve had, I don’t think it’s bad
> > > (especially how the last year has gone). A good JVM-based ActivityStreams
> > > implementation and related tooling is something that’s needed so the goal
> > > is still good IMO.
> > > >
> > > > A lot of stuff has changed though since the project started. Working
> > > with social data has become a lot more complicated for both business and
> > > ethical reasons, there are tons of frameworks (ASF and otherwise) that do
> > > ingest and persistence, the Java ecosystem has evolved, ActivityStreams
> > > 2.0
> > > exists.
> > > >
> > > > So on those days where I find myself poking around on Streams, I think
> > > about what could a 1.0 or breaking release would be like if we were able
> > > to
> > > pick up the pace again:
> > > >
> > > > - A JSON-LD processor and ActivityStreams 2.0 support would be huge.
> > > >
> > > > - We need to think through what to do about the runtimes and
> > > persistence. I don’t think there’s a big need for them anymore; or at
> > > least
> > > as they are. However, I think the database schema generation stuff Steve
> > > previously started very much is and would be cool if it had wide coverage.
> > > >
> > > > - We’re still on Java 1.8 (minus some baby steps for 11), as is a lot of
> > > the world. 17-LTS is due out next month.
> > > >
> > > > - There’s probably community building stuff we could do.
> > > >
> > > > Anywho, I was just thinking about what might be fun to work on and
> > > wondering whether anyone else has ideas you’ve been thinking about.
> > > >
> > > > Best,
> > > >
> > > > -joey
> > > >
> > >
