Hi All,
Sample code to see how records tagging will be handled in
Flink is posted on [1]. The main class to run the same is MockHudi.java
with a sample path for checkpointing.
As of now this is just a sample to know we should ke caching in Flink
states with bare minimum configs.
As per my experience I have cached around 10s of TBs in Flink rocksDB state
with the right configs. So I'm sure it should work here as well.
1:
https://github.com/taherk77/FlinkHudi/tree/master/FlinkHudiExample/src/main/java/org/apache/hudi
Regards,
Taher Koitawala
On Sun, Sep 22, 2019, 7:34 PM Vinoth Chandar <vin...@apache.org> wrote:
> It wont be much different than the HBaseIndex we have today. Would like to
> have always have an option like BloomIndex that does not need any external
> dependencies.
> The moment you bring an external data store in, someone becomes a DBA. :)
>
> On Sun, Sep 22, 2019 at 6:46 AM Semantic Beeng <n...@semanticbeeng.com>
> wrote:
>
> > @vc can you see how ApacheCrail could be used to implement this at scale
> > but also in a way that abstracts over both Spark and Flink?
> >
> > "Crail Store implements a hierarchical namespace across a cluster of RDMA
> > interconnected storage resources such as DRAM or flash"
> >
> > https://crail.incubator.apache.org/overview/
> >
> > + 2 cents
> > https://twitter.com/semanticbeeng/status/1175767500790915072?s=20
> >
> > Cheers
> >
> > Nick
> >
> > On September 22, 2019 at 9:28 AM Vinoth Chandar <vin...@apache.org>
> wrote:
> >
> >
> > It could be much larger. :) imagine billions of keys each 32 bytes,
> mapped
> > to another 32 byte
> >
> > The advantage of the current bloom index is that its effectively stored
> > with data itself and this reduces complexity in terms of keeping index
> and
> > data consistent etc
> >
> > One orthogonal idea from long time ago that moves indexing out of data
> > storage and is generalizable
> >
> > https://github.com/apache/incubator-hudi/wiki/HashMap-Index
> >
> > If someone here knows flink well and can implement some standalone flink
> > code to mimic tagLocation() functionality and share with the group, that
> > would be great. Lets worry about performance once we have a flink DAG. I
> > think this is a critical and most tricky piece in supporting flink.
> >
> > On Sat, Sep 21, 2019 at 4:17 AM Vinay Patil <vinay18.pa...@gmail.com>
> > wrote:
> >
> > Hi Taher,
> >
> > I agree with this , if the state is becoming too large we should have an
> > option of storing it in external state like File System or RocksDb.
> >
> > @Vinoth Chandar <vin...@apache.org> can the state of HoodieBloomIndex go
> > beyond 10-15 GB
> >
> > Regards,
> > Vinay Patil
> >
> > >
> >
> > On Fri, Sep 20, 2019 at 11:37 AM Taher Koitawala <taher...@gmail.com>
> > wrote:
> >
> > >> Hey Guys, Any thoughts on the above idea? To handle HoodieBloomIndex
> > with
> > >> HeapState, RocksDBState and FsState but on Spark.
> > >>
> > >> On Tue, Sep 17, 2019 at 1:41 PM Taher Koitawala <taher...@gmail.com>
> > >> wrote:
> > >>
> > >> > Hi Vinoth,
> > >> > Having seen the doc and code. I understand the
> > >> > HoodieBloomIndex mainly caches key and partition path. Can we
> address
> > >> how
> > >> > Flink does it? Like, have HeapState where the user chooses to cache
> > the
> > >> > Index on heap, RockDBState where indexes are written to RocksDB and
> > >> finally
> > >> > FsState where indexes can be written to HDFS, S3, Azure Fs. And on
> > top,
> > >> we
> > >> > can do an index Time To Live.
> > >> >
> > >> > Regards,
> > >> > Taher Koitawala
> > >> >
> > >> > On Mon, Sep 16, 2019 at 11:43 PM Vinoth Chandar <vin...@apache.org>
> > >> wrote:
> > >> >
> > >> >> I still feel the key thing here is reimplementing HoodieBloomIndex
> > >> without
> > >> >> needing spark caching.
> > >> >>
> > >> >>
> > >>
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=103093742#Design&Architecture-BloomIndex(non-global
> > )
> > >> >> documents the spark DAG in detail.
> > >> >>
> > >> >> If everyone feels, it's best for me to scope the work out, then
> happy
> > >> to
> > >> >> do
> > >> >> it!
> > >> >>
> > >> >> On Mon, Sep 16, 2019 at 10:23 AM Taher Koitawala <
> taher...@gmail.com
> > >
> > >> >> wrote:
> > >> >>
> > >> >> > Guys I think we are slowing down on this again. We need to start
> > >> >> planning
> > >> >> > small small tasks towards this VC please can you help fast track
> > >> this?
> > >> >> >
> > >> >> > Regards,
> > >> >> > Taher Koitawala
> > >> >> >
> > >> >> > On Thu, Aug 15, 2019, 10:07 AM Vinoth Chandar <vin...@apache.org
> >
> > >> >> wrote:
> > >> >> >
> > >> >> > > Look forward to the analysis. A key class to read would be
> > >> >> > > HoodieBloomIndex, which uses a lot of spark caching and
> shuffles.
> > >> >> > >
> > >> >> > > On Tue, Aug 13, 2019 at 7:52 PM vino yang <
> yanghua1...@gmail.com
> > >
> > >> >> wrote:
> > >> >> > >
> > >> >> > > > >> Currently Spark Streaming micro batching fits well with
> > Hudi,
> > >> >> since
> > >> >> > it
> > >> >> > > > amortizes the cost of indexing, workload profiling etc. 1
> spark
> > >> >> micro
> > >> >> > > batch
> > >> >> > > > = 1 hudi commit
> > >> >> > > > With the per-record model in Flink, I am not sure how useful
> it
> > >> >> will be
> > >> >> > > to
> > >> >> > > > support hudi.. for e.g, 1 input record cannot be 1 hudi
> commit,
> > >> it
> > >> >> will
> > >> >> > > be
> > >> >> > > > inefficient..
> > >> >> > > >
> > >> >> > > > Yes, if 1 input record = 1 hudi commit, it would be
> > inefficient.
> > >> >> About
> > >> >> > > > Flink streaming, we can also implement the "batch" and
> > >> "micro-batch"
> > >> >> > > model
> > >> >> > > > when process data. For example:
> > >> >> > > >
> > >> >> > > > - aggregation: use flexibility window mechanism;
> > >> >> > > > - non-aggregation: use Flink stateful state API cache a batch
> > >> >> data
> > >> >> > > >
> > >> >> > > >
> > >> >> > > > >> On first focussing on decoupling of Spark and Hudi alone,
> > yes
> > >> a
> > >> >> full
> > >> >> > > > summary of how Spark is being used in a wiki page is a good
> > start
> > >> >> IMO.
> > >> >> > We
> > >> >> > > > can then hash out what can be generalized and what cannot be
> > and
> > >> >> needs
> > >> >> > to
> > >> >> > > > be left in hudi-client-spark vs hudi-client-core
> > >> >> > > >
> > >> >> > > > agree
> > >> >> > > >
> > >> >> > > > Vinoth Chandar <vin...@apache.org> 于2019年8月14日周三 上午8:35写道：
> > >> >> > > >
> > >> >> > > > > >> We should only stick to Flink Streaming. Furthermore if
> > >> there
> > >> >> is a
> > >> >> > > > > requirement for batch then users
> > >> >> > > > > >> should use Spark or then we will anyway have a beam
> > >> integration
> > >> >> > > coming
> > >> >> > > > > up.
> > >> >> > > > >
> > >> >> > > > > Currently Spark Streaming micro batching fits well with
> Hudi,
> > >> >> since
> > >> >> > it
> > >> >> > > > > amortizes the cost of indexing, workload profiling etc. 1
> > spark
> > >> >> micro
> > >> >> > > > batch
> > >> >> > > > > = 1 hudi commit
> > >> >> > > > > With the per-record model in Flink, I am not sure how
> useful
> > it
> > >> >> will
> > >> >> > be
> > >> >> > > > to
> > >> >> > > > > support hudi.. for e.g, 1 input record cannot be 1 hudi
> > >> commit, it
> > >> >> > will
> > >> >> > > > be
> > >> >> > > > > inefficient..
> > >> >> > > > >
> > >> >> > > > > On first focussing on decoupling of Spark and Hudi alone,
> > yes a
> > >> >> full
> > >> >> > > > > summary of how Spark is being used in a wiki page is a good
> > >> start
> > >> >> > IMO.
> > >> >> > > We
> > >> >> > > > > can then hash out what can be generalized and what cannot
> be
> > >> and
> > >> >> > needs
> > >> >> > > to
> > >> >> > > > > be left in hudi-client-spark vs hudi-client-core
> > >> >> > > > >
> > >> >> > > > >
> > >> >> > > > >
> > >> >> > > > > On Tue, Aug 13, 2019 at 3:57 AM vino yang <
> > >> yanghua1...@gmail.com>
> > >> >> > > wrote:
> > >> >> > > > >
> > >> >> > > > > > Hi Nick and Taher,
> > >> >> > > > > >
> > >> >> > > > > > I just want to answer Nishith's question. Reference his
> old
> > >> >> > > description
> > >> >> > > > > > here:
> > >> >> > > > > >
> > >> >> > > > > > > You can do a parallel investigation while we are
> deciding
> > >> on
> > >> >> the
> > >> >> > > > module
> > >> >> > > > > > structure. You could be looking at all the patterns in
> > >> Hudi's
> > >> >> > Spark
> > >> >> > > > APIs
> > >> >> > > > > > usage (RDD/DataSource/SparkContext) and see if such
> support
> > >> can
> > >> >> be
> > >> >> > > > > achieved
> > >> >> > > > > > in theory with Flink. If not, what is the workaround.
> > >> >> Documenting
> > >> >> > > such
> > >> >> > > > > > patterns would be valuable when multiple engineers are
> > >> working
> > >> >> on
> > >> >> > it.
> > >> >> > > > For
> > >> >> > > > > > e:g, Hudi relies on (a) custom partitioning logic for
> > >> >> upserts,
> > >> >> > > > > (b)
> > >> >> > > > > > caching RDDs to avoid reruns of costly stages (c) A Spark
> > >> >> > upsert
> > >> >> > > > task
> > >> >> > > > > > knowing its spark partition/task/attempt ids
> > >> >> > > > > >
> > >> >> > > > > > And just like the title of this thread, we are going to
> try
> > >> to
> > >> >> > > decouple
> > >> >> > > > > > Hudi and Spark. That means we can run the whole Hudi
> > without
> > >> >> > > depending
> > >> >> > > > > > Spark. So we need to analyze all the usage of Spark in
> > Hudi.
> > >> >> > > > > >
> > >> >> > > > > > Here we are not discussing the integration of Hudi and
> > Flink
> > >> in
> > >> >> the
> > >> >> > > > > > application layer. Instead, I want Hudi to be decoupled
> > from
> > >> >> Spark
> > >> >> > > and
> > >> >> > > > > > allow other engines (such as Flink) to replace Spark.
> > >> >> > > > > >
> > >> >> > > > > > It can be divided into long-term goals and short-term
> > goals.
> > >> As
> > >> >> > > Nishith
> > >> >> > > > > > stated in a recent email.
> > >> >> > > > > >
> > >> >> > > > > > I mentioned the Flink Batch API here because Hudi can
> > connect
> > >> >> with
> > >> >> > > many
> > >> >> > > > > > different Source/Sinks. Some file-based reads are not
> > >> >> appropriate
> > >> >> > for
> > >> >> > > > > Flink
> > >> >> > > > > > Streaming.
> > >> >> > > > > >
> > >> >> > > > > > Therefore, this is a comprehensive survey of the use of
> > >> Spark in
> > >> >> > > Hudi.
> > >> >> > > > > >
> > >> >> > > > > > Best,
> > >> >> > > > > > Vino
> > >> >> > > > > >
> > >> >> > > > > >
> > >> >> > > > > > taher koitawala <taher...@gmail.com> 于2019年8月13日周二
> > 下午5:43写道：
> > >> >> > > > > >
> > >> >> > > > > > > Hi Vino,
> > >> >> > > > > > > According to what I've seen Hudi has a lot of spark
> > >> >> > component
> > >> >> > > > > > flowing
> > >> >> > > > > > > throwing it. Like Taskcontexts, JavaSparkContexts etc.
> > The
> > >> >> main
> > >> >> > > > > classes I
> > >> >> > > > > > > guess we should focus upon is HoodieTable and Hoodie
> > write
> > >> >> > clients.
> > >> >> > > > > > >
> > >> >> > > > > > > Also Vino, I don't think we should be providing Flink
> > >> dataset
> > >> >> > > > > > > implementation. We should only stick to Flink
> Streaming.
> > >> >> > > > > > > Furthermore if there is a requirement for
> > >> batch
> > >> >> > then
> > >> >> > > > > users
> > >> >> > > > > > > should use Spark or then we will anyway have a beam
> > >> >> integration
> > >> >> > > > coming
> > >> >> > > > > > up.
> > >> >> > > > > > >
> > >> >> > > > > > > As of cache, How about we write our stateful Flink
> > function
> > >> >> and
> > >> >> > use
> > >> >> > > > > > > RocksDbStateBackend with some state TTL.
> > >> >> > > > > > >
> > >> >> > > > > > > On Tue, Aug 13, 2019, 2:28 PM vino yang <
> > >> >> yanghua1...@gmail.com>
> > >> >> > > > wrote:
> > >> >> > > > > > >
> > >> >> > > > > > > > Hi all,
> > >> >> > > > > > > >
> > >> >> > > > > > > > After doing some research, let me share my
> information:
> > >> >> > > > > > > >
> > >> >> > > > > > > >
> > >> >> > > > > > > > - Limitation of computing engine capabilities: Hudi
> > >> uses
> > >> >> > > Spark's
> > >> >> > > > > > > > RDD#persist, and Flink currently has no API to cache
> > >> >> > datasets.
> > >> >> > > > > Maybe
> > >> >> > > > > > > we
> > >> >> > > > > > > > can
> > >> >> > > > > > > > only choose to use external storage or do not use
> > >> cache?
> > >> >> For
> > >> >> > > the
> > >> >> > > > > use
> > >> >> > > > > > > of
> > >> >> > > > > > > > other APIs, the two currently offer almost equivalent
> > >> >> > > > > capabilities.
> > >> >> > > > > > > > - The abstraction of the computing engine is
> > >> different:
> > >> >> > > > > Considering
> > >> >> > > > > > > the
> > >> >> > > > > > > > different usage scenarios of the computing engine in
> > >> >> Hudi,
> > >> >> > > Flink
> > >> >> > > > > has
> > >> >> > > > > > > not
> > >> >> > > > > > > > yet implemented stream batch unification, so we may
> > >> use
> > >> >> both
> > >> >> > > > > Flink's
> > >> >> > > > > > > > DataSet API (batch processing) and DataStream API
> > >> (stream
> > >> >> > > > > > processing).
> > >> >> > > > > > > >
> > >> >> > > > > > > > Best,
> > >> >> > > > > > > > Vino
> > >> >> > > > > > > >
> > >> >> > > > > > > > nishith agarwal <n3.nas...@gmail.com> 于2019年8月8日周四
> > >> >> 上午12:57写道：
> > >> >> > > > > > > >
> > >> >> > > > > > > > > Nick,
> > >> >> > > > > > > > >
> > >> >> > > > > > > > > You bring up a good point about the non-trivial
> > >> >> programming
> > >> >> > > model
> > >> >> > > > > > > > > differences between these different technologies.
> > From
> > >> a
> > >> >> > > > > theoretical
> > >> >> > > > > > > > > perspective, I'd say considering a higher level
> > >> >> abstraction
> > >> >> > > makes
> > >> >> > > > > > > sense.
> > >> >> > > > > > > > I
> > >> >> > > > > > > > > think we have to decouple some objectives and
> > concerns
> > >> >> here.
> > >> >> > > > > > > > >
> > >> >> > > > > > > > > a) The immediate desire is to have Hudi be able to
> > run
> > >> on
> > >> >> a
> > >> >> > > Flink
> > >> >> > > > > (or
> > >> >> > > > > > > > > non-spark) engine. This naturally begs the question
> > of
> > >> >> > > decoupling
> > >> >> > > > > > Hudi
> > >> >> > > > > > > > > concepts from direct Spark dependencies.
> > >> >> > > > > > > > >
> > >> >> > > > > > > > > b) If we do want to initiate the above effort,
> would
> > it
> > >> >> make
> > >> >> > > > sense
> > >> >> > > > > to
> > >> >> > > > > > > > just
> > >> >> > > > > > > > > have a higher level abstraction, building on other
> > >> >> > technologies
> > >> >> > > > > like
> > >> >> > > > > > > beam
> > >> >> > > > > > > > > (euphoria etc) and provide single, clean API's that
> > >> may be
> > >> >> > more
> > >> >> > > > > > > > > maintainable from a code perspective. But at the
> same
> > >> time
> > >> >> > this
> > >> >> > > > > will
> > >> >> > > > > > > > > introduce challenges on how to maintain efficiency
> > and
> > >> >> > > optimized
> > >> >> > > > > > > runtime
> > >> >> > > > > > > > > dags for Hudi (since the code would move away from
> > >> point
> > >> >> > > > > integrations
> > >> >> > > > > > > and
> > >> >> > > > > > > > > whenever this happens, tuning natively for specific
> > >> >> engines
> > >> >> > > > becomes
> > >> >> > > > > > > more
> > >> >> > > > > > > > > and more difficult).
> > >> >> > > > > > > > >
> > >> >> > > > > > > > > My general opinion is that, as the community grows
> > over
> > >> >> time
> > >> >> > > with
> > >> >> > > > > > more
> > >> >> > > > > > > > > folks having an in-depth understanding of Hudi,
> going
> > >> from
> > >> >> > > > > > > current_state
> > >> >> > > > > > > > ->
> > >> >> > > > > > > > > (a) -> (b) might be the most reliable and adoptable
> > >> path
> > >> >> for
> > >> >> > > this
> > >> >> > > > > > > > project.
> > >> >> > > > > > > > >
> > >> >> > > > > > > > > Thanks,
> > >> >> > > > > > > > > Nishith
> > >> >> > > > > > > > >
> > >> >> > > > > > > > > On Tue, Aug 6, 2019 at 1:30 PM Semantic Beeng <
> > >> >> > > > > > n...@semanticbeeng.com>
> > >> >> > > > > > > > > wrote:
> > >> >> > > > > > > > >
> > >> >> > > > > > > > > > There are some not trivial difference between
> > >> >> programming
> > >> >> > > model
> > >> >> > > > > and
> > >> >> > > > > > > > > > runtime semantics between Beam, Spark and Flink.
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > >
> > >> >> > > > > > > >
> > >> >> > > > > > >
> > >> >> > > > > >
> > >> >> > > > >
> > >> >> > > >
> > >> >> > >
> > >> >> >
> > >> >>
> > >>
> >
> https://beam.apache.org/documentation/runners/capability-matrix/#cap-full-how
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > > Nitish, Vino - thoughts?
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > > Does it feel to consider a higher level
> > abstraction /
> > >> >> DSL
> > >> >> > > > instead
> > >> >> > > > > > of
> > >> >> > > > > > > > > > maintaining different code with same
> functionality
> > >> but
> > >> >> > > > different
> > >> >> > > > > > > > > > programming models ?
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > >
> > >> >> https://beam.apache.org/documentation/sdks/java/euphoria/
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > > Nick
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > > On August 6, 2019 at 4:04 PM nishith agarwal <
> > >> >> > > > > n3.nas...@gmail.com>
> > >> >> > > > > > > > > wrote:
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > > +1 for Approach 1 Point integration with each
> > >> framework.
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > > Pros for point integration
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > > - Hudi community is already familiar with spark
> > >> and
> > >> >> > spark
> > >> >> > > > > based
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > > actions/shuffles etc. Since both modules can be
> > >> >> decoupled,
> > >> >> > > this
> > >> >> > > > > > > enables
> > >> >> > > > > > > > > us
> > >> >> > > > > > > > > > to have a steady release for Hudi for 1 execution
> > >> engine
> > >> >> > > > (spark)
> > >> >> > > > > > > while
> > >> >> > > > > > > > we
> > >> >> > > > > > > > > > hone our skills and iterate on making flink dag
> > >> >> optimized,
> > >> >> > > > > > performant
> > >> >> > > > > > > > > with
> > >> >> > > > > > > > > > the right configuration.
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > > - This might be a stepping stone towards
> rewriting
> > >> >> the
> > >> >> > > > entire
> > >> >> > > > > > code
> > >> >> > > > > > > > > base
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > > being agnostic of spark/flink. This approach will
> > >> help
> > >> >> us
> > >> >> > fix
> > >> >> > > > > > tests,
> > >> >> > > > > > > > > > intricacies and help make the code base ready
> for a
> > >> >> larger
> > >> >> > > > > rework.
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > > - Seems like the easiest way to add flink support
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > > Cons
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > > - More code paths to maintain and reason since
> the
> > >> >> spark
> > >> >> > > and
> > >> >> > > > > > flink
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > > integrations will naturally diverge over time.
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > > Theoretically, I do like the idea of being able
> to
> > >> run
> > >> >> the
> > >> >> > > hudi
> > >> >> > > > > dag
> > >> >> > > > > > > on
> > >> >> > > > > > > > > beam
> > >> >> > > > > > > > > > more than point integrations, where there is one
> > >> >> API/logic
> > >> >> > to
> > >> >> > > > > > reason
> > >> >> > > > > > > > > about.
> > >> >> > > > > > > > > > But practically, that may not be the right
> > direction.
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > > Pros
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > > - Lesser cognitive burden in maintaining,
> evolving
> > >> >> and
> > >> >> > > > > releasing
> > >> >> > > > > > > the
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > > project with one API to reason with.
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > > - Theoretically, going forward assuming beam is
> > >> >> adopted
> > >> >> > > as a
> > >> >> > > > > > > > standard
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > > programming paradigm for stream/batch, this would
> > >> enable
> > >> >> > > > > consumers
> > >> >> > > > > > > > > leverage
> > >> >> > > > > > > > > > the power of hudi more easily.
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > > Cons
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > > - Massive rewrite of the code base. Additionally,
> > >> >> since
> > >> >> > we
> > >> >> > > > > would
> > >> >> > > > > > > > have
> > >> >> > > > > > > > > > moved
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > > away from directly using spark APIs, there is a
> > >> bigger
> > >> >> risk
> > >> >> > > of
> > >> >> > > > > > > > > regression.
> > >> >> > > > > > > > > > We would have to be very thorough with all the
> > >> >> intricacies
> > >> >> > > and
> > >> >> > > > > > ensure
> > >> >> > > > > > > > the
> > >> >> > > > > > > > > > same stability of new releases.
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > > - Managing future features (which may be very
> > >> spark
> > >> >> > > driven)
> > >> >> > > > > will
> > >> >> > > > > > > > > either
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > > clash or pause or will need to be reworked.
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > > - Tuning jobs for Spark/Flink type execution
> > >> >> frameworks
> > >> >> > > > > > > individually
> > >> >> > > > > > > > > > might
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > > be difficult and will get difficult over time as
> > the
> > >> >> > project
> > >> >> > > > > > evolves,
> > >> >> > > > > > > > > where
> > >> >> > > > > > > > > > some beam integrations with spark/flink may not
> > work
> > >> as
> > >> >> > > > expected.
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > > - Also, as pointed above, need to probably
> support
> > >> >> the
> > >> >> > > > > > > hoodie-spark
> > >> >> > > > > > > > > > module
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > > as a first-class.
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > > Thank,
> > >> >> > > > > > > > > > Nishith
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > > On Tue, Aug 6, 2019 at 9:48 AM taher koitawala <
> > >> >> > > > > taher...@gmail.com
> > >> >> > > > > > >
> > >> >> > > > > > > > > wrote:
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > > Hi Vinoth,
> > >> >> > > > > > > > > > Are there some tasks I can take up to ramp up the
> > >> code?
> > >> >> > Want
> > >> >> > > to
> > >> >> > > > > get
> > >> >> > > > > > > > > > more used to the code and understand the existing
> > >> >> > > > implementation
> > >> >> > > > > > > > better.
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > > Thanks,
> > >> >> > > > > > > > > > Taher Koitawala
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > > On Tue, Aug 6, 2019, 10:02 PM Vinoth Chandar <
> > >> >> > > > vin...@apache.org>
> > >> >> > > > > > > > wrote:
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > > Let's see if others have any thoughts as well. We
> > can
> > >> >> plan
> > >> >> > to
> > >> >> > > > fix
> > >> >> > > > > > the
> > >> >> > > > > > > > > > approach by EOW.
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > > On Mon, Aug 5, 2019 at 7:06 PM vino yang <
> > >> >> > > > yanghua1...@gmail.com>
> > >> >> > > > > > > > wrote:
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > > Hi guys,
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > > Also, +1 for Approach 1 like Taher.
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > > If we can do a comprehensive analysis of this
> model
> > >> and
> > >> >> > come
> > >> >> > > up
> > >> >> > > > > > with.
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > > means
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > > to refactor this cleanly, this would be
> promising.
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > > Yes, when we get the conclusion, we could start
> > this
> > >> >> work.
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > > Best,
> > >> >> > > > > > > > > > Vino
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > > >
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > > taher koitawala <taher...@gmail.com>
> 于2019年8月6日周二
> > >> >> > 上午12:28写道：
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > > +1 for Approch 1 Point integration with each
> > >> framework
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > > Approach 2 has a problem as you said "Developers
> > >> need to
> > >> >> > > think
> > >> >> > > > > > about
> > >> >> > > > > > > > > >
> what-if-this-piece-of-code-ran-as-spark-vs-flink..
> > >> So in
> > >> >> > the
> > >> >> > > > end,
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > > this
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > > may
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > > not be the panacea that it seems to be"
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > > We have seen various pipelines in the beam dag
> > being
> > >> >> > > expressed
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > > differently
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > > then we had them in our original usecase. And
> also
> > >> >> > switching
> > >> >> > > > > > between
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > > spark
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > > and Flink runners in beam have various impact on
> > the
> > >> >> > > pipelines
> > >> >> > > > > like
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > > some
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > > features available in Flink are not available on
> > the
> > >> >> spark
> > >> >> > > > runner
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > > etc.
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > > Refer to this compatible matrix ->
> > >> >> > > > > > > > > >
> > >> >> > > >
> > https://beam.apache.org/documentation/runners/capability-matrix/
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > > Hence my vote on Approch 1 let's decouple and
> build
> > >> the
> > >> >> > > > abstract
> > >> >> > > > > > for
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > > each
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > > framework. That is a much better option. We will
> > also
> > >> >> have
> > >> >> > > more
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > > control
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > > over each framework's implement.
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > > On Mon, Aug 5, 2019, 9:28 PM Vinoth Chandar <
> > >> >> > > vin...@apache.org
> > >> >> > > > >
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > > wrote:
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > > Would like to highlight that there are two
> distinct
> > >> >> > > approaches
> > >> >> > > > > here
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > > with
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > > different tradeoffs. Think of this as my
> braindump,
> > >> as I
> > >> >> > have
> > >> >> > > > > been
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > > thinking
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > > about this quite a bit in the past.
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > > >
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > > *Approach 1 : Point integration with each
> > framework *
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > > We may need a pure client module named for
> example
> > >> >> > > > > > > > > > hoodie-client-core(common)
> > >> >> > > > > > > > > > >> Then we could have: hoodie-client-spark,
> > >> >> > > hoodie-client-flink
> > >> >> > > > > > > > > > and hoodie-client-beam
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > > (+) This is the safest to do IMO, since we can
> > >> isolate
> > >> >> the
> > >> >> > > > > current
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > > Spark
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > > execution (hoodie-spark, hoodie-client-spark)
> from
> > >> the
> > >> >> > > changes
> > >> >> > > > > for
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > > flink,
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > > while it stabilizes over few releases.
> > >> >> > > > > > > > > > (-) Downside is that the utilities needs to be
> > >> redone :
> > >> >> > > > > > > > > > hoodie-utilities-spark and hoodie-utilities-flink
> > and
> > >> >> > > > > > > > > > hoodie-utilities-core ? hoodie-cli?
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > > If we can do a comprehensive analysis of this
> model
> > >> and
> > >> >> > come
> > >> >> > > up
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > > with.
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > > means
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > > to refactor this cleanly, this would be
> promising.
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > > >
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > > *Approach 2: Beam as the compute abstraction*
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > > Another more drastic approach is to remove Spark
> as
> > >> the
> > >> >> > > compute
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > > abstraction
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > > for writing data and replace it with Beam.
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > > (+) All of the code remains more or less similar
> > and
> > >> >> there
> > >> >> > is
> > >> >> > > > one
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > > compute
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > > API to reason about.
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > > (-) The (very big) assumption here is that we are
> > >> able
> > >> >> to
> > >> >> > > tune
> > >> >> > > > > the
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > > spark
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > > runtime the same way using Beam : custom
> > >> partitioners,
> > >> >> > > support
> > >> >> > > > > for
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > > all
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > > RDD
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > > operations we invoke, caching etc etc.
> > >> >> > > > > > > > > > (-) It will be a massive rewrite and testing of
> > such
> > >> a
> > >> >> > large
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > > rewrite
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > > would
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > > also be really challenging, since we need to pay
> > >> >> attention
> > >> >> > to
> > >> >> > > > all
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > > intricate
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > > details to ensure the spark users today
> experience
> > no
> > >> >> > > > > > > > > > regressions/side-effects
> > >> >> > > > > > > > > > (-) Note that we still need to probably support
> the
> > >> >> > > > hoodie-spark
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > > module
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > > and
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > > may be a first-class such integration with flink,
> > for
> > >> >> > native
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > > flink/spark
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > > pipeline authoring. Users of say DeltaStreamer
> need
> > >> to
> > >> >> pass
> > >> >> > > in
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > > Spark
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > > or
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > > Flink configs anyway.. Developers need to think
> > about
> > >> >> > > > > > > > > >
> what-if-this-piece-of-code-ran-as-spark-vs-flink..
> > >> So in
> > >> >> > the
> > >> >> > > > end,
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > > this
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > > may
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > > not be the panacea that it seems to be.
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > > >
> > >> >> > > > > > > > > > >
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > > One goal for the HIP is to get us all to agree
> as a
> > >> >> > community
> > >> >> > > > > which
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > > one
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > > to
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > > pick, with sufficient investigation, testing,
> > >> >> > benchmarking..
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > > On Sat, Aug 3, 2019 at 7:56 PM vino yang <
> > >> >> > > > yanghua1...@gmail.com>
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > > wrote:
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > > +1 for both Beam and Flink
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > > First step here is to probably draw out current
> > >> >> hierrarchy
> > >> >> > > and
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > > figure
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > > out
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > > what the abstraction points are..
> > >> >> > > > > > > > > > In my opinion, the runtime (spark, flink) should
> be
> > >> >> done at
> > >> >> > > the
> > >> >> > > > > > > > > > hoodie-client level and just used by
> > hoodie-utilties
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > > seamlessly..
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > > +1 for Vinoth's opinion, it should be the first
> > step.
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > > No matter we hope Hudi to integrate with which
> > >> computing
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > > framework.
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > > We need to decouple Hudi client and Spark.
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > > We may need a pure client module named for
> example
> > >> >> > > > > > > > > > hoodie-client-core(common)
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > > Then we could have: hoodie-client-spark,
> > >> >> > hoodie-client-flink
> > >> >> > > > and
> > >> >> > > > > > > > > > hoodie-client-beam
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > > Suneel Marthi <smar...@apache.org> 于2019年8月4日周日
> > >> >> 上午10:45写道：
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > > +1 for Beam -- agree with Semantic Beeng's
> > analysis.
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > > On Sat, Aug 3, 2019 at 10:30 PM taher koitawala <
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > > taher...@gmail.com>
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > > wrote:
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > > So the way to go around this is that file a hip.
> > >> Chalk
> > >> >> all
> > >> >> > th
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > > classes
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > > our
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > > and start moving towards Pure client.
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > > Secondly should we want to try beam?
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > > I think there is to much going on here and I'm
> not
> > >> able
> > >> >> to
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > > follow.
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > > If
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > > we
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > > want to try out beam all along I don't think it
> > makes
> > >> >> sense
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > > to
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > > do
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > > anything
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > > on Flink then.
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > > On Sun, Aug 4, 2019, 2:30 AM Semantic Beeng <
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > > n...@semanticbeeng.com>
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > > wrote:
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > > >> +1 My money is on this approach.
> > >> >> > > > > > > > > > >>
> > >> >> > > > > > > > > > >> The existing abstractions from Beam seem
> enough
> > >> for
> > >> >> the
> > >> >> > > use
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > > cases
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > > as I
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > > imagine them.
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > > >> Flink also has "dynamic table", "table source"
> > and
> > >> >> > "table
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > > sink"
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > > which
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > > seem very useful abstractions where Hudi might
> fit
> > >> >> nicely.
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > > >>
> > >> >> > > > > > > > > > >>
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > >
> > >> >> > > > > > > >
> > >> >> > > > > > >
> > >> >> > > > > >
> > >> >> > > > >
> > >> >> > > >
> > >> >> > >
> > >> >> >
> > >> >>
> > >>
> >
> https://ci.apache.org/projects/flink/flink-docs-stable/dev/table/streaming/dynamic_tables.html
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > > >>
> > >> >> > > > > > > > > > >> Attached a screen shot.
> > >> >> > > > > > > > > > >>
> > >> >> > > > > > > > > > >> This seems to fit with the original premise of
> > >> Hudi
> > >> >> as
> > >> >> > > well.
> > >> >> > > > > > > > > > >>
> > >> >> > > > > > > > > > >> Am exploring this venue with a use case that
> > >> involves
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > > "temporal
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > > joins
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > > on
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > > streams" which I need for feature extraction.
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > > >> Anyone is interested in this or has concrete
> > >> enough
> > >> >> > needs
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > > and
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > > use
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > > cases
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > > please let me know.
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > > >> Best to go from an agreed upon set of 2-3 use
> > >> cases.
> > >> >> > > > > > > > > > >>
> > >> >> > > > > > > > > > >> Cheers
> > >> >> > > > > > > > > > >>
> > >> >> > > > > > > > > > >> Nick
> > >> >> > > > > > > > > > >>
> > >> >> > > > > > > > > > >>
> > >> >> > > > > > > > > > >> > Also, we do have some Beam experts on the
> > >> mailing
> > >> >> > list..
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > > Can
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > > you
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > > please
> > >> >> > > > > > > > > > >> weigh on viability of using Beam as the
> > >> intermediate
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > > abstraction
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > > here
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > > between Spark/Flink?
> > >> >> > > > > > > > > > Hudi uses RDD apis like groupBy, mapToPair,
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > > sortAndRepartition,
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > > reduceByKey, countByKey and also does custom
> > >> >> partitioning a
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > > lot.>
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > > >> >
> > >> >> > > > > > > > > > >>
> > >> >> > > > > > > > > > >
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > >
> > >> >> > > > > > > >
> > >> >> > > > > > >
> > >> >> > > > > >
> > >> >> > > > >
> > >> >> > > >
> > >> >> > >
> > >> >> >
> > >> >>
> > >> >
> > >>
> > >
> >
> >
>
