Re: [DISCUSS] Introduce a write committed callback hook

+1. That would be great to have a communication mechanism between downstream CDC applications chain. e.g. A->B->C->D. Right now I am using the commit timestamp to identify whether there is a new commit came in. But if I need to recompute app B, it’s difficult for C and D to aware they have to

Re: [DISCUSS] Publishing benchmarks for releases

Lucene has nightly runs even https://home.apache.org/~mikemccand/lucenebench/ We can do something like this? In any case, raising a Jira under performance component seems like a good idea? On Sun, Jun 21, 2020 at 6:41 PM vino yang wrote: > +1 as well, > > it would be helpful to measure the

Re: [DISCUSS] Publishing benchmarks for releases

+1 as well, it would be helpful to measure the performance between different versions. Shiyan Xu 于2020年6月22日周一 上午8:37写道： > +1 definitely useful info. > > On Sun, Jun 21, 2020 at 4:56 PM Sivabalan wrote: > > > Hey folks, > > Is it a common practise to publish benchmarks for releases? I

Re: [DISCUSS] Introduce a write committed callback hook

+1. a great feature. Sivabalan 于2020年6月22日周一 上午7:50写道： > +1. would be a nice addition. > > On Sun, Jun 21, 2020 at 12:02 PM vbal...@apache.org > wrote: > > > > > +1. This would be a really good feature to have when building dependent > > ETL pipelines. > > > > On Friday, June 19, 2020,

Re: [DISCUSS] Publishing benchmarks for releases

+1 definitely useful info. On Sun, Jun 21, 2020 at 4:56 PM Sivabalan wrote: > Hey folks, > Is it a common practise to publish benchmarks for releases? I have put > up an initial PR to add jmh > benchmark support to a couple of Hudi operations. If

Re: Re:Re: [DISCUSS] Regarding nightly builds

+1 very helpful to accelerate the adoption. On Sun, Jun 21, 2020 at 4:51 PM Sivabalan wrote: > +1 > > On Sun, Jun 21, 2020 at 11:58 AM vbal...@apache.org > wrote: > > > +1. It is a good idea to run hudi-test-suite on a daily basis with > > expanded tests. > > Balaji.VOn Sunday, June 21,

[DISCUSS] Publishing benchmarks for releases

Hey folks, Is it a common practise to publish benchmarks for releases? I have put up an initial PR to add jmh benchmark support to a couple of Hudi operations. If the community feels positive on publishing benchmarks, we can add support for more

Re: Re:Re: [DISCUSS] Regarding nightly builds

+1 On Sun, Jun 21, 2020 at 11:58 AM vbal...@apache.org wrote: > +1. It is a good idea to run hudi-test-suite on a daily basis with > expanded tests. > Balaji.VOn Sunday, June 21, 2020, 08:16:39 AM PDT, Trevor-zhang < > 957029...@qq.com> wrote: > > +1 as well. > > -- 原始邮件

Re: [DISCUSS] Introduce a write committed callback hook

+1. would be a nice addition. On Sun, Jun 21, 2020 at 12:02 PM vbal...@apache.org wrote: > > +1. This would be a really good feature to have when building dependent > ETL pipelines. > > On Friday, June 19, 2020, 05:13:45 PM PDT, vino yang < > vinoy...@apache.org> wrote: > > Hi all, > >

Re: [DISCUSS] Introduce a write committed callback hook

+1. This would be a really good feature to have when building dependent ETL pipelines. On Friday, June 19, 2020, 05:13:45 PM PDT, vino yang wrote: Hi all, Currently, we have a need to incrementally process and build a new table based on an original hoodie table. We expect that

Re: Re:Re: [DISCUSS] Regarding nightly builds

+1. It is a good idea to run hudi-test-suite on a daily basis with expanded tests. Balaji.VOn Sunday, June 21, 2020, 08:16:39 AM PDT, Trevor-zhang <957029...@qq.com> wrote: +1 as well. -- 原始邮件 -- 发件人:"vino yang" https://github.com/apachehudi-ci [2]:

[ANNOUNCE] Hudi community Weekly Update(2020-06-14 ~ 2020-06-21)

Nice to share Hudi community weekly update for 2020-06-14 ~ 2020-06-21 with updates on discussion, bug fix and tests. === Discussion [Release] Hudi 0.5.3 has released, it is the first release after graduating. [1] [CI] A discussion regarding nightly builds to

Re:Re: [DISCUSS] Regarding nightly builds

+1 as well. -- 原始邮件 -- 发件人:"vino yang" https://github.com/apachehudi-ci [2]: https://cwiki.apache.org/confluence/display/FLINK/2020/03/22/Migrating+Flink%27s+CI+Infrastructure+from+Travis+CI+to+Azure+Pipelines Vinoth Chandar

Re: [DISCUSS] Introduce a write committed callback hook

+1. It is a great complement to the pull model; helpful to fan-out scenarios On Sun, Jun 21, 2020 at 8:07 AM Bhavani Sudha wrote: > +1 . I think this is a valid use case and would be useful in general. > > On Sun, Jun 21, 2020 at 7:11 AM Vinoth Chandar wrote: > > > +1 as well > > > > > We

Re: IllegalStateException: Hudi File Id (...) has more than 1 pending compactions. Hudi 0.5.3 + S3

If you are running inline compaction it should not cause two pending compactions on the same file group. Along with above details, can you please open a [SUPPORT] git issue with full stack trace and also a `ls` of you .hoodie folder if possible? Thanks, Sudha On Thu, Jun 18, 2020 at 9:57 PM

Re: [DISCUSS] Introduce a write committed callback hook

+1 . I think this is a valid use case and would be useful in general. On Sun, Jun 21, 2020 at 7:11 AM Vinoth Chandar wrote: > +1 as well > > > We expect to introduce a proactive notification(event callback) > mechanism. For example, a hook can be introduced after a successful commit. > > This

Re: [DISCUSS] Regarding nightly builds

+1 as well, Currently, I am waiting for hudi-test-suite to be merged into the master branch, so that when we have a new PR merged into the master branch, this will cause the "hudi-test-suite" that is also on the master branch to be triggered on Azure Pipeline " easier. Sharing more information

Re: [DISCUSS] Regarding nightly builds

Hi Sudha, Thanks for getting this kicked off.. +1 on a new nightly build process.. This will help us more easily make the bleeding edge testable.. My initial thoughts here are - Figure out a way to get Azure Pipelines enabled for Hudi - Setup the nightly there (this will also help us

Re: [DISCUSS] Introduce a write committed callback hook

+1 as well > We expect to introduce a proactive notification(event callback) mechanism. For example, a hook can be introduced after a successful commit. This would be very useful. We could write to a variety of event bus-es and notify new data arrival. On Sat, Jun 20, 2020 at 2:51 AM