Re: Need clarity on these test cases in TestHoodieDeltaStreamer
@Balaji As part of bug fix, the addition of 200 records has been removed from our code base. I guess there is no need of documenting this. If you still feel there is a need, please let me know. Will do the needful. On Wed, Mar 4, 2020 at 1:07 PM Pratyaksh Sharma wrote: > Sure. I will share my updated changes with Siva today so that he can take > a look. Then we will discuss further. > > On Tue, Mar 3, 2020 at 8:19 PM Vinoth Chandar wrote: > >> Please keep discussions on the mailing list, as much as possible :) >> >> On Mon, Mar 2, 2020 at 11:04 PM Sivabalan wrote: >> >>> I will sync up with Pratyaksh offline on this. >>> >>> On Thu, Feb 27, 2020 at 11:24 PM Pratyaksh Sharma >> > >>> wrote: >>> >>> > Hi Balaji, >>> > >>> > Right now I am facing some different issue in the same test case. The >>> > number of records are not matching and assertion is failing. Once I am >>> able >>> > to fix that as well, I will open the PR for sure. :) >>> > >>> > On Thu, Feb 27, 2020 at 11:17 PM Balaji Varadarajan >>> > wrote: >>> > >>> >> >>> >> Awesome Pratyaksh, would you mind opening a PR to documenting it. >>> >> Balaji.V >>> >> >>> >> Sent from Yahoo Mail for iPhone >>> >> >>> >> >>> >> On Wednesday, February 26, 2020, 11:14 PM, Pratyaksh Sharma < >>> >> pratyaks...@gmail.com> wrote: >>> >> >>> >> Hi, >>> >> >>> >> I figured out the issue yesterday. Thank you for helping me out. >>> >> >>> >> On Thu, Feb 27, 2020 at 12:01 AM vbal...@apache.org < >>> vbal...@apache.org> >>> >> wrote: >>> >> >>> >> > >>> >> > This change was done as part of adding delete API support : >>> >> > >>> >> >>> https://github.com/apache/incubator-hudi/commit/7031445eb3cae5a4557786c7eb080944320609aa >>> >> > >>> >> > I don't remember the reason behind this. >>> >> > Sivabalan, Can you explain the reason when you get a chance. >>> >> > Thanks,Balaji.V >>> >> >On Wednesday, February 26, 2020, 06:03:53 AM PST, Pratyaksh >>> Sharma < >>> >> > pratyaks...@gmail.com> wrote: >>> >> > >>> >> > Anybody got a chance to look at this? >>> >> > >>> >> > On Mon, Feb 24, 2020 at 1:04 AM Pratyaksh Sharma < >>> pratyaks...@gmail.com >>> >> > >>> >> > wrote: >>> >> > >>> >> > > Hi, >>> >> > > >>> >> > > While working on one of my PRs, I am stuck with the following test >>> >> cases >>> >> > > in TestHoodieDeltaStreamer - >>> >> > > 1. testUpsertsCOWContinuousMode >>> >> > > 2. testUpsertsMORContinuousMode >>> >> > > >>> >> > > For both of them, at line [1] and [2], we are adding 200 to >>> >> totalRecords >>> >> > > while asserting record count and distance count respectively. I am >>> >> unable >>> >> > > to understand what do these 200 records correspond to. Any leads >>> are >>> >> > > appreciated. >>> >> > > >>> >> > > I feel probably I am missing some piece of code where I need to do >>> >> > changes >>> >> > > for the above tests to pass. >>> >> > > >>> >> > > [1] >>> >> > > >>> >> > >>> >> >>> https://github.com/apache/incubator-hudi/blob/078d4825d909b2c469398f31c97d2290687321a8/hudi-utilities/src/test/java/org/apache/hudi/utilities/TestHoodieDeltaStreamer.java#L425 >>> >> > > . >>> >> > > [2] >>> >> > > >>> >> > >>> >> >>> https://github.com/apache/incubator-hudi/blob/078d4825d909b2c469398f31c97d2290687321a8/hudi-utilities/src/test/java/org/apache/hudi/utilities/TestHoodieDeltaStreamer.java#L426 >>> >> > > . >>> >> > > >>> >> > > >>> >> > >>> >> >>> >> >>> >> >>> >> >>> >>> -- >>> Regards, >>> -Sivabalan >>> >>
Re: Need clarity on these test cases in TestHoodieDeltaStreamer
Sure. I will share my updated changes with Siva today so that he can take a look. Then we will discuss further. On Tue, Mar 3, 2020 at 8:19 PM Vinoth Chandar wrote: > Please keep discussions on the mailing list, as much as possible :) > > On Mon, Mar 2, 2020 at 11:04 PM Sivabalan wrote: > >> I will sync up with Pratyaksh offline on this. >> >> On Thu, Feb 27, 2020 at 11:24 PM Pratyaksh Sharma >> wrote: >> >> > Hi Balaji, >> > >> > Right now I am facing some different issue in the same test case. The >> > number of records are not matching and assertion is failing. Once I am >> able >> > to fix that as well, I will open the PR for sure. :) >> > >> > On Thu, Feb 27, 2020 at 11:17 PM Balaji Varadarajan >> > wrote: >> > >> >> >> >> Awesome Pratyaksh, would you mind opening a PR to documenting it. >> >> Balaji.V >> >> >> >> Sent from Yahoo Mail for iPhone >> >> >> >> >> >> On Wednesday, February 26, 2020, 11:14 PM, Pratyaksh Sharma < >> >> pratyaks...@gmail.com> wrote: >> >> >> >> Hi, >> >> >> >> I figured out the issue yesterday. Thank you for helping me out. >> >> >> >> On Thu, Feb 27, 2020 at 12:01 AM vbal...@apache.org < >> vbal...@apache.org> >> >> wrote: >> >> >> >> > >> >> > This change was done as part of adding delete API support : >> >> > >> >> >> https://github.com/apache/incubator-hudi/commit/7031445eb3cae5a4557786c7eb080944320609aa >> >> > >> >> > I don't remember the reason behind this. >> >> > Sivabalan, Can you explain the reason when you get a chance. >> >> > Thanks,Balaji.V >> >> >On Wednesday, February 26, 2020, 06:03:53 AM PST, Pratyaksh >> Sharma < >> >> > pratyaks...@gmail.com> wrote: >> >> > >> >> > Anybody got a chance to look at this? >> >> > >> >> > On Mon, Feb 24, 2020 at 1:04 AM Pratyaksh Sharma < >> pratyaks...@gmail.com >> >> > >> >> > wrote: >> >> > >> >> > > Hi, >> >> > > >> >> > > While working on one of my PRs, I am stuck with the following test >> >> cases >> >> > > in TestHoodieDeltaStreamer - >> >> > > 1. testUpsertsCOWContinuousMode >> >> > > 2. testUpsertsMORContinuousMode >> >> > > >> >> > > For both of them, at line [1] and [2], we are adding 200 to >> >> totalRecords >> >> > > while asserting record count and distance count respectively. I am >> >> unable >> >> > > to understand what do these 200 records correspond to. Any leads >> are >> >> > > appreciated. >> >> > > >> >> > > I feel probably I am missing some piece of code where I need to do >> >> > changes >> >> > > for the above tests to pass. >> >> > > >> >> > > [1] >> >> > > >> >> > >> >> >> https://github.com/apache/incubator-hudi/blob/078d4825d909b2c469398f31c97d2290687321a8/hudi-utilities/src/test/java/org/apache/hudi/utilities/TestHoodieDeltaStreamer.java#L425 >> >> > > . >> >> > > [2] >> >> > > >> >> > >> >> >> https://github.com/apache/incubator-hudi/blob/078d4825d909b2c469398f31c97d2290687321a8/hudi-utilities/src/test/java/org/apache/hudi/utilities/TestHoodieDeltaStreamer.java#L426 >> >> > > . >> >> > > >> >> > > >> >> > >> >> >> >> >> >> >> >> >> >> -- >> Regards, >> -Sivabalan >> >
Re: Need clarity on these test cases in TestHoodieDeltaStreamer
Please keep discussions on the mailing list, as much as possible :) On Mon, Mar 2, 2020 at 11:04 PM Sivabalan wrote: > I will sync up with Pratyaksh offline on this. > > On Thu, Feb 27, 2020 at 11:24 PM Pratyaksh Sharma > wrote: > > > Hi Balaji, > > > > Right now I am facing some different issue in the same test case. The > > number of records are not matching and assertion is failing. Once I am > able > > to fix that as well, I will open the PR for sure. :) > > > > On Thu, Feb 27, 2020 at 11:17 PM Balaji Varadarajan > > wrote: > > > >> > >> Awesome Pratyaksh, would you mind opening a PR to documenting it. > >> Balaji.V > >> > >> Sent from Yahoo Mail for iPhone > >> > >> > >> On Wednesday, February 26, 2020, 11:14 PM, Pratyaksh Sharma < > >> pratyaks...@gmail.com> wrote: > >> > >> Hi, > >> > >> I figured out the issue yesterday. Thank you for helping me out. > >> > >> On Thu, Feb 27, 2020 at 12:01 AM vbal...@apache.org > > >> wrote: > >> > >> > > >> > This change was done as part of adding delete API support : > >> > > >> > https://github.com/apache/incubator-hudi/commit/7031445eb3cae5a4557786c7eb080944320609aa > >> > > >> > I don't remember the reason behind this. > >> > Sivabalan, Can you explain the reason when you get a chance. > >> > Thanks,Balaji.V > >> >On Wednesday, February 26, 2020, 06:03:53 AM PST, Pratyaksh Sharma > < > >> > pratyaks...@gmail.com> wrote: > >> > > >> > Anybody got a chance to look at this? > >> > > >> > On Mon, Feb 24, 2020 at 1:04 AM Pratyaksh Sharma < > pratyaks...@gmail.com > >> > > >> > wrote: > >> > > >> > > Hi, > >> > > > >> > > While working on one of my PRs, I am stuck with the following test > >> cases > >> > > in TestHoodieDeltaStreamer - > >> > > 1. testUpsertsCOWContinuousMode > >> > > 2. testUpsertsMORContinuousMode > >> > > > >> > > For both of them, at line [1] and [2], we are adding 200 to > >> totalRecords > >> > > while asserting record count and distance count respectively. I am > >> unable > >> > > to understand what do these 200 records correspond to. Any leads are > >> > > appreciated. > >> > > > >> > > I feel probably I am missing some piece of code where I need to do > >> > changes > >> > > for the above tests to pass. > >> > > > >> > > [1] > >> > > > >> > > >> > https://github.com/apache/incubator-hudi/blob/078d4825d909b2c469398f31c97d2290687321a8/hudi-utilities/src/test/java/org/apache/hudi/utilities/TestHoodieDeltaStreamer.java#L425 > >> > > . > >> > > [2] > >> > > > >> > > >> > https://github.com/apache/incubator-hudi/blob/078d4825d909b2c469398f31c97d2290687321a8/hudi-utilities/src/test/java/org/apache/hudi/utilities/TestHoodieDeltaStreamer.java#L426 > >> > > . > >> > > > >> > > > >> > > >> > >> > >> > >> > > -- > Regards, > -Sivabalan >
Re: Need clarity on these test cases in TestHoodieDeltaStreamer
I will sync up with Pratyaksh offline on this. On Thu, Feb 27, 2020 at 11:24 PM Pratyaksh Sharma wrote: > Hi Balaji, > > Right now I am facing some different issue in the same test case. The > number of records are not matching and assertion is failing. Once I am able > to fix that as well, I will open the PR for sure. :) > > On Thu, Feb 27, 2020 at 11:17 PM Balaji Varadarajan > wrote: > >> >> Awesome Pratyaksh, would you mind opening a PR to documenting it. >> Balaji.V >> >> Sent from Yahoo Mail for iPhone >> >> >> On Wednesday, February 26, 2020, 11:14 PM, Pratyaksh Sharma < >> pratyaks...@gmail.com> wrote: >> >> Hi, >> >> I figured out the issue yesterday. Thank you for helping me out. >> >> On Thu, Feb 27, 2020 at 12:01 AM vbal...@apache.org >> wrote: >> >> > >> > This change was done as part of adding delete API support : >> > >> https://github.com/apache/incubator-hudi/commit/7031445eb3cae5a4557786c7eb080944320609aa >> > >> > I don't remember the reason behind this. >> > Sivabalan, Can you explain the reason when you get a chance. >> > Thanks,Balaji.V >> >On Wednesday, February 26, 2020, 06:03:53 AM PST, Pratyaksh Sharma < >> > pratyaks...@gmail.com> wrote: >> > >> > Anybody got a chance to look at this? >> > >> > On Mon, Feb 24, 2020 at 1:04 AM Pratyaksh Sharma > > >> > wrote: >> > >> > > Hi, >> > > >> > > While working on one of my PRs, I am stuck with the following test >> cases >> > > in TestHoodieDeltaStreamer - >> > > 1. testUpsertsCOWContinuousMode >> > > 2. testUpsertsMORContinuousMode >> > > >> > > For both of them, at line [1] and [2], we are adding 200 to >> totalRecords >> > > while asserting record count and distance count respectively. I am >> unable >> > > to understand what do these 200 records correspond to. Any leads are >> > > appreciated. >> > > >> > > I feel probably I am missing some piece of code where I need to do >> > changes >> > > for the above tests to pass. >> > > >> > > [1] >> > > >> > >> https://github.com/apache/incubator-hudi/blob/078d4825d909b2c469398f31c97d2290687321a8/hudi-utilities/src/test/java/org/apache/hudi/utilities/TestHoodieDeltaStreamer.java#L425 >> > > . >> > > [2] >> > > >> > >> https://github.com/apache/incubator-hudi/blob/078d4825d909b2c469398f31c97d2290687321a8/hudi-utilities/src/test/java/org/apache/hudi/utilities/TestHoodieDeltaStreamer.java#L426 >> > > . >> > > >> > > >> > >> >> >> >> -- Regards, -Sivabalan
Re: Need clarity on these test cases in TestHoodieDeltaStreamer
Hi Balaji, Right now I am facing some different issue in the same test case. The number of records are not matching and assertion is failing. Once I am able to fix that as well, I will open the PR for sure. :) On Thu, Feb 27, 2020 at 11:17 PM Balaji Varadarajan wrote: > > Awesome Pratyaksh, would you mind opening a PR to documenting it. > Balaji.V > > Sent from Yahoo Mail for iPhone > > > On Wednesday, February 26, 2020, 11:14 PM, Pratyaksh Sharma < > pratyaks...@gmail.com> wrote: > > Hi, > > I figured out the issue yesterday. Thank you for helping me out. > > On Thu, Feb 27, 2020 at 12:01 AM vbal...@apache.org > wrote: > > > > > This change was done as part of adding delete API support : > > > https://github.com/apache/incubator-hudi/commit/7031445eb3cae5a4557786c7eb080944320609aa > > > > I don't remember the reason behind this. > > Sivabalan, Can you explain the reason when you get a chance. > > Thanks,Balaji.V > >On Wednesday, February 26, 2020, 06:03:53 AM PST, Pratyaksh Sharma < > > pratyaks...@gmail.com> wrote: > > > > Anybody got a chance to look at this? > > > > On Mon, Feb 24, 2020 at 1:04 AM Pratyaksh Sharma > > wrote: > > > > > Hi, > > > > > > While working on one of my PRs, I am stuck with the following test > cases > > > in TestHoodieDeltaStreamer - > > > 1. testUpsertsCOWContinuousMode > > > 2. testUpsertsMORContinuousMode > > > > > > For both of them, at line [1] and [2], we are adding 200 to > totalRecords > > > while asserting record count and distance count respectively. I am > unable > > > to understand what do these 200 records correspond to. Any leads are > > > appreciated. > > > > > > I feel probably I am missing some piece of code where I need to do > > changes > > > for the above tests to pass. > > > > > > [1] > > > > > > https://github.com/apache/incubator-hudi/blob/078d4825d909b2c469398f31c97d2290687321a8/hudi-utilities/src/test/java/org/apache/hudi/utilities/TestHoodieDeltaStreamer.java#L425 > > > . > > > [2] > > > > > > https://github.com/apache/incubator-hudi/blob/078d4825d909b2c469398f31c97d2290687321a8/hudi-utilities/src/test/java/org/apache/hudi/utilities/TestHoodieDeltaStreamer.java#L426 > > > . > > > > > > > > > > > >
Re: Need clarity on these test cases in TestHoodieDeltaStreamer
Awesome Pratyaksh, would you mind opening a PR to documenting it. Balaji.V Sent from Yahoo Mail for iPhone On Wednesday, February 26, 2020, 11:14 PM, Pratyaksh Sharma wrote: Hi, I figured out the issue yesterday. Thank you for helping me out. On Thu, Feb 27, 2020 at 12:01 AM vbal...@apache.org wrote: > > This change was done as part of adding delete API support : > https://github.com/apache/incubator-hudi/commit/7031445eb3cae5a4557786c7eb080944320609aa > > I don't remember the reason behind this. > Sivabalan, Can you explain the reason when you get a chance. > Thanks,Balaji.V > On Wednesday, February 26, 2020, 06:03:53 AM PST, Pratyaksh Sharma < > pratyaks...@gmail.com> wrote: > > Anybody got a chance to look at this? > > On Mon, Feb 24, 2020 at 1:04 AM Pratyaksh Sharma > wrote: > > > Hi, > > > > While working on one of my PRs, I am stuck with the following test cases > > in TestHoodieDeltaStreamer - > > 1. testUpsertsCOWContinuousMode > > 2. testUpsertsMORContinuousMode > > > > For both of them, at line [1] and [2], we are adding 200 to totalRecords > > while asserting record count and distance count respectively. I am unable > > to understand what do these 200 records correspond to. Any leads are > > appreciated. > > > > I feel probably I am missing some piece of code where I need to do > changes > > for the above tests to pass. > > > > [1] > > > https://github.com/apache/incubator-hudi/blob/078d4825d909b2c469398f31c97d2290687321a8/hudi-utilities/src/test/java/org/apache/hudi/utilities/TestHoodieDeltaStreamer.java#L425 > > . > > [2] > > > https://github.com/apache/incubator-hudi/blob/078d4825d909b2c469398f31c97d2290687321a8/hudi-utilities/src/test/java/org/apache/hudi/utilities/TestHoodieDeltaStreamer.java#L426 > > . > > > > >
Re: Need clarity on these test cases in TestHoodieDeltaStreamer
Hi, I figured out the issue yesterday. Thank you for helping me out. On Thu, Feb 27, 2020 at 12:01 AM vbal...@apache.org wrote: > > This change was done as part of adding delete API support : > https://github.com/apache/incubator-hudi/commit/7031445eb3cae5a4557786c7eb080944320609aa > > I don't remember the reason behind this. > Sivabalan, Can you explain the reason when you get a chance. > Thanks,Balaji.V > On Wednesday, February 26, 2020, 06:03:53 AM PST, Pratyaksh Sharma < > pratyaks...@gmail.com> wrote: > > Anybody got a chance to look at this? > > On Mon, Feb 24, 2020 at 1:04 AM Pratyaksh Sharma > wrote: > > > Hi, > > > > While working on one of my PRs, I am stuck with the following test cases > > in TestHoodieDeltaStreamer - > > 1. testUpsertsCOWContinuousMode > > 2. testUpsertsMORContinuousMode > > > > For both of them, at line [1] and [2], we are adding 200 to totalRecords > > while asserting record count and distance count respectively. I am unable > > to understand what do these 200 records correspond to. Any leads are > > appreciated. > > > > I feel probably I am missing some piece of code where I need to do > changes > > for the above tests to pass. > > > > [1] > > > https://github.com/apache/incubator-hudi/blob/078d4825d909b2c469398f31c97d2290687321a8/hudi-utilities/src/test/java/org/apache/hudi/utilities/TestHoodieDeltaStreamer.java#L425 > > . > > [2] > > > https://github.com/apache/incubator-hudi/blob/078d4825d909b2c469398f31c97d2290687321a8/hudi-utilities/src/test/java/org/apache/hudi/utilities/TestHoodieDeltaStreamer.java#L426 > > . > > > > >
Re: Need clarity on these test cases in TestHoodieDeltaStreamer
This change was done as part of adding delete API support : https://github.com/apache/incubator-hudi/commit/7031445eb3cae5a4557786c7eb080944320609aa I don't remember the reason behind this. Sivabalan, Can you explain the reason when you get a chance. Thanks,Balaji.V On Wednesday, February 26, 2020, 06:03:53 AM PST, Pratyaksh Sharma wrote: Anybody got a chance to look at this? On Mon, Feb 24, 2020 at 1:04 AM Pratyaksh Sharma wrote: > Hi, > > While working on one of my PRs, I am stuck with the following test cases > in TestHoodieDeltaStreamer - > 1. testUpsertsCOWContinuousMode > 2. testUpsertsMORContinuousMode > > For both of them, at line [1] and [2], we are adding 200 to totalRecords > while asserting record count and distance count respectively. I am unable > to understand what do these 200 records correspond to. Any leads are > appreciated. > > I feel probably I am missing some piece of code where I need to do changes > for the above tests to pass. > > [1] > https://github.com/apache/incubator-hudi/blob/078d4825d909b2c469398f31c97d2290687321a8/hudi-utilities/src/test/java/org/apache/hudi/utilities/TestHoodieDeltaStreamer.java#L425 > . > [2] > https://github.com/apache/incubator-hudi/blob/078d4825d909b2c469398f31c97d2290687321a8/hudi-utilities/src/test/java/org/apache/hudi/utilities/TestHoodieDeltaStreamer.java#L426 > . > >
Re: Need clarity on these test cases in TestHoodieDeltaStreamer
Anybody got a chance to look at this? On Mon, Feb 24, 2020 at 1:04 AM Pratyaksh Sharma wrote: > Hi, > > While working on one of my PRs, I am stuck with the following test cases > in TestHoodieDeltaStreamer - > 1. testUpsertsCOWContinuousMode > 2. testUpsertsMORContinuousMode > > For both of them, at line [1] and [2], we are adding 200 to totalRecords > while asserting record count and distance count respectively. I am unable > to understand what do these 200 records correspond to. Any leads are > appreciated. > > I feel probably I am missing some piece of code where I need to do changes > for the above tests to pass. > > [1] > https://github.com/apache/incubator-hudi/blob/078d4825d909b2c469398f31c97d2290687321a8/hudi-utilities/src/test/java/org/apache/hudi/utilities/TestHoodieDeltaStreamer.java#L425 > . > [2] > https://github.com/apache/incubator-hudi/blob/078d4825d909b2c469398f31c97d2290687321a8/hudi-utilities/src/test/java/org/apache/hudi/utilities/TestHoodieDeltaStreamer.java#L426 > . > >
Need clarity on these test cases in TestHoodieDeltaStreamer
Hi, While working on one of my PRs, I am stuck with the following test cases in TestHoodieDeltaStreamer - 1. testUpsertsCOWContinuousMode 2. testUpsertsMORContinuousMode For both of them, at line [1] and [2], we are adding 200 to totalRecords while asserting record count and distance count respectively. I am unable to understand what do these 200 records correspond to. Any leads are appreciated. I feel probably I am missing some piece of code where I need to do changes for the above tests to pass. [1] https://github.com/apache/incubator-hudi/blob/078d4825d909b2c469398f31c97d2290687321a8/hudi-utilities/src/test/java/org/apache/hudi/utilities/TestHoodieDeltaStreamer.java#L425 . [2] https://github.com/apache/incubator-hudi/blob/078d4825d909b2c469398f31c97d2290687321a8/hudi-utilities/src/test/java/org/apache/hudi/utilities/TestHoodieDeltaStreamer.java#L426 .
