[jira] [Commented] (HUDI-760) Remove Rolling Stat management from Hudi Writer
[ https://issues.apache.org/jira/browse/HUDI-760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17136791#comment-17136791 ] Balaji Varadarajan commented on HUDI-760: - [~baobaoyeye]: No worries. [~shivnarayan] is already started to working on this issue as this is getting targeted for 0.6 launch. I would encourage you to look at other open tickets to see if you can contribute on any one :) > Remove Rolling Stat management from Hudi Writer > --- > > Key: HUDI-760 > URL: https://issues.apache.org/jira/browse/HUDI-760 > Project: Apache Hudi > Issue Type: Improvement > Components: Writer Core >Reporter: Balaji Varadarajan >Assignee: sivabalan narayanan >Priority: Blocker > Labels: bug-bash-0.6.0, help-requested, help-wanted, newbie,, > pull-request-available > Fix For: 0.6.0 > > > Current implementation of rolling stat is not scalable. As Consolidated > Metadata will be implemented eventually, we can have one design to manage > file-level stats too. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HUDI-760) Remove Rolling Stat management from Hudi Writer
[ https://issues.apache.org/jira/browse/HUDI-760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17136502#comment-17136502 ] renyi.bao commented on HUDI-760: [~shivnarayan] so sorry. I just saw these replies. Recently, I was obsessed with the company's affairs. Now, does anyone follow this issue? If there's no one, I would finish it in two days > Remove Rolling Stat management from Hudi Writer > --- > > Key: HUDI-760 > URL: https://issues.apache.org/jira/browse/HUDI-760 > Project: Apache Hudi > Issue Type: Improvement > Components: Writer Core >Reporter: Balaji Varadarajan >Assignee: sivabalan narayanan >Priority: Blocker > Labels: bug-bash-0.6.0, help-requested, help-wanted, newbie, > Fix For: 0.6.0 > > > Current implementation of rolling stat is not scalable. As Consolidated > Metadata will be implemented eventually, we can have one design to manage > file-level stats too. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HUDI-760) Remove Rolling Stat management from Hudi Writer
[ https://issues.apache.org/jira/browse/HUDI-760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17136210#comment-17136210 ] sivabalan narayanan commented on HUDI-760: -- Synced up w/ Balaji offline. Rolling stat is never read in the read path and hence we could remove it safely. > Remove Rolling Stat management from Hudi Writer > --- > > Key: HUDI-760 > URL: https://issues.apache.org/jira/browse/HUDI-760 > Project: Apache Hudi > Issue Type: Improvement > Components: Writer Core >Reporter: Balaji Varadarajan >Assignee: sivabalan narayanan >Priority: Blocker > Labels: bug-bash-0.6.0, help-requested, help-wanted, newbie, > Fix For: 0.6.0 > > > Current implementation of rolling stat is not scalable. As Consolidated > Metadata will be implemented eventually, we can have one design to manage > file-level stats too. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HUDI-760) Remove Rolling Stat management from Hudi Writer
[ https://issues.apache.org/jira/browse/HUDI-760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17133881#comment-17133881 ] sivabalan narayanan commented on HUDI-760: -- [~vbalaji]: I have some doubts on this ticket. IIUC, we should remove the usage of HoodieRollingStat? But within AbstractHoodieWriteClient#updateMetadataAndRollingStats(), I don't see any config under which we disable or not write rolling stat. Can you clarify your statement above "We had earlier disabled it by not writing this data. But, the code has still references to Rolling stats.", or in other words, can you point me to code where the rolling stats are not written. Guess I am missing something. > Remove Rolling Stat management from Hudi Writer > --- > > Key: HUDI-760 > URL: https://issues.apache.org/jira/browse/HUDI-760 > Project: Apache Hudi > Issue Type: Improvement > Components: Writer Core >Reporter: Balaji Varadarajan >Assignee: sivabalan narayanan >Priority: Blocker > Labels: bug-bash-0.6.0, help-requested, help-wanted, newbie, > Fix For: 0.6.0 > > > Current implementation of rolling stat is not scalable. As Consolidated > Metadata will be implemented eventually, we can have one design to manage > file-level stats too. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HUDI-760) Remove Rolling Stat management from Hudi Writer
[ https://issues.apache.org/jira/browse/HUDI-760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17133873#comment-17133873 ] sivabalan narayanan commented on HUDI-760: -- [~baobaoyeye]: I am taking this up as this ticket is marked as a blocker for next release. > Remove Rolling Stat management from Hudi Writer > --- > > Key: HUDI-760 > URL: https://issues.apache.org/jira/browse/HUDI-760 > Project: Apache Hudi > Issue Type: Improvement > Components: Writer Core >Reporter: Balaji Varadarajan >Assignee: renyi.bao >Priority: Blocker > Labels: bug-bash-0.6.0, help-requested, help-wanted, newbie, > Fix For: 0.6.0 > > > Current implementation of rolling stat is not scalable. As Consolidated > Metadata will be implemented eventually, we can have one design to manage > file-level stats too. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HUDI-760) Remove Rolling Stat management from Hudi Writer
[ https://issues.apache.org/jira/browse/HUDI-760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17114947#comment-17114947 ] sivabalan narayanan commented on HUDI-760: -- [~baobaoyeye]: is there any progress made on this regards. Do link to any PR if available. > Remove Rolling Stat management from Hudi Writer > --- > > Key: HUDI-760 > URL: https://issues.apache.org/jira/browse/HUDI-760 > Project: Apache Hudi > Issue Type: Improvement > Components: Writer Core >Reporter: Balaji Varadarajan >Assignee: renyi.bao >Priority: Major > Labels: bug-bash-0.6.0, help-requested, help-wanted, newbie, > Fix For: 0.6.0 > > > Current implementation of rolling stat is not scalable. As Consolidated > Metadata will be implemented eventually, we can have one design to manage > file-level stats too. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HUDI-760) Remove Rolling Stat management from Hudi Writer
[ https://issues.apache.org/jira/browse/HUDI-760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17092650#comment-17092650 ] leesf commented on HUDI-760: [~baobaoyeye] assign the ticket to you, feel free to send a PR. > Remove Rolling Stat management from Hudi Writer > --- > > Key: HUDI-760 > URL: https://issues.apache.org/jira/browse/HUDI-760 > Project: Apache Hudi (incubating) > Issue Type: Improvement > Components: Writer Core >Reporter: Balaji Varadarajan >Priority: Major > Labels: help-wanted, newbie, > Fix For: 0.6.0 > > > Current implementation of rolling stat is not scalable. As Consolidated > Metadata will be implemented eventually, we can have one design to manage > file-level stats too. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HUDI-760) Remove Rolling Stat management from Hudi Writer
[ https://issues.apache.org/jira/browse/HUDI-760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17088169#comment-17088169 ] renyi.bao commented on HUDI-760: [~vbalaji] thanks for your guidance, if I understand it correctly, this issue's main purpose is to clean up the related code about rolling stat from the existing logic. I'm interested in trying to solve it > Remove Rolling Stat management from Hudi Writer > --- > > Key: HUDI-760 > URL: https://issues.apache.org/jira/browse/HUDI-760 > Project: Apache Hudi (incubating) > Issue Type: Improvement > Components: Writer Core >Reporter: Balaji Varadarajan >Priority: Major > Labels: help-wanted, newbie, > Fix For: 0.6.0 > > > Current implementation of rolling stat is not scalable. As Consolidated > Metadata will be implemented eventually, we can have one design to manage > file-level stats too. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HUDI-760) Remove Rolling Stat management from Hudi Writer
[ https://issues.apache.org/jira/browse/HUDI-760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17087835#comment-17087835 ] Balaji Varadarajan commented on HUDI-760: - Hi [~baobaoyeye] : Sorry for the delay. Here are extra information. Hudi had an earlier implementation of keeping all consolidated metadata in every "commit" files. We had earlier disabled it by not writing this data. But, the code has still references to Rolling stats. 1. org.apache.hudi.common.model.HoodieRollingStatMetadata 2. hudi-client/src/main/java/org/apache/hudi/client/AbstractHoodieWriteClient.java 3. hudi-client/src/test/java/org/apache/hudi/table/TestMergeOnReadTable.java 4. hudi-client/src/test/java/org/apache/hudi/client/TestHoodieClientOnCopyOnWriteStorage.java Hope this helps !! Let us know if you plan to take it up. Thanks for offering to contribute. Balaji.V > Remove Rolling Stat management from Hudi Writer > --- > > Key: HUDI-760 > URL: https://issues.apache.org/jira/browse/HUDI-760 > Project: Apache Hudi (incubating) > Issue Type: Improvement > Components: Writer Core >Reporter: Balaji Varadarajan >Priority: Major > Labels: help-wanted, newbie, > Fix For: 0.6.0 > > > Current implementation of rolling stat is not scalable. As Consolidated > Metadata will be implemented eventually, we can have one design to manage > file-level stats too. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HUDI-760) Remove Rolling Stat management from Hudi Writer
[ https://issues.apache.org/jira/browse/HUDI-760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17083742#comment-17083742 ] renyi.bao commented on HUDI-760: hi,bv would you describe the problem in detail? I want to get involved > Remove Rolling Stat management from Hudi Writer > --- > > Key: HUDI-760 > URL: https://issues.apache.org/jira/browse/HUDI-760 > Project: Apache Hudi (incubating) > Issue Type: Improvement > Components: Writer Core >Reporter: Balaji Varadarajan >Priority: Major > Labels: help-wanted, newbie, > Fix For: 0.6.0 > > > Current implementation of rolling stat is not scalable. As Consolidated > Metadata will be implemented eventually, we can have one design to manage > file-level stats too. -- This message was sent by Atlassian Jira (v8.3.4#803005)
