[jira] [Commented] (HUDI-417) Refactor HoodieWriteClient so that commit logic can be shareable by both bootstrap and normal write operations
[ https://issues.apache.org/jira/browse/HUDI-417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17010298#comment-17010298 ] Nicholas Jiang commented on HUDI-417: - [~vbalaji]OK. I will try Spark datasource. > Refactor HoodieWriteClient so that commit logic can be shareable by both > bootstrap and normal write operations > -- > > Key: HUDI-417 > URL: https://issues.apache.org/jira/browse/HUDI-417 > Project: Apache Hudi (incubating) > Issue Type: Sub-task > Components: Writer Core >Reporter: Balaji Varadarajan >Assignee: Balaji Varadarajan >Priority: Major > Labels: pull-request-available > Fix For: 0.5.1 > > Time Spent: 20m > Remaining Estimate: 0h > > > Basic Code Changes are present in the fork : > [https://github.com/bvaradar/hudi/tree/vb_bootstrap] > > The current implementation of HoodieBootstrapClient has duplicate code for > committing bootstrap. > [https://github.com/bvaradar/hudi/blob/vb_bootstrap/hudi-client/src/main/java/org/apache/hudi/bootstrap/HoodieBootstrapClient.java] > > > We can have an independent PR which would move these commit functionality > from HoodieWriteClient to a new base class AbstractHoodieWriteClient which > HoodieBootstrapClient can inherit. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HUDI-417) Refactor HoodieWriteClient so that commit logic can be shareable by both bootstrap and normal write operations
[ https://issues.apache.org/jira/browse/HUDI-417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17004624#comment-17004624 ] Balaji Varadarajan commented on HUDI-417: - [~nicholasjiang]: Sorry, I didnt get a chance to respond to your question. I am back from vacation. This is an initial step as it would make it easy to have further changes done with less conflicts. I will take it up as I have cycles tonight to get it done. As per earlier discussion, the spark datasource support task is not in the critical path for now but it would be great if you can start looking into it right away. > Refactor HoodieWriteClient so that commit logic can be shareable by both > bootstrap and normal write operations > -- > > Key: HUDI-417 > URL: https://issues.apache.org/jira/browse/HUDI-417 > Project: Apache Hudi (incubating) > Issue Type: Sub-task > Components: Writer Core >Reporter: Balaji Varadarajan >Priority: Major > Fix For: 0.5.1 > > > > Basic Code Changes are present in the fork : > [https://github.com/bvaradar/hudi/tree/vb_bootstrap] > > The current implementation of HoodieBootstrapClient has duplicate code for > committing bootstrap. > [https://github.com/bvaradar/hudi/blob/vb_bootstrap/hudi-client/src/main/java/org/apache/hudi/bootstrap/HoodieBootstrapClient.java] > > > We can have an independent PR which would move these commit functionality > from HoodieWriteClient to a new base class AbstractHoodieWriteClient which > HoodieBootstrapClient can inherit. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HUDI-417) Refactor HoodieWriteClient so that commit logic can be shareable by both bootstrap and normal write operations
[ https://issues.apache.org/jira/browse/HUDI-417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16999150#comment-16999150 ] Nicholas Jiang commented on HUDI-417: - [~vbalaji]What about the priority of this task？ > Refactor HoodieWriteClient so that commit logic can be shareable by both > bootstrap and normal write operations > -- > > Key: HUDI-417 > URL: https://issues.apache.org/jira/browse/HUDI-417 > Project: Apache Hudi (incubating) > Issue Type: Sub-task > Components: Write Client >Reporter: Balaji Varadarajan >Priority: Major > Fix For: 0.5.1 > > > > Basic Code Changes are present in the fork : > [https://github.com/bvaradar/hudi/tree/vb_bootstrap] > > The current implementation of HoodieBootstrapClient has duplicate code for > committing bootstrap. > [https://github.com/bvaradar/hudi/blob/vb_bootstrap/hudi-client/src/main/java/org/apache/hudi/bootstrap/HoodieBootstrapClient.java] > > > We can have an independent PR which would move these commit functionality > from HoodieWriteClient to a new base class AbstractHoodieWriteClient which > HoodieBootstrapClient can inherit. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HUDI-417) Refactor HoodieWriteClient so that commit logic can be shareable by both bootstrap and normal write operations
[ https://issues.apache.org/jira/browse/HUDI-417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16998868#comment-16998868 ] Balaji Varadarajan commented on HUDI-417: - [~nicholasjiang]: I may not get a chance to work on it for a week. It would be of great help if you can take a crack at this task (meaning put a PR) by 27th Dec. You can take a look at my fork and check both HoodieBootstrapClient and HoodieWriteClient. I have copied a bunch of code related to committing from HoodieWriteClient.java to [https://github.com/bvaradar/hudi/blob/vb_bootstrap/hudi-client/src/main/java/org/apache/hudi/bootstrap/HoodieBootstrapClient.java] If you can independently put up a PR (no need to depend on my changes) and create a new AbstractHoodieWriteClient which takes care of handling commit() and is inherited by HoodieWriteClient, it would be great help. As a subsequent step, you can try moving clean and compaction functionality to separate classes to reduce bloating in HoodieWriteClient. THanks, Balaji.V > Refactor HoodieWriteClient so that commit logic can be shareable by both > bootstrap and normal write operations > -- > > Key: HUDI-417 > URL: https://issues.apache.org/jira/browse/HUDI-417 > Project: Apache Hudi (incubating) > Issue Type: Sub-task > Components: Write Client >Reporter: Balaji Varadarajan >Priority: Major > Fix For: 0.5.1 > > > > Basic Code Changes are present in the fork : > [https://github.com/bvaradar/hudi/tree/vb_bootstrap] > > The current implementation of HoodieBootstrapClient has duplicate code for > committing bootstrap. > [https://github.com/bvaradar/hudi/blob/vb_bootstrap/hudi-client/src/main/java/org/apache/hudi/bootstrap/HoodieBootstrapClient.java] > > > We can have an independent PR which would move these commit functionality > from HoodieWriteClient to a new base class AbstractHoodieWriteClient which > HoodieBootstrapClient can inherit. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HUDI-417) Refactor HoodieWriteClient so that commit logic can be shareable by both bootstrap and normal write operations
[ https://issues.apache.org/jira/browse/HUDI-417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16998847#comment-16998847 ] Nicholas Jiang commented on HUDI-417: - [~vbalaji]Do you have free time to move above commit functionality from HoodieWriteClient? If you have no time, I would like to claim this issue to myself. > Refactor HoodieWriteClient so that commit logic can be shareable by both > bootstrap and normal write operations > -- > > Key: HUDI-417 > URL: https://issues.apache.org/jira/browse/HUDI-417 > Project: Apache Hudi (incubating) > Issue Type: Sub-task > Components: Write Client >Reporter: Balaji Varadarajan >Priority: Major > Fix For: 0.5.1 > > > > Basic Code Changes are present in the fork : > [https://github.com/bvaradar/hudi/tree/vb_bootstrap] > > The current implementation of HoodieBootstrapClient has duplicate code for > committing bootstrap. > [https://github.com/bvaradar/hudi/blob/vb_bootstrap/hudi-client/src/main/java/org/apache/hudi/bootstrap/HoodieBootstrapClient.java] > > > We can have an independent PR which would move these commit functionality > from HoodieWriteClient to a new base class AbstractHoodieWriteClient which > HoodieBootstrapClient can inherit. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
