[jira] [Commented] (HUDI-5426) Create Flink Catalog support for Hudi
[ https://issues.apache.org/jira/browse/HUDI-5426?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17649859#comment-17649859 ] Samrat Deb commented on HUDI-5426: -- cc : [~danny0405] > Create Flink Catalog support for Hudi > -- > > Key: HUDI-5426 > URL: https://issues.apache.org/jira/browse/HUDI-5426 > Project: Apache Hudi > Issue Type: Wish > Components: flink, flink-sql >Reporter: Samrat Deb >Priority: Minor > > Currently there is no support for Hudi flink glue catalog for sql-client > Iceberg provides support for catalog in sql-client > [https://iceberg.apache.org/docs/latest/flink/#creating-catalogs-and-using-catalogs] > . > This issue created to add similar support with flink hudi -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HUDI-5426) Create Flink Glue Catalog support for Hudi
[ https://issues.apache.org/jira/browse/HUDI-5426?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Samrat Deb updated HUDI-5426: - Summary: Create Flink Glue Catalog support for Hudi (was: Create Flink Catalog support for Hudi ) > Create Flink Glue Catalog support for Hudi > --- > > Key: HUDI-5426 > URL: https://issues.apache.org/jira/browse/HUDI-5426 > Project: Apache Hudi > Issue Type: Wish > Components: flink, flink-sql >Reporter: Samrat Deb >Priority: Minor > > Currently there is no support for Hudi flink glue catalog for sql-client > Iceberg provides support for catalog in sql-client > [https://iceberg.apache.org/docs/latest/flink/#creating-catalogs-and-using-catalogs] > . > This issue created to add similar support with flink hudi -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HUDI-5426) Create Flink Catalog support for Hudi
Samrat Deb created HUDI-5426: Summary: Create Flink Catalog support for Hudi Key: HUDI-5426 URL: https://issues.apache.org/jira/browse/HUDI-5426 Project: Apache Hudi Issue Type: Wish Components: flink, flink-sql Reporter: Samrat Deb Currently there is no support for Hudi flink glue catalog for sql-client Iceberg provides support for catalog in sql-client [https://iceberg.apache.org/docs/latest/flink/#creating-catalogs-and-using-catalogs] . This issue created to add similar support with flink hudi -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HUDI-2189) Delete partition support in HoodieDeltaStreamer
[ https://issues.apache.org/jira/browse/HUDI-2189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17388488#comment-17388488 ] Samrat Deb commented on HUDI-2189: -- [~codope] so it is kind of truncate table operation , partition wise ? there is also a delete op , do we need to support that as well ? > Delete partition support in HoodieDeltaStreamer > > > Key: HUDI-2189 > URL: https://issues.apache.org/jira/browse/HUDI-2189 > Project: Apache Hudi > Issue Type: Task >Reporter: Samrat Deb >Assignee: Samrat Deb >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (HUDI-2189) Delete partition support in HoodieDeltaStreamer
[ https://issues.apache.org/jira/browse/HUDI-2189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17386648#comment-17386648 ] Samrat Deb edited comment on HUDI-2189 at 7/24/21, 9:38 AM: I was trying the follow-up tasks suggested in [https://github.com/apache/hudi/pull/3184#discussion_r670990427] . (support delete_partition on DeltaStreamer) [clarification ] # As per what I understood is it will be a deltastreamer , all the messages coming through the stream will go and delete the row from the hudi table . (please let me know my understanding is correct ) # What happens when the message comes and that doesn’t matches any record in the table? # will DeltaStreamer job started with `op` as `delete_partition` for only delete purpose? [~codope] , [~shivnarayan] please help me with the clarity was (Author: samrat007): I was trying the follow-up tasks suggested in [https://github.com/apache/hudi/pull/3184#discussion_r670990427] . (support delete_partition on DeltaStreamer) [clarification ] # As per what I understood is it will be a deltastreamer , all the messages coming through the stream will go and delete the row from the hudi table . (please let me know my understanding is correct ) # What happens when the message comes and that doesn’t matches any record in the table? # will DeltaStreamer job started with `op` as `delete_partition` for only delete purpose? [~codope] , [~shivnarayan] > Delete partition support in HoodieDeltaStreamer > > > Key: HUDI-2189 > URL: https://issues.apache.org/jira/browse/HUDI-2189 > Project: Apache Hudi > Issue Type: Task >Reporter: Samrat Deb >Assignee: Samrat Deb >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HUDI-2189) Delete partition support in HoodieDeltaStreamer
[ https://issues.apache.org/jira/browse/HUDI-2189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17386648#comment-17386648 ] Samrat Deb commented on HUDI-2189: -- I was trying the follow-up tasks suggested in [https://github.com/apache/hudi/pull/3184#discussion_r670990427] . (support delete_partition on DeltaStreamer) [clarification ] # As per what I understood is it will be a deltastreamer , all the messages coming through the stream will go and delete the row from the hudi table . (please let me know my understanding is correct ) # What happens when the message comes and that doesn’t matches any record in the table? # will DeltaStreamer job started with `op` as `delete_partition` for only delete purpose? [~codope] , [~shivnarayan] > Delete partition support in HoodieDeltaStreamer > > > Key: HUDI-2189 > URL: https://issues.apache.org/jira/browse/HUDI-2189 > Project: Apache Hudi > Issue Type: Task >Reporter: Samrat Deb >Assignee: Samrat Deb >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HUDI-2188) Improve test for the insert_overwrite and insert_overwrite_table in hoodieDeltaStreamer
[ https://issues.apache.org/jira/browse/HUDI-2188?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Samrat Deb reassigned HUDI-2188: Assignee: Samrat Deb > Improve test for the insert_overwrite and insert_overwrite_table in > hoodieDeltaStreamer > --- > > Key: HUDI-2188 > URL: https://issues.apache.org/jira/browse/HUDI-2188 > Project: Apache Hudi > Issue Type: Test >Reporter: Samrat Deb >Assignee: Samrat Deb >Priority: Major > > InsertOverwrite overwrites only the partitions matching the incoming records. > need to add a test that verifies insert_overwrite does not overwrite > mismatched partitions. > reference -: https://github.com/apache/hudi/pull/3184/files#r670993094 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HUDI-2189) Delete partition support in HoodieDeltaStreamer
[ https://issues.apache.org/jira/browse/HUDI-2189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Samrat Deb reassigned HUDI-2189: Assignee: Samrat Deb > Delete partition support in HoodieDeltaStreamer > > > Key: HUDI-2189 > URL: https://issues.apache.org/jira/browse/HUDI-2189 > Project: Apache Hudi > Issue Type: Task >Reporter: Samrat Deb >Assignee: Samrat Deb >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HUDI-2188) Improve test for the insert_overwrite and insert_overwrite_table in hoodieDeltaStreamer
[ https://issues.apache.org/jira/browse/HUDI-2188?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Samrat Deb updated HUDI-2188: - Description: InsertOverwrite overwrites only the partitions matching the incoming records. need to add a test that verifies insert_overwrite does not overwrite mismatched partitions. reference -: https://github.com/apache/hudi/pull/3184/files#r670993094 > Improve test for the insert_overwrite and insert_overwrite_table in > hoodieDeltaStreamer > --- > > Key: HUDI-2188 > URL: https://issues.apache.org/jira/browse/HUDI-2188 > Project: Apache Hudi > Issue Type: Test >Reporter: Samrat Deb >Priority: Major > > InsertOverwrite overwrites only the partitions matching the incoming records. > need to add a test that verifies insert_overwrite does not overwrite > mismatched partitions. > reference -: https://github.com/apache/hudi/pull/3184/files#r670993094 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HUDI-2189) Delete partition support in HoodieDeltaStreamer
Samrat Deb created HUDI-2189: Summary: Delete partition support in HoodieDeltaStreamer Key: HUDI-2189 URL: https://issues.apache.org/jira/browse/HUDI-2189 Project: Apache Hudi Issue Type: Task Reporter: Samrat Deb -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HUDI-2188) Improve test for the insert_overwrite and insert_overwrite_table in hoodieDeltaStreamer
Samrat Deb created HUDI-2188: Summary: Improve test for the insert_overwrite and insert_overwrite_table in hoodieDeltaStreamer Key: HUDI-2188 URL: https://issues.apache.org/jira/browse/HUDI-2188 Project: Apache Hudi Issue Type: Test Reporter: Samrat Deb -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HUDI-2148) Add Statsd Support for monitoring
Samrat Deb created HUDI-2148: Summary: Add Statsd Support for monitoring Key: HUDI-2148 URL: https://issues.apache.org/jira/browse/HUDI-2148 Project: Apache Hudi Issue Type: New Feature Components: metrics Reporter: Samrat Deb Assignee: Samrat Deb Provide support for Statsd monitoring in hudi -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HUDI-1860) Add INSERT_OVERWRITE support to DeltaStreamer
[ https://issues.apache.org/jira/browse/HUDI-1860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Samrat Deb reassigned HUDI-1860: Assignee: Samrat Deb (was: Sagar Sumit) > Add INSERT_OVERWRITE support to DeltaStreamer > - > > Key: HUDI-1860 > URL: https://issues.apache.org/jira/browse/HUDI-1860 > Project: Apache Hudi > Issue Type: Sub-task >Reporter: Sagar Sumit >Assignee: Samrat Deb >Priority: Major > Labels: pull-request-available > Original Estimate: 72h > Remaining Estimate: 72h > > As discussed in [this > RFC|https://cwiki.apache.org/confluence/display/HUDI/RFC+-+14+%3A+JDBC+incremental+puller], > having full fetch mode use the inser_overwrite to write to sync would be > better as it can handle schema changes. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HUDI-1860) Add INSERT_OVERWRITE support to DeltaStreamer
[ https://issues.apache.org/jira/browse/HUDI-1860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17369632#comment-17369632 ] Samrat Deb commented on HUDI-1860: -- [~codope], tried to assign the sub-task to me, don't have permission to do that. Already Sent mail to [dev-subscr...@hudi.apache.org|mailto:to%c2%a0dev-subscr...@hudi.apache.org] for adding my Jira-id to the community. > Add INSERT_OVERWRITE support to DeltaStreamer > - > > Key: HUDI-1860 > URL: https://issues.apache.org/jira/browse/HUDI-1860 > Project: Apache Hudi > Issue Type: Sub-task >Reporter: Sagar Sumit >Assignee: Sagar Sumit >Priority: Major > Original Estimate: 72h > Remaining Estimate: 72h > > As discussed in [this > RFC|https://cwiki.apache.org/confluence/display/HUDI/RFC+-+14+%3A+JDBC+incremental+puller], > having full fetch mode use the inser_overwrite to write to sync would be > better as it can handle schema changes. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HUDI-1860) Add INSERT_OVERWRITE support to DeltaStreamer
[ https://issues.apache.org/jira/browse/HUDI-1860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17368540#comment-17368540 ] Samrat Deb commented on HUDI-1860: -- Hi [~codope], Greetings! I want to pick this subtask. If you are not working on this sub-task actively, please help me with the context. > Add INSERT_OVERWRITE support to DeltaStreamer > - > > Key: HUDI-1860 > URL: https://issues.apache.org/jira/browse/HUDI-1860 > Project: Apache Hudi > Issue Type: Sub-task >Reporter: Sagar Sumit >Assignee: Sagar Sumit >Priority: Major > Original Estimate: 72h > Remaining Estimate: 72h > > As discussed in [this > RFC|https://cwiki.apache.org/confluence/display/HUDI/RFC+-+14+%3A+JDBC+incremental+puller], > having full fetch mode use the inser_overwrite to write to sync would be > better as it can handle schema changes. -- This message was sent by Atlassian Jira (v8.3.4#803005)
