[jira] [Created] (HUDI-3658) Add Hudi Uber Meetup on March 1st

2022-03-18 Thread Nishith Agarwal (Jira)
Nishith Agarwal created HUDI-3658: - Summary: Add Hudi Uber Meetup on March 1st Key: HUDI-3658 URL: https://issues.apache.org/jira/browse/HUDI-3658 Project: Apache Hudi Issue Type: Task

[jira] [Created] (HUDI-3256) Add Links to Hudi Meetup Jan 2022

2022-01-16 Thread Nishith Agarwal (Jira)
Nishith Agarwal created HUDI-3256: - Summary: Add Links to Hudi Meetup Jan 2022 Key: HUDI-3256 URL: https://issues.apache.org/jira/browse/HUDI-3256 Project: Apache Hudi Issue Type: Task

[jira] [Commented] (HUDI-1576) Add ability to perform archival synchronously

2022-01-14 Thread Nishith Agarwal (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17476506#comment-17476506 ] Nishith Agarwal commented on HUDI-1576: --- [~guoyihua] Yes, the idea was to detach archiving from

[jira] [Commented] (HUDI-2275) HoodieDeltaStreamerException when using OCC and a second concurrent writer

2021-10-07 Thread Nishith Agarwal (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17425912#comment-17425912 ] Nishith Agarwal commented on HUDI-2275: --- [~dave_hagman] To ensure that the checkpoints from

[jira] [Commented] (HUDI-2146) Concurrent writes loss data

2021-07-17 Thread Nishith Agarwal (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17382691#comment-17382691 ] Nishith Agarwal commented on HUDI-2146: --- [~wenningd] I see that there is a conflict thrown when both

[jira] [Updated] (HUDI-1824) Spark Datasource V2/V1 (Dataset) integration with ORC

2021-07-15 Thread Nishith Agarwal (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1824?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nishith Agarwal updated HUDI-1824: -- Summary: Spark Datasource V2/V1 (Dataset) integration with ORC (was: Spark Integration with

[jira] [Updated] (HUDI-765) Implement OrcReaderIterator

2021-07-15 Thread Nishith Agarwal (Jira)
[ https://issues.apache.org/jira/browse/HUDI-765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nishith Agarwal updated HUDI-765: - Fix Version/s: 0.9.0 > Implement OrcReaderIterator > --- > >

[jira] [Updated] (HUDI-764) Implement HoodieOrcWriter

2021-07-15 Thread Nishith Agarwal (Jira)
[ https://issues.apache.org/jira/browse/HUDI-764?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nishith Agarwal updated HUDI-764: - Status: Closed (was: Patch Available) > Implement HoodieOrcWriter > - > >

[jira] [Updated] (HUDI-765) Implement OrcReaderIterator

2021-07-15 Thread Nishith Agarwal (Jira)
[ https://issues.apache.org/jira/browse/HUDI-765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nishith Agarwal updated HUDI-765: - Status: Closed (was: Patch Available) > Implement OrcReaderIterator > ---

[jira] [Assigned] (HUDI-764) Implement HoodieOrcWriter

2021-07-15 Thread Nishith Agarwal (Jira)
[ https://issues.apache.org/jira/browse/HUDI-764?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nishith Agarwal reassigned HUDI-764: Assignee: (was: Teresa Kang) > Implement HoodieOrcWriter > - >

[jira] [Assigned] (HUDI-764) Implement HoodieOrcWriter

2021-07-15 Thread Nishith Agarwal (Jira)
[ https://issues.apache.org/jira/browse/HUDI-764?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nishith Agarwal reassigned HUDI-764: Assignee: Teresa Kang > Implement HoodieOrcWriter > - > >

[jira] [Commented] (HUDI-2159) Supporting Clustering and Metadata Table together

2021-07-12 Thread Nishith Agarwal (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17379423#comment-17379423 ] Nishith Agarwal commented on HUDI-2159: --- Thanks for the detailed analysis [~pwason]. I think it is

[jira] [Commented] (HUDI-2146) Concurrent writes loss data

2021-07-08 Thread Nishith Agarwal (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17377455#comment-17377455 ] Nishith Agarwal commented on HUDI-2146: --- [~wenningd] Thanks for the detailed description. Few

[jira] [Updated] (HUDI-2146) Concurrent writes loss data

2021-07-08 Thread Nishith Agarwal (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2146?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nishith Agarwal updated HUDI-2146: -- Fix Version/s: 0.9.0 > Concurrent writes loss data > > >

[jira] [Updated] (HUDI-2146) Concurrent writes loss data

2021-07-08 Thread Nishith Agarwal (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2146?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nishith Agarwal updated HUDI-2146: -- Priority: Blocker (was: Major) > Concurrent writes loss data > >

[jira] [Created] (HUDI-2091) Add Uber's grafana dashboard to OSS

2021-06-28 Thread Nishith Agarwal (Jira)
Nishith Agarwal created HUDI-2091: - Summary: Add Uber's grafana dashboard to OSS Key: HUDI-2091 URL: https://issues.apache.org/jira/browse/HUDI-2091 Project: Apache Hudi Issue Type: New

[jira] [Commented] (HUDI-1537) Move validation of file listings to something that happens before each write

2021-06-22 Thread Nishith Agarwal (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17367807#comment-17367807 ] Nishith Agarwal commented on HUDI-1537: --- This logic is being removed. Additionally, falling back to

[jira] [Commented] (HUDI-1542) Fix Flaky test : TestHoodieMetadata#testSync

2021-06-22 Thread Nishith Agarwal (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17367808#comment-17367808 ] Nishith Agarwal commented on HUDI-1542: --- [~pwason] Will take this up next week. > Fix Flaky test :

[jira] [Commented] (HUDI-1492) Handle DeltaWriteStat correctly for storage schemes that support appends

2021-06-22 Thread Nishith Agarwal (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17367806#comment-17367806 ] Nishith Agarwal commented on HUDI-1492: --- Confirmed with [~pwason] that this does not affect

[jira] [Assigned] (HUDI-1077) Integration tests to validate clustering

2021-06-22 Thread Nishith Agarwal (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1077?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nishith Agarwal reassigned HUDI-1077: - Assignee: satish > Integration tests to validate clustering >

[jira] [Updated] (HUDI-1839) FSUtils getAllPartitions broken by NotSerializableException: org.apache.hadoop.fs.Path

2021-06-22 Thread Nishith Agarwal (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1839?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nishith Agarwal updated HUDI-1839: -- Priority: Blocker (was: Major) > FSUtils getAllPartitions broken by NotSerializableException:

[jira] [Updated] (HUDI-1839) FSUtils getAllPartitions broken by NotSerializableException: org.apache.hadoop.fs.Path

2021-06-22 Thread Nishith Agarwal (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1839?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nishith Agarwal updated HUDI-1839: -- Fix Version/s: 0.9.0 > FSUtils getAllPartitions broken by NotSerializableException: >

[jira] [Commented] (HUDI-1839) FSUtils getAllPartitions broken by NotSerializableException: org.apache.hadoop.fs.Path

2021-06-22 Thread Nishith Agarwal (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1839?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17367647#comment-17367647 ] Nishith Agarwal commented on HUDI-1839: --- [~pwason] Is this something that we have identified the

[jira] [Updated] (HUDI-1047) Support asynchronize clustering in CoW mode

2021-06-15 Thread Nishith Agarwal (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1047?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nishith Agarwal updated HUDI-1047: -- Fix Version/s: 0.9.0 > Support asynchronize clustering in CoW mode >

[jira] [Updated] (HUDI-1048) Support Asynchronize clustering in MoR mode

2021-06-15 Thread Nishith Agarwal (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nishith Agarwal updated HUDI-1048: -- Fix Version/s: 0.9.0 > Support Asynchronize clustering in MoR mode >

[jira] [Updated] (HUDI-1048) Support Asynchronize clustering in MoR mode

2021-06-15 Thread Nishith Agarwal (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nishith Agarwal updated HUDI-1048: -- Priority: Blocker (was: Major) > Support Asynchronize clustering in MoR mode >

[jira] [Updated] (HUDI-1706) Test flakiness w/ multiwriter test

2021-06-15 Thread Nishith Agarwal (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1706?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nishith Agarwal updated HUDI-1706: -- Priority: Blocker (was: Major) > Test flakiness w/ multiwriter test >

[jira] [Updated] (HUDI-1706) Test flakiness w/ multiwriter test

2021-06-15 Thread Nishith Agarwal (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1706?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nishith Agarwal updated HUDI-1706: -- Fix Version/s: 0.9.0 > Test flakiness w/ multiwriter test > --

[jira] [Created] (HUDI-2026) Add documentation for GlobalDeleteKeyGenerator

2021-06-15 Thread Nishith Agarwal (Jira)
Nishith Agarwal created HUDI-2026: - Summary: Add documentation for GlobalDeleteKeyGenerator Key: HUDI-2026 URL: https://issues.apache.org/jira/browse/HUDI-2026 Project: Apache Hudi Issue

[jira] [Commented] (HUDI-1975) Upgrade java-prometheus-client from 3.1.2 to 4.x

2021-06-15 Thread Nishith Agarwal (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17363911#comment-17363911 ] Nishith Agarwal commented on HUDI-1975: --- [~vinaypatil18] I think there are 2 options :  # Shade the

[jira] [Updated] (HUDI-2003) Auto Compute Compression ratio for input data to output parquet/orc file size

2021-06-14 Thread Nishith Agarwal (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2003?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nishith Agarwal updated HUDI-2003: -- Summary: Auto Compute Compression ratio for input data to output parquet/orc file size (was:

[jira] [Updated] (HUDI-2003) Auto Compute Compression ratio for input data to output parquet/orc file size

2021-06-14 Thread Nishith Agarwal (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2003?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nishith Agarwal updated HUDI-2003: -- Issue Type: Improvement (was: Bug) > Auto Compute Compression ratio for input data to output

[jira] [Commented] (HUDI-1910) Supporting Kafka based checkpointing for HoodieDeltaStreamer

2021-06-14 Thread Nishith Agarwal (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17363067#comment-17363067 ] Nishith Agarwal commented on HUDI-1910: --- [~vinaypatil18] Yes, that makes sense, please go ahead. >

[jira] [Comment Edited] (HUDI-2005) Audit and remove references of fs.listStatus()

2021-06-13 Thread Nishith Agarwal (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17362662#comment-17362662 ] Nishith Agarwal edited comment on HUDI-2005 at 6/13/21, 10:54 PM: -- 1. 

[jira] [Comment Edited] (HUDI-2005) Audit and remove references of fs.listStatus()

2021-06-13 Thread Nishith Agarwal (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17362662#comment-17362662 ] Nishith Agarwal edited comment on HUDI-2005 at 6/13/21, 10:49 PM: -- 1. 

[jira] [Assigned] (HUDI-2005) Audit and remove references of fs.listStatus()

2021-06-13 Thread Nishith Agarwal (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2005?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nishith Agarwal reassigned HUDI-2005: - Assignee: Prashant Wason (was: Nishith Agarwal) > Audit and remove references of

[jira] [Commented] (HUDI-2005) Audit and remove references of fs.listStatus()

2021-06-13 Thread Nishith Agarwal (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17362662#comment-17362662 ] Nishith Agarwal commented on HUDI-2005: --- 1. 

[jira] [Created] (HUDI-2005) Audit and remove references of fs.listStatus()

2021-06-13 Thread Nishith Agarwal (Jira)
Nishith Agarwal created HUDI-2005: - Summary: Audit and remove references of fs.listStatus() Key: HUDI-2005 URL: https://issues.apache.org/jira/browse/HUDI-2005 Project: Apache Hudi Issue

[jira] [Updated] (HUDI-1457) Add multi writing to Hudi tables using DFS based locking (only HDFS atomic renames)

2021-06-13 Thread Nishith Agarwal (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nishith Agarwal updated HUDI-1457: -- Summary: Add multi writing to Hudi tables using DFS based locking (only HDFS atomic renames)

[jira] [Updated] (HUDI-1457) Add multi writing to Hudi tables using DFS based locking

2021-06-13 Thread Nishith Agarwal (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nishith Agarwal updated HUDI-1457: -- Summary: Add multi writing to Hudi tables using DFS based locking (was: Add parallel writing

[jira] [Resolved] (HUDI-1679) Add example to docker for optimistic lock use

2021-06-13 Thread Nishith Agarwal (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nishith Agarwal resolved HUDI-1679. --- Fix Version/s: 0.8.0 Resolution: Fixed > Add example to docker for optimistic lock use

[jira] [Updated] (HUDI-1623) Support start_commit_time & end_commit_times for serializable incremental pull

2021-06-13 Thread Nishith Agarwal (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1623?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nishith Agarwal updated HUDI-1623: -- Fix Version/s: 0.10.0 > Support start_commit_time & end_commit_times for serializable

[jira] [Updated] (HUDI-1575) Early detection by periodically checking last written commit

2021-06-13 Thread Nishith Agarwal (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1575?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nishith Agarwal updated HUDI-1575: -- Summary: Early detection by periodically checking last written commit (was: Early detection,

[jira] [Updated] (HUDI-1575) Early detection by periodically checking last written commit

2021-06-13 Thread Nishith Agarwal (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1575?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nishith Agarwal updated HUDI-1575: -- Description: Check if there are more commits, try to do resolution, and abort for a currently

[jira] [Commented] (HUDI-944) Support more complete concurrency control when writing data

2021-06-13 Thread Nishith Agarwal (Jira)
[ https://issues.apache.org/jira/browse/HUDI-944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17362627#comment-17362627 ] Nishith Agarwal commented on HUDI-944: -- Concurrent writing to HUDI tables is now supported. Closing

[jira] [Resolved] (HUDI-944) Support more complete concurrency control when writing data

2021-06-13 Thread Nishith Agarwal (Jira)
[ https://issues.apache.org/jira/browse/HUDI-944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nishith Agarwal resolved HUDI-944. -- Fix Version/s: 0.8.0 Resolution: Fixed > Support more complete concurrency control when

[jira] [Updated] (HUDI-1577) Document that multi-writer cannot be used within the same write client

2021-06-13 Thread Nishith Agarwal (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1577?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nishith Agarwal updated HUDI-1577: -- Fix Version/s: 0.9.0 > Document that multi-writer cannot be used within the same write client >

[jira] [Updated] (HUDI-1577) Document that multi-writer cannot be used within the same write client

2021-06-13 Thread Nishith Agarwal (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1577?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nishith Agarwal updated HUDI-1577: -- Priority: Blocker (was: Major) > Document that multi-writer cannot be used within the same

[jira] [Updated] (HUDI-1706) Test flakiness w/ multiwriter test

2021-06-13 Thread Nishith Agarwal (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1706?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nishith Agarwal updated HUDI-1706: -- Parent: HUDI-1456 Issue Type: Sub-task (was: Bug) > Test flakiness w/ multiwriter test

[jira] [Updated] (HUDI-1456) [UMBRELLA] Concurrent Writing (multiwriter) to Hudi tables

2021-06-13 Thread Nishith Agarwal (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1456?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nishith Agarwal updated HUDI-1456: -- Summary: [UMBRELLA] Concurrent Writing (multiwriter) to Hudi tables (was: [UMBRELLA]

[jira] [Updated] (HUDI-1698) Multiwriting for Flink / Java

2021-06-13 Thread Nishith Agarwal (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1698?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nishith Agarwal updated HUDI-1698: -- Parent: HUDI-1456 Issue Type: Sub-task (was: Improvement) > Multiwriting for Flink /

[jira] [Updated] (HUDI-1047) Support asynchronize clustering in CoW mode

2021-06-13 Thread Nishith Agarwal (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1047?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nishith Agarwal updated HUDI-1047: -- Summary: Support asynchronize clustering in CoW mode (was: Support synchronize clustering in

[jira] [Updated] (HUDI-1048) Support Asynchronize clustering in MoR mode

2021-06-13 Thread Nishith Agarwal (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nishith Agarwal updated HUDI-1048: -- Summary: Support Asynchronize clustering in MoR mode (was: Support synchronize clustering in

[jira] [Updated] (HUDI-1077) Integration tests to validate clustering

2021-06-13 Thread Nishith Agarwal (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1077?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nishith Agarwal updated HUDI-1077: -- Fix Version/s: 0.9.0 > Integration tests to validate clustering >

[jira] [Updated] (HUDI-1353) Incremental timeline support for pending clustering operations

2021-06-13 Thread Nishith Agarwal (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1353?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nishith Agarwal updated HUDI-1353: -- Priority: Blocker (was: Major) > Incremental timeline support for pending clustering

[jira] [Assigned] (HUDI-1468) incremental read support with clustering

2021-06-13 Thread Nishith Agarwal (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1468?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nishith Agarwal reassigned HUDI-1468: - Assignee: liwei > incremental read support with clustering >

[jira] [Updated] (HUDI-1468) incremental read support with clustering

2021-06-13 Thread Nishith Agarwal (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1468?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nishith Agarwal updated HUDI-1468: -- Priority: Blocker (was: Major) > incremental read support with clustering >

[jira] [Updated] (HUDI-1077) Integration tests to validate clustering

2021-06-13 Thread Nishith Agarwal (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1077?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nishith Agarwal updated HUDI-1077: -- Priority: Blocker (was: Major) > Integration tests to validate clustering >

[jira] [Updated] (HUDI-1482) async clustering for spark streaming

2021-06-13 Thread Nishith Agarwal (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1482?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nishith Agarwal updated HUDI-1482: -- Priority: Blocker (was: Major) > async clustering for spark streaming >

[jira] [Updated] (HUDI-1483) async clustering for deltastreamer

2021-06-13 Thread Nishith Agarwal (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1483?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nishith Agarwal updated HUDI-1483: -- Fix Version/s: 0.9.0 > async clustering for deltastreamer > --

[jira] [Updated] (HUDI-1482) async clustering for spark streaming

2021-06-13 Thread Nishith Agarwal (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1482?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nishith Agarwal updated HUDI-1482: -- Fix Version/s: 0.9.0 > async clustering for spark streaming >

[jira] [Updated] (HUDI-1500) support incremental read clustering commit in deltastreamer

2021-06-13 Thread Nishith Agarwal (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1500?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nishith Agarwal updated HUDI-1500: -- Priority: Blocker (was: Major) > support incremental read clustering commit in deltastreamer

[jira] [Updated] (HUDI-1500) support incremental read clustering commit in deltastreamer

2021-06-13 Thread Nishith Agarwal (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1500?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nishith Agarwal updated HUDI-1500: -- Fix Version/s: 0.9.0 > support incremental read clustering commit in deltastreamer >

[jira] [Updated] (HUDI-1483) async clustering for deltastreamer

2021-06-13 Thread Nishith Agarwal (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1483?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nishith Agarwal updated HUDI-1483: -- Priority: Blocker (was: Major) > async clustering for deltastreamer >

[jira] [Assigned] (HUDI-1500) support incremental read clustering commit in deltastreamer

2021-06-13 Thread Nishith Agarwal (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1500?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nishith Agarwal reassigned HUDI-1500: - Assignee: satish > support incremental read clustering commit in deltastreamer >

[jira] [Assigned] (HUDI-1937) When clustering fail, generating unfinished replacecommit timeline.

2021-06-13 Thread Nishith Agarwal (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nishith Agarwal reassigned HUDI-1937: - Assignee: liwei > When clustering fail, generating unfinished replacecommit timeline. >

[jira] [Updated] (HUDI-1937) When clustering fail, generating unfinished replacecommit timeline.

2021-06-13 Thread Nishith Agarwal (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nishith Agarwal updated HUDI-1937: -- Parent: HUDI-1042 Issue Type: Sub-task (was: Bug) > When clustering fail, generating

[jira] [Updated] (HUDI-1937) When clustering fail, generating unfinished replacecommit timeline.

2021-06-13 Thread Nishith Agarwal (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nishith Agarwal updated HUDI-1937: -- Priority: Blocker (was: Critical) > When clustering fail, generating unfinished replacecommit

[jira] [Commented] (HUDI-1937) When clustering fail, generating unfinished replacecommit timeline.

2021-06-13 Thread Nishith Agarwal (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17362624#comment-17362624 ] Nishith Agarwal commented on HUDI-1937: --- [~satish] [~309637554] Can one of you take a look at this ?

[jira] [Updated] (HUDI-1937) When clustering fail, generating unfinished replacecommit timeline.

2021-06-13 Thread Nishith Agarwal (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nishith Agarwal updated HUDI-1937: -- Fix Version/s: 0.9.0 > When clustering fail, generating unfinished replacecommit timeline. >

[jira] [Commented] (HUDI-1309) Listing Metadata unreadable in S3 as the log block is deemed corrupted

2021-06-13 Thread Nishith Agarwal (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1309?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17362623#comment-17362623 ] Nishith Agarwal commented on HUDI-1309: --- [~vbalaji] Is this something you still see ?  > Listing

[jira] [Commented] (HUDI-1537) Move validation of file listings to something that happens before each write

2021-06-13 Thread Nishith Agarwal (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17362622#comment-17362622 ] Nishith Agarwal commented on HUDI-1537: --- [~pwason] Is validation of file listing applicable ? >

[jira] [Updated] (HUDI-1309) Listing Metadata unreadable in S3 as the log block is deemed corrupted

2021-06-13 Thread Nishith Agarwal (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1309?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nishith Agarwal updated HUDI-1309: -- Priority: Blocker (was: Major) > Listing Metadata unreadable in S3 as the log block is deemed

[jira] [Commented] (HUDI-1649) Bugs with Metadata Table in 0.7 release

2021-06-13 Thread Nishith Agarwal (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17362621#comment-17362621 ] Nishith Agarwal commented on HUDI-1649: --- [~pwason] Are you going to open a PR to address all of

[jira] [Updated] (HUDI-1537) Move validation of file listings to something that happens before each write

2021-06-13 Thread Nishith Agarwal (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1537?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nishith Agarwal updated HUDI-1537: -- Priority: Blocker (was: Major) > Move validation of file listings to something that happens

[jira] [Updated] (HUDI-1542) Fix Flaky test : TestHoodieMetadata#testSync

2021-06-13 Thread Nishith Agarwal (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1542?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nishith Agarwal updated HUDI-1542: -- Priority: Blocker (was: Major) > Fix Flaky test : TestHoodieMetadata#testSync >

[jira] [Resolved] (HUDI-1962) Add a blog/docs for shuffle paralelism

2021-06-13 Thread Nishith Agarwal (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1962?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nishith Agarwal resolved HUDI-1962. --- Resolution: Fixed > Add a blog/docs for shuffle paralelism >

[jira] [Commented] (HUDI-1962) Add a blog/docs for shuffle paralelism

2021-06-13 Thread Nishith Agarwal (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17362619#comment-17362619 ] Nishith Agarwal commented on HUDI-1962: --- Added a FAQ -> 

[jira] [Commented] (HUDI-1959) Add links to small file handling and clustering to the config section

2021-06-13 Thread Nishith Agarwal (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17362615#comment-17362615 ] Nishith Agarwal commented on HUDI-1959: --- Added a FAQ here -> 

[jira] [Resolved] (HUDI-1959) Add links to small file handling and clustering to the config section

2021-06-13 Thread Nishith Agarwal (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1959?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nishith Agarwal resolved HUDI-1959. --- Fix Version/s: 0.9.0 Resolution: Fixed > Add links to small file handling and

[jira] [Resolved] (HUDI-1960) Add documentation to be able to disable parquet configs

2021-06-13 Thread Nishith Agarwal (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1960?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nishith Agarwal resolved HUDI-1960. --- Fix Version/s: 0.9.0 Resolution: Fixed > Add documentation to be able to disable

[jira] [Commented] (HUDI-1960) Add documentation to be able to disable parquet configs

2021-06-13 Thread Nishith Agarwal (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1960?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17362607#comment-17362607 ] Nishith Agarwal commented on HUDI-1960: --- Added a FAQ here -> 

[jira] [Commented] (HUDI-1975) Upgrade java-prometheus-client from 3.1.2 to 4.x

2021-06-13 Thread Nishith Agarwal (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17362604#comment-17362604 ] Nishith Agarwal commented on HUDI-1975: --- [~vinaypatil18] It looks like even the latest prometheus

[jira] [Updated] (HUDI-1975) Upgrade java-prometheus-client from 3.1.2 to 4.x

2021-06-13 Thread Nishith Agarwal (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1975?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nishith Agarwal updated HUDI-1975: -- Description: Find more details here ->  https://github.com/apache/hudi/issues/2774 > Upgrade

[jira] [Deleted] (HUDI-1945) Support Hudi to read from Kafka Consumer Group Offset

2021-06-13 Thread Nishith Agarwal (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1945?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nishith Agarwal deleted HUDI-1945: -- > Support Hudi to read from Kafka Consumer Group Offset >

[jira] [Commented] (HUDI-1910) Supporting Kafka based checkpointing for HoodieDeltaStreamer

2021-06-13 Thread Nishith Agarwal (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17362600#comment-17362600 ] Nishith Agarwal commented on HUDI-1910: --- [~vinaypatil18] Thanks for sharing your approach. The first

[jira] [Updated] (HUDI-1909) Skip the commits with empty files for flink streaming reader

2021-06-11 Thread Nishith Agarwal (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1909?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nishith Agarwal updated HUDI-1909: -- Description: Log warnings instead of throwing to make the reader more robust.  

[jira] [Created] (HUDI-1998) Provide a way to find list of commits through a pythonic API

2021-06-11 Thread Nishith Agarwal (Jira)
Nishith Agarwal created HUDI-1998: - Summary: Provide a way to find list of commits through a pythonic API Key: HUDI-1998 URL: https://issues.apache.org/jira/browse/HUDI-1998 Project: Apache Hudi

[jira] [Created] (HUDI-1997) Fix hoodie.datasource.hive_sync.auto_create_database documentation

2021-06-11 Thread Nishith Agarwal (Jira)
Nishith Agarwal created HUDI-1997: - Summary: Fix hoodie.datasource.hive_sync.auto_create_database documentation Key: HUDI-1997 URL: https://issues.apache.org/jira/browse/HUDI-1997 Project: Apache

[jira] [Commented] (HUDI-1138) Re-implement marker files via timeline server

2021-06-08 Thread Nishith Agarwal (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17359760#comment-17359760 ] Nishith Agarwal commented on HUDI-1138: --- Okay, thanks for sharing this info.  > Re-implement marker

[jira] [Commented] (HUDI-1827) Add ORC support in Bootstrap Op

2021-06-06 Thread Nishith Agarwal (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17358358#comment-17358358 ] Nishith Agarwal commented on HUDI-1827: --- [~manasaks] You approach sounds good to me. For marking the

[jira] [Comment Edited] (HUDI-1827) Add ORC support in Bootstrap Op

2021-06-06 Thread Nishith Agarwal (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17358358#comment-17358358 ] Nishith Agarwal edited comment on HUDI-1827 at 6/7/21, 5:03 AM:

[jira] [Created] (HUDI-1977) Fix Hudi-CLI show table spark-sql

2021-06-05 Thread Nishith Agarwal (Jira)
Nishith Agarwal created HUDI-1977: - Summary: Fix Hudi-CLI show table spark-sql Key: HUDI-1977 URL: https://issues.apache.org/jira/browse/HUDI-1977 Project: Apache Hudi Issue Type: Task

[jira] [Created] (HUDI-1976) Upgrade hive, jackson, log4j, hadoop to remove vulnerability

2021-06-05 Thread Nishith Agarwal (Jira)
Nishith Agarwal created HUDI-1976: - Summary: Upgrade hive, jackson, log4j, hadoop to remove vulnerability Key: HUDI-1976 URL: https://issues.apache.org/jira/browse/HUDI-1976 Project: Apache Hudi

[jira] [Updated] (HUDI-1976) Upgrade hive, jackson, log4j, hadoop to remove vulnerability

2021-06-05 Thread Nishith Agarwal (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nishith Agarwal updated HUDI-1976: -- Fix Version/s: 0.9.0 Priority: Blocker (was: Major) > Upgrade hive, jackson, log4j,

[jira] [Updated] (HUDI-1592) Metadata listing fails for non partitoned dataset

2021-06-05 Thread Nishith Agarwal (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1592?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nishith Agarwal updated HUDI-1592: -- Fix Version/s: 0.9.0 Priority: Blocker (was: Major) > Metadata listing fails for non

[jira] [Updated] (HUDI-1975) Upgrade java-prometheus-client from 3.1.2 to 4.x

2021-06-05 Thread Nishith Agarwal (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1975?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nishith Agarwal updated HUDI-1975: -- Fix Version/s: 0.9.0 Priority: Blocker (was: Major) > Upgrade java-prometheus-client

[jira] [Created] (HUDI-1975) Upgrade java-prometheus-client from 3.1.2 to 4.x

2021-06-05 Thread Nishith Agarwal (Jira)
Nishith Agarwal created HUDI-1975: - Summary: Upgrade java-prometheus-client from 3.1.2 to 4.x Key: HUDI-1975 URL: https://issues.apache.org/jira/browse/HUDI-1975 Project: Apache Hudi Issue

[jira] [Updated] (HUDI-1974) Run pyspark and validate that it works correctly with all hudi versions

2021-06-05 Thread Nishith Agarwal (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1974?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nishith Agarwal updated HUDI-1974: -- Fix Version/s: 0.9.0 > Run pyspark and validate that it works correctly with all hudi versions

[jira] [Updated] (HUDI-1974) Run pyspark and validate that it works correctly with all hudi versions

2021-06-05 Thread Nishith Agarwal (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1974?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nishith Agarwal updated HUDI-1974: -- Priority: Blocker (was: Major) > Run pyspark and validate that it works correctly with all

  1   2   3   4   5   >