[jira] [Updated] (HUDI-4722) Add support for metrics for locking infra
[ https://issues.apache.org/jira/browse/HUDI-4722?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jagmeet bali updated HUDI-4722: --- Description: Added metrics for following # Lock request latency # Count of Lock success # Count of failed to acquire the lock # Duration of locks held with support for re-entrancy # Conflict resolution metrics. Succes vs Failure > Add support for metrics for locking infra > - > > Key: HUDI-4722 > URL: https://issues.apache.org/jira/browse/HUDI-4722 > Project: Apache Hudi > Issue Type: Improvement >Reporter: Jagmeet bali >Priority: Minor > > Added metrics for following > # Lock request latency > # Count of Lock success > # Count of failed to acquire the lock > # Duration of locks held with support for re-entrancy > # Conflict resolution metrics. Succes vs Failure -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HUDI-4722) Add support for metrics for locking infra
[ https://issues.apache.org/jira/browse/HUDI-4722?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jagmeet bali updated HUDI-4722: --- Priority: Minor (was: Major) > Add support for metrics for locking infra > - > > Key: HUDI-4722 > URL: https://issues.apache.org/jira/browse/HUDI-4722 > Project: Apache Hudi > Issue Type: Improvement >Reporter: Jagmeet bali >Priority: Minor > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HUDI-4722) Add support for metrics for locking infra
Jagmeet bali created HUDI-4722: -- Summary: Add support for metrics for locking infra Key: HUDI-4722 URL: https://issues.apache.org/jira/browse/HUDI-4722 Project: Apache Hudi Issue Type: Improvement Reporter: Jagmeet bali -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HUDI-2711) Fallback to full table scan for IncrementalRelation and HoodieIncrSource when data file is missing.
[ https://issues.apache.org/jira/browse/HUDI-2711?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jagmeet Bali updated HUDI-2711: --- Priority: Minor (was: Major) > Fallback to full table scan for IncrementalRelation and HoodieIncrSource when > data file is missing. > --- > > Key: HUDI-2711 > URL: https://issues.apache.org/jira/browse/HUDI-2711 > Project: Apache Hudi > Issue Type: New Feature >Reporter: Jagmeet Bali >Priority: Minor > > Fallback to full table scan for incremental readers if underlying file has > been moved or deleted due to cleaner. > For more info https://github.com/apache/hudi/issues/2934 -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Created] (HUDI-2711) Fallback to full table scan for IncrementalRelation and HoodieIncrSource when data file is missing.
Jagmeet Bali created HUDI-2711: -- Summary: Fallback to full table scan for IncrementalRelation and HoodieIncrSource when data file is missing. Key: HUDI-2711 URL: https://issues.apache.org/jira/browse/HUDI-2711 Project: Apache Hudi Issue Type: New Feature Reporter: Jagmeet Bali Fallback to full table scan for incremental readers if underlying file has been moved or deleted due to cleaner. For more info https://github.com/apache/hudi/issues/2934 -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Created] (HUDI-2248) Unable to shutdown local metastore client
Jagmeet Bali created HUDI-2248: -- Summary: Unable to shutdown local metastore client Key: HUDI-2248 URL: https://issues.apache.org/jira/browse/HUDI-2248 Project: Apache Hudi Issue Type: New Feature Reporter: Jagmeet Bali https://github.com/apache/hudi/issues/3187 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HUDI-1848) Add support for HMS in Hive-sync-tool
Jagmeet Bali created HUDI-1848: -- Summary: Add support for HMS in Hive-sync-tool Key: HUDI-1848 URL: https://issues.apache.org/jira/browse/HUDI-1848 Project: Apache Hudi Issue Type: New Feature Reporter: Jagmeet Bali Add support for HMS in Hive-sync-tool Currently there are two ways to sun DDL queries in hive-sync-tool. This work adds on top of [https://github.com/apache/hudi/pull/2532|https://github.com/apache/hudi/pull/2532/files] and adds a pluggable way to support new way to run DDL queries using HMS. Different DDL executors can be selected via diff syncConfig options useJDBC true -> JDBCExecutor will be used useJDBC false -> QlHiveQueryExecutor will be used useHMS true -> HMSDDLExecutor will be used. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HUDI-1789) Support snapshot queries with maxInstantTimes
[ https://issues.apache.org/jira/browse/HUDI-1789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17319485#comment-17319485 ] Jagmeet Bali commented on HUDI-1789: https://github.com/apache/hudi/pull/2809 > Support snapshot queries with maxInstantTimes > - > > Key: HUDI-1789 > URL: https://issues.apache.org/jira/browse/HUDI-1789 > Project: Apache Hudi > Issue Type: New Feature >Reporter: Jagmeet Bali >Priority: Minor > Labels: pull-request-available > > In HoodieParquetInoutFormat we currently default to the latest version of > base files. > This PR attempts to add a new jobConf > hoodie.%s.consume.snapshot.time > This new config will allow us to read older snapshots. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HUDI-1789) Support snapshot queries with maxInstantTimes
Jagmeet Bali created HUDI-1789: -- Summary: Support snapshot queries with maxInstantTimes Key: HUDI-1789 URL: https://issues.apache.org/jira/browse/HUDI-1789 Project: Apache Hudi Issue Type: New Feature Reporter: Jagmeet Bali In HoodieParquetInoutFormat we currently default to the latest version of base files. This PR attempts to add a new jobConf hoodie.%s.consume.snapshot.time This new config will allow us to read older snapshots. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HUDI-1780) Refactoring of parts of HoodieMetadataArchiveLog have changed behaviour of Archival
Jagmeet Bali created HUDI-1780: -- Summary: Refactoring of parts of HoodieMetadataArchiveLog have changed behaviour of Archival Key: HUDI-1780 URL: https://issues.apache.org/jira/browse/HUDI-1780 Project: Apache Hudi Issue Type: Bug Reporter: Jagmeet Bali The refactoring of HoodieMetadataArchiveLog to MetadataConverionUtils has changed the behaviour of how replacecommit.* files will be handled. Before the logic was [TimelineArchiveLog|https://github.com/apache/hudi/blob/c4a66324cdd3e289e0bf18bdd150b95ee6e4c66c/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/HoodieTimelineArchiveLog.java#L402] but now it has been changed to [MetadaConversionUtils.|https://github.com/apache/hudi/blob/master/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/utils/MetadataConversionUtils.java#L70] The issue with the code is replacecommit.* can be generated from different places and not from clustering alone. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HUDI-1746) Added support for replace commits in hudi-cli
[ https://issues.apache.org/jira/browse/HUDI-1746?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17312583#comment-17312583 ] Jagmeet Bali commented on HUDI-1746: https://github.com/apache/hudi/pull/2678 > Added support for replace commits in hudi-cli > - > > Key: HUDI-1746 > URL: https://issues.apache.org/jira/browse/HUDI-1746 > Project: Apache Hudi > Issue Type: New Feature >Reporter: Jagmeet Bali >Priority: Minor > > Currently, hudi-cli doesn't support replace commits in the commit show* > functions. This adds the foundation for that. > This PR still doesn't support the extraMetadata of the replace commit which > will be added in subsequent PR's. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HUDI-1746) Added support for replace commits in hudi-cli
Jagmeet Bali created HUDI-1746: -- Summary: Added support for replace commits in hudi-cli Key: HUDI-1746 URL: https://issues.apache.org/jira/browse/HUDI-1746 Project: Apache Hudi Issue Type: New Feature Reporter: Jagmeet Bali Currently, hudi-cli doesn't support replace commits in the commit show* functions. This adds the foundation for that. This PR still doesn't support the extraMetadata of the replace commit which will be added in subsequent PR's. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HUDI-1740) insert_overwrite_table and insert_overwrite first replacecommit has empty partitionToReplaceFileIds
Jagmeet Bali created HUDI-1740: -- Summary: insert_overwrite_table and insert_overwrite first replacecommit has empty partitionToReplaceFileIds Key: HUDI-1740 URL: https://issues.apache.org/jira/browse/HUDI-1740 Project: Apache Hudi Issue Type: Bug Reporter: Jagmeet Bali insert_overwrite_table and insert_overwrite first replacecommit has empty partitionToReplaceFileIds which messes up archival code. Fix The code needs to only proceed if partitionToReplaceFileIds is not Empty. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HUDI-1739) insert_overwrite_table and insert_overwrite create empty replacecommit.requested file which breaks archival
Jagmeet Bali created HUDI-1739: -- Summary: insert_overwrite_table and insert_overwrite create empty replacecommit.requested file which breaks archival Key: HUDI-1739 URL: https://issues.apache.org/jira/browse/HUDI-1739 Project: Apache Hudi Issue Type: Bug Reporter: Jagmeet Bali Fixes can be to # Ignore empty replacecommit.requested files. # Standardise the replacecommit.requested format across all invocations be it from clustering or this use case. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HUDI-1714) Improve code coverage of TestHoodieTimelineArchiveLog
Jagmeet Bali created HUDI-1714: -- Summary: Improve code coverage of TestHoodieTimelineArchiveLog Key: HUDI-1714 URL: https://issues.apache.org/jira/browse/HUDI-1714 Project: Apache Hudi Issue Type: Test Reporter: Jagmeet Bali Add tests for the newly added code which supports the archival of clean and rollback commits specifically around the getCleanInstantsToArchive codepath within HoodieTimelineArchiveLog -- This message was sent by Atlassian Jira (v8.3.4#803005)
