[GitHub] [hudi] danny0405 commented on pull request #2867: [HUDI-1836] Logging consuming instant to StreamReadOperator#processSplits
danny0405 commented on pull request #2867: URL: https://github.com/apache/hudi/pull/2867#issuecomment-827329997 Hi @yanghua ,can you take a look for this PR again, thanks ~ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HUDI-1853) Add flink index and compaction options to document
[ https://issues.apache.org/jira/browse/HUDI-1853?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HUDI-1853: - Labels: pull-request-available (was: ) > Add flink index and compaction options to document > -- > > Key: HUDI-1853 > URL: https://issues.apache.org/jira/browse/HUDI-1853 > Project: Apache Hudi > Issue Type: Task > Components: Flink Integration >Reporter: Danny Chen >Assignee: Danny Chen >Priority: Major > Labels: pull-request-available > Fix For: 0.9.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hudi] danny0405 opened a new pull request #2885: [HUDI-1853] Add flink index and compaction options to document
danny0405 opened a new pull request #2885: URL: https://github.com/apache/hudi/pull/2885 ## *Tips* - *Thank you very much for contributing to Apache Hudi.* - *Please review https://hudi.apache.org/contributing.html before opening a pull request.* ## What is the purpose of the pull request *(For example: This pull request adds quick-start document.)* ## Brief change log *(for example:)* - *Modify AnnotationLocation checkstyle rule in checkstyle.xml* ## Verify this pull request *(Please pick either of the following options)* This pull request is a trivial rework / code cleanup without any test coverage. *(or)* This pull request is already covered by existing tests, such as *(please describe tests)*. (or) This change added tests and can be verified as follows: *(example:)* - *Added integration tests for end-to-end.* - *Added HoodieClientWriteTest to verify the change.* - *Manually verified the change by running a job locally.* ## Committer checklist - [ ] Has a corresponding JIRA in PR title & commit - [ ] Commit message is descriptive of the change - [ ] CI is green - [ ] Necessary doc changes done or have another open PR - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #2643: DO NOT MERGE (Azure CI) test branch ci
hudi-bot edited a comment on pull request #2643: URL: https://github.com/apache/hudi/pull/2643#issuecomment-792368481 ## CI report: * 9831a6c50e9f49f8a71c02fc6ac50ae1446f7c1f UNKNOWN * a569dbe9409910fbb83b3764b300574c0e52612e Azure: [FAILURE](https://dev.azure.com/XUSH0012/0ef433cc-d4b4-47cc-b6a1-03d032ef546c/_build/results?buildId=142) * e6e9f1f1554a1474dd6c20338215030cad23a2e0 UNKNOWN * 2a6690a256c8cd8efe9ed2b1984b896fb27ef077 UNKNOWN * d8b7cca55e057a52a2e229d81e8cb52b60dc275f UNKNOWN * 3bce301333cc78194d13a702598b46e04fe9f85f UNKNOWN * f07f345baa450f3fec7eab59caa76b0fbda1e132 UNKNOWN * 869d2ce3fad330af93c1bb3b576824f519c6e68b UNKNOWN * fa86907f7522bc8dbe512d48b5a87e4a6b13f035 UNKNOWN * 4ebe53016ce3e0648992dbe14d04f71a92f116e6 UNKNOWN * 682ae9985f591f6d0c30ee2ef9b159403c1e46de UNKNOWN * d80397fcfeaa2996ab550bcdab4524be7420a364 UNKNOWN * bfe3a803e19540578b94f778f7ba7551db0f86f1 UNKNOWN * a632e58390eb94fcc7e757bd7580780cf184f9a8 UNKNOWN * 2e413d601c80b123269c2fc3fc6aa9a8bd0d746a UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] n3nash commented on issue #2609: [SUPPORT] Presto hudi query slow when compared to parquet
n3nash commented on issue #2609: URL: https://github.com/apache/hudi/issues/2609#issuecomment-827323716 I've filed a ticket here -> https://issues.apache.org/jira/browse/HUDI-1856. Let's move the conversation to the JIRA. Happy to guide you @GaruGaru or @njalan if you would like to upstream the change in Trino. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] n3nash closed issue #2609: [SUPPORT] Presto hudi query slow when compared to parquet
n3nash closed issue #2609: URL: https://github.com/apache/hudi/issues/2609 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (HUDI-1856) Upstream changes made in PrestoDB to eliminate file listing to Trino
Nishith Agarwal created HUDI-1856: - Summary: Upstream changes made in PrestoDB to eliminate file listing to Trino Key: HUDI-1856 URL: https://issues.apache.org/jira/browse/HUDI-1856 Project: Apache Hudi Issue Type: Wish Reporter: Nishith Agarwal inputFormat.getSplits() code was optimized for PrestoDB code base. This change is not implemented / upstreamed in Trino. Additionally, there are other changes that need to be upstreamed in Trino. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HUDI-1856) Upstream changes made in PrestoDB to eliminate file listing to Trino
[ https://issues.apache.org/jira/browse/HUDI-1856?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nishith Agarwal updated HUDI-1856: -- Labels: sev:high sev:triage (was: ) > Upstream changes made in PrestoDB to eliminate file listing to Trino > > > Key: HUDI-1856 > URL: https://issues.apache.org/jira/browse/HUDI-1856 > Project: Apache Hudi > Issue Type: Wish >Reporter: Nishith Agarwal >Priority: Major > Labels: sev:high, sev:triage > > inputFormat.getSplits() code was optimized for PrestoDB code base. This > change is not implemented / upstreamed in Trino. > > Additionally, there are other changes that need to be upstreamed in Trino. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hudi] n3nash commented on issue #2620: [SUPPORT] Performance Tuning: Slow stages (Building Workload Profile & Getting Small files from partitions) during Hudi Writes
n3nash commented on issue #2620: URL: https://github.com/apache/hudi/issues/2620#issuecomment-827321400 @kimberlyamandalu @njalan @codejoyan There are a few problems when using BLOOM_INDEX 1. Depending on the number of entries in the parquet file, if the BLOOM_INDEX num_entries is not configured correctly, it will lead to lots of false positives that results in bloom index spending more time looking up data. You can check the default bloom index entries here -> https://github.com/apache/hudi/blob/5be3997f70415e1752a0b5214f9398880fc8fd1f/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieIndexConfig.java#L47. You can either increase this or use dynamic bloom filter. We are working on adding metrics to emit how many such false positives happened. 2. The BLOOM_INDEX step needs to perform a "listing" of the partitions to find the candidate files. On S3 without `hoodie.metadata.table` being enabled, this listing can take time. Enable the config to eliminate these file listings. 3. Depending on your workload, BLOOM_INDEX could, in some cases not be the ideal choice. For example, if you have updates across all your partitions, then using SIMPLE_INDEX is better since bloom will just do some extra work and then do the work that SIMPLE_INDEX would have done anyways. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] n3nash commented on issue #2623: org.apache.hudi.exception.HoodieDependentSystemUnavailableException:System HBASE unavailable.
n3nash commented on issue #2623: URL: https://github.com/apache/hudi/issues/2623#issuecomment-827317890 @hj2016 Thanks for the update. @root18039532923 Is your issue resolved now ? Were you able to package the jar and try it out -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] n3nash closed issue #2639: [SUPPORT] Spark 3.0.1 upgrade cause severe increase in Hudi write time
n3nash closed issue #2639: URL: https://github.com/apache/hudi/issues/2639 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] n3nash commented on issue #2639: [SUPPORT] Spark 3.0.1 upgrade cause severe increase in Hudi write time
n3nash commented on issue #2639: URL: https://github.com/apache/hudi/issues/2639#issuecomment-827313973 @afeldman1 I think this issue is related to the AWS EMR changes for Hudi 0.6. As @rubenssoto described, the Hudi 0.7 works fine. Closing this ticket, feel free to re-open if issue still persists for you. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] n3nash commented on issue #2641: [SUPPORT]Build Hudi source with spark 3.1.1 compile error
n3nash commented on issue #2641: URL: https://github.com/apache/hudi/issues/2641#issuecomment-827313469 @garyli1019 Are you able to take a look at this one as well ? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] n3nash commented on issue #2647: [SUPPORT] hive2.1.1 MR cannot be queried based on hive partition
n3nash commented on issue #2647: URL: https://github.com/apache/hudi/issues/2647#issuecomment-827313315 @liujinhui1994 Are you able to provide information that @nsivabalan and @bvaradar have asked ? We can help triage this issue, if not, I will close this ticket soon. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HUDI-1855) RejectedExecutionException FutureTask rejected from ThreadPoolExecutor[Terminated...]
[ https://issues.apache.org/jira/browse/HUDI-1855?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17332927#comment-17332927 ] 李建康 commented on HUDI-1855: --- [[SUPPORT]RejectedExecutionException FutureTask rejected from ThreadPoolExecutor[Terminated...] · Issue #2723 · apache/hudi (github.com)|https://github.com/apache/hudi/issues/2723] > RejectedExecutionException FutureTask rejected from > ThreadPoolExecutor[Terminated...] > -- > > Key: HUDI-1855 > URL: https://issues.apache.org/jira/browse/HUDI-1855 > Project: Apache Hudi > Issue Type: Bug > Components: Cleaner >Affects Versions: 0.6.0 >Reporter: 李建康 >Priority: Major > > *Describe the problem you faced* > We use Structured Streaming to subscribe to the data in Kafka, and then write > the data to the hoodie, the program will stop abnormally after running for a > period of time. > *Environment Description* > * Hudi version :0.6.0 > * Spark version :2.4.0 > * Hive version :2.1.1 > * Hadoop version :3.0.0 > * Storage (HDFS/S3/GCS..) :HDFS > * Running on Docker? (yes/no) :no -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Issue Comment Deleted] (HUDI-1855) RejectedExecutionException FutureTask rejected from ThreadPoolExecutor[Terminated...]
[ https://issues.apache.org/jira/browse/HUDI-1855?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] 李建康 updated HUDI-1855: -- Comment: was deleted (was: [[SUPPORT]RejectedExecutionException FutureTask rejected from ThreadPoolExecutor[Terminated...] · Issue #2723 · apache/hudi (github.com)|https://github.com/apache/hudi/issues/2723]) > RejectedExecutionException FutureTask rejected from > ThreadPoolExecutor[Terminated...] > -- > > Key: HUDI-1855 > URL: https://issues.apache.org/jira/browse/HUDI-1855 > Project: Apache Hudi > Issue Type: Bug > Components: Cleaner >Affects Versions: 0.6.0 >Reporter: 李建康 >Priority: Major > > *Describe the problem you faced* > We use Structured Streaming to subscribe to the data in Kafka, and then write > the data to the hoodie, the program will stop abnormally after running for a > period of time. > *Environment Description* > * Hudi version :0.6.0 > * Spark version :2.4.0 > * Hive version :2.1.1 > * Hadoop version :3.0.0 > * Storage (HDFS/S3/GCS..) :HDFS > * Running on Docker? (yes/no) :no -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hudi] n3nash commented on issue #2648: [SUPPORT] a NPE error when reading MOR table in spark datasource
n3nash commented on issue #2648: URL: https://github.com/apache/hudi/issues/2648#issuecomment-827312511 @hushenmin Thanks for the cross comment. Yes, it looks like for _rt view, for indexType = global when there are no base data files, this exception is thrown. We will take this up soon. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HUDI-1855) RejectedExecutionException FutureTask rejected from ThreadPoolExecutor[Terminated...]
[ https://issues.apache.org/jira/browse/HUDI-1855?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17332926#comment-17332926 ] 李建康 commented on HUDI-1855: --- [[SUPPORT]RejectedExecutionException FutureTask rejected from ThreadPoolExecutor[Terminated...] · Issue #2723 · apache/hudi (github.com)|https://github.com/apache/hudi/issues/2723] > RejectedExecutionException FutureTask rejected from > ThreadPoolExecutor[Terminated...] > -- > > Key: HUDI-1855 > URL: https://issues.apache.org/jira/browse/HUDI-1855 > Project: Apache Hudi > Issue Type: Bug > Components: Cleaner >Affects Versions: 0.6.0 >Reporter: 李建康 >Priority: Major > > *Describe the problem you faced* > We use Structured Streaming to subscribe to the data in Kafka, and then write > the data to the hoodie, the program will stop abnormally after running for a > period of time. > *Environment Description* > * Hudi version :0.6.0 > * Spark version :2.4.0 > * Hive version :2.1.1 > * Hadoop version :3.0.0 > * Storage (HDFS/S3/GCS..) :HDFS > * Running on Docker? (yes/no) :no -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Issue Comment Deleted] (HUDI-1855) RejectedExecutionException FutureTask rejected from ThreadPoolExecutor[Terminated...]
[ https://issues.apache.org/jira/browse/HUDI-1855?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] 李建康 updated HUDI-1855: -- Comment: was deleted (was: [[SUPPORT]RejectedExecutionException FutureTask rejected from ThreadPoolExecutor[Terminated...] · Issue #2723 · apache/hudi (github.com)|https://github.com/apache/hudi/issues/2723]) > RejectedExecutionException FutureTask rejected from > ThreadPoolExecutor[Terminated...] > -- > > Key: HUDI-1855 > URL: https://issues.apache.org/jira/browse/HUDI-1855 > Project: Apache Hudi > Issue Type: Bug > Components: Cleaner >Affects Versions: 0.6.0 >Reporter: 李建康 >Priority: Major > > *Describe the problem you faced* > We use Structured Streaming to subscribe to the data in Kafka, and then write > the data to the hoodie, the program will stop abnormally after running for a > period of time. > *Environment Description* > * Hudi version :0.6.0 > * Spark version :2.4.0 > * Hive version :2.1.1 > * Hadoop version :3.0.0 > * Storage (HDFS/S3/GCS..) :HDFS > * Running on Docker? (yes/no) :no -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hudi] n3nash commented on issue #2657: [SUPPORT] SparkSQL/Hive query fails if there are two or more record array fields in MOR table.
n3nash commented on issue #2657: URL: https://github.com/apache/hudi/issues/2657#issuecomment-827312061 @garyli1019 Are you able to pick this up now ? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] liijiankang commented on issue #2723: [SUPPORT]RejectedExecutionException FutureTask rejected from ThreadPoolExecutor[Terminated...]
liijiankang commented on issue #2723: URL: https://github.com/apache/hudi/issues/2723#issuecomment-827312076 @n3nash https://issues.apache.org/jira/browse/HUDI-1855 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (HUDI-1855) RejectedExecutionException FutureTask rejected from ThreadPoolExecutor[Terminated...]
李建康 created HUDI-1855: - Summary: RejectedExecutionException FutureTask rejected from ThreadPoolExecutor[Terminated...] Key: HUDI-1855 URL: https://issues.apache.org/jira/browse/HUDI-1855 Project: Apache Hudi Issue Type: Bug Components: Cleaner Affects Versions: 0.6.0 Reporter: 李建康 *Describe the problem you faced* We use Structured Streaming to subscribe to the data in Kafka, and then write the data to the hoodie, the program will stop abnormally after running for a period of time. *Environment Description* * Hudi version :0.6.0 * Spark version :2.4.0 * Hive version :2.1.1 * Hadoop version :3.0.0 * Storage (HDFS/S3/GCS..) :HDFS * Running on Docker? (yes/no) :no -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hudi] n3nash commented on issue #2680: [SUPPORT]Hive sync error by using run_sync_tool.sh
n3nash commented on issue #2680: URL: https://github.com/apache/hudi/issues/2680#issuecomment-827311874 @ztcheck Since this issue is not reported by other users, my guess is that this has something to do with your environment. If you are able to pin point which jar helped resolved the issue, that will be helpful to provide a longer term fix, if not, we can close this ticket. Let me know -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] n3nash commented on issue #2688: [SUPPORT] Sync to Hive using Metastore
n3nash commented on issue #2688: URL: https://github.com/apache/hudi/issues/2688#issuecomment-827309433 @rubenssoto Gentle ping -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] codecov-commenter edited a comment on pull request #2822: [Hotfix][hudi-sync] Refactor method up to parent-class
codecov-commenter edited a comment on pull request #2822: URL: https://github.com/apache/hudi/pull/2822#issuecomment-821787385 # [Codecov](https://codecov.io/gh/apache/hudi/pull/2822?src=pr=h1_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) Report > Merging [#2822](https://codecov.io/gh/apache/hudi/pull/2822?src=pr=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (afd686c) into [master](https://codecov.io/gh/apache/hudi/commit/ab4a7b0b4afc66d2123c7f63fdab77b925a8a7f1?el=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (ab4a7b0) will **decrease** coverage by `8.97%`. > The diff coverage is `50.00%`. [](https://codecov.io/gh/apache/hudi/pull/2822?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master#2822 +/- ## - Coverage 61.96% 52.99% -8.98% - Complexity 335 3744+3409 Files54 488 +434 Lines 199323525 +21532 Branches235 2501+2266 + Hits 123512467 +11232 - Misses 638 9957+9319 - Partials120 1101 +981 ``` | Flag | Coverage Δ | Complexity Δ | | |---|---|---|---| | hudicli | `39.53% <ø> (?)` | `220.00 <ø> (?)` | | | hudiclient | `∅ <ø> (?)` | `0.00 <ø> (?)` | | | hudicommon | `50.37% <ø> (?)` | `1975.00 <ø> (?)` | | | hudiflink | `59.65% <ø> (?)` | `536.00 <ø> (?)` | | | hudihadoopmr | `33.33% <ø> (?)` | `198.00 <ø> (?)` | | | hudisparkdatasource | `73.33% <ø> (?)` | `237.00 <ø> (?)` | | | hudisync | `46.39% <50.00%> (?)` | `142.00 <0.00> (?)` | | | huditimelineservice | `64.36% <ø> (?)` | `62.00 <ø> (?)` | | | hudiutilities | `69.70% <ø> (+7.73%)` | `374.00 <ø> (+39.00)` | | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/2822?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [...main/java/org/apache/hudi/dla/HoodieDLAClient.java](https://codecov.io/gh/apache/hudi/pull/2822/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1zeW5jL2h1ZGktZGxhLXN5bmMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvZGxhL0hvb2RpZURMQUNsaWVudC5qYXZh) | `0.00% <0.00%> (ø)` | `0.00 <0.00> (?)` | | | [...in/java/org/apache/hudi/hive/HoodieHiveClient.java](https://codecov.io/gh/apache/hudi/pull/2822/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvSG9vZGllSGl2ZUNsaWVudC5qYXZh) | `71.25% <100.00%> (ø)` | `41.00 <0.00> (?)` | | | [...org/apache/hudi/utilities/HoodieClusteringJob.java](https://codecov.io/gh/apache/hudi/pull/2822/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL0hvb2RpZUNsdXN0ZXJpbmdKb2IuamF2YQ==) | `62.50% <0.00%> (-2.72%)` | `9.00% <0.00%> (ø%)` | | | [...apache/hudi/utilities/deltastreamer/DeltaSync.java](https://codecov.io/gh/apache/hudi/pull/2822/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2RlbHRhc3RyZWFtZXIvRGVsdGFTeW5jLmphdmE=) | `71.08% <0.00%> (-0.35%)` | `55.00% <0.00%> (-1.00%)` | | | [...org/apache/hudi/common/model/HoodieFileFormat.java](https://codecov.io/gh/apache/hudi/pull/2822/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL21vZGVsL0hvb2RpZUZpbGVGb3JtYXQuamF2YQ==) | `100.00% <0.00%> (ø)` | `3.00% <0.00%> (?%)` | | |
[GitHub] [hudi] codecov-commenter edited a comment on pull request #2822: [Hotfix][hudi-sync] Refactor method up to parent-class
codecov-commenter edited a comment on pull request #2822: URL: https://github.com/apache/hudi/pull/2822#issuecomment-821787385 # [Codecov](https://codecov.io/gh/apache/hudi/pull/2822?src=pr=h1_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) Report > Merging [#2822](https://codecov.io/gh/apache/hudi/pull/2822?src=pr=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (afd686c) into [master](https://codecov.io/gh/apache/hudi/commit/ab4a7b0b4afc66d2123c7f63fdab77b925a8a7f1?el=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (ab4a7b0) will **decrease** coverage by `1.04%`. > The diff coverage is `50.00%`. [](https://codecov.io/gh/apache/hudi/pull/2822?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master#2822 +/- ## - Coverage 61.96% 60.92% -1.05% - Complexity 335 516 +181 Files54 72 +18 Lines 1993 3204+1211 Branches235 377 +142 + Hits 1235 1952 +717 - Misses 638 1058 +420 - Partials120 194 +74 ``` | Flag | Coverage Δ | Complexity Δ | | |---|---|---|---| | hudisync | `46.39% <50.00%> (?)` | `142.00 <0.00> (?)` | | | hudiutilities | `69.70% <ø> (+7.73%)` | `374.00 <ø> (+39.00)` | | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/2822?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [...main/java/org/apache/hudi/dla/HoodieDLAClient.java](https://codecov.io/gh/apache/hudi/pull/2822/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1zeW5jL2h1ZGktZGxhLXN5bmMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvZGxhL0hvb2RpZURMQUNsaWVudC5qYXZh) | `0.00% <0.00%> (ø)` | `0.00 <0.00> (?)` | | | [...in/java/org/apache/hudi/hive/HoodieHiveClient.java](https://codecov.io/gh/apache/hudi/pull/2822/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvSG9vZGllSGl2ZUNsaWVudC5qYXZh) | `71.25% <100.00%> (ø)` | `41.00 <0.00> (?)` | | | [...org/apache/hudi/utilities/HoodieClusteringJob.java](https://codecov.io/gh/apache/hudi/pull/2822/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL0hvb2RpZUNsdXN0ZXJpbmdKb2IuamF2YQ==) | `62.50% <0.00%> (-2.72%)` | `9.00% <0.00%> (ø%)` | | | [...apache/hudi/utilities/deltastreamer/DeltaSync.java](https://codecov.io/gh/apache/hudi/pull/2822/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2RlbHRhc3RyZWFtZXIvRGVsdGFTeW5jLmphdmE=) | `71.08% <0.00%> (-0.35%)` | `55.00% <0.00%> (-1.00%)` | | | [.../org/apache/hudi/hive/NonPartitionedExtractor.java](https://codecov.io/gh/apache/hudi/pull/2822/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvTm9uUGFydGl0aW9uZWRFeHRyYWN0b3IuamF2YQ==) | `100.00% <0.00%> (ø)` | `2.00% <0.00%> (?%)` | | | [...ache/hudi/hive/HiveMetastoreBasedLockProvider.java](https://codecov.io/gh/apache/hudi/pull/2822/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvSGl2ZU1ldGFzdG9yZUJhc2VkTG9ja1Byb3ZpZGVyLmphdmE=) | `0.00% <0.00%> (ø)` | `0.00% <0.00%> (?%)` | | |
[jira] [Updated] (HUDI-1854) Corrupt blocks in GCS log files
[ https://issues.apache.org/jira/browse/HUDI-1854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nishith Agarwal updated HUDI-1854: -- Labels: sev:critical sev:triage (was: ) > Corrupt blocks in GCS log files > --- > > Key: HUDI-1854 > URL: https://issues.apache.org/jira/browse/HUDI-1854 > Project: Apache Hudi > Issue Type: Bug > Components: Common Core >Reporter: Nishith Agarwal >Priority: Major > Labels: sev:critical, sev:triage > > Details on how to reproduce this can be found here -> > [https://github.com/apache/hudi/issues/2692] > > We need a GCS, google data proc environment to reproduce this. > > [~vburenin] Would you be able to help try out hudi 0.7 and follow the steps > mentioned in this ticket to help reproduce this issue and find the root cause > ? -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HUDI-1854) Corrupt blocks in GCS log files
Nishith Agarwal created HUDI-1854: - Summary: Corrupt blocks in GCS log files Key: HUDI-1854 URL: https://issues.apache.org/jira/browse/HUDI-1854 Project: Apache Hudi Issue Type: Bug Components: Common Core Reporter: Nishith Agarwal Details on how to reproduce this can be found here -> [https://github.com/apache/hudi/issues/2692] We need a GCS, google data proc environment to reproduce this. [~vburenin] Would you be able to help try out hudi 0.7 and follow the steps mentioned in this ticket to help reproduce this issue and find the root cause ? -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hudi] n3nash commented on issue #2723: [SUPPORT]RejectedExecutionException FutureTask rejected from ThreadPoolExecutor[Terminated...]
n3nash commented on issue #2723: URL: https://github.com/apache/hudi/issues/2723#issuecomment-827307848 @liijiankang Does this issue come back when you turn on async cleaning ? If yes, can you file a JIRA ticket and ping back the ticket here so we can look into this ? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] n3nash commented on issue #2774: [SUPPORT] NoSuchMethodError on exposing metrics
n3nash commented on issue #2774: URL: https://github.com/apache/hudi/issues/2774#issuecomment-827306636 @vobukh Closing this ticket since this seems to have resolved your issue. Feel free to re-open if otherwise -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] n3nash closed issue #2774: [SUPPORT] NoSuchMethodError on exposing metrics
n3nash closed issue #2774: URL: https://github.com/apache/hudi/issues/2774 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] codecov-commenter edited a comment on pull request #2822: [Hotfix][hudi-sync] Refactor method up to parent-class
codecov-commenter edited a comment on pull request #2822: URL: https://github.com/apache/hudi/pull/2822#issuecomment-821787385 # [Codecov](https://codecov.io/gh/apache/hudi/pull/2822?src=pr=h1_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) Report > Merging [#2822](https://codecov.io/gh/apache/hudi/pull/2822?src=pr=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (afd686c) into [master](https://codecov.io/gh/apache/hudi/commit/ab4a7b0b4afc66d2123c7f63fdab77b925a8a7f1?el=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (ab4a7b0) will **increase** coverage by `7.73%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/hudi/pull/2822?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master#2822 +/- ## + Coverage 61.96% 69.70% +7.73% - Complexity 335 374 +39 Files54 54 Lines 1993 1997 +4 Branches235 236 +1 + Hits 1235 1392 +157 + Misses 638 473 -165 - Partials120 132 +12 ``` | Flag | Coverage Δ | Complexity Δ | | |---|---|---|---| | hudiutilities | `69.70% <ø> (+7.73%)` | `374.00 <ø> (+39.00)` | | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/2822?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [...org/apache/hudi/utilities/HoodieClusteringJob.java](https://codecov.io/gh/apache/hudi/pull/2822/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL0hvb2RpZUNsdXN0ZXJpbmdKb2IuamF2YQ==) | `62.50% <0.00%> (-2.72%)` | `9.00% <0.00%> (ø%)` | | | [...apache/hudi/utilities/deltastreamer/DeltaSync.java](https://codecov.io/gh/apache/hudi/pull/2822/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2RlbHRhc3RyZWFtZXIvRGVsdGFTeW5jLmphdmE=) | `71.08% <0.00%> (-0.35%)` | `55.00% <0.00%> (-1.00%)` | | | [...s/deltastreamer/HoodieMultiTableDeltaStreamer.java](https://codecov.io/gh/apache/hudi/pull/2822/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2RlbHRhc3RyZWFtZXIvSG9vZGllTXVsdGlUYWJsZURlbHRhU3RyZWFtZXIuamF2YQ==) | `78.52% <0.00%> (+0.13%)` | `19.00% <0.00%> (+1.00%)` | | | [...in/java/org/apache/hudi/utilities/UtilHelpers.java](https://codecov.io/gh/apache/hudi/pull/2822/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL1V0aWxIZWxwZXJzLmphdmE=) | `64.53% <0.00%> (+23.83%)` | `33.00% <0.00%> (+6.00%)` | | | [...he/hudi/utilities/transform/AWSDmsTransformer.java](https://codecov.io/gh/apache/hudi/pull/2822/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3RyYW5zZm9ybS9BV1NEbXNUcmFuc2Zvcm1lci5qYXZh) | `66.66% <0.00%> (+66.66%)` | `2.00% <0.00%> (+2.00%)` | | | [...hudi/utilities/schema/JdbcbasedSchemaProvider.java](https://codecov.io/gh/apache/hudi/pull/2822/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NjaGVtYS9KZGJjYmFzZWRTY2hlbWFQcm92aWRlci5qYXZh) | `72.22% <0.00%> (+72.22%)` | `2.00% <0.00%> (+2.00%)` | | |
[jira] [Updated] (HUDI-1853) Add flink index and compaction options to document
[ https://issues.apache.org/jira/browse/HUDI-1853?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Danny Chen updated HUDI-1853: - Fix Version/s: 0.9.0 > Add flink index and compaction options to document > -- > > Key: HUDI-1853 > URL: https://issues.apache.org/jira/browse/HUDI-1853 > Project: Apache Hudi > Issue Type: Task > Components: Flink Integration >Reporter: Danny Chen >Assignee: Danny Chen >Priority: Major > Fix For: 0.9.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HUDI-1853) Add flink index and compaction options to document
Danny Chen created HUDI-1853: Summary: Add flink index and compaction options to document Key: HUDI-1853 URL: https://issues.apache.org/jira/browse/HUDI-1853 Project: Apache Hudi Issue Type: Task Components: Flink Integration Reporter: Danny Chen -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HUDI-1853) Add flink index and compaction options to document
[ https://issues.apache.org/jira/browse/HUDI-1853?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Danny Chen reassigned HUDI-1853: Assignee: Danny Chen > Add flink index and compaction options to document > -- > > Key: HUDI-1853 > URL: https://issues.apache.org/jira/browse/HUDI-1853 > Project: Apache Hudi > Issue Type: Task > Components: Flink Integration >Reporter: Danny Chen >Assignee: Danny Chen >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hudi] n3nash edited a comment on issue #2874: [SUPPORT] Can not query mor `_rt` table by FlinkSQL Writing .
n3nash edited a comment on issue #2874: URL: https://github.com/apache/hudi/issues/2874#issuecomment-827298091 @danny0405 Thanks for that explanation. This is a hive reader not the spark reader, did you mean hive ? Is there already a JIRA filed for this ? Does all implementation of flink use global index or is there a non-global index implementation for merge on read tables ? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] n3nash edited a comment on issue #2874: [SUPPORT] Can not query mor `_rt` table by FlinkSQL Writing .
n3nash edited a comment on issue #2874: URL: https://github.com/apache/hudi/issues/2874#issuecomment-827298091 @danny0405 Thanks for that explanation. This is a hive path not the spark reader, did you mean hive ? Is there already a JIRA filed for this ? Does all implementation of flink use global index ? Or is there a non-global index implementation for merge on read tables ? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] n3nash commented on issue #2874: [SUPPORT] Can not query mor `_rt` table by FlinkSQL Writing .
n3nash commented on issue #2874: URL: https://github.com/apache/hudi/issues/2874#issuecomment-827298091 @danny0405 Thanks for that explanation. Is there already a JIRA filed for this ? Does all implementation of flink use global index ? Or is there a non-global index implementation for merge on read tables ? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] RocMarshal commented on a change in pull request #2822: [Hotfix][hudi-sync] Refactor method up to parent-class
RocMarshal commented on a change in pull request #2822:
URL: https://github.com/apache/hudi/pull/2822#discussion_r620836739
##
File path:
hudi-sync/hudi-sync-common/src/main/java/org/apache/hudi/sync/common/AbstractSyncHoodieClient.java
##
@@ -29,20 +29,25 @@
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
+import org.apache.hudi.common.util.ValidationUtils;
import org.apache.log4j.LogManager;
import org.apache.log4j.Logger;
import org.apache.parquet.schema.MessageType;
+import java.io.Serializable;
import java.sql.ResultSet;
import java.sql.SQLException;
import java.sql.Statement;
import java.util.List;
import java.util.Map;
+import java.util.Objects;
public abstract class AbstractSyncHoodieClient {
private static final Logger LOG =
LogManager.getLogger(AbstractSyncHoodieClient.class);
+ public static final TypeConverter DECIMAL_TYPE_CONVERTOR = new
TypeConverter() {};
Review comment:
@leesf Done. Thank you for your suggestions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hudi] RocMarshal commented on a change in pull request #2822: [Hotfix][hudi-sync] Refactor method up to parent-class
RocMarshal commented on a change in pull request #2822:
URL: https://github.com/apache/hudi/pull/2822#discussion_r620411008
##
File path:
hudi-sync/hudi-sync-common/src/main/java/org/apache/hudi/sync/common/AbstractSyncHoodieClient.java
##
@@ -29,20 +29,25 @@
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
+import org.apache.hudi.common.util.ValidationUtils;
import org.apache.log4j.LogManager;
import org.apache.log4j.Logger;
import org.apache.parquet.schema.MessageType;
+import java.io.Serializable;
import java.sql.ResultSet;
import java.sql.SQLException;
import java.sql.Statement;
import java.util.List;
import java.util.Map;
+import java.util.Objects;
public abstract class AbstractSyncHoodieClient {
private static final Logger LOG =
LogManager.getLogger(AbstractSyncHoodieClient.class);
+ public static final TypeConverter DECIMAL_TYPE_CONVERTOR = new
TypeConverter() {};
Review comment:
IMO, The current default implementation is the default implementation of
the decimal type convertor. And we would do a special implementation by
extending `TypeConverter` in the specific scene. So, keeping
`DECIMAL_TYPE_CONVERTOR` would be a fine choice in the case...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hudi] codecov-commenter edited a comment on pull request #2884: [HUDI-1852]Add SCHEMA_REGISTRY_SOURCE_URL_SUFFIX and SCHEMA_REGISTRY_TARGET_URL_SUFFIX property
codecov-commenter edited a comment on pull request #2884: URL: https://github.com/apache/hudi/pull/2884#issuecomment-827288106 # [Codecov](https://codecov.io/gh/apache/hudi/pull/2884?src=pr=h1_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) Report > Merging [#2884](https://codecov.io/gh/apache/hudi/pull/2884?src=pr=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (6f9f3ec) into [master](https://codecov.io/gh/apache/hudi/commit/9bbb458e888273af95b04f72cc5b428524b41487?el=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (9bbb458) will **increase** coverage by `0.04%`. > The diff coverage is `0.00%`. [](https://codecov.io/gh/apache/hudi/pull/2884?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master#2884 +/- ## + Coverage 52.93% 52.97% +0.04% - Complexity 3742 3745 +3 Files 488 488 Lines 2351123544 +33 Branches 2501 2504 +3 + Hits 1244612473 +27 - Misses 9966 9971 +5 - Partials 1099 1100 +1 ``` | Flag | Coverage Δ | Complexity Δ | | |---|---|---|---| | hudicli | `39.53% <ø> (ø)` | `220.00 <ø> (ø)` | | | hudiclient | `∅ <ø> (∅)` | `0.00 <ø> (ø)` | | | hudicommon | `50.38% <ø> (+0.01%)` | `1975.00 <ø> (ø)` | | | hudiflink | `59.65% <ø> (+0.19%)` | `536.00 <ø> (+2.00)` | | | hudihadoopmr | `33.33% <ø> (ø)` | `198.00 <ø> (ø)` | | | hudisparkdatasource | `73.33% <ø> (ø)` | `237.00 <ø> (ø)` | | | hudisync | `46.10% <ø> (ø)` | `142.00 <ø> (ø)` | | | huditimelineservice | `64.36% <ø> (ø)` | `62.00 <ø> (ø)` | | | hudiutilities | `69.58% <0.00%> (-0.13%)` | `375.00 <0.00> (+1.00)` | :arrow_down: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/2884?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [...s/deltastreamer/HoodieMultiTableDeltaStreamer.java](https://codecov.io/gh/apache/hudi/pull/2884/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2RlbHRhc3RyZWFtZXIvSG9vZGllTXVsdGlUYWJsZURlbHRhU3RyZWFtZXIuamF2YQ==) | `76.19% <0.00%> (-2.34%)` | `19.00 <0.00> (ø)` | | | [...java/org/apache/hudi/sink/StreamWriteFunction.java](https://codecov.io/gh/apache/hudi/pull/2884/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1mbGluay9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaHVkaS9zaW5rL1N0cmVhbVdyaXRlRnVuY3Rpb24uamF2YQ==) | `79.24% <0.00%> (-1.68%)` | `24.00% <0.00%> (ø%)` | | | [...va/org/apache/hudi/configuration/FlinkOptions.java](https://codecov.io/gh/apache/hudi/pull/2884/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1mbGluay9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaHVkaS9jb25maWd1cmF0aW9uL0ZsaW5rT3B0aW9ucy5qYXZh) | `90.48% <0.00%> (+0.28%)` | `11.00% <0.00%> (ø%)` | | | [...apache/hudi/utilities/deltastreamer/DeltaSync.java](https://codecov.io/gh/apache/hudi/pull/2884/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2RlbHRhc3RyZWFtZXIvRGVsdGFTeW5jLmphdmE=) | `71.42% <0.00%> (+0.34%)` | `56.00% <0.00%> (+1.00%)` | | | [...c/main/java/org/apache/hudi/util/StreamerUtil.java](https://codecov.io/gh/apache/hudi/pull/2884/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1mbGluay9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaHVkaS91dGlsL1N0cmVhbWVyVXRpbC5qYXZh) | `58.33% <0.00%> (+0.35%)` | `18.00% <0.00%> (ø%)` | | |
[GitHub] [hudi] codecov-commenter edited a comment on pull request #2884: [HUDI-1852]Add SCHEMA_REGISTRY_SOURCE_URL_SUFFIX and SCHEMA_REGISTRY_TARGET_URL_SUFFIX property
codecov-commenter edited a comment on pull request #2884: URL: https://github.com/apache/hudi/pull/2884#issuecomment-827288106 # [Codecov](https://codecov.io/gh/apache/hudi/pull/2884?src=pr=h1_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) Report > Merging [#2884](https://codecov.io/gh/apache/hudi/pull/2884?src=pr=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (6f9f3ec) into [master](https://codecov.io/gh/apache/hudi/commit/9bbb458e888273af95b04f72cc5b428524b41487?el=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (9bbb458) will **increase** coverage by `0.04%`. > The diff coverage is `0.00%`. [](https://codecov.io/gh/apache/hudi/pull/2884?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master#2884 +/- ## + Coverage 52.93% 52.97% +0.04% - Complexity 3742 3745 +3 Files 488 488 Lines 2351123544 +33 Branches 2501 2504 +3 + Hits 1244612473 +27 - Misses 9966 9971 +5 - Partials 1099 1100 +1 ``` | Flag | Coverage Δ | Complexity Δ | | |---|---|---|---| | hudicli | `39.53% <ø> (ø)` | `220.00 <ø> (ø)` | | | hudiclient | `∅ <ø> (∅)` | `0.00 <ø> (ø)` | | | hudicommon | `50.38% <ø> (+0.01%)` | `1975.00 <ø> (ø)` | | | hudiflink | `59.65% <ø> (+0.19%)` | `536.00 <ø> (+2.00)` | | | hudihadoopmr | `33.33% <ø> (ø)` | `198.00 <ø> (ø)` | | | hudisparkdatasource | `73.33% <ø> (ø)` | `237.00 <ø> (ø)` | | | hudisync | `46.10% <ø> (ø)` | `142.00 <ø> (ø)` | | | huditimelineservice | `64.36% <ø> (ø)` | `62.00 <ø> (ø)` | | | hudiutilities | `69.58% <0.00%> (-0.13%)` | `375.00 <0.00> (+1.00)` | :arrow_down: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/2884?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [...s/deltastreamer/HoodieMultiTableDeltaStreamer.java](https://codecov.io/gh/apache/hudi/pull/2884/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2RlbHRhc3RyZWFtZXIvSG9vZGllTXVsdGlUYWJsZURlbHRhU3RyZWFtZXIuamF2YQ==) | `76.19% <0.00%> (-2.34%)` | `19.00 <0.00> (ø)` | | | [...java/org/apache/hudi/sink/StreamWriteFunction.java](https://codecov.io/gh/apache/hudi/pull/2884/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1mbGluay9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaHVkaS9zaW5rL1N0cmVhbVdyaXRlRnVuY3Rpb24uamF2YQ==) | `79.24% <0.00%> (-1.68%)` | `24.00% <0.00%> (ø%)` | | | [...va/org/apache/hudi/configuration/FlinkOptions.java](https://codecov.io/gh/apache/hudi/pull/2884/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1mbGluay9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaHVkaS9jb25maWd1cmF0aW9uL0ZsaW5rT3B0aW9ucy5qYXZh) | `90.48% <0.00%> (+0.28%)` | `11.00% <0.00%> (ø%)` | | | [...apache/hudi/utilities/deltastreamer/DeltaSync.java](https://codecov.io/gh/apache/hudi/pull/2884/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2RlbHRhc3RyZWFtZXIvRGVsdGFTeW5jLmphdmE=) | `71.42% <0.00%> (+0.34%)` | `56.00% <0.00%> (+1.00%)` | | | [...c/main/java/org/apache/hudi/util/StreamerUtil.java](https://codecov.io/gh/apache/hudi/pull/2884/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1mbGluay9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaHVkaS91dGlsL1N0cmVhbWVyVXRpbC5qYXZh) | `58.33% <0.00%> (+0.35%)` | `18.00% <0.00%> (ø%)` | | |
[GitHub] [hudi] codecov-commenter edited a comment on pull request #2884: [HUDI-1852]Add SCHEMA_REGISTRY_SOURCE_URL_SUFFIX and SCHEMA_REGISTRY_TARGET_URL_SUFFIX property
codecov-commenter edited a comment on pull request #2884: URL: https://github.com/apache/hudi/pull/2884#issuecomment-827288106 # [Codecov](https://codecov.io/gh/apache/hudi/pull/2884?src=pr=h1_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) Report > Merging [#2884](https://codecov.io/gh/apache/hudi/pull/2884?src=pr=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (6f9f3ec) into [master](https://codecov.io/gh/apache/hudi/commit/9bbb458e888273af95b04f72cc5b428524b41487?el=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (9bbb458) will **decrease** coverage by `1.64%`. > The diff coverage is `0.00%`. [](https://codecov.io/gh/apache/hudi/pull/2884?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master#2884 +/- ## - Coverage 52.93% 51.29% -1.65% + Complexity 3742 3304 -438 Files 488 425 -63 Lines 2351120055-3456 Branches 2501 2084 -417 - Hits 1244610287-2159 + Misses 9966 8914-1052 + Partials 1099 854 -245 ``` | Flag | Coverage Δ | Complexity Δ | | |---|---|---|---| | hudicli | `39.53% <ø> (ø)` | `220.00 <ø> (ø)` | | | hudiclient | `∅ <ø> (∅)` | `0.00 <ø> (ø)` | | | hudicommon | `50.38% <ø> (+0.01%)` | `1975.00 <ø> (ø)` | | | hudiflink | `59.65% <ø> (+0.19%)` | `536.00 <ø> (+2.00)` | | | hudihadoopmr | `33.33% <ø> (ø)` | `198.00 <ø> (ø)` | | | hudisparkdatasource | `?` | `?` | | | hudisync | `?` | `?` | | | huditimelineservice | `?` | `?` | | | hudiutilities | `69.58% <0.00%> (-0.13%)` | `375.00 <0.00> (+1.00)` | :arrow_down: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/2884?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [...s/deltastreamer/HoodieMultiTableDeltaStreamer.java](https://codecov.io/gh/apache/hudi/pull/2884/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2RlbHRhc3RyZWFtZXIvSG9vZGllTXVsdGlUYWJsZURlbHRhU3RyZWFtZXIuamF2YQ==) | `76.19% <0.00%> (-2.34%)` | `19.00 <0.00> (ø)` | | | [...java/org/apache/hudi/sink/StreamWriteFunction.java](https://codecov.io/gh/apache/hudi/pull/2884/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1mbGluay9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaHVkaS9zaW5rL1N0cmVhbVdyaXRlRnVuY3Rpb24uamF2YQ==) | `79.24% <0.00%> (-1.68%)` | `24.00% <0.00%> (ø%)` | | | [.../src/main/java/org/apache/hudi/dla/util/Utils.java](https://codecov.io/gh/apache/hudi/pull/2884/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1zeW5jL2h1ZGktZGxhLXN5bmMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvZGxhL3V0aWwvVXRpbHMuamF2YQ==) | | | | | [...g/apache/hudi/timeline/service/RequestHandler.java](https://codecov.io/gh/apache/hudi/pull/2884/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS10aW1lbGluZS1zZXJ2aWNlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL3RpbWVsaW5lL3NlcnZpY2UvUmVxdWVzdEhhbmRsZXIuamF2YQ==) | | | | | [...src/main/java/org/apache/hudi/dla/DLASyncTool.java](https://codecov.io/gh/apache/hudi/pull/2884/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1zeW5jL2h1ZGktZGxhLXN5bmMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvZGxhL0RMQVN5bmNUb29sLmphdmE=) | | | | |
[jira] [Resolved] (HUDI-1415) Read Hoodie Table As Spark DataSource Table
[
https://issues.apache.org/jira/browse/HUDI-1415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
pengzhiwei resolved HUDI-1415.
--
Resolution: Fixed
> Read Hoodie Table As Spark DataSource Table
>
>
> Key: HUDI-1415
> URL: https://issues.apache.org/jira/browse/HUDI-1415
> Project: Apache Hudi
> Issue Type: Improvement
> Components: Spark Integration
>Affects Versions: 0.9.0
>Reporter: pengzhiwei
>Assignee: pengzhiwei
>Priority: Major
> Labels: pull-request-available, user-support-issues
> Fix For: 0.9.0
>
>
> Currently hudi can sync the meta data to hive meta store using HiveSyncTool.
> The table description synced to hive just like this:
> {code:java}
> CREATE EXTERNAL TABLE `tbl_price_insert0`(
> `_hoodie_commit_time` string,
> `_hoodie_commit_seqno` string,
> `_hoodie_record_key` string,
> `_hoodie_partition_path` string,
> `_hoodie_file_name` string,
> `id` int,
> `name` string,
> `price` double,
> `version` int,
> `dt` string)
> ROW FORMAT SERDE
> 'org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe'
> STORED AS INPUTFORMAT
> 'org.apache.hudi.hadoop.HoodieParquetInputFormat'
> OUTPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat'
> LOCATION
> 'file:/tmp/hudi/tbl_price_insert0'
> TBLPROPERTIES (
> 'last_commit_time_sync'='20201124105009',
> 'transient_lastDdlTime'='1606186222')
> {code}
> When we query this table using spark sql, it trait it as a Hive Table, not a
> spark data source table and convert it to parquet LogicalRelation in
> HiveStrategies#RelationConversions. As a result, spark sql read the hudi
> table just like a parquet data source. This lead to an incorrect query
> result if user missing set the spark.sql.hive.convertMetastoreParquet=false.
> Inorder to query hudi table as data source table in spark, more table
> properties and serde properties must be added to the hive meta,just like the
> follow:
> {code:java}
> CREATE EXTERNAL TABLE `tbl_price_cow0`(
> `_hoodie_commit_time` string,
> `_hoodie_commit_seqno` string,
> `_hoodie_record_key` string,
> `_hoodie_partition_path` string,
> `_hoodie_file_name` string,
> `id` int,
> `name` string,
> `price` double,
> `version` int)
> ROW FORMAT SERDE
> 'org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe'
> WITH SERDEPROPERTIES (
> 'path'='/tmp/hudi/tbl_price_cow0')
> STORED AS INPUTFORMAT
> 'org.apache.hudi.hadoop.HoodieParquetInputFormat'
> OUTPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat'
> LOCATION
> 'file:/tmp/hudi/tbl_price_cow0'
> TBLPROPERTIES (
> 'last_commit_time_sync'='20201124120532',
> 'spark.sql.sources.provider'='hudi',
> 'spark.sql.sources.schema.numParts'='1',
>
> 'spark.sql.sources.schema.part.0'='{"type":"struct","fields":[{"name":"id","type":"integer","nullable":false,"metadata":{}},{"name":"name","type":"string","nullable":true,"metadata":{}},{"name":"price","type":"double","nullable":false,"metadata":{}},{"name":"version","type":"integer","nullable":false,"metadata":{}}]}',
>
> 'transient_lastDdlTime'='1606190729')
> {code}
> These are the missing table properties:
> {code:java}
> spark.sql.sources.provider= 'hudi'
> spark.sql.sources.schema.numParts = 'xx'
> spark.sql.sources.schema.part.{num} ='xx'
> spark.sql.sources.schema.numPartCols = 'xx'
> spark.sql.sources.schema.partCol.{num} = 'xx'{code}
> and serde property:
> {code:java}
> 'path'='/path/to/hudi'
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hudi] codecov-commenter commented on pull request #2884: [HUDI-1852]Add SCHEMA_REGISTRY_SOURCE_URL_SUFFIX and SCHEMA_REGISTRY_TARGET_URL_SUFFIX property
codecov-commenter commented on pull request #2884: URL: https://github.com/apache/hudi/pull/2884#issuecomment-827288106 # [Codecov](https://codecov.io/gh/apache/hudi/pull/2884?src=pr=h1_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) Report > Merging [#2884](https://codecov.io/gh/apache/hudi/pull/2884?src=pr=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (6f9f3ec) into [master](https://codecov.io/gh/apache/hudi/commit/9bbb458e888273af95b04f72cc5b428524b41487?el=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (9bbb458) will **increase** coverage by `16.64%`. > The diff coverage is `0.00%`. [](https://codecov.io/gh/apache/hudi/pull/2884?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master#2884 +/- ## = + Coverage 52.93% 69.58% +16.64% + Complexity 3742 375 -3367 = Files 488 54 -434 Lines 23511 2002-21509 Branches 2501 237 -2264 = - Hits 12446 1393-11053 + Misses 9966 478 -9488 + Partials 1099 131 -968 ``` | Flag | Coverage Δ | Complexity Δ | | |---|---|---|---| | hudicli | `?` | `?` | | | hudiclient | `?` | `?` | | | hudicommon | `?` | `?` | | | hudiflink | `?` | `?` | | | hudihadoopmr | `?` | `?` | | | hudisparkdatasource | `?` | `?` | | | hudisync | `?` | `?` | | | huditimelineservice | `?` | `?` | | | hudiutilities | `69.58% <0.00%> (-0.13%)` | `375.00 <0.00> (+1.00)` | :arrow_down: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/2884?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [...s/deltastreamer/HoodieMultiTableDeltaStreamer.java](https://codecov.io/gh/apache/hudi/pull/2884/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2RlbHRhc3RyZWFtZXIvSG9vZGllTXVsdGlUYWJsZURlbHRhU3RyZWFtZXIuamF2YQ==) | `76.19% <0.00%> (-2.34%)` | `19.00 <0.00> (ø)` | | | [.../java/org/apache/hudi/HoodieDataSourceHelpers.java](https://codecov.io/gh/apache/hudi/pull/2884/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1zcGFyay1kYXRhc291cmNlL2h1ZGktc3Bhcmsvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvSG9vZGllRGF0YVNvdXJjZUhlbHBlcnMuamF2YQ==) | | | | | [.../org/apache/hudi/hadoop/utils/HoodieHiveUtils.java](https://codecov.io/gh/apache/hudi/pull/2884/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1oYWRvb3AtbXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvaGFkb29wL3V0aWxzL0hvb2RpZUhpdmVVdGlscy5qYXZh) | | | | | [.../hudi/common/bloom/InternalDynamicBloomFilter.java](https://codecov.io/gh/apache/hudi/pull/2884/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL2Jsb29tL0ludGVybmFsRHluYW1pY0Jsb29tRmlsdGVyLmphdmE=) | | | | | [.../java/org/apache/hudi/common/util/RateLimiter.java](https://codecov.io/gh/apache/hudi/pull/2884/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL3V0aWwvUmF0ZUxpbWl0ZXIuamF2YQ==) | | | | |
[GitHub] [hudi] hudi-bot edited a comment on pull request #2643: DO NOT MERGE (Azure CI) test branch ci
hudi-bot edited a comment on pull request #2643: URL: https://github.com/apache/hudi/pull/2643#issuecomment-792368481 ## CI report: * 9831a6c50e9f49f8a71c02fc6ac50ae1446f7c1f UNKNOWN * a569dbe9409910fbb83b3764b300574c0e52612e Azure: [FAILURE](https://dev.azure.com/XUSH0012/0ef433cc-d4b4-47cc-b6a1-03d032ef546c/_build/results?buildId=142) * e6e9f1f1554a1474dd6c20338215030cad23a2e0 UNKNOWN * 2a6690a256c8cd8efe9ed2b1984b896fb27ef077 UNKNOWN * d8b7cca55e057a52a2e229d81e8cb52b60dc275f UNKNOWN * 3bce301333cc78194d13a702598b46e04fe9f85f UNKNOWN * f07f345baa450f3fec7eab59caa76b0fbda1e132 UNKNOWN * 869d2ce3fad330af93c1bb3b576824f519c6e68b UNKNOWN * fa86907f7522bc8dbe512d48b5a87e4a6b13f035 UNKNOWN * 4ebe53016ce3e0648992dbe14d04f71a92f116e6 UNKNOWN * 682ae9985f591f6d0c30ee2ef9b159403c1e46de UNKNOWN * d80397fcfeaa2996ab550bcdab4524be7420a364 UNKNOWN * bfe3a803e19540578b94f778f7ba7551db0f86f1 UNKNOWN * a632e58390eb94fcc7e757bd7580780cf184f9a8 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HUDI-1852) Add SCHEMA_REGISTRY_SOURCE_URL_SUFFIX SCHEMA_REGISTRY_TARGET_URL_SUFFIX property
[ https://issues.apache.org/jira/browse/HUDI-1852?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HUDI-1852: - Labels: pull-request-available (was: ) > Add SCHEMA_REGISTRY_SOURCE_URL_SUFFIX SCHEMA_REGISTRY_TARGET_URL_SUFFIX > property > > > Key: HUDI-1852 > URL: https://issues.apache.org/jira/browse/HUDI-1852 > Project: Apache Hudi > Issue Type: New Feature > Components: Utilities >Reporter: NickYoung >Priority: Major > Labels: pull-request-available > Fix For: 0.9.0 > > > Before, only schemaRegistrySuffix did not distinguish between > targetSchemaRegistrySuffix and sourceSchemaRegistrySuffix, which is not > suitable for our needs. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hudi] NickYoungPeng opened a new pull request #2884: [HUDI-1852]Add SCHEMA_REGISTRY_SOURCE_URL_SUFFIX SCHEMA_REGISTRY_TARGET_URL_SUFFIX property
NickYoungPeng opened a new pull request #2884: URL: https://github.com/apache/hudi/pull/2884 ## What is the purpose of the pull request *(For example: This pull request adds quick-start document.)* ## Brief change log Before, only schemaRegistrySuffix did not distinguish between targetSchemaRegistrySuffix and sourceSchemaRegistrySuffix, which is not suitable for our needs. ## Verify this pull request This pull request is a trivial rework / code cleanup without any test coverage. ## Committer checklist - [ ] Has a corresponding JIRA in PR title & commit - [ ] Commit message is descriptive of the change - [ ] CI is green - [ ] Necessary doc changes done or have another open PR - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Closed] (HUDI-1841) Tweak the min max commits to keep when setting up cleaning retain commits for Flink
[ https://issues.apache.org/jira/browse/HUDI-1841?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] vinoyang closed HUDI-1841. -- Resolution: Done 5be3997f70415e1752a0b5214f9398880fc8fd1f > Tweak the min max commits to keep when setting up cleaning retain commits for > Flink > --- > > Key: HUDI-1841 > URL: https://issues.apache.org/jira/browse/HUDI-1841 > Project: Apache Hudi > Issue Type: Improvement > Components: Flink Integration >Reporter: Danny Chen >Assignee: Danny Chen >Priority: Major > Labels: pull-request-available > Fix For: 0.9.0 > > > If the retain commits is greater that min commits to keep, reset the options: > 1. min commits to keep: retain commits + 10 > 2. max commits to keep: retain commits + 20 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hudi] yanghua merged pull request #2875: [HUDI-1841] Tweak the min max commits to keep when setting up cleanin…
yanghua merged pull request #2875: URL: https://github.com/apache/hudi/pull/2875 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[hudi] branch master updated (9bbb458 -> 5be3997)
This is an automated email from the ASF dual-hosted git repository. vinoyang pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/hudi.git. from 9bbb458 [MINOR] Remove redundant method-calling. (#2881) add 5be3997 [HUDI-1841] Tweak the min max commits to keep when setting up cleaning retain commits for Flink (#2875) No new revisions were added by this update. Summary of changes: .../apache/hudi/configuration/FlinkOptions.java| 12 + .../org/apache/hudi/sink/StreamWriteFunction.java | 36 +- .../org/apache/hudi/table/HoodieTableFactory.java | 18 +++ .../java/org/apache/hudi/util/StreamerUtil.java| 1 + .../apache/hudi/table/TestHoodieTableFactory.java | 56 ++ 5 files changed, 112 insertions(+), 11 deletions(-)
[jira] [Updated] (HUDI-1852) Add SCHEMA_REGISTRY_SOURCE_URL_SUFFIX SCHEMA_REGISTRY_TARGET_URL_SUFFIX property
[ https://issues.apache.org/jira/browse/HUDI-1852?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] NickYoung updated HUDI-1852: Summary: Add SCHEMA_REGISTRY_SOURCE_URL_SUFFIX SCHEMA_REGISTRY_TARGET_URL_SUFFIX property (was: Add SCHEMA_REGISTRY_SOURCE_URL_SUFFIX SCHEMA_REGISTRY_TARGET_URL_SUFFIX variable) > Add SCHEMA_REGISTRY_SOURCE_URL_SUFFIX SCHEMA_REGISTRY_TARGET_URL_SUFFIX > property > > > Key: HUDI-1852 > URL: https://issues.apache.org/jira/browse/HUDI-1852 > Project: Apache Hudi > Issue Type: New Feature > Components: Utilities >Reporter: NickYoung >Priority: Major > Fix For: 0.9.0 > > > Before, only schemaRegistrySuffix did not distinguish between > targetSchemaRegistrySuffix and sourceSchemaRegistrySuffix, which is not > suitable for our needs. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HUDI-1852) Add SCHEMA_REGISTRY_SOURCE_URL_SUFFIX SCHEMA_REGISTRY_TARGET_URL_SUFFIX variable
NickYoung created HUDI-1852: --- Summary: Add SCHEMA_REGISTRY_SOURCE_URL_SUFFIX SCHEMA_REGISTRY_TARGET_URL_SUFFIX variable Key: HUDI-1852 URL: https://issues.apache.org/jira/browse/HUDI-1852 Project: Apache Hudi Issue Type: New Feature Components: Utilities Reporter: NickYoung Fix For: 0.9.0 Before, only schemaRegistrySuffix did not distinguish between targetSchemaRegistrySuffix and sourceSchemaRegistrySuffix, which is not suitable for our needs. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HUDI-1742) improve table level config priority in HoodieMultiTableDeltaStreamer
[ https://issues.apache.org/jira/browse/HUDI-1742?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] NickYoung resolved HUDI-1742. - Fix Version/s: 0.9.0 Resolution: Done > improve table level config priority in HoodieMultiTableDeltaStreamer > > > Key: HUDI-1742 > URL: https://issues.apache.org/jira/browse/HUDI-1742 > Project: Apache Hudi > Issue Type: Wish > Components: DeltaStreamer >Reporter: NickYoung >Priority: Major > Labels: pull-request-available > Fix For: 0.9.0 > > > I hope that when the table-level configuration file and the public l > configuration file have the same configuration, the table-level configuration > file configuration is used。 > But now if the table-level configuration file and the public configuration > file have the same configuration, the configuration in the public > configuration file will be adopted。 > https://hudi.apache.org/blog/ingest-multiple-tables-using-hudi/ -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hudi] codecov-commenter edited a comment on pull request #2882: [HUDI-1806] Honoring skipROSuffix in spark ds
codecov-commenter edited a comment on pull request #2882: URL: https://github.com/apache/hudi/pull/2882#issuecomment-827054323 # [Codecov](https://codecov.io/gh/apache/hudi/pull/2882?src=pr=h1_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) Report > Merging [#2882](https://codecov.io/gh/apache/hudi/pull/2882?src=pr=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (7999033) into [master](https://codecov.io/gh/apache/hudi/commit/3e4fa170cfd2c198599c3bed3982f2f643c7fbe8?el=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (3e4fa17) will **increase** coverage by `8.77%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/hudi/pull/2882?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master#2882 +/- ## + Coverage 53.06% 61.84% +8.77% + Complexity 3746 335-3411 Files 488 54 -434 Lines 23466 1997 -21469 Branches 2503 236-2267 - Hits 12453 1235 -11218 + Misses 9914 640-9274 + Partials 1099 122 -977 ``` | Flag | Coverage Δ | Complexity Δ | | |---|---|---|---| | hudicli | `?` | `?` | | | hudiclient | `∅ <ø> (∅)` | `0.00 <ø> (ø)` | | | hudicommon | `?` | `?` | | | hudiflink | `?` | `?` | | | hudihadoopmr | `?` | `?` | | | hudisparkdatasource | `?` | `?` | | | hudisync | `?` | `?` | | | huditimelineservice | `?` | `?` | | | hudiutilities | `61.84% <ø> (-7.90%)` | `335.00 <ø> (-39.00)` | | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/2882?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [...ies/exception/HoodieSnapshotExporterException.java](https://codecov.io/gh/apache/hudi/pull/2882/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2V4Y2VwdGlvbi9Ib29kaWVTbmFwc2hvdEV4cG9ydGVyRXhjZXB0aW9uLmphdmE=) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-1.00%)` | | | [.../apache/hudi/utilities/HoodieSnapshotExporter.java](https://codecov.io/gh/apache/hudi/pull/2882/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL0hvb2RpZVNuYXBzaG90RXhwb3J0ZXIuamF2YQ==) | `5.17% <0.00%> (-83.63%)` | `0.00% <0.00%> (-28.00%)` | | | [...hudi/utilities/schema/JdbcbasedSchemaProvider.java](https://codecov.io/gh/apache/hudi/pull/2882/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NjaGVtYS9KZGJjYmFzZWRTY2hlbWFQcm92aWRlci5qYXZh) | `0.00% <0.00%> (-72.23%)` | `0.00% <0.00%> (-2.00%)` | | | [...he/hudi/utilities/transform/AWSDmsTransformer.java](https://codecov.io/gh/apache/hudi/pull/2882/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3RyYW5zZm9ybS9BV1NEbXNUcmFuc2Zvcm1lci5qYXZh) | `0.00% <0.00%> (-66.67%)` | `0.00% <0.00%> (-2.00%)` | | | [...in/java/org/apache/hudi/utilities/UtilHelpers.java](https://codecov.io/gh/apache/hudi/pull/2882/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL1V0aWxIZWxwZXJzLmphdmE=) | `40.69% <0.00%> (-23.84%)` | `27.00% <0.00%> (-6.00%)` | | |
[hudi] branch master updated (f4e3b94 -> 9bbb458)
This is an automated email from the ASF dual-hosted git repository. vinoyang pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/hudi.git. from f4e3b94 [HUDI-1742] Improve table level config priority for HoodieMultiTableDeltaStreamer (#2744) add 9bbb458 [MINOR] Remove redundant method-calling. (#2881) No new revisions were added by this update. Summary of changes: .../org/apache/hudi/table/action/commit/BaseCommitActionExecutor.java | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
[GitHub] [hudi] yanghua merged pull request #2881: [MINOR] Remove redundant method-calling.
yanghua merged pull request #2881: URL: https://github.com/apache/hudi/pull/2881 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HUDI-1817) when query incr view of hudi table by using spark-sql. the result is wrong
[
https://issues.apache.org/jira/browse/HUDI-1817?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
sivabalan narayanan updated HUDI-1817:
--
Labels: pull-request-available sev:critical (was: pull-request-available)
> when query incr view of hudi table by using spark-sql. the result is wrong
> --
>
> Key: HUDI-1817
> URL: https://issues.apache.org/jira/browse/HUDI-1817
> Project: Apache Hudi
> Issue Type: Bug
> Components: Hive Integration
>Affects Versions: 0.8.0
> Environment: spark2.4.5 hive 3.1.1 hadoop 3.1.1
>Reporter: tao meng
>Priority: Major
> Labels: pull-request-available, sev:critical
> Fix For: 0.9.0
>
>
> create hudi table (mor or cow)
>
> val base_data = spark.read.parquet("/tmp/tb_base")
> val upsert_data = spark.read.parquet("/tmp/tb_upsert")
> base_data.write.format("hudi").option(TABLE_TYPE_OPT_KEY,
> MOR_TABLE_TYPE_OPT_VAL).option(PRECOMBINE_FIELD_OPT_KEY,
> "col2").option(RECORDKEY_FIELD_OPT_KEY,
> "primary_key").option(PARTITIONPATH_FIELD_OPT_KEY,
> "col0").option(KEYGENERATOR_CLASS_OPT_KEY,
> "org.apache.hudi.keygen.SimpleKeyGenerator").option(OPERATION_OPT_KEY,
> "bulk_insert").option(HIVE_SYNC_ENABLED_OPT_KEY,
> "true").option(HIVE_PARTITION_FIELDS_OPT_KEY,
> "col0").option(HIVE_PARTITION_EXTRACTOR_CLASS_OPT_KEY,
> "org.apache.hudi.hive.MultiPartKeysValueExtractor").option(HIVE_DATABASE_OPT_KEY,
> "testdb").option(HIVE_TABLE_OPT_KEY,
> "tb_test_mor_par").option(HIVE_USE_JDBC_OPT_KEY,
> "false").option("hoodie.bulkinsert.shuffle.parallelism",
> 4).option("hoodie.insert.shuffle.parallelism",
> 4).option("hoodie.upsert.shuffle.parallelism",
> 4).option("hoodie.delete.shuffle.parallelism",
> 4).option("hoodie.datasource.write.hive_style_partitioning",
> "true").option(TABLE_NAME,
> "tb_test_mor_par").mode(Overwrite).save(s"/tmp/testdb/tb_test_mor_par")
> upsert_data.write.format("hudi").option(TABLE_TYPE_OPT_KEY,
> MOR_TABLE_TYPE_OPT_VAL).option(PRECOMBINE_FIELD_OPT_KEY,
> "col2").option(RECORDKEY_FIELD_OPT_KEY,
> "primary_key").option(PARTITIONPATH_FIELD_OPT_KEY,
> "col0").option(KEYGENERATOR_CLASS_OPT_KEY,
> "org.apache.hudi.keygen.SimpleKeyGenerator").option(OPERATION_OPT_KEY,
> "upsert").option(HIVE_SYNC_ENABLED_OPT_KEY,
> "true").option(HIVE_PARTITION_FIELDS_OPT_KEY,
> "col0").option(HIVE_PARTITION_EXTRACTOR_CLASS_OPT_KEY,
> "org.apache.hudi.hive.MultiPartKeysValueExtractor").option(HIVE_DATABASE_OPT_KEY,
> "testdb").option(HIVE_TABLE_OPT_KEY,
> "tb_test_mor_par").option(HIVE_USE_JDBC_OPT_KEY,
> "false").option("hoodie.bulkinsert.shuffle.parallelism",
> 4).option("hoodie.insert.shuffle.parallelism",
> 4).option("hoodie.upsert.shuffle.parallelism",
> 4).option("hoodie.delete.shuffle.parallelism",
> 4).option("hoodie.datasource.write.hive_style_partitioning",
> "true").option(TABLE_NAME,
> "tb_test_mor_par").mode(Append).save(s"/tmp/testdb/tb_test_mor_par")
> query incr view by sparksql:
> set hoodie.tb_test_mor_par.consume.mode=INCREMENTAL;
> set hoodie.tb_test_mor_par.consume.start.timestamp=20210420145330;
> set hoodie.tb_test_mor_par.consume.max.commits=3;
> select
> _hoodie_commit_time,primary_key,col0,col1,col2,col3,col4,col5,col6,col7 from
> testdb.tb_test_mor_par_rt where _hoodie_commit_time > '20210420145330' order
> by primary_key;
> +---+---+++++
> |_hoodie_commit_time|primary_key|col0|col1|col6 |col7|
> +---+---+++++
> |20210420155738 |20 |77 |sC |158788760400|739 |
> |20210420155738 |21 |66 |ps |160979049700|61 |
> |20210420155738 |22 |47 |1P |158460042900|835 |
> |20210420155738 |23 |36 |5K |160763480800|538 |
> |20210420155738 |24 |1 |BA |160685711300|775 |
> |20210420155738 |24 |101 |BA |160685711300|775 |
> |20210420155738 |24 |100 |BA |160685711300|775 |
> |20210420155738 |24 |102 |BA |160685711300|775 |
> +---+---+++++
>
> the primary_key is repeated.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HUDI-1833) pending cluster rollback when there are greater commits
[ https://issues.apache.org/jira/browse/HUDI-1833?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-1833: -- Labels: pull-request-available sev:high (was: pull-request-available) > pending cluster rollback when there are greater commits > --- > > Key: HUDI-1833 > URL: https://issues.apache.org/jira/browse/HUDI-1833 > Project: Apache Hudi > Issue Type: Sub-task >Reporter: satish >Assignee: satish >Priority: Major > Labels: pull-request-available, sev:high > > pending clustering can be rolled back even if there are greater commits. Fix > rollback logic -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HUDI-1845) Exception Throws When Sync Non-Partitioned Table To Hive With MultiPartKeysValueExtractor
[
https://issues.apache.org/jira/browse/HUDI-1845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
sivabalan narayanan updated HUDI-1845:
--
Labels: pull-request-available sev:critical (was: pull-request-available)
> Exception Throws When Sync Non-Partitioned Table To Hive With
> MultiPartKeysValueExtractor
> ---
>
> Key: HUDI-1845
> URL: https://issues.apache.org/jira/browse/HUDI-1845
> Project: Apache Hudi
> Issue Type: Bug
> Components: Hive Integration
>Reporter: pengzhiwei
>Assignee: pengzhiwei
>Priority: Major
> Labels: pull-request-available, sev:critical
>
> When sync a non-partitioned table to hive with MultiPartKeysValueExtractor,
> There is an exception will throw out.
> {code:java}
> IllegalArgumentException: Partition key parts [] does not match with
> partition values{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HUDI-1850) Read on table fails if the first write to table failed
[
https://issues.apache.org/jira/browse/HUDI-1850?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
sivabalan narayanan updated HUDI-1850:
--
Labels: sev:critical sev:high spark (was: sev:critical sev:triage spark)
> Read on table fails if the first write to table failed
> --
>
> Key: HUDI-1850
> URL: https://issues.apache.org/jira/browse/HUDI-1850
> Project: Apache Hudi
> Issue Type: Bug
>Affects Versions: 0.8.0
>Reporter: Vaibhav Sinha
>Priority: Major
> Labels: sev:critical, sev:high, spark
> Attachments: Screenshot 2021-04-24 at 7.53.22 PM.png
>
>
> {code:java}
> ava.util.NoSuchElementException: No value present in Option
> at org.apache.hudi.common.util.Option.get(Option.java:88)
> ~[hudi-spark3-bundle_2.12-0.8.0.jar:0.8.0]
> at
> org.apache.hudi.common.table.TableSchemaResolver.getTableSchemaFromCommitMetadata(TableSchemaResolver.java:215)
> ~[hudi-spark3-bundle_2.12-0.8.0.jar:0.8.0]
> at
> org.apache.hudi.common.table.TableSchemaResolver.getTableAvroSchema(TableSchemaResolver.java:166)
> ~[hudi-spark3-bundle_2.12-0.8.0.jar:0.8.0]
> at
> org.apache.hudi.common.table.TableSchemaResolver.getTableAvroSchema(TableSchemaResolver.java:155)
> ~[hudi-spark3-bundle_2.12-0.8.0.jar:0.8.0]
> at
> org.apache.hudi.MergeOnReadSnapshotRelation.(MergeOnReadSnapshotRelation.scala:65)
> ~[hudi-spark3-bundle_2.12-0.8.0.jar:0.8.0]
> at org.apache.hudi.DefaultSource.createRelation(DefaultSource.scala:99)
> ~[hudi-spark3-bundle_2.12-0.8.0.jar:0.8.0]
> at org.apache.hudi.DefaultSource.createRelation(DefaultSource.scala:63)
> ~[hudi-spark3-bundle_2.12-0.8.0.jar:0.8.0]
> at
> org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:354)
> ~[spark-sql_2.12-3.1.1.jar:3.1.1]
> at
> org.apache.spark.sql.DataFrameReader.loadV1Source(DataFrameReader.scala:326)
> ~[spark-sql_2.12-3.1.1.jar:3.1.1]
> at
> org.apache.spark.sql.DataFrameReader.$anonfun$load$3(DataFrameReader.scala:308)
> ~[spark-sql_2.12-3.1.1.jar:3.1.1]
> at scala.Option.getOrElse(Option.scala:189)
> ~[scala-library-2.12.10.jar:?]
> at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:308)
> ~[spark-sql_2.12-3.1.1.jar:3.1.1]
> at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:240)
> ~[spark-sql_2.12-3.1.1.jar:3.1.1]
> {code}
> The screenshot shows the files that got created before the write had failed.
>
> !Screenshot 2021-04-24 at 7.53.22 PM.png!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HUDI-1806) Honor "skipROSuffix" in spark ds
[ https://issues.apache.org/jira/browse/HUDI-1806?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-1806: -- Labels: pull-request-available sev:normal (was: pull-request-available sev:critical) > Honor "skipROSuffix" in spark ds > > > Key: HUDI-1806 > URL: https://issues.apache.org/jira/browse/HUDI-1806 > Project: Apache Hudi > Issue Type: Bug > Components: Hive Integration >Reporter: sivabalan narayanan >Priority: Major > Labels: pull-request-available, sev:normal > Fix For: 0.9.0 > > Original Estimate: 1m > Remaining Estimate: 1m > > In HoodieSparkSqlWriter#buildSyncConfig(), we don't set skipROSuffix based on > configs. This needs fixing. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hudi] cdmikechen commented on issue #2544: [SUPPORT]failed to read timestamp column in version 0.7.0 even when HIVE_SUPPORT_TIMESTAMP is enabled
cdmikechen commented on issue #2544: URL: https://github.com/apache/hudi/issues/2544#issuecomment-827219576 @nsivabalan Sure ~ I'll deal with it next week. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Comment Edited] (HUDI-1138) Re-implement marker files via timeline server
[ https://issues.apache.org/jira/browse/HUDI-1138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17332798#comment-17332798 ] Vinoth Chandar edited comment on HUDI-1138 at 4/26/21, 10:48 PM: - [~balajeeUber] For the issue you faced with tasks still running during finalization, we also block any running tasks from creating new files, once a commit is about to be finalized. This will be a more general solution that is very nice, since we will be using timeline server anyway. cc [~nagarwal] [~nagarwal] [~guoyihua] also expressed interested in doing this btw. So lets please decide soon, who is going to take this :) was (Author: vc): [~balajeeUber] We also block any running tasks from creating new files, once a commit is about to be finalized. This will be a more general solution that is very nice, since we will be using timeline server anyway. cc [~nagarwal] [~nagarwal] [~guoyihua] also expressed interested in doing this btw. So lets please decide soon, who is going to take this :) > Re-implement marker files via timeline server > - > > Key: HUDI-1138 > URL: https://issues.apache.org/jira/browse/HUDI-1138 > Project: Apache Hudi > Issue Type: Improvement > Components: Writer Core >Affects Versions: 0.9.0 >Reporter: Vinoth Chandar >Priority: Blocker > Fix For: 0.9.0 > > > Even as you can argue that RFC-15/consolidated metadata, removes the need for > deleting partial files written due to spark task failures/stage retries. It > will still leave extra files inside the table (and users will pay for it > every month) and we need the marker mechanism to be able to delete these > partial files. > Here we explore if we can improve the current marker file mechanism, that > creates one marker file per data file written, by > Delegating the createMarker() call to the driver/timeline server, and have it > create marker metadata into a single file handle, that is flushed for > durability guarantees > > P.S: I was tempted to think Spark listener mechanism can help us deal with > failed tasks, but it has no guarantees. the writer job could die without > deleting a partial file. i.e it can improve things, but cant provide > guarantees -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HUDI-1138) Re-implement marker files via timeline server
[ https://issues.apache.org/jira/browse/HUDI-1138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17332798#comment-17332798 ] Vinoth Chandar commented on HUDI-1138: -- [~balajeeUber] We also block any running tasks from creating new files, once a commit is about to be finalized. This will be a more general solution that is very nice, since we will be using timeline server anyway. cc [~nagarwal] [~nagarwal] [~guoyihua] also expressed interested in doing this btw. So lets please decide soon, who is going to take this :) > Re-implement marker files via timeline server > - > > Key: HUDI-1138 > URL: https://issues.apache.org/jira/browse/HUDI-1138 > Project: Apache Hudi > Issue Type: Improvement > Components: Writer Core >Affects Versions: 0.9.0 >Reporter: Vinoth Chandar >Priority: Blocker > Fix For: 0.9.0 > > > Even as you can argue that RFC-15/consolidated metadata, removes the need for > deleting partial files written due to spark task failures/stage retries. It > will still leave extra files inside the table (and users will pay for it > every month) and we need the marker mechanism to be able to delete these > partial files. > Here we explore if we can improve the current marker file mechanism, that > creates one marker file per data file written, by > Delegating the createMarker() call to the driver/timeline server, and have it > create marker metadata into a single file handle, that is flushed for > durability guarantees > > P.S: I was tempted to think Spark listener mechanism can help us deal with > failed tasks, but it has no guarantees. the writer job could die without > deleting a partial file. i.e it can improve things, but cant provide > guarantees -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HUDI-1850) Read on table fails if the first write to table failed
[
https://issues.apache.org/jira/browse/HUDI-1850?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nishith Agarwal updated HUDI-1850:
--
Labels: sev:critical sev:triage spark (was: )
> Read on table fails if the first write to table failed
> --
>
> Key: HUDI-1850
> URL: https://issues.apache.org/jira/browse/HUDI-1850
> Project: Apache Hudi
> Issue Type: Bug
>Affects Versions: 0.8.0
>Reporter: Vaibhav Sinha
>Priority: Major
> Labels: sev:critical, sev:triage, spark
> Attachments: Screenshot 2021-04-24 at 7.53.22 PM.png
>
>
> {code:java}
> ava.util.NoSuchElementException: No value present in Option
> at org.apache.hudi.common.util.Option.get(Option.java:88)
> ~[hudi-spark3-bundle_2.12-0.8.0.jar:0.8.0]
> at
> org.apache.hudi.common.table.TableSchemaResolver.getTableSchemaFromCommitMetadata(TableSchemaResolver.java:215)
> ~[hudi-spark3-bundle_2.12-0.8.0.jar:0.8.0]
> at
> org.apache.hudi.common.table.TableSchemaResolver.getTableAvroSchema(TableSchemaResolver.java:166)
> ~[hudi-spark3-bundle_2.12-0.8.0.jar:0.8.0]
> at
> org.apache.hudi.common.table.TableSchemaResolver.getTableAvroSchema(TableSchemaResolver.java:155)
> ~[hudi-spark3-bundle_2.12-0.8.0.jar:0.8.0]
> at
> org.apache.hudi.MergeOnReadSnapshotRelation.(MergeOnReadSnapshotRelation.scala:65)
> ~[hudi-spark3-bundle_2.12-0.8.0.jar:0.8.0]
> at org.apache.hudi.DefaultSource.createRelation(DefaultSource.scala:99)
> ~[hudi-spark3-bundle_2.12-0.8.0.jar:0.8.0]
> at org.apache.hudi.DefaultSource.createRelation(DefaultSource.scala:63)
> ~[hudi-spark3-bundle_2.12-0.8.0.jar:0.8.0]
> at
> org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:354)
> ~[spark-sql_2.12-3.1.1.jar:3.1.1]
> at
> org.apache.spark.sql.DataFrameReader.loadV1Source(DataFrameReader.scala:326)
> ~[spark-sql_2.12-3.1.1.jar:3.1.1]
> at
> org.apache.spark.sql.DataFrameReader.$anonfun$load$3(DataFrameReader.scala:308)
> ~[spark-sql_2.12-3.1.1.jar:3.1.1]
> at scala.Option.getOrElse(Option.scala:189)
> ~[scala-library-2.12.10.jar:?]
> at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:308)
> ~[spark-sql_2.12-3.1.1.jar:3.1.1]
> at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:240)
> ~[spark-sql_2.12-3.1.1.jar:3.1.1]
> {code}
> The screenshot shows the files that got created before the write had failed.
>
> !Screenshot 2021-04-24 at 7.53.22 PM.png!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hudi] jiegzhan opened a new issue #2883: [SUPPORT] Does Hudi support flink DataStream API?
jiegzhan opened a new issue #2883:
URL: https://github.com/apache/hudi/issues/2883
Besides
[flink-sql-client](https://hudi.apache.org/docs/flink-quick-start-guide.html#setup),
does hudi support [flink DataStream
API](https://ci.apache.org/projects/flink/flink-docs-stable/dev/connectors/kafka.html#kafka-consumer)?
```
// Read kafka topic with DataStream API
val props = new Properties()
props.put("bootstrap.servers", "localhost:9092")
props.put("group.id", "flink-kafka-consumer")
val consumer = new FlinkKafkaConsumer[String]("device_logs", new
SimpleStringSchema(), props)
val stream = senv.addSource(consumer)
```
Given DataStream[String] above ^^^, is there a way to publish it to hudi
table? If yes, any scala/java code snippet? Thanks for your help.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hudi] codecov-commenter commented on pull request #2882: [HUDI-1806] Honoring skipROSuffix in spark ds
codecov-commenter commented on pull request #2882: URL: https://github.com/apache/hudi/pull/2882#issuecomment-827054323 # [Codecov](https://codecov.io/gh/apache/hudi/pull/2882?src=pr=h1_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) Report > Merging [#2882](https://codecov.io/gh/apache/hudi/pull/2882?src=pr=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (7999033) into [master](https://codecov.io/gh/apache/hudi/commit/3e4fa170cfd2c198599c3bed3982f2f643c7fbe8?el=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (3e4fa17) will **increase** coverage by `8.77%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/hudi/pull/2882?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master#2882 +/- ## + Coverage 53.06% 61.84% +8.77% + Complexity 3746 335-3411 Files 488 54 -434 Lines 23466 1997 -21469 Branches 2503 236-2267 - Hits 12453 1235 -11218 + Misses 9914 640-9274 + Partials 1099 122 -977 ``` | Flag | Coverage Δ | Complexity Δ | | |---|---|---|---| | hudicli | `?` | `?` | | | hudiclient | `?` | `?` | | | hudicommon | `?` | `?` | | | hudiflink | `?` | `?` | | | hudihadoopmr | `?` | `?` | | | hudisparkdatasource | `?` | `?` | | | hudisync | `?` | `?` | | | huditimelineservice | `?` | `?` | | | hudiutilities | `61.84% <ø> (-7.90%)` | `335.00 <ø> (-39.00)` | | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/2882?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [...ies/exception/HoodieSnapshotExporterException.java](https://codecov.io/gh/apache/hudi/pull/2882/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2V4Y2VwdGlvbi9Ib29kaWVTbmFwc2hvdEV4cG9ydGVyRXhjZXB0aW9uLmphdmE=) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-1.00%)` | | | [.../apache/hudi/utilities/HoodieSnapshotExporter.java](https://codecov.io/gh/apache/hudi/pull/2882/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL0hvb2RpZVNuYXBzaG90RXhwb3J0ZXIuamF2YQ==) | `5.17% <0.00%> (-83.63%)` | `0.00% <0.00%> (-28.00%)` | | | [...hudi/utilities/schema/JdbcbasedSchemaProvider.java](https://codecov.io/gh/apache/hudi/pull/2882/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NjaGVtYS9KZGJjYmFzZWRTY2hlbWFQcm92aWRlci5qYXZh) | `0.00% <0.00%> (-72.23%)` | `0.00% <0.00%> (-2.00%)` | | | [...he/hudi/utilities/transform/AWSDmsTransformer.java](https://codecov.io/gh/apache/hudi/pull/2882/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3RyYW5zZm9ybS9BV1NEbXNUcmFuc2Zvcm1lci5qYXZh) | `0.00% <0.00%> (-66.67%)` | `0.00% <0.00%> (-2.00%)` | | | [...in/java/org/apache/hudi/utilities/UtilHelpers.java](https://codecov.io/gh/apache/hudi/pull/2882/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL1V0aWxIZWxwZXJzLmphdmE=) | `40.69% <0.00%> (-23.84%)` | `27.00% <0.00%> (-6.00%)` | | |
[GitHub] [hudi] satishkotha commented on a change in pull request #2809: [HUDI-1789] Support reading older snapshots
satishkotha commented on a change in pull request #2809:
URL: https://github.com/apache/hudi/pull/2809#discussion_r620508454
##
File path:
hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/utils/HoodieHiveUtils.java
##
@@ -122,19 +135,48 @@ public static Path getNthParent(Path path, int n) {
return result;
}
+ /**
+ * Depending on the configs hoodie.%s.consume.pending.commits and
hoodie.%s.consume.commit of job
+ *
+ * (hoodie..consume.pending.commits,
hoodie..consume.commit) ->
+ * (true, validCommit) -> returns activeTimeline filtered until
validCommit
+ * (true, InValidCommit) -> Raises HoodieIOException
+ * (true, notSet)-> Raises HoodieIOException
+ * (false, validCommit) -> returns compeltedTimeline filtered
until validCommit
+ * (false, InValidCommit)-> Raises HoodieIOException
+ * (false or notSet, notSet) -> returns completedTimeline unfiltered
+ *
+ * validCommit is one which exists in the timeline being checked and
vice versa
+ *
+ * @param tableName
+ * @param job
+ * @param metaClient
+ * @return
+ */
public static HoodieTimeline getTableTimeline(final String tableName, final
JobConf job, final HoodieTableMetaClient metaClient) {
+HoodieTimeline timeline =
metaClient.getActiveTimeline().getCommitsTimeline();
+
boolean includePendingCommits =
job.getBoolean(String.format(HOODIE_CONSUME_PENDING_COMMITS, tableName), false);
+String maxCommit = job.get(String.format(HOODIE_CONSUME_COMMIT,
tableName));
+
if (includePendingCommits) {
- HoodieTimeline timeline =
metaClient.getActiveTimeline().getCommitsTimeline();
- String maxCommit = job.get(String.format(HOODIE_CONSUME_COMMIT,
tableName));
- if (maxCommit == null || !timeline.containsInstant(maxCommit)) {
-LOG.info("Timestamp configured for validation: " + maxCommit + "
commits timeline:" + timeline + " table: " + tableName);
-throw new HoodieIOException("Valid timestamp is required for " +
HOODIE_CONSUME_COMMIT + " in validate mode");
- }
- return timeline.findInstantsBeforeOrEquals(maxCommit);
+ return filterIfInstantExists(tableName, timeline, maxCommit);
+}
+
+timeline = timeline.filterCompletedInstants();
+if (maxCommit != null) {
+ return filterIfInstantExists(tableName, timeline, maxCommit);
}
// by default return all completed commits.
-return
metaClient.getActiveTimeline().getCommitsTimeline().filterCompletedInstants();
+return timeline;
+ }
+
+ private static HoodieTimeline filterIfInstantExists(String tableName,
HoodieTimeline timeline, String maxCommit) {
+if (maxCommit == null || !timeline.containsInstant(maxCommit)) {
+ LOG.info("Timestamp configured for validation: " + maxCommit + " commits
timeline:" + timeline + " table: " + tableName);
Review comment:
minor: could you update log message and below error message (This is no
longer related to validate?)
##
File path:
hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/utils/HoodieInputFormatUtils.java
##
@@ -438,11 +437,20 @@ public static HoodieMetadataConfig
buildMetadataConfig(Configuration conf) {
if (LOG.isDebugEnabled()) {
LOG.debug("Hoodie Metadata initialized with completed commit instant
as :" + metaClient);
}
-
HoodieTimeline timeline =
HoodieHiveUtils.getTableTimeline(metaClient.getTableConfig().getTableName(),
job, metaClient);
+
Review comment:
minor: this file seems to only have new line changes, can we reset this
file to look like master (unless you have strong opinion on new lines here)?
##
File path:
hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/utils/HoodieHiveUtils.java
##
@@ -122,19 +135,48 @@ public static Path getNthParent(Path path, int n) {
return result;
}
+ /**
+ * Depending on the configs hoodie.%s.consume.pending.commits and
hoodie.%s.consume.commit of job
+ *
+ * (hoodie..consume.pending.commits,
hoodie..consume.commit) ->
+ * (true, validCommit) -> returns activeTimeline filtered until
validCommit
+ * (true, InValidCommit) -> Raises HoodieIOException
+ * (true, notSet)-> Raises HoodieIOException
+ * (false, validCommit) -> returns compeltedTimeline filtered
until validCommit
+ * (false, InValidCommit)-> Raises HoodieIOException
+ * (false or notSet, notSet) -> returns completedTimeline unfiltered
+ *
+ * validCommit is one which exists in the timeline being checked and
vice versa
+ *
+ * @param tableName
+ * @param job
+ * @param metaClient
+ * @return
+ */
public static HoodieTimeline getTableTimeline(final String tableName, final
JobConf job, final HoodieTableMetaClient metaClient) {
+HoodieTimeline timeline =
metaClient.getActiveTimeline().getCommitsTimeline();
+
[jira] [Updated] (HUDI-1806) Honor "skipROSuffix" in spark ds
[ https://issues.apache.org/jira/browse/HUDI-1806?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HUDI-1806: - Labels: pull-request-available sev:critical (was: sev:critical) > Honor "skipROSuffix" in spark ds > > > Key: HUDI-1806 > URL: https://issues.apache.org/jira/browse/HUDI-1806 > Project: Apache Hudi > Issue Type: Bug > Components: Hive Integration >Reporter: sivabalan narayanan >Priority: Major > Labels: pull-request-available, sev:critical > Fix For: 0.9.0 > > Original Estimate: 1m > Remaining Estimate: 1m > > In HoodieSparkSqlWriter#buildSyncConfig(), we don't set skipROSuffix based on > configs. This needs fixing. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hudi] nsivabalan opened a new pull request #2882: [HUDI-1806] Honoring skipROSuffix in spark ds
nsivabalan opened a new pull request #2882: URL: https://github.com/apache/hudi/pull/2882 # What is the purpose of the pull request *Honoring skipROSuffix in spark ds ## Brief change log - Honoring skipROSuffix in spark ds ## Verify this pull request This change added tests and can be verified as follows: - Tested via HoodieSparkSqlWriterSuite.Test build sync config for spark sql ## Committer checklist - [ ] Has a corresponding JIRA in PR title & commit - [ ] Commit message is descriptive of the change - [ ] CI is green - [ ] Necessary doc changes done or have another open PR - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] vinothchandar commented on pull request #2645: [HUDI-1659] Basic Implementation Of Spark Sql Support
vinothchandar commented on pull request #2645: URL: https://github.com/apache/hudi/pull/2645#issuecomment-826999206 @pengzhiwei2018 Will re-review this again today. Also have a test env now. So will start running some tests as well and report results here. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] codecov-commenter edited a comment on pull request #2881: [Hotfix][minor] Remove redundant method-calling.
codecov-commenter edited a comment on pull request #2881: URL: https://github.com/apache/hudi/pull/2881#issuecomment-826968742 # [Codecov](https://codecov.io/gh/apache/hudi/pull/2881?src=pr=h1_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) Report > Merging [#2881](https://codecov.io/gh/apache/hudi/pull/2881?src=pr=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (4632790) into [master](https://codecov.io/gh/apache/hudi/commit/f4e3b949714aceaf8823fd1659e44d3b7e98089a?el=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (f4e3b94) will **increase** coverage by `16.75%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/hudi/pull/2881?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master#2881 +/- ## = + Coverage 52.94% 69.70% +16.75% + Complexity 3743 374 -3369 = Files 488 54 -434 Lines 23511 1997-21514 Branches 2501 236 -2265 = - Hits 12449 1392-11057 + Misses 9965 473 -9492 + Partials 1097 132 -965 ``` | Flag | Coverage Δ | Complexity Δ | | |---|---|---|---| | hudicli | `?` | `?` | | | hudiclient | `?` | `?` | | | hudicommon | `?` | `?` | | | hudiflink | `?` | `?` | | | hudihadoopmr | `?` | `?` | | | hudisparkdatasource | `?` | `?` | | | hudisync | `?` | `?` | | | huditimelineservice | `?` | `?` | | | hudiutilities | `69.70% <ø> (-0.06%)` | `374.00 <ø> (-1.00)` | | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/2881?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [...apache/hudi/utilities/deltastreamer/DeltaSync.java](https://codecov.io/gh/apache/hudi/pull/2881/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2RlbHRhc3RyZWFtZXIvRGVsdGFTeW5jLmphdmE=) | `71.08% <0.00%> (-0.35%)` | `55.00% <0.00%> (-1.00%)` | | | [.../apache/hudi/hadoop/RecordReaderValueIterator.java](https://codecov.io/gh/apache/hudi/pull/2881/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1oYWRvb3AtbXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvaGFkb29wL1JlY29yZFJlYWRlclZhbHVlSXRlcmF0b3IuamF2YQ==) | | | | | [...main/java/org/apache/hudi/hive/HiveSyncConfig.java](https://codecov.io/gh/apache/hudi/pull/2881/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvSGl2ZVN5bmNDb25maWcuamF2YQ==) | | | | | [...i/common/table/view/FileSystemViewStorageType.java](https://codecov.io/gh/apache/hudi/pull/2881/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL3RhYmxlL3ZpZXcvRmlsZVN5c3RlbVZpZXdTdG9yYWdlVHlwZS5qYXZh) | | | | | [.../hadoop/realtime/RealtimeUnmergedRecordReader.java](https://codecov.io/gh/apache/hudi/pull/2881/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1oYWRvb3AtbXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvaGFkb29wL3JlYWx0aW1lL1JlYWx0aW1lVW5tZXJnZWRSZWNvcmRSZWFkZXIuamF2YQ==) | | | | | [.../org/apache/hudi/common/model/HoodieFileGroup.java](https://codecov.io/gh/apache/hudi/pull/2881/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL21vZGVsL0hvb2RpZUZpbGVHcm91cC5qYXZh) | | | | |
[GitHub] [hudi] codecov-commenter edited a comment on pull request #2880: [HUDI-1851] Automate suite of tests suites w/ docker and integ test
codecov-commenter edited a comment on pull request #2880: URL: https://github.com/apache/hudi/pull/2880#issuecomment-826968162 # [Codecov](https://codecov.io/gh/apache/hudi/pull/2880?src=pr=h1_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) Report > Merging [#2880](https://codecov.io/gh/apache/hudi/pull/2880?src=pr=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (718fa28) into [master](https://codecov.io/gh/apache/hudi/commit/a5789c40673b36d40adab696706acc1446a286f8?el=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (a5789c4) will **decrease** coverage by `0.06%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/hudi/pull/2880?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master#2880 +/- ## - Coverage 53.00% 52.93% -0.07% + Complexity 3743 3742 -1 Files 488 488 Lines 2343523511 +76 Branches 2500 2501 +1 + Hits 1242212446 +24 - Misses 9913 9966 +53 + Partials 1100 1099 -1 ``` | Flag | Coverage Δ | Complexity Δ | | |---|---|---|---| | hudicli | `39.53% <ø> (ø)` | `220.00 <ø> (ø)` | | | hudiclient | `∅ <ø> (∅)` | `0.00 <ø> (ø)` | | | hudicommon | `50.37% <ø> (-0.29%)` | `1975.00 <ø> (-1.00)` | | | hudiflink | `59.45% <ø> (+0.33%)` | `534.00 <ø> (-1.00)` | :arrow_up: | | hudihadoopmr | `33.33% <ø> (ø)` | `198.00 <ø> (ø)` | | | hudisparkdatasource | `73.33% <ø> (ø)` | `237.00 <ø> (ø)` | | | hudisync | `46.10% <ø> (ø)` | `142.00 <ø> (ø)` | | | huditimelineservice | `64.36% <ø> (ø)` | `62.00 <ø> (ø)` | | | hudiutilities | `69.70% <ø> (+0.01%)` | `374.00 <ø> (+1.00)` | | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/2880?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [...common/table/log/HoodieMergedLogRecordScanner.java](https://codecov.io/gh/apache/hudi/pull/2880/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL3RhYmxlL2xvZy9Ib29kaWVNZXJnZWRMb2dSZWNvcmRTY2FubmVyLmphdmE=) | `82.35% <0.00%> (-5.15%)` | `9.00% <0.00%> (-1.00%)` | | | [...mmon/table/log/AbstractHoodieLogRecordScanner.java](https://codecov.io/gh/apache/hudi/pull/2880/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL3RhYmxlL2xvZy9BYnN0cmFjdEhvb2RpZUxvZ1JlY29yZFNjYW5uZXIuamF2YQ==) | `80.00% <0.00%> (-1.62%)` | `34.00% <0.00%> (ø%)` | | | [...pache/hudi/metadata/HoodieBackedTableMetadata.java](https://codecov.io/gh/apache/hudi/pull/2880/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvbWV0YWRhdGEvSG9vZGllQmFja2VkVGFibGVNZXRhZGF0YS5qYXZh) | `0.00% <0.00%> (ø)` | `0.00% <0.00%> (ø%)` | | | [...e/hudi/table/format/mor/MergeOnReadInputSplit.java](https://codecov.io/gh/apache/hudi/pull/2880/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1mbGluay9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaHVkaS90YWJsZS9mb3JtYXQvbW9yL01lcmdlT25SZWFkSW5wdXRTcGxpdC5qYXZh) | `94.44% <0.00%> (ø)` | `8.00% <0.00%> (ø%)` | | | [...metadata/HoodieMetadataMergedLogRecordScanner.java](https://codecov.io/gh/apache/hudi/pull/2880/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvbWV0YWRhdGEvSG9vZGllTWV0YWRhdGFNZXJnZWRMb2dSZWNvcmRTY2FubmVyLmphdmE=) | `0.00%
[GitHub] [hudi] codecov-commenter commented on pull request #2881: [Hotfix][minor] Remove redundant method-calling.
codecov-commenter commented on pull request #2881: URL: https://github.com/apache/hudi/pull/2881#issuecomment-826968742 # [Codecov](https://codecov.io/gh/apache/hudi/pull/2881?src=pr=h1_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) Report > Merging [#2881](https://codecov.io/gh/apache/hudi/pull/2881?src=pr=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (4632790) into [master](https://codecov.io/gh/apache/hudi/commit/f4e3b949714aceaf8823fd1659e44d3b7e98089a?el=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (f4e3b94) will **decrease** coverage by `43.58%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/hudi/pull/2881?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master #2881 +/- ## - Coverage 52.94% 9.36% -43.59% + Complexity 3743 48 -3695 Files 488 54 -434 Lines 235111997-21514 Branches 2501 236 -2265 - Hits 12449 187-12262 + Misses 99651797 -8168 + Partials 1097 13 -1084 ``` | Flag | Coverage Δ | Complexity Δ | | |---|---|---|---| | hudicli | `?` | `?` | | | hudiclient | `?` | `?` | | | hudicommon | `?` | `?` | | | hudiflink | `?` | `?` | | | hudihadoopmr | `?` | `?` | | | hudisparkdatasource | `?` | `?` | | | hudisync | `?` | `?` | | | huditimelineservice | `?` | `?` | | | hudiutilities | `9.36% <ø> (-60.40%)` | `48.00 <ø> (-327.00)` | | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/2881?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [...va/org/apache/hudi/utilities/IdentitySplitter.java](https://codecov.io/gh/apache/hudi/pull/2881/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL0lkZW50aXR5U3BsaXR0ZXIuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-2.00%)` | | | [...va/org/apache/hudi/utilities/schema/SchemaSet.java](https://codecov.io/gh/apache/hudi/pull/2881/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NjaGVtYS9TY2hlbWFTZXQuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-3.00%)` | | | [...a/org/apache/hudi/utilities/sources/RowSource.java](https://codecov.io/gh/apache/hudi/pull/2881/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvUm93U291cmNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-4.00%)` | | | [.../org/apache/hudi/utilities/sources/AvroSource.java](https://codecov.io/gh/apache/hudi/pull/2881/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvQXZyb1NvdXJjZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-1.00%)` | | | [.../org/apache/hudi/utilities/sources/JsonSource.java](https://codecov.io/gh/apache/hudi/pull/2881/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvblNvdXJjZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-1.00%)` | | |
[GitHub] [hudi] codecov-commenter edited a comment on pull request #2880: [HUDI-1851] Automate suite of tests suites w/ docker and integ test
codecov-commenter edited a comment on pull request #2880: URL: https://github.com/apache/hudi/pull/2880#issuecomment-826968162 # [Codecov](https://codecov.io/gh/apache/hudi/pull/2880?src=pr=h1_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) Report > Merging [#2880](https://codecov.io/gh/apache/hudi/pull/2880?src=pr=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (718fa28) into [master](https://codecov.io/gh/apache/hudi/commit/a5789c40673b36d40adab696706acc1446a286f8?el=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (a5789c4) will **decrease** coverage by `0.06%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/hudi/pull/2880?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master#2880 +/- ## - Coverage 53.00% 52.93% -0.07% + Complexity 3743 3742 -1 Files 488 488 Lines 2343523511 +76 Branches 2500 2501 +1 + Hits 1242212446 +24 - Misses 9913 9966 +53 + Partials 1100 1099 -1 ``` | Flag | Coverage Δ | Complexity Δ | | |---|---|---|---| | hudicli | `39.53% <ø> (ø)` | `220.00 <ø> (ø)` | | | hudiclient | `∅ <ø> (∅)` | `0.00 <ø> (ø)` | | | hudicommon | `50.37% <ø> (-0.29%)` | `1975.00 <ø> (-1.00)` | | | hudiflink | `59.45% <ø> (+0.33%)` | `534.00 <ø> (-1.00)` | :arrow_up: | | hudihadoopmr | `33.33% <ø> (ø)` | `198.00 <ø> (ø)` | | | hudisparkdatasource | `73.33% <ø> (ø)` | `237.00 <ø> (ø)` | | | hudisync | `46.10% <ø> (ø)` | `142.00 <ø> (ø)` | | | huditimelineservice | `64.36% <ø> (ø)` | `62.00 <ø> (ø)` | | | hudiutilities | `69.70% <ø> (+0.01%)` | `374.00 <ø> (+1.00)` | | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/2880?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [...common/table/log/HoodieMergedLogRecordScanner.java](https://codecov.io/gh/apache/hudi/pull/2880/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL3RhYmxlL2xvZy9Ib29kaWVNZXJnZWRMb2dSZWNvcmRTY2FubmVyLmphdmE=) | `82.35% <0.00%> (-5.15%)` | `9.00% <0.00%> (-1.00%)` | | | [...mmon/table/log/AbstractHoodieLogRecordScanner.java](https://codecov.io/gh/apache/hudi/pull/2880/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL3RhYmxlL2xvZy9BYnN0cmFjdEhvb2RpZUxvZ1JlY29yZFNjYW5uZXIuamF2YQ==) | `80.00% <0.00%> (-1.62%)` | `34.00% <0.00%> (ø%)` | | | [...pache/hudi/metadata/HoodieBackedTableMetadata.java](https://codecov.io/gh/apache/hudi/pull/2880/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvbWV0YWRhdGEvSG9vZGllQmFja2VkVGFibGVNZXRhZGF0YS5qYXZh) | `0.00% <0.00%> (ø)` | `0.00% <0.00%> (ø%)` | | | [...e/hudi/table/format/mor/MergeOnReadInputSplit.java](https://codecov.io/gh/apache/hudi/pull/2880/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1mbGluay9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaHVkaS90YWJsZS9mb3JtYXQvbW9yL01lcmdlT25SZWFkSW5wdXRTcGxpdC5qYXZh) | `94.44% <0.00%> (ø)` | `8.00% <0.00%> (ø%)` | | | [...metadata/HoodieMetadataMergedLogRecordScanner.java](https://codecov.io/gh/apache/hudi/pull/2880/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvbWV0YWRhdGEvSG9vZGllTWV0YWRhdGFNZXJnZWRMb2dSZWNvcmRTY2FubmVyLmphdmE=) | `0.00%
[GitHub] [hudi] codecov-commenter commented on pull request #2880: [HUDI-1851] Automate suite of tests suites w/ docker and integ test
codecov-commenter commented on pull request #2880: URL: https://github.com/apache/hudi/pull/2880#issuecomment-826968162 # [Codecov](https://codecov.io/gh/apache/hudi/pull/2880?src=pr=h1_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) Report > Merging [#2880](https://codecov.io/gh/apache/hudi/pull/2880?src=pr=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (718fa28) into [master](https://codecov.io/gh/apache/hudi/commit/a5789c40673b36d40adab696706acc1446a286f8?el=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (a5789c4) will **decrease** coverage by `0.06%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/hudi/pull/2880?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master#2880 +/- ## - Coverage 53.00% 52.93% -0.07% + Complexity 3743 3742 -1 Files 488 488 Lines 2343523511 +76 Branches 2500 2501 +1 + Hits 1242212446 +24 - Misses 9913 9966 +53 + Partials 1100 1099 -1 ``` | Flag | Coverage Δ | Complexity Δ | | |---|---|---|---| | hudicli | `39.53% <ø> (ø)` | `220.00 <ø> (ø)` | | | hudiclient | `∅ <ø> (∅)` | `0.00 <ø> (ø)` | | | hudicommon | `50.37% <ø> (-0.29%)` | `1975.00 <ø> (-1.00)` | | | hudiflink | `59.45% <ø> (+0.33%)` | `534.00 <ø> (-1.00)` | :arrow_up: | | hudihadoopmr | `33.33% <ø> (ø)` | `198.00 <ø> (ø)` | | | hudisparkdatasource | `73.33% <ø> (ø)` | `237.00 <ø> (ø)` | | | hudisync | `46.10% <ø> (ø)` | `142.00 <ø> (ø)` | | | huditimelineservice | `64.36% <ø> (ø)` | `62.00 <ø> (ø)` | | | hudiutilities | `69.70% <ø> (+0.01%)` | `374.00 <ø> (+1.00)` | | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/2880?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [...common/table/log/HoodieMergedLogRecordScanner.java](https://codecov.io/gh/apache/hudi/pull/2880/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL3RhYmxlL2xvZy9Ib29kaWVNZXJnZWRMb2dSZWNvcmRTY2FubmVyLmphdmE=) | `82.35% <0.00%> (-5.15%)` | `9.00% <0.00%> (-1.00%)` | | | [...mmon/table/log/AbstractHoodieLogRecordScanner.java](https://codecov.io/gh/apache/hudi/pull/2880/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL3RhYmxlL2xvZy9BYnN0cmFjdEhvb2RpZUxvZ1JlY29yZFNjYW5uZXIuamF2YQ==) | `80.00% <0.00%> (-1.62%)` | `34.00% <0.00%> (ø%)` | | | [...pache/hudi/metadata/HoodieBackedTableMetadata.java](https://codecov.io/gh/apache/hudi/pull/2880/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvbWV0YWRhdGEvSG9vZGllQmFja2VkVGFibGVNZXRhZGF0YS5qYXZh) | `0.00% <0.00%> (ø)` | `0.00% <0.00%> (ø%)` | | | [...e/hudi/table/format/mor/MergeOnReadInputSplit.java](https://codecov.io/gh/apache/hudi/pull/2880/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1mbGluay9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaHVkaS90YWJsZS9mb3JtYXQvbW9yL01lcmdlT25SZWFkSW5wdXRTcGxpdC5qYXZh) | `94.44% <0.00%> (ø)` | `8.00% <0.00%> (ø%)` | | | [...metadata/HoodieMetadataMergedLogRecordScanner.java](https://codecov.io/gh/apache/hudi/pull/2880/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvbWV0YWRhdGEvSG9vZGllTWV0YWRhdGFNZXJnZWRMb2dSZWNvcmRTY2FubmVyLmphdmE=) | `0.00% <0.00%>
[GitHub] [hudi] RocMarshal opened a new pull request #2881: [Hotfix][minor] Remove redundant method-calling.
RocMarshal opened a new pull request #2881: URL: https://github.com/apache/hudi/pull/2881 ## *Tips* - *Thank you very much for contributing to Apache Hudi.* - *Please review https://hudi.apache.org/contributing.html before opening a pull request.* ## What is the purpose of the pull request *(For example: This pull request adds quick-start document.)* ## Brief change log *(for example:)* - *Modify AnnotationLocation checkstyle rule in checkstyle.xml* ## Verify this pull request *(Please pick either of the following options)* This pull request is a trivial rework / code cleanup without any test coverage. *(or)* This pull request is already covered by existing tests, such as *(please describe tests)*. (or) This change added tests and can be verified as follows: *(example:)* - *Added integration tests for end-to-end.* - *Added HoodieClientWriteTest to verify the change.* - *Manually verified the change by running a job locally.* ## Committer checklist - [ ] Has a corresponding JIRA in PR title & commit - [ ] Commit message is descriptive of the change - [ ] CI is green - [ ] Necessary doc changes done or have another open PR - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] RocMarshal commented on a change in pull request #2822: [Hotfix][hudi-sync] Refactor method up to parent-class
RocMarshal commented on a change in pull request #2822:
URL: https://github.com/apache/hudi/pull/2822#discussion_r620411008
##
File path:
hudi-sync/hudi-sync-common/src/main/java/org/apache/hudi/sync/common/AbstractSyncHoodieClient.java
##
@@ -29,20 +29,25 @@
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
+import org.apache.hudi.common.util.ValidationUtils;
import org.apache.log4j.LogManager;
import org.apache.log4j.Logger;
import org.apache.parquet.schema.MessageType;
+import java.io.Serializable;
import java.sql.ResultSet;
import java.sql.SQLException;
import java.sql.Statement;
import java.util.List;
import java.util.Map;
+import java.util.Objects;
public abstract class AbstractSyncHoodieClient {
private static final Logger LOG =
LogManager.getLogger(AbstractSyncHoodieClient.class);
+ public static final TypeConverter DECIMAL_TYPE_CONVERTOR = new
TypeConverter() {};
Review comment:
IMO, The current default implementation is the default implementation of
the decimal type convertor. And we would do a special implementation by
extending `TypeConverter` in the specific scene. So, keeping
`DECIMAL_TYPE_CONVERTOR` would be a fine choice in the case...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Updated] (HUDI-1851) Automate suite of tests suites w/ docker and integ test
[ https://issues.apache.org/jira/browse/HUDI-1851?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HUDI-1851: - Labels: pull-request-available (was: ) > Automate suite of tests suites w/ docker and integ test > --- > > Key: HUDI-1851 > URL: https://issues.apache.org/jira/browse/HUDI-1851 > Project: Apache Hudi > Issue Type: Improvement > Components: Testing >Reporter: sivabalan narayanan >Priority: Major > Labels: pull-request-available > > Automate suite of tests suites w/ docker and integ test -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hudi] nsivabalan opened a new pull request #2880: [HUDI-1851] Automate suite of tests suites w/ docker and integ test
nsivabalan opened a new pull request #2880: URL: https://github.com/apache/hudi/pull/2880 ## What is the purpose of the pull request *Automate suite of tests suites w/ docker and integ test ## Brief change log - *Added templates and script to automate tests w/ integ test suite job in docker* ## Verify this pull request Ran locally to verify ./generate_test_suite.sh ./generate_test_suite.sh --include_cluster_yaml true ./generate_test_suite.sh --num_iterations 50 --intermittent_delay_mins 1 --table_type MERGE_ON_READ ## Committer checklist - [ ] Has a corresponding JIRA in PR title & commit - [ ] Commit message is descriptive of the change - [ ] CI is green - [ ] Necessary doc changes done or have another open PR - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (HUDI-1850) Read on table fails if the first write to table failed
Vaibhav Sinha created HUDI-1850:
---
Summary: Read on table fails if the first write to table failed
Key: HUDI-1850
URL: https://issues.apache.org/jira/browse/HUDI-1850
Project: Apache Hudi
Issue Type: Bug
Affects Versions: 0.8.0
Reporter: Vaibhav Sinha
Attachments: Screenshot 2021-04-24 at 7.53.22 PM.png
{code:java}
ava.util.NoSuchElementException: No value present in Option
at org.apache.hudi.common.util.Option.get(Option.java:88)
~[hudi-spark3-bundle_2.12-0.8.0.jar:0.8.0]
at
org.apache.hudi.common.table.TableSchemaResolver.getTableSchemaFromCommitMetadata(TableSchemaResolver.java:215)
~[hudi-spark3-bundle_2.12-0.8.0.jar:0.8.0]
at
org.apache.hudi.common.table.TableSchemaResolver.getTableAvroSchema(TableSchemaResolver.java:166)
~[hudi-spark3-bundle_2.12-0.8.0.jar:0.8.0]
at
org.apache.hudi.common.table.TableSchemaResolver.getTableAvroSchema(TableSchemaResolver.java:155)
~[hudi-spark3-bundle_2.12-0.8.0.jar:0.8.0]
at
org.apache.hudi.MergeOnReadSnapshotRelation.(MergeOnReadSnapshotRelation.scala:65)
~[hudi-spark3-bundle_2.12-0.8.0.jar:0.8.0]
at org.apache.hudi.DefaultSource.createRelation(DefaultSource.scala:99)
~[hudi-spark3-bundle_2.12-0.8.0.jar:0.8.0]
at org.apache.hudi.DefaultSource.createRelation(DefaultSource.scala:63)
~[hudi-spark3-bundle_2.12-0.8.0.jar:0.8.0]
at
org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:354)
~[spark-sql_2.12-3.1.1.jar:3.1.1]
at
org.apache.spark.sql.DataFrameReader.loadV1Source(DataFrameReader.scala:326)
~[spark-sql_2.12-3.1.1.jar:3.1.1]
at
org.apache.spark.sql.DataFrameReader.$anonfun$load$3(DataFrameReader.scala:308)
~[spark-sql_2.12-3.1.1.jar:3.1.1]
at scala.Option.getOrElse(Option.scala:189)
~[scala-library-2.12.10.jar:?]
at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:308)
~[spark-sql_2.12-3.1.1.jar:3.1.1]
at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:240)
~[spark-sql_2.12-3.1.1.jar:3.1.1]
{code}
The screenshot shows the files that got created before the write had failed.
!Screenshot 2021-04-24 at 7.53.22 PM.png!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hudi] abhijeetkushe commented on issue #2850: [SUPPORT] S3 files skipped by HoodieDeltaStreamer on s3 bucket in continuous mode
abhijeetkushe commented on issue #2850: URL: https://github.com/apache/hudi/issues/2850#issuecomment-826883701 Thanks your help.Closing this issue -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] leesf commented on a change in pull request #2822: [Hotfix][hudi-sync] Refactor method up to parent-class
leesf commented on a change in pull request #2822:
URL: https://github.com/apache/hudi/pull/2822#discussion_r620331369
##
File path:
hudi-sync/hudi-sync-common/src/main/java/org/apache/hudi/sync/common/AbstractSyncHoodieClient.java
##
@@ -29,20 +29,25 @@
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
+import org.apache.hudi.common.util.ValidationUtils;
import org.apache.log4j.LogManager;
import org.apache.log4j.Logger;
import org.apache.parquet.schema.MessageType;
+import java.io.Serializable;
import java.sql.ResultSet;
import java.sql.SQLException;
import java.sql.Statement;
import java.util.List;
import java.util.Map;
+import java.util.Objects;
public abstract class AbstractSyncHoodieClient {
private static final Logger LOG =
LogManager.getLogger(AbstractSyncHoodieClient.class);
+ public static final TypeConverter DECIMAL_TYPE_CONVERTOR = new
TypeConverter() {};
Review comment:
rename to `TYPE_CONVERTOR`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[hudi] branch master updated: [HUDI-1742] Improve table level config priority for HoodieMultiTableDeltaStreamer (#2744)
This is an automated email from the ASF dual-hosted git repository.
leesf pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/hudi.git
The following commit(s) were added to refs/heads/master by this push:
new f4e3b94 [HUDI-1742] Improve table level config priority for
HoodieMultiTableDeltaStreamer (#2744)
f4e3b94 is described below
commit f4e3b949714aceaf8823fd1659e44d3b7e98089a
Author: Nick Young <72905543+nickyoungp...@users.noreply.github.com>
AuthorDate: Mon Apr 26 22:05:06 2021 +0800
[HUDI-1742] Improve table level config priority for
HoodieMultiTableDeltaStreamer (#2744)
---
.../deltastreamer/HoodieMultiTableDeltaStreamer.java | 4 +++-
.../utilities/functional/TestHoodieDeltaStreamer.java | 7 +++
.../functional/TestHoodieMultiTableDeltaStreamer.java | 18 ++
.../short_trip_uber_config.properties | 3 ++-
4 files changed, 30 insertions(+), 2 deletions(-)
diff --git
a/hudi-utilities/src/main/java/org/apache/hudi/utilities/deltastreamer/HoodieMultiTableDeltaStreamer.java
b/hudi-utilities/src/main/java/org/apache/hudi/utilities/deltastreamer/HoodieMultiTableDeltaStreamer.java
index a39b973..8e557f1 100644
---
a/hudi-utilities/src/main/java/org/apache/hudi/utilities/deltastreamer/HoodieMultiTableDeltaStreamer.java
+++
b/hudi-utilities/src/main/java/org/apache/hudi/utilities/deltastreamer/HoodieMultiTableDeltaStreamer.java
@@ -118,7 +118,9 @@ public class HoodieMultiTableDeltaStreamer {
checkIfTableConfigFileExists(configFolder, fs, configFilePath);
TypedProperties tableProperties = UtilHelpers.readConfig(fs, new
Path(configFilePath), new ArrayList<>()).getConfig();
properties.forEach((k, v) -> {
-tableProperties.setProperty(k.toString(), v.toString());
+if (tableProperties.get(k) == null) {
+ tableProperties.setProperty(k.toString(), v.toString());
+}
});
final HoodieDeltaStreamer.Config cfg = new HoodieDeltaStreamer.Config();
//copy all the values from config to cfg
diff --git
a/hudi-utilities/src/test/java/org/apache/hudi/utilities/functional/TestHoodieDeltaStreamer.java
b/hudi-utilities/src/test/java/org/apache/hudi/utilities/functional/TestHoodieDeltaStreamer.java
index 7d4db2c..362a294 100644
---
a/hudi-utilities/src/test/java/org/apache/hudi/utilities/functional/TestHoodieDeltaStreamer.java
+++
b/hudi-utilities/src/test/java/org/apache/hudi/utilities/functional/TestHoodieDeltaStreamer.java
@@ -1631,6 +1631,13 @@ public class TestHoodieDeltaStreamer extends
UtilitiesTestBase {
}
}
+ public static class TestTableLevelGenerator extends SimpleKeyGenerator {
+
+public TestTableLevelGenerator(TypedProperties props) {
+ super(props);
+}
+ }
+
public static class DummyAvroPayload extends OverwriteWithLatestAvroPayload {
public DummyAvroPayload(GenericRecord gr, Comparable orderingVal) {
diff --git
a/hudi-utilities/src/test/java/org/apache/hudi/utilities/functional/TestHoodieMultiTableDeltaStreamer.java
b/hudi-utilities/src/test/java/org/apache/hudi/utilities/functional/TestHoodieMultiTableDeltaStreamer.java
index 7b5ce9d..17450a0 100644
---
a/hudi-utilities/src/test/java/org/apache/hudi/utilities/functional/TestHoodieMultiTableDeltaStreamer.java
+++
b/hudi-utilities/src/test/java/org/apache/hudi/utilities/functional/TestHoodieMultiTableDeltaStreamer.java
@@ -213,6 +213,24 @@ public class TestHoodieMultiTableDeltaStreamer extends
TestHoodieDeltaStreamer {
}
}
+ @Test
+ public void testTableLevelProperties() throws IOException {
+HoodieMultiTableDeltaStreamer.Config cfg =
TestHelpers.getConfig(PROPS_FILENAME_TEST_SOURCE1, dfsBasePath + "/config",
TestDataSource.class.getName(), false);
+HoodieMultiTableDeltaStreamer streamer = new
HoodieMultiTableDeltaStreamer(cfg, jsc);
+List tableExecutionContexts =
streamer.getTableExecutionContexts();
+tableExecutionContexts.forEach(tableExecutionContext -> {
+ switch (tableExecutionContext.getTableName()) {
+case "dummy_table_short_trip":
+ String tableLevelKeyGeneratorClass =
tableExecutionContext.getProperties().getString(DataSourceWriteOptions.KEYGENERATOR_CLASS_OPT_KEY());
+
assertEquals(TestHoodieDeltaStreamer.TestTableLevelGenerator.class.getName(),
tableLevelKeyGeneratorClass);
+ break;
+default:
+ String defaultKeyGeneratorClass =
tableExecutionContext.getProperties().getString(DataSourceWriteOptions.KEYGENERATOR_CLASS_OPT_KEY());
+ assertEquals(TestHoodieDeltaStreamer.TestGenerator.class.getName(),
defaultKeyGeneratorClass);
+ }
+});
+ }
+
private String populateCommonPropsAndWriteToFile() throws IOException {
TypedProperties commonProps = new TypedProperties();
populateCommonProps(commonProps);
diff --git
a/hudi-utilities/src/test/resources/delta-streamer-config/short_trip_uber_config.properties
[GitHub] [hudi] leesf merged pull request #2744: [HUDI-1742] improve table level config priority for HoodieMultiTableDeltaStreamer
leesf merged pull request #2744: URL: https://github.com/apache/hudi/pull/2744 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] liujinhui1994 commented on pull request #2710: [HUDI-648][RFC-20] Implement error log/table for Datasource/DeltaStreamer/WriteClient/Compaction writes
liujinhui1994 commented on pull request #2710: URL: https://github.com/apache/hudi/pull/2710#issuecomment-826771631 Thanks for the suggestion，I will take the time to deal with -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] codecov-commenter edited a comment on pull request #2875: [HUDI-1841] Tweak the min max commits to keep when setting up cleanin…
codecov-commenter edited a comment on pull request #2875: URL: https://github.com/apache/hudi/pull/2875#issuecomment-826288360 # [Codecov](https://codecov.io/gh/apache/hudi/pull/2875?src=pr=h1_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) Report > Merging [#2875](https://codecov.io/gh/apache/hudi/pull/2875?src=pr=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (3cb3128) into [master](https://codecov.io/gh/apache/hudi/commit/3e4fa170cfd2c198599c3bed3982f2f643c7fbe8?el=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (3e4fa17) will **decrease** coverage by `0.08%`. > The diff coverage is `90.90%`. [](https://codecov.io/gh/apache/hudi/pull/2875?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master#2875 +/- ## - Coverage 53.06% 52.98% -0.09% + Complexity 3746 3743 -3 Files 488 488 Lines 2346623538 +72 Branches 2503 2503 + Hits 1245312471 +18 - Misses 9914 9966 +52 - Partials 1099 1101 +2 ``` | Flag | Coverage Δ | Complexity Δ | | |---|---|---|---| | hudicli | `39.53% <ø> (ø)` | `220.00 <ø> (ø)` | | | hudiclient | `∅ <ø> (∅)` | `0.00 <ø> (ø)` | | | hudicommon | `50.38% <ø> (-0.27%)` | `1975.00 <ø> (-1.00)` | | | hudiflink | `59.65% <90.90%> (+0.22%)` | `536.00 <3.00> (-1.00)` | :arrow_up: | | hudihadoopmr | `33.33% <ø> (ø)` | `198.00 <ø> (ø)` | | | hudisparkdatasource | `73.33% <ø> (ø)` | `237.00 <ø> (ø)` | | | hudisync | `46.10% <ø> (ø)` | `142.00 <ø> (ø)` | | | huditimelineservice | `64.36% <ø> (ø)` | `62.00 <ø> (ø)` | | | hudiutilities | `69.68% <ø> (-0.06%)` | `373.00 <ø> (-1.00)` | | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/2875?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [...java/org/apache/hudi/sink/StreamWriteFunction.java](https://codecov.io/gh/apache/hudi/pull/2875/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1mbGluay9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaHVkaS9zaW5rL1N0cmVhbVdyaXRlRnVuY3Rpb24uamF2YQ==) | `79.24% <76.92%> (-1.68%)` | `24.00 <1.00> (ø)` | | | [...va/org/apache/hudi/configuration/FlinkOptions.java](https://codecov.io/gh/apache/hudi/pull/2875/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1mbGluay9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaHVkaS9jb25maWd1cmF0aW9uL0ZsaW5rT3B0aW9ucy5qYXZh) | `90.48% <100.00%> (+0.42%)` | `11.00 <0.00> (ø)` | | | [...java/org/apache/hudi/table/HoodieTableFactory.java](https://codecov.io/gh/apache/hudi/pull/2875/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1mbGluay9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaHVkaS90YWJsZS9Ib29kaWVUYWJsZUZhY3RvcnkuamF2YQ==) | `89.83% <100.00%> (+2.07%)` | `17.00 <2.00> (+2.00)` | | | [...c/main/java/org/apache/hudi/util/StreamerUtil.java](https://codecov.io/gh/apache/hudi/pull/2875/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1mbGluay9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaHVkaS91dGlsL1N0cmVhbWVyVXRpbC5qYXZh) | `58.33% <100.00%> (+0.70%)` | `18.00 <0.00> (ø)` | | | [...common/table/log/HoodieMergedLogRecordScanner.java](https://codecov.io/gh/apache/hudi/pull/2875/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL3RhYmxlL2xvZy9Ib29kaWVNZXJnZWRMb2dSZWNvcmRTY2FubmVyLmphdmE=) | `82.35% <0.00%> (-5.15%)` | `9.00% <0.00%> (-1.00%)` | | |
[GitHub] [hudi] PavelPetukhov edited a comment on issue #2856: [SUPPORT] Metrics Prometheus pushgateway
PavelPetukhov edited a comment on issue #2856: URL: https://github.com/apache/hudi/issues/2856#issuecomment-826758533 Thanks a lot for the reply I my case I am getting the following error: 1/04/26 13:29:52 WARN prometheus.PushGatewayReporter: Can't push monitoring information to pushGateway java.net.UnknownHostException: https at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:184) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:607) at sun.net.NetworkClient.doConnect(NetworkClient.java:175) at sun.net.www.http.HttpClient.openServer(HttpClient.java:463) at sun.net.www.http.HttpClient.openServer(HttpClient.java:558) at sun.net.www.http.HttpClient.(HttpClient.java:242) at sun.net.www.http.HttpClient.New(HttpClient.java:339) at sun.net.www.http.HttpClient.New(HttpClient.java:357) at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:1226) at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1162) at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1056) at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:990) at io.prometheus.client.exporter.PushGateway.doRequest(PushGateway.java:305) at io.prometheus.client.exporter.PushGateway.pushAdd(PushGateway.java:160) at org.apache.hudi.metrics.prometheus.PushGatewayReporter.report(PushGatewayReporter.java:72) at org.apache.hudi.metrics.prometheus.PushGatewayMetricsReporter.report(PushGatewayMetricsReporter.java:64) at org.apache.hudi.metrics.Metrics.reportAndCloseReporter(Metrics.java:64) at org.apache.hudi.metrics.Metrics.lambda$new$0(Metrics.java:57) at java.lang.Thread.run(Thread.java:748) my parameters are --hoodie-conf hoodie.metrics.on=true --hoodie-conf hoodie.metrics.reporter.type=PROMETHEUS_PUSHGATEWAY --hoodie-conf hoodie.metrics.pushgateway.host=https://pushgateway_url.com/ --hoodie-conf hoodie.metrics.pushgateway.port=8080 --hoodie-conf hoodie.metrics.pushgateway.job.name=test_job_1 --hoodie-conf hoodie.metrics.pushgateway.random.job.name.suffix=false --hoodie-conf hoodie.metrics.pushgateway.delete.on.shutdown=false -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] PavelPetukhov commented on issue #2856: [SUPPORT] Metrics Prometheus pushgateway
PavelPetukhov commented on issue #2856: URL: https://github.com/apache/hudi/issues/2856#issuecomment-826758533 Thanks a lot for the reply I my case I am getting the following error: 1/04/26 13:29:52 WARN prometheus.PushGatewayReporter: Can't push monitoring information to pushGateway java.net.UnknownHostException: https at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:184) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:607) at sun.net.NetworkClient.doConnect(NetworkClient.java:175) at sun.net.www.http.HttpClient.openServer(HttpClient.java:463) at sun.net.www.http.HttpClient.openServer(HttpClient.java:558) at sun.net.www.http.HttpClient.(HttpClient.java:242) at sun.net.www.http.HttpClient.New(HttpClient.java:339) at sun.net.www.http.HttpClient.New(HttpClient.java:357) at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:1226) at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1162) at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1056) at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:990) at io.prometheus.client.exporter.PushGateway.doRequest(PushGateway.java:305) at io.prometheus.client.exporter.PushGateway.pushAdd(PushGateway.java:160) at org.apache.hudi.metrics.prometheus.PushGatewayReporter.report(PushGatewayReporter.java:72) at org.apache.hudi.metrics.prometheus.PushGatewayMetricsReporter.report(PushGatewayMetricsReporter.java:64) at org.apache.hudi.metrics.Metrics.reportAndCloseReporter(Metrics.java:64) at org.apache.hudi.metrics.Metrics.lambda$new$0(Metrics.java:57) at java.lang.Thread.run(Thread.java:748) my parameters are --hoodie-conf hoodie.metrics.on=true --hoodie-conf hoodie.metrics.reporter.type=PROMETHEUS_PUSHGATEWAY --hoodie-conf hoodie.metrics.pushgateway.host=https://pushgateway_url.com/ --hoodie-conf hoodie.metrics.pushgateway.port=8081 \ --hoodie-conf hoodie.metrics.pushgateway.job.name=test_job_1 \ --hoodie-conf hoodie.metrics.pushgateway.random.job.name.suffix=false \ --hoodie-conf hoodie.metrics.pushgateway.delete.on.shutdown=false \ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] codecov-commenter edited a comment on pull request #2875: [HUDI-1841] Tweak the min max commits to keep when setting up cleanin…
codecov-commenter edited a comment on pull request #2875: URL: https://github.com/apache/hudi/pull/2875#issuecomment-826288360 # [Codecov](https://codecov.io/gh/apache/hudi/pull/2875?src=pr=h1_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) Report > Merging [#2875](https://codecov.io/gh/apache/hudi/pull/2875?src=pr=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (3cb3128) into [master](https://codecov.io/gh/apache/hudi/commit/3e4fa170cfd2c198599c3bed3982f2f643c7fbe8?el=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (3e4fa17) will **increase** coverage by `16.62%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/hudi/pull/2875?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master#2875 +/- ## = + Coverage 53.06% 69.68% +16.62% + Complexity 3746 373 -3373 = Files 488 54 -434 Lines 23466 1996-21470 Branches 2503 236 -2267 = - Hits 12453 1391-11062 + Misses 9914 473 -9441 + Partials 1099 132 -967 ``` | Flag | Coverage Δ | Complexity Δ | | |---|---|---|---| | hudicli | `?` | `?` | | | hudiclient | `?` | `?` | | | hudicommon | `?` | `?` | | | hudiflink | `?` | `?` | | | hudihadoopmr | `?` | `?` | | | hudisparkdatasource | `?` | `?` | | | hudisync | `?` | `?` | | | huditimelineservice | `?` | `?` | | | hudiutilities | `69.68% <ø> (-0.06%)` | `373.00 <ø> (-1.00)` | | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/2875?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [...apache/hudi/utilities/deltastreamer/DeltaSync.java](https://codecov.io/gh/apache/hudi/pull/2875/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2RlbHRhc3RyZWFtZXIvRGVsdGFTeW5jLmphdmE=) | `71.08% <0.00%> (-0.35%)` | `55.00% <0.00%> (-1.00%)` | | | [...odie/hadoop/hive/HoodieCombineHiveInputFormat.java](https://codecov.io/gh/apache/hudi/pull/2875/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1oYWRvb3AtbXIvc3JjL21haW4vamF2YS9jb20vdWJlci9ob29kaWUvaGFkb29wL2hpdmUvSG9vZGllQ29tYmluZUhpdmVJbnB1dEZvcm1hdC5qYXZh) | | | | | [.../hudi/table/format/mor/MergeOnReadInputFormat.java](https://codecov.io/gh/apache/hudi/pull/2875/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1mbGluay9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaHVkaS90YWJsZS9mb3JtYXQvbW9yL01lcmdlT25SZWFkSW5wdXRGb3JtYXQuamF2YQ==) | | | | | [...til/jvm/HotSpotMemoryLayoutSpecification32bit.java](https://codecov.io/gh/apache/hudi/pull/2875/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL3V0aWwvanZtL0hvdFNwb3RNZW1vcnlMYXlvdXRTcGVjaWZpY2F0aW9uMzJiaXQuamF2YQ==) | | | | | [...in/java/org/apache/hudi/common/model/BaseFile.java](https://codecov.io/gh/apache/hudi/pull/2875/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL21vZGVsL0Jhc2VGaWxlLmphdmE=) | | | | | [...rg/apache/hudi/sink/utils/CoordinatorExecutor.java](https://codecov.io/gh/apache/hudi/pull/2875/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1mbGluay9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaHVkaS9zaW5rL3V0aWxzL0Nvb3JkaW5hdG9yRXhlY3V0b3IuamF2YQ==) | | | | |
[jira] [Assigned] (HUDI-1849) Remove the unit from the Flink sql options
[ https://issues.apache.org/jira/browse/HUDI-1849?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] 谢波 reassigned HUDI-1849: Assignee: 谢波 > Remove the unit from the Flink sql options > -- > > Key: HUDI-1849 > URL: https://issues.apache.org/jira/browse/HUDI-1849 > Project: Apache Hudi > Issue Type: Improvement > Components: Flink Integration >Reporter: Danny Chen >Assignee: 谢波 >Priority: Major > Fix For: 0.9.0 > > > Remove the unit suffix because it is verbose. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hudi] aditiwari01 commented on issue #2802: Hive read issues when different partition have different schemas.
aditiwari01 commented on issue #2802: URL: https://github.com/apache/hudi/issues/2802#issuecomment-826671636 Have quite a busy sprint. I'll try to get back. If not, I'll update by weekend. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] codecov-commenter edited a comment on pull request #2875: [HUDI-1841] Tweak the min max commits to keep when setting up cleanin…
codecov-commenter edited a comment on pull request #2875: URL: https://github.com/apache/hudi/pull/2875#issuecomment-826288360 # [Codecov](https://codecov.io/gh/apache/hudi/pull/2875?src=pr=h1_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) Report > Merging [#2875](https://codecov.io/gh/apache/hudi/pull/2875?src=pr=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (371c2e9) into [master](https://codecov.io/gh/apache/hudi/commit/3e4fa170cfd2c198599c3bed3982f2f643c7fbe8?el=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (3e4fa17) will **decrease** coverage by `43.69%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/hudi/pull/2875?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master #2875 +/- ## - Coverage 53.06% 9.36% -43.70% + Complexity 3746 48 -3698 Files 488 54 -434 Lines 234661996-21470 Branches 2503 236 -2267 - Hits 12453 187-12266 + Misses 99141796 -8118 + Partials 1099 13 -1086 ``` | Flag | Coverage Δ | Complexity Δ | | |---|---|---|---| | hudicli | `?` | `?` | | | hudiclient | `?` | `?` | | | hudicommon | `?` | `?` | | | hudiflink | `?` | `?` | | | hudihadoopmr | `?` | `?` | | | hudisparkdatasource | `?` | `?` | | | hudisync | `?` | `?` | | | huditimelineservice | `?` | `?` | | | hudiutilities | `9.36% <ø> (-60.38%)` | `48.00 <ø> (-326.00)` | | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/2875?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [...va/org/apache/hudi/utilities/IdentitySplitter.java](https://codecov.io/gh/apache/hudi/pull/2875/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL0lkZW50aXR5U3BsaXR0ZXIuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-2.00%)` | | | [...va/org/apache/hudi/utilities/schema/SchemaSet.java](https://codecov.io/gh/apache/hudi/pull/2875/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NjaGVtYS9TY2hlbWFTZXQuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-3.00%)` | | | [...a/org/apache/hudi/utilities/sources/RowSource.java](https://codecov.io/gh/apache/hudi/pull/2875/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvUm93U291cmNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-4.00%)` | | | [.../org/apache/hudi/utilities/sources/AvroSource.java](https://codecov.io/gh/apache/hudi/pull/2875/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvQXZyb1NvdXJjZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-1.00%)` | | | [.../org/apache/hudi/utilities/sources/JsonSource.java](https://codecov.io/gh/apache/hudi/pull/2875/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvblNvdXJjZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-1.00%)` | | |
[GitHub] [hudi] codecov-commenter commented on pull request #2879: [HUDI-1848] Adding support for HMS for running DDL queries in hive-sy…
codecov-commenter commented on pull request #2879: URL: https://github.com/apache/hudi/pull/2879#issuecomment-82454 # [Codecov](https://codecov.io/gh/apache/hudi/pull/2879?src=pr=h1_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) Report > Merging [#2879](https://codecov.io/gh/apache/hudi/pull/2879?src=pr=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (56a2a21) into [master](https://codecov.io/gh/apache/hudi/commit/a5789c40673b36d40adab696706acc1446a286f8?el=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (a5789c4) will **decrease** coverage by `43.63%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/hudi/pull/2879?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master #2879 +/- ## - Coverage 53.00% 9.36% -43.64% + Complexity 3743 48 -3695 Files 488 54 -434 Lines 234351996-21439 Branches 2500 236 -2264 - Hits 12422 187-12235 + Misses 99131796 -8117 + Partials 1100 13 -1087 ``` | Flag | Coverage Δ | Complexity Δ | | |---|---|---|---| | hudicli | `?` | `?` | | | hudiclient | `?` | `?` | | | hudicommon | `?` | `?` | | | hudiflink | `?` | `?` | | | hudihadoopmr | `?` | `?` | | | hudisparkdatasource | `?` | `?` | | | hudisync | `?` | `?` | | | huditimelineservice | `?` | `?` | | | hudiutilities | `9.36% <ø> (-60.33%)` | `48.00 <ø> (-325.00)` | | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/2879?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [...va/org/apache/hudi/utilities/IdentitySplitter.java](https://codecov.io/gh/apache/hudi/pull/2879/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL0lkZW50aXR5U3BsaXR0ZXIuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-2.00%)` | | | [...va/org/apache/hudi/utilities/schema/SchemaSet.java](https://codecov.io/gh/apache/hudi/pull/2879/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NjaGVtYS9TY2hlbWFTZXQuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-3.00%)` | | | [...a/org/apache/hudi/utilities/sources/RowSource.java](https://codecov.io/gh/apache/hudi/pull/2879/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvUm93U291cmNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-4.00%)` | | | [.../org/apache/hudi/utilities/sources/AvroSource.java](https://codecov.io/gh/apache/hudi/pull/2879/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvQXZyb1NvdXJjZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-1.00%)` | | | [.../org/apache/hudi/utilities/sources/JsonSource.java](https://codecov.io/gh/apache/hudi/pull/2879/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvblNvdXJjZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-1.00%)` | | |
[jira] [Closed] (HUDI-1837) Add optional instant range to log record scanner for log block filtering
[ https://issues.apache.org/jira/browse/HUDI-1837?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] vinoyang closed HUDI-1837. -- Resolution: Implemented d047e91d86d6905c0f66dbdb24f2e9ecd3684430 > Add optional instant range to log record scanner for log block filtering > - > > Key: HUDI-1837 > URL: https://issues.apache.org/jira/browse/HUDI-1837 > Project: Apache Hudi > Issue Type: Improvement > Components: Flink Integration >Reporter: Danny Chen >Assignee: Danny Chen >Priority: Major > Labels: pull-request-available > Fix For: 0.9.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HUDI-1837) Add optional instant range to log record scanner for log block filtering
[ https://issues.apache.org/jira/browse/HUDI-1837?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] vinoyang updated HUDI-1837: --- Fix Version/s: 0.9.0 > Add optional instant range to log record scanner for log block filtering > - > > Key: HUDI-1837 > URL: https://issues.apache.org/jira/browse/HUDI-1837 > Project: Apache Hudi > Issue Type: Improvement > Components: Flink Integration >Reporter: Danny Chen >Assignee: Danny Chen >Priority: Major > Labels: pull-request-available > Fix For: 0.9.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[hudi] branch master updated (3e4fa17 -> d047e91)
This is an automated email from the ASF dual-hosted git repository.
vinoyang pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/hudi.git.
from 3e4fa17 [HUDI-1835] Fixing kafka native config param for auto offset
reset (#2864)
add d047e91 [HUDI-1837] Add optional instant range to log record scanner
for log (#2870)
No new revisions were added by this update.
Summary of changes:
...SparkExecuteClusteringCommitActionExecutor.java | 17 +++--
.../table/log/AbstractHoodieLogRecordScanner.java | 18 -
.../table/log/HoodieMergedLogRecordScanner.java| 52 --
.../table/log/HoodieUnMergedLogRecordScanner.java | 8 +--
.../hudi/common/table/log}/InstantRange.java | 4 +-
.../hudi/metadata/HoodieBackedTableMetadata.java | 12 +++-
.../HoodieMetadataMergedLogRecordScanner.java | 79 +-
.../apache/hudi/configuration/FlinkOptions.java| 6 ++
.../hudi/source/StreamReadMonitoringFunction.java | 2 +-
.../org/apache/hudi/table/format/FormatUtils.java | 1 +
.../table/format/mor/MergeOnReadInputFormat.java | 12
.../table/format/mor/MergeOnReadInputSplit.java| 1 +
.../java/org/apache/hudi/util/StreamerUtil.java| 1 +
13 files changed, 159 insertions(+), 54 deletions(-)
rename {hudi-flink/src/main/java/org/apache/hudi/table/format/mor =>
hudi-common/src/main/java/org/apache/hudi/common/table/log}/InstantRange.java
(97%)
[GitHub] [hudi] yanghua merged pull request #2870: [HUDI-1837] Add optional instant range to log record scanner for log block filtering
yanghua merged pull request #2870: URL: https://github.com/apache/hudi/pull/2870 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (HUDI-1849) Remove the unit from the Flink sql options
Danny Chen created HUDI-1849: Summary: Remove the unit from the Flink sql options Key: HUDI-1849 URL: https://issues.apache.org/jira/browse/HUDI-1849 Project: Apache Hudi Issue Type: Improvement Components: Flink Integration Reporter: Danny Chen Fix For: 0.9.0 Remove the unit suffix because it is verbose. -- This message was sent by Atlassian Jira (v8.3.4#803005)
