[jira] [Created] (HUDI-3132) Minor fixes for HoodieCatalog

2021-12-29 Thread Danny Chen (Jira)
Danny Chen created HUDI-3132: Summary: Minor fixes for HoodieCatalog Key: HUDI-3132 URL: https://issues.apache.org/jira/browse/HUDI-3132 Project: Apache Hudi Issue Type: Bug

[jira] [Assigned] (HUDI-2661) java.lang.NoSuchMethodError: org.apache.spark.sql.catalyst.catalog.CatalogTable.copy

2021-12-29 Thread Yann Byron (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2661?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yann Byron reassigned HUDI-2661: Assignee: Forward Xu (was: Yann Byron) > java.lang.NoSuchMethodError: >

[jira] [Assigned] (HUDI-1850) Read on table fails if the first write to table failed

2021-12-29 Thread Yann Byron (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1850?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yann Byron reassigned HUDI-1850: Assignee: sivabalan narayanan (was: Yann Byron) > Read on table fails if the first write to table

[GitHub] [hudi] vinothchandar commented on pull request #3173: [HUDI-1951] Add bucket hash index, compatible with the hive bucket

2021-12-29 Thread GitBox
vinothchandar commented on pull request #3173: URL: https://github.com/apache/hudi/pull/3173#issuecomment-1002904375 ``` [INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 125.714 s - in org.apache.hudi.integ.command.ITTestHoodieSyncCommand [ERROR] Tests run:

[jira] [Updated] (HUDI-2901) Fixed the bug clustering jobs are not running in parallel

2021-12-29 Thread Raymond Xu (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2901?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raymond Xu updated HUDI-2901: - Sprint: Hudi-Sprint-0.10.1 > Fixed the bug clustering jobs are not running in parallel >

[jira] [Updated] (HUDI-2901) Fixed the bug clustering jobs are not running in parallel

2021-12-29 Thread Raymond Xu (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2901?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raymond Xu updated HUDI-2901: - Story Points: 1 > Fixed the bug clustering jobs are not running in parallel >

[jira] [Updated] (HUDI-2938) Code Refactor: Metadata util to get latest file slices for readers and writers

2021-12-29 Thread Raymond Xu (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raymond Xu updated HUDI-2938: - Issue Type: Improvement (was: Task) > Code Refactor: Metadata util to get latest file slices for readers

[jira] [Assigned] (HUDI-281) HiveSync failure through Spark when useJdbc is set to false

2021-12-29 Thread Raymond Xu (Jira)
[ https://issues.apache.org/jira/browse/HUDI-281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raymond Xu reassigned HUDI-281: --- Assignee: (was: Raymond Xu) > HiveSync failure through Spark when useJdbc is set to false >

[GitHub] [hudi] harsh1231 commented on a change in pull request #4404: [HUDI-2558] Fixing Clustering w/ sort columns with null values fails

2021-12-29 Thread GitBox
harsh1231 commented on a change in pull request #4404: URL: https://github.com/apache/hudi/pull/4404#discussion_r776596935 ## File path: hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/execution/bulkinsert/RDDCustomColumnsSortPartitioner.java ## @@ -55,8 +55,17 @@

[GitHub] [hudi] harsh1231 commented on a change in pull request #4404: [HUDI-2558] Fixing Clustering w/ sort columns with null values fails

2021-12-29 Thread GitBox
harsh1231 commented on a change in pull request #4404: URL: https://github.com/apache/hudi/pull/4404#discussion_r776596753 ## File path: hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/execution/bulkinsert/RDDCustomColumnsSortPartitioner.java ## @@ -55,8 +55,17 @@

[jira] [Updated] (HUDI-2946) Upgrade maven plugin to make Hudi be compatible with higher Java versions

2021-12-29 Thread Raymond Xu (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raymond Xu updated HUDI-2946: - Fix Version/s: (was: 0.10.1) > Upgrade maven plugin to make Hudi be compatible with higher Java

[jira] [Updated] (HUDI-2426) spark sql extensions breaks read.table from metastore

2021-12-29 Thread Raymond Xu (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2426?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raymond Xu updated HUDI-2426: - Story Points: 1 > spark sql extensions breaks read.table from metastore >

[jira] [Updated] (HUDI-2611) `create table if not exists` should print message instead of throwing error

2021-12-29 Thread Raymond Xu (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raymond Xu updated HUDI-2611: - Story Points: 1 > `create table if not exists` should print message instead of throwing error >

[jira] [Updated] (HUDI-2661) java.lang.NoSuchMethodError: org.apache.spark.sql.catalyst.catalog.CatalogTable.copy

2021-12-29 Thread Raymond Xu (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2661?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raymond Xu updated HUDI-2661: - Story Points: 1 > java.lang.NoSuchMethodError: > org.apache.spark.sql.catalyst.catalog.CatalogTable.copy

[jira] [Updated] (HUDI-2915) Fix field not found in record error for spark-sql

2021-12-29 Thread Raymond Xu (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2915?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raymond Xu updated HUDI-2915: - Story Points: 1 (was: 2) > Fix field not found in record error for spark-sql >

[jira] [Updated] (HUDI-1850) Read on table fails if the first write to table failed

2021-12-29 Thread Raymond Xu (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1850?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raymond Xu updated HUDI-1850: - Story Points: 1 > Read on table fails if the first write to table failed >

[jira] [Updated] (HUDI-3100) Hive Conditional sync cannot be set from deltastreamer

2021-12-29 Thread Raymond Xu (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3100?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raymond Xu updated HUDI-3100: - Story Points: 1 > Hive Conditional sync cannot be set from deltastreamer >

[jira] [Updated] (HUDI-2915) Fix field not found in record error for spark-sql

2021-12-29 Thread Raymond Xu (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2915?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raymond Xu updated HUDI-2915: - Story Points: 2 > Fix field not found in record error for spark-sql >

[jira] [Updated] (HUDI-2966) Add TaskCompletionListener for HoodieMergeOnReadRDD to close logScanner

2021-12-29 Thread Raymond Xu (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2966?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raymond Xu updated HUDI-2966: - Story Points: 1 > Add TaskCompletionListener for HoodieMergeOnReadRDD to close logScanner >

[jira] [Updated] (HUDI-3107) Fix HiveSyncTool drop partitions using JDBC

2021-12-29 Thread Raymond Xu (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3107?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raymond Xu updated HUDI-3107: - Story Points: 1 > Fix HiveSyncTool drop partitions using JDBC >

[jira] [Updated] (HUDI-281) HiveSync failure through Spark when useJdbc is set to false

2021-12-29 Thread Raymond Xu (Jira)
[ https://issues.apache.org/jira/browse/HUDI-281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raymond Xu updated HUDI-281: Story Points: 1 > HiveSync failure through Spark when useJdbc is set to false >

[jira] [Assigned] (HUDI-281) HiveSync failure through Spark when useJdbc is set to false

2021-12-29 Thread Raymond Xu (Jira)
[ https://issues.apache.org/jira/browse/HUDI-281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raymond Xu reassigned HUDI-281: --- Assignee: Raymond Xu > HiveSync failure through Spark when useJdbc is set to false >

[jira] [Updated] (HUDI-3104) Hudi-kafka-connect can not scan hadoop config files by HADOOP_CONF_DIR

2021-12-29 Thread Raymond Xu (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3104?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raymond Xu updated HUDI-3104: - Status: In Progress (was: Open) > Hudi-kafka-connect can not scan hadoop config files by HADOOP_CONF_DIR

[jira] [Assigned] (HUDI-3104) Hudi-kafka-connect can not scan hadoop config files by HADOOP_CONF_DIR

2021-12-29 Thread Raymond Xu (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3104?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raymond Xu reassigned HUDI-3104: Assignee: cdmikechen > Hudi-kafka-connect can not scan hadoop config files by HADOOP_CONF_DIR >

[jira] [Updated] (HUDI-3125) Spark SQL writing timestamp type don't need to disable `spark.sql.datetime.java8API.enabled` manually

2021-12-29 Thread Raymond Xu (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3125?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raymond Xu updated HUDI-3125: - Story Points: 1 > Spark SQL writing timestamp type don't need to disable >

[jira] [Updated] (HUDI-3131) Spark3.1.1 CTAS error

2021-12-29 Thread Raymond Xu (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raymond Xu updated HUDI-3131: - Story Points: 1 > Spark3.1.1 CTAS error > - > > Key: HUDI-3131 >

[jira] [Updated] (HUDI-3125) Spark SQL writing timestamp type don't need to disable `spark.sql.datetime.java8API.enabled` manually

2021-12-29 Thread Raymond Xu (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3125?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raymond Xu updated HUDI-3125: - Issue Type: Bug (was: Improvement) > Spark SQL writing timestamp type don't need to disable >

[jira] [Updated] (HUDI-3100) Hive Conditional sync cannot be set from deltastreamer

2021-12-29 Thread Raymond Xu (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3100?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raymond Xu updated HUDI-3100: - Status: In Progress (was: Open) > Hive Conditional sync cannot be set from deltastreamer >

[jira] [Updated] (HUDI-2987) event time not recorded in commit metadata when insert or bulk insert

2021-12-29 Thread Raymond Xu (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2987?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raymond Xu updated HUDI-2987: - Fix Version/s: (was: 0.10.1) > event time not recorded in commit metadata when insert or bulk insert

[GitHub] [hudi] LuPan2015 commented on issue #4475: [SUPPORT] Hudi and aws S3 integration exception

2021-12-29 Thread GitBox
LuPan2015 commented on issue #4475: URL: https://github.com/apache/hudi/issues/4475#issuecomment-1002895931 solved #4474 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

[GitHub] [hudi] LuPan2015 closed issue #4475: [SUPPORT] Hudi and aws S3 integration exception

2021-12-29 Thread GitBox
LuPan2015 closed issue #4475: URL: https://github.com/apache/hudi/issues/4475 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[GitHub] [hudi] danny0405 commented on pull request #4065: [HUDI-2817] Sync the configuration inference for HoodieFlinkStreamer

2021-12-29 Thread GitBox
danny0405 commented on pull request #4065: URL: https://github.com/apache/hudi/pull/4065#issuecomment-1002894948 Can we sync up all the inference logic ? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above

[GitHub] [hudi] danny0405 commented on pull request #4189: [HUDI-2913] Disable auto clean in writer task

2021-12-29 Thread GitBox
danny0405 commented on pull request #4189: URL: https://github.com/apache/hudi/pull/4189#issuecomment-1002894548 Close because it is not necessary ~ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

[GitHub] [hudi] danny0405 closed pull request #4189: [HUDI-2913] Disable auto clean in writer task

2021-12-29 Thread GitBox
danny0405 closed pull request #4189: URL: https://github.com/apache/hudi/pull/4189 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[GitHub] [hudi] danny0405 closed pull request #3386: [HUDI-2270] Remove corrupted clean action

2021-12-29 Thread GitBox
danny0405 closed pull request #3386: URL: https://github.com/apache/hudi/pull/3386 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[GitHub] [hudi] danny0405 commented on pull request #3386: [HUDI-2270] Remove corrupted clean action

2021-12-29 Thread GitBox
danny0405 commented on pull request #3386: URL: https://github.com/apache/hudi/pull/3386#issuecomment-1002893737 Close because #4016 solves the problem. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above

[GitHub] [hudi] LuPan2015 commented on issue #4474: [SUPPORT] Should we shade all aws dependencies to avoid class conflicts?

2021-12-29 Thread GitBox
LuPan2015 commented on issue #4474: URL: https://github.com/apache/hudi/issues/4474#issuecomment-1002893633 yes. But it works fine。 Next I need to store the metadata in glue。 Thanks. -- This is an automated message from the Apache Git Service. To respond to the message, please

[GitHub] [hudi] boneanxs commented on issue #4474: [SUPPORT] Should we shade all aws dependencies to avoid class conflicts?

2021-12-29 Thread GitBox
boneanxs commented on issue #4474: URL: https://github.com/apache/hudi/issues/4474#issuecomment-1002890635 > Error in query: Specified schema in create table statement is not equal to the table schema.You should not specify the schema for an exist table: `default`.`hudi_mor_s32`

[GitHub] [hudi] YannByron commented on issue #4429: [SUPPORT] Spark SQL CTAS command doesn't work with 0.10.0 version and Spark 3.1.1

2021-12-29 Thread GitBox
YannByron commented on issue #4429: URL: https://github.com/apache/hudi/issues/4429#issuecomment-1002887429 @vingov as the picture I mentioned above, need to `set spark.sql.datetime.java8API.enabled=false;` manually at this time. And i also try to improve it that don't need set by

[GitHub] [hudi] LuPan2015 edited a comment on issue #4474: [SUPPORT] Should we shade all aws dependencies to avoid class conflicts?

2021-12-29 Thread GitBox
LuPan2015 edited a comment on issue #4474: URL: https://github.com/apache/hudi/issues/4474#issuecomment-1002882980 I tried it, but the following exception was still thrown。 ``` spark/bin/spark-sql --packages org.apache.hadoop:hadoop-aws:3.2.0,com.amazonaws:aws-java-sdk:1.12.22 --jars

[GitHub] [hudi] LuPan2015 commented on issue #4474: [SUPPORT] Should we shade all aws dependencies to avoid class conflicts?

2021-12-29 Thread GitBox
LuPan2015 commented on issue #4474: URL: https://github.com/apache/hudi/issues/4474#issuecomment-1002882980 I tried it, but the following exception was still thrown。 ``` spark/bin/spark-sql --packages org.apache.hadoop:hadoop-aws:3.2.0,com.amazonaws:aws-java-sdk:1.12.22 --jars

[GitHub] [hudi] vingov edited a comment on issue #4429: [SUPPORT] Spark SQL CTAS command doesn't work with 0.10.0 version and Spark 3.1.1

2021-12-29 Thread GitBox
vingov edited a comment on issue #4429: URL: https://github.com/apache/hudi/issues/4429#issuecomment-1002881175 @YannByron - Thanks for the quick turnaround, I appreciate it! @xushiyan - There are more errors with Spark 3.1.2 as well, see below: ``` spark-sql> create table

[jira] [Updated] (HUDI-3120) Cache compactionPlan in buffer

2021-12-29 Thread Danny Chen (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Danny Chen updated HUDI-3120: - Fix Version/s: 0.11.0 0.10.1 > Cache compactionPlan in buffer >

[GitHub] [hudi] danny0405 commented on a change in pull request #4463: [HUDI-3120] Cache compactionPlan in buffer

2021-12-29 Thread GitBox
danny0405 commented on a change in pull request #4463: URL: https://github.com/apache/hudi/pull/4463#discussion_r776575452 ## File path: hudi-flink/src/main/java/org/apache/hudi/sink/compact/CompactionCommitSink.java ## @@ -108,8 +124,15 @@ public void

[GitHub] [hudi] danny0405 commented on a change in pull request #4463: [HUDI-3120] Cache compactionPlan in buffer

2021-12-29 Thread GitBox
danny0405 commented on a change in pull request #4463: URL: https://github.com/apache/hudi/pull/4463#discussion_r776575452 ## File path: hudi-flink/src/main/java/org/apache/hudi/sink/compact/CompactionCommitSink.java ## @@ -108,8 +124,15 @@ public void

[jira] [Assigned] (HUDI-3107) Fix HiveSyncTool drop partitions using JDBC

2021-12-29 Thread Raymond Xu (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3107?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raymond Xu reassigned HUDI-3107: Assignee: Yue Zhang (was: Yue Zhang) > Fix HiveSyncTool drop partitions using JDBC >

[jira] [Commented] (HUDI-3107) Fix HiveSyncTool drop partitions using JDBC

2021-12-29 Thread Raymond Xu (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17466690#comment-17466690 ] Raymond Xu commented on HUDI-3107: -- [~danielzhang]  > Fix HiveSyncTool drop partitions using JDBC >

[jira] [Comment Edited] (HUDI-3107) Fix HiveSyncTool drop partitions using JDBC

2021-12-29 Thread Raymond Xu (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17466690#comment-17466690 ] Raymond Xu edited comment on HUDI-3107 at 12/30/21, 5:41 AM: - [~danielzhang]

[GitHub] [hudi] vingov commented on issue #4429: [SUPPORT] Spark SQL CTAS command doesn't work with 0.10.0 version and Spark 3.1.1

2021-12-29 Thread GitBox
vingov commented on issue #4429: URL: https://github.com/apache/hudi/issues/4429#issuecomment-1002881175 @YannByron - Thanks for the quick turnaround, I appreciate it! @xushiyan - There are more errors with Spark 3.1.2 as well, see below: ``` spark-sql> create table h0_p

[jira] [Commented] (HUDI-3107) Fix HiveSyncTool drop partitions using JDBC

2021-12-29 Thread Yue Zhang (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17466689#comment-17466689 ] Yue Zhang commented on HUDI-3107: - Hi Raymond, this is the wrong GitHub id. -- Yue (Daniel) Zhang,

[jira] [Updated] (HUDI-3106) Fix HiveSyncTool not sync schema

2021-12-29 Thread Raymond Xu (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3106?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raymond Xu updated HUDI-3106: - Reviewers: Raymond Xu > Fix HiveSyncTool not sync schema > > >

[jira] [Updated] (HUDI-2990) Sync to HMS when deleting partitions

2021-12-29 Thread Raymond Xu (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raymond Xu updated HUDI-2990: - Reviewers: Raymond Xu > Sync to HMS when deleting partitions > > >

[jira] [Updated] (HUDI-3107) Fix HiveSyncTool drop partitions using JDBC

2021-12-29 Thread Raymond Xu (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3107?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raymond Xu updated HUDI-3107: - Reviewers: Raymond Xu > Fix HiveSyncTool drop partitions using JDBC >

[jira] [Updated] (HUDI-2426) spark sql extensions breaks read.table from metastore

2021-12-29 Thread Raymond Xu (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2426?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raymond Xu updated HUDI-2426: - Reviewers: Raymond Xu > spark sql extensions breaks read.table from metastore >

[jira] [Updated] (HUDI-2611) `create table if not exists` should print message instead of throwing error

2021-12-29 Thread Raymond Xu (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raymond Xu updated HUDI-2611: - Reviewers: Raymond Xu > `create table if not exists` should print message instead of throwing error >

[jira] [Updated] (HUDI-2915) Fix field not found in record error for spark-sql

2021-12-29 Thread Raymond Xu (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2915?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raymond Xu updated HUDI-2915: - Reviewers: Raymond Xu > Fix field not found in record error for spark-sql >

[jira] [Updated] (HUDI-1850) Read on table fails if the first write to table failed

2021-12-29 Thread Raymond Xu (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1850?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raymond Xu updated HUDI-1850: - Reviewers: Raymond Xu > Read on table fails if the first write to table failed >

[jira] [Updated] (HUDI-2661) java.lang.NoSuchMethodError: org.apache.spark.sql.catalyst.catalog.CatalogTable.copy

2021-12-29 Thread Raymond Xu (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2661?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raymond Xu updated HUDI-2661: - Reviewers: Raymond Xu > java.lang.NoSuchMethodError: >

[jira] [Assigned] (HUDI-3125) Spark SQL writing timestamp type don't need to disable `spark.sql.datetime.java8API.enabled` manually

2021-12-29 Thread Raymond Xu (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3125?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raymond Xu reassigned HUDI-3125: Assignee: Yann Byron > Spark SQL writing timestamp type don't need to disable >

[jira] [Updated] (HUDI-3125) Spark SQL writing timestamp type don't need to disable `spark.sql.datetime.java8API.enabled` manually

2021-12-29 Thread Raymond Xu (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3125?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raymond Xu updated HUDI-3125: - Reviewers: Raymond Xu > Spark SQL writing timestamp type don't need to disable >

[jira] [Updated] (HUDI-3104) Hudi-kafka-connect can not scan hadoop config files by HADOOP_CONF_DIR

2021-12-29 Thread Raymond Xu (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3104?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raymond Xu updated HUDI-3104: - Fix Version/s: 0.11.0 > Hudi-kafka-connect can not scan hadoop config files by HADOOP_CONF_DIR >

[jira] [Updated] (HUDI-3112) KafkaConnect can not sync to Hive

2021-12-29 Thread Raymond Xu (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3112?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raymond Xu updated HUDI-3112: - Fix Version/s: 0.11.0 > KafkaConnect can not sync to Hive > - > >

[jira] [Updated] (HUDI-3131) Spark3.1.1 CTAS error

2021-12-29 Thread Raymond Xu (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raymond Xu updated HUDI-3131: - Sprint: Hudi-Sprint-0.10.1 > Spark3.1.1 CTAS error > - > > Key:

[jira] [Updated] (HUDI-3131) Spark3.1.1 CTAS error

2021-12-29 Thread Raymond Xu (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raymond Xu updated HUDI-3131: - Reviewers: Raymond Xu > Spark3.1.1 CTAS error > - > > Key: HUDI-3131

[jira] [Assigned] (HUDI-3131) Spark3.1.1 CTAS error

2021-12-29 Thread Raymond Xu (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raymond Xu reassigned HUDI-3131: Assignee: Yann Byron > Spark3.1.1 CTAS error > - > > Key:

[jira] [Commented] (HUDI-1079) Cannot upsert on schema with Array of Record with single field

2021-12-29 Thread Raymond Xu (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17466687#comment-17466687 ] Raymond Xu commented on HUDI-1079: -- This should be resolved by having parquet 1.12, which will be the

[jira] [Commented] (HUDI-2323) Upsert of Case Class with single field causes SchemaParseException

2021-12-29 Thread Raymond Xu (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17466686#comment-17466686 ] Raymond Xu commented on HUDI-2323: -- This should be resolved by having parquet 1.12, which will be the

[GitHub] [hudi] hudi-bot removed a comment on pull request #4476: [HUDI-3131] fix ctas error in spark3.1.1

2021-12-29 Thread GitBox
hudi-bot removed a comment on pull request #4476: URL: https://github.com/apache/hudi/pull/4476#issuecomment-1002862870 ## CI report: * 037786516e6622a5e60d1543bef8ad77ed39b490 Azure:

[GitHub] [hudi] hudi-bot commented on pull request #4476: [HUDI-3131] fix ctas error in spark3.1.1

2021-12-29 Thread GitBox
hudi-bot commented on pull request #4476: URL: https://github.com/apache/hudi/pull/4476#issuecomment-1002873623 ## CI report: * 037786516e6622a5e60d1543bef8ad77ed39b490 Azure:

[jira] [Updated] (HUDI-2735) Fix archival of commits in Java client for Kafka Connect

2021-12-29 Thread Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo updated HUDI-2735: Story Points: 2 > Fix archival of commits in Java client for Kafka Connect >

[GitHub] [hudi] zuyanton commented on issue #4457: [SUPPORT] Hudi archive stopped working

2021-12-29 Thread GitBox
zuyanton commented on issue #4457: URL: https://github.com/apache/hudi/issues/4457#issuecomment-1002870461 @nsivabalan do you mean content of .hoodie folder ? - its bunch of old commit files plus following subfolders ".aux", ".temp", "metadata", "archived"

[GitHub] [hudi] yuzhaojing commented on a change in pull request #4463: [HUDI-3120] Cache compactionPlan in buffer

2021-12-29 Thread GitBox
yuzhaojing commented on a change in pull request #4463: URL: https://github.com/apache/hudi/pull/4463#discussion_r776565278 ## File path: hudi-flink/src/main/java/org/apache/hudi/sink/compact/CompactionCommitSink.java ## @@ -108,8 +124,15 @@ public void

[GitHub] [hudi] danny0405 commented on a change in pull request #4016: [HUDI-2675] Fix the exception 'Not an Avro data file' when archive and clean

2021-12-29 Thread GitBox
danny0405 commented on a change in pull request #4016: URL: https://github.com/apache/hudi/pull/4016#discussion_r776562829 ## File path: hudi-common/src/main/java/org/apache/hudi/common/table/timeline/HoodieActiveTimeline.java ## @@ -190,12 +190,36 @@ public static void

[jira] [Closed] (HUDI-2658) When disable auto clean, do not check if MIN_COMMITS_TO_KEEP was larger CLEANER_COMMITS_RETAINED

2021-12-29 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2658?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan closed HUDI-2658. - Resolution: Invalid > When disable auto clean, do not check if MIN_COMMITS_TO_KEEP was

[jira] [Commented] (HUDI-2658) When disable auto clean, do not check if MIN_COMMITS_TO_KEEP was larger CLEANER_COMMITS_RETAINED

2021-12-29 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2658?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17466674#comment-17466674 ] sivabalan narayanan commented on HUDI-2658: --- Closing as invalid.  comment from the PR I think

[GitHub] [hudi] danny0405 commented on a change in pull request #4016: [HUDI-2675] Fix the exception 'Not an Avro data file' when archive and clean

2021-12-29 Thread GitBox
danny0405 commented on a change in pull request #4016: URL: https://github.com/apache/hudi/pull/4016#discussion_r776561623 ## File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/HoodieTimelineArchiveLog.java ## @@ -321,10 +321,19 @@ public void

[GitHub] [hudi] hudi-bot removed a comment on pull request #4476: [HUDI-3131] fix ctas error in spark3.1.1

2021-12-29 Thread GitBox
hudi-bot removed a comment on pull request #4476: URL: https://github.com/apache/hudi/pull/4476#issuecomment-1002862236 ## CI report: * 037786516e6622a5e60d1543bef8ad77ed39b490 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot

[GitHub] [hudi] hudi-bot commented on pull request #4476: [HUDI-3131] fix ctas error in spark3.1.1

2021-12-29 Thread GitBox
hudi-bot commented on pull request #4476: URL: https://github.com/apache/hudi/pull/4476#issuecomment-1002862870 ## CI report: * 037786516e6622a5e60d1543bef8ad77ed39b490 Azure:

[GitHub] [hudi] nsivabalan commented on issue #3879: [SUPPORT] Incomplete Table Migration

2021-12-29 Thread GitBox
nsivabalan commented on issue #3879: URL: https://github.com/apache/hudi/issues/3879#issuecomment-1002862574 sure. let us know once you have the dataset available to share. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub

[GitHub] [hudi] hudi-bot commented on pull request #4476: [HUDI-3131] fix ctas error in spark3.1.1

2021-12-29 Thread GitBox
hudi-bot commented on pull request #4476: URL: https://github.com/apache/hudi/pull/4476#issuecomment-1002862236 ## CI report: * 037786516e6622a5e60d1543bef8ad77ed39b490 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure`

[GitHub] [hudi] danny0405 commented on a change in pull request #4463: [HUDI-3120] Cache compactionPlan in buffer

2021-12-29 Thread GitBox
danny0405 commented on a change in pull request #4463: URL: https://github.com/apache/hudi/pull/4463#discussion_r776560447 ## File path: hudi-flink/src/main/java/org/apache/hudi/sink/compact/CompactionCommitSink.java ## @@ -108,8 +124,15 @@ public void

[GitHub] [hudi] YannByron commented on issue #4429: [SUPPORT] Spark SQL CTAS command doesn't work with 0.10.0 version and Spark 3.1.1

2021-12-29 Thread GitBox
YannByron commented on issue #4429: URL: https://github.com/apache/hudi/issues/4429#issuecomment-1002861916 @vingov @xushiyan I can reproduce this in spark3.1.1. It's a real bug, and i've opened a ticket and committed a pr for this, #4476 . -- This is an automated message from the

[jira] [Updated] (HUDI-3131) Spark3.1.1 CTAS error

2021-12-29 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HUDI-3131: - Labels: pull-request-available (was: ) > Spark3.1.1 CTAS error > - > >

[jira] [Commented] (HUDI-3124) Bootstrap when timeline have completed instant

2021-12-29 Thread Danny Chen (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17466672#comment-17466672 ] Danny Chen commented on HUDI-3124: -- Fixed via master branch: 0f0088fe4b740c4acec0cb25988250db8fb483b6 >

[GitHub] [hudi] nsivabalan closed issue #4432: [SUPPORT] The parquet file size exceeds the configured value

2021-12-29 Thread GitBox
nsivabalan closed issue #4432: URL: https://github.com/apache/hudi/issues/4432 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[GitHub] [hudi] nsivabalan commented on issue #4432: [SUPPORT] The parquet file size exceeds the configured value

2021-12-29 Thread GitBox
nsivabalan commented on issue #4432: URL: https://github.com/apache/hudi/issues/4432#issuecomment-1002861656 please re-open if the proposed solution does not work.thanks -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

[GitHub] [hudi] YannByron opened a new pull request #4476: [HUDI-3131] fix ctas error in spark3.1.1

2021-12-29 Thread GitBox
YannByron opened a new pull request #4476: URL: https://github.com/apache/hudi/pull/4476 ## *Tips* - *Thank you very much for contributing to Apache Hudi.* - *Please review https://hudi.apache.org/contribute/how-to-contribute before opening a pull request.* ## What is the

[jira] [Updated] (HUDI-3124) Bootstrap when timeline have completed instant

2021-12-29 Thread Danny Chen (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3124?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Danny Chen updated HUDI-3124: - Fix Version/s: 0.11.0 > Bootstrap when timeline have completed instant >

[jira] [Created] (HUDI-3131) Spark3.1.1 CTAS error

2021-12-29 Thread Yann Byron (Jira)
Yann Byron created HUDI-3131: Summary: Spark3.1.1 CTAS error Key: HUDI-3131 URL: https://issues.apache.org/jira/browse/HUDI-3131 Project: Apache Hudi Issue Type: Bug Components: Spark

[jira] [Resolved] (HUDI-3124) Bootstrap when timeline have completed instant

2021-12-29 Thread Danny Chen (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3124?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Danny Chen resolved HUDI-3124. -- > Bootstrap when timeline have completed instant > -- > >

[hudi] branch master updated (436becf -> 0f0088f)

2021-12-29 Thread danny0405
This is an automated email from the ASF dual-hosted git repository. danny0405 pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/hudi.git. from 436becf [HUDI-2675] Fix the exception 'Not an Avro data file' when archive and clean (#4016) add 0f0088f

[GitHub] [hudi] danny0405 merged pull request #4467: [HUDI-3124] Bootstrap when timeline have completed instant

2021-12-29 Thread GitBox
danny0405 merged pull request #4467: URL: https://github.com/apache/hudi/pull/4467 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[GitHub] [hudi] nsivabalan commented on issue #4466: [SUPPORT]ERROR table.HoodieTimelineArchiveLog: Failed to archive commits,Not an Avro data file

2021-12-29 Thread GitBox
nsivabalan commented on issue #4466: URL: https://github.com/apache/hudi/issues/4466#issuecomment-1002861348 We have a fix [here](https://github.com/apache/hudi/pull/4016). Can you try out the patch. -- This is an automated message from the Apache Git Service. To respond to the

[hudi] branch master updated (674c149 -> 436becf)

2021-12-29 Thread sivabalan
This is an automated email from the ASF dual-hosted git repository. sivabalan pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/hudi.git. from 674c149 [HUDI-3083] Support component data types for flink bulk_insert (#4470) add 436becf [HUDI-2675] Fix

[GitHub] [hudi] nsivabalan merged pull request #4016: [HUDI-2675] Fix the exception 'Not an Avro data file' when archive and clean

2021-12-29 Thread GitBox
nsivabalan merged pull request #4016: URL: https://github.com/apache/hudi/pull/4016 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[GitHub] [hudi] hudi-bot commented on pull request #4473: [HUDI-2590] Adding tests to validate different key generators

2021-12-29 Thread GitBox
hudi-bot commented on pull request #4473: URL: https://github.com/apache/hudi/pull/4473#issuecomment-1002860504 ## CI report: * 35841cdbffb0edd8d7e1f114147b12ee3daf0872 UNKNOWN * b91d9c4a42a05e01ee5a75449e861d9bf88b69c7 Azure:

[GitHub] [hudi] hudi-bot removed a comment on pull request #4473: [HUDI-2590] Adding tests to validate different key generators

2021-12-29 Thread GitBox
hudi-bot removed a comment on pull request #4473: URL: https://github.com/apache/hudi/pull/4473#issuecomment-1002849419 ## CI report: * 35841cdbffb0edd8d7e1f114147b12ee3daf0872 UNKNOWN * b91d9c4a42a05e01ee5a75449e861d9bf88b69c7 Azure:

[GitHub] [hudi] boneanxs commented on issue #4474: [SUPPORT] Should we shade all aws dependencies to avoid class conflicts?

2021-12-29 Thread GitBox
boneanxs commented on issue #4474: URL: https://github.com/apache/hudi/issues/4474#issuecomment-1002859639 For our internal hudi version, we shade aws dependencies, you can add new relocation and build a new bundle package: For example, to shade aws dependencies in spark, add

[GitHub] [hudi] BruceKellan closed issue #4247: [SUPPORT] Unsupport operation exception occur when using flink+hudi in bulk_insert mode

2021-12-29 Thread GitBox
BruceKellan closed issue #4247: URL: https://github.com/apache/hudi/issues/4247 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[GitHub] [hudi] BruceKellan commented on issue #4247: [SUPPORT] Unsupport operation exception occur when using flink+hudi in bulk_insert mode

2021-12-29 Thread GitBox
BruceKellan commented on issue #4247: URL: https://github.com/apache/hudi/issues/4247#issuecomment-1002858083 Thanks. I have seen the PR. [Link](https://github.com/apache/hudi/pull/4470) I will close it. -- This is an automated message from the Apache Git Service. To respond to

  1   2   3   >