n3nash edited a comment on pull request #2424:
URL: https://github.com/apache/hudi/pull/2424#issuecomment-758475039
@pratyakshsharma in that case, can you review this PR ? @prashantwason Had
missed to push some local changes, can you take another pass, I think it should
address all your
yanghua edited a comment on pull request #2433:
URL: https://github.com/apache/hudi/pull/2433#issuecomment-758474842
> The file check of each task is useless because even if a task of the
source has no data for some time interval, the checkpoint still can trigger
normally. So all task
n3nash commented on a change in pull request #2424:
URL: https://github.com/apache/hudi/pull/2424#discussion_r71734
##
File path: hudi-common/src/main/java/org/apache/hudi/avro/HoodieAvroUtils.java
##
@@ -292,53 +284,57 @@ public static GenericRecord
n3nash commented on pull request #2424:
URL: https://github.com/apache/hudi/pull/2424#issuecomment-758475039
@pratyakshsharma in that case, can you review this PR ?
This is an automated message from the Apache Git Service.
yanghua commented on pull request #2433:
URL: https://github.com/apache/hudi/pull/2433#issuecomment-758474842
> The file check of each task is useless because even if a task of the
source has no data for some time interval, the checkpoint still can trigger
normally. So all task checkpoint
pratyakshsharma commented on pull request #2424:
URL: https://github.com/apache/hudi/pull/2424#issuecomment-758461542
@n3nash In my previous org, we were dealing with a similar scenario where
fields were getting deleted from few tables in production. Yeah parquet-avro
reader will throw
loukey-lj opened a new pull request #2434:
URL: https://github.com/apache/hudi/pull/2434
InstantGenerateOperator support multiple parallelism.
When InstantGenerateOperator subtask size greater than 1 we can set subtask
0 as a main subtask, only main task create new instant.
The
n3nash commented on pull request #2424:
URL: https://github.com/apache/hudi/pull/2424#issuecomment-758449461
@pratyakshsharma Do you have a use-case of deleting fields ? What is the
reason for supporting deleting fields. Has deleting fields case been tested for
all types of cases such as
loukey-lj closed pull request #2433:
URL: https://github.com/apache/hudi/pull/2433
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to
bvaradar closed issue #2414:
URL: https://github.com/apache/hudi/issues/2414
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the
bvaradar edited a comment on issue #2423:
URL: https://github.com/apache/hudi/issues/2423#issuecomment-758433327
Hudi does not synchronize on partition path creation. Instead, each executor
task which is about to write to a parquet file ensures the directory path
exists by issuing
bvaradar commented on issue #2423:
URL: https://github.com/apache/hudi/issues/2423#issuecomment-758433327
Hudi does not synchronize on partition path creation. Instead, each executor
task which is about to write to a parquet file ensures the directory path
exists by issuing fs.mkdirs
Balaji Varadarajan created HUDI-1523:
Summary: Avoid excessive mkdir calls when creating new files
Key: HUDI-1523
URL: https://issues.apache.org/jira/browse/HUDI-1523
Project: Apache Hudi
Karl-WangSK commented on pull request #2260:
URL: https://github.com/apache/hudi/pull/2260#issuecomment-758424635
@wangxianghu
This is an automated message from the Apache Git Service.
To respond to the message, please log
bvaradar commented on issue #2432:
URL: https://github.com/apache/hudi/issues/2432#issuecomment-758403084
@quitozang : Binding to port 0 should ensure that OS assigns a random free
port. I am not sure why you are seeing the error. You can workaround by setting
danny0405 commented on pull request #2433:
URL: https://github.com/apache/hudi/pull/2433#issuecomment-758372866
The file check of each task is useless because even if a task of the source
has no data for some time interval, the checkpoint still can trigger normally.
So all task checkpoint
jtmzheng commented on issue #2408:
URL: https://github.com/apache/hudi/issues/2408#issuecomment-758360941
Thanks Udit! I'd tried setting `hoodie.commits.archival.batch` to 5 earlier
today after going through the source code - that got my application back and
running again.
The
garyli1019 commented on a change in pull request #2412:
URL: https://github.com/apache/hudi/pull/2412#discussion_r555477972
##
File path: pom.xml
##
@@ -1361,6 +1363,7 @@
${fasterxml.spark3.version}
${fasterxml.spark3.version}
lw309637554 commented on pull request #2418:
URL: https://github.com/apache/hudi/pull/2418#issuecomment-758337679
LGTM
This is an automated message from the Apache Git Service.
To respond to the message, please log on to
lw309637554 commented on a change in pull request #2418:
URL: https://github.com/apache/hudi/pull/2418#discussion_r555456066
##
File path:
hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/table/action/rollback/HoodieClientRollbackTestBase.java
##
@@ -96,4 +99,61 @@
[
https://issues.apache.org/jira/browse/HUDI-1520?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
liwei resolved HUDI-1520.
-
Resolution: Fixed
> add configure for spark sql overwrite use replace
>
yanghua commented on pull request #2433:
URL: https://github.com/apache/hudi/pull/2433#issuecomment-758327239
@danny0405 wdyt about this optimization?
This is an automated message from the Apache Git Service.
To respond to
[
https://issues.apache.org/jira/browse/HUDI-1511?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HUDI-1511:
-
Labels: pull-request-available (was: )
> InstantGenerateOperator support multiple parallelism
>
yanghua commented on pull request #2433:
URL: https://github.com/apache/hudi/pull/2433#issuecomment-758321935
@loukey-lj thanks for your contribution! Can you please:
1) Fix the Travis issue? It's red now;
2) Update the RFC-13 and describe your optimization.
umehrot2 commented on issue #2408:
URL: https://github.com/apache/hudi/issues/2408#issuecomment-758321313
For now, I would suggest to archive at smaller intervals. May be try out
something like:
- `hoodie.keep.max.commits`: 10
- `hoodie.keep.min.commits`: 10
umehrot2 commented on issue #2408:
URL: https://github.com/apache/hudi/issues/2408#issuecomment-758320870
I took a deeper look at this. For you this seems to be happening in the
archival code path:
```
at
This is an automated email from the ASF dual-hosted git repository.
vinoth pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/hudi.git
The following commit(s) were added to refs/heads/master by this push:
new e3d3677 [HUDI-1502] MOR rollback and restore
vinothchandar merged pull request #2421:
URL: https://github.com/apache/hudi/pull/2421
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go
codecov-io edited a comment on pull request #2412:
URL: https://github.com/apache/hudi/pull/2412#issuecomment-755726635
This is an automated message from the Apache Git Service.
To respond to the message, please log on to
codecov-io edited a comment on pull request #2412:
URL: https://github.com/apache/hudi/pull/2412#issuecomment-755726635
# [Codecov](https://codecov.io/gh/apache/hudi/pull/2412?src=pr=h1) Report
> Merging
[#2412](https://codecov.io/gh/apache/hudi/pull/2412?src=pr=desc) (9b9a5c9)
into
codecov-io edited a comment on pull request #2421:
URL: https://github.com/apache/hudi/pull/2421#issuecomment-757112911
# [Codecov](https://codecov.io/gh/apache/hudi/pull/2421?src=pr=h1) Report
> Merging
[#2421](https://codecov.io/gh/apache/hudi/pull/2421?src=pr=desc) (c2647c3)
into
satishkotha commented on issue #2346:
URL: https://github.com/apache/hudi/issues/2346#issuecomment-758169722
@sumihehe Did you get a chance to look at above? It'll be helpful if you
can provide more information.
This is an
pratyakshsharma commented on pull request #2424:
URL: https://github.com/apache/hudi/pull/2424#issuecomment-758162231
@n3nash Just a high level thought before going through the changes
thoroughly. How about keeping the old changes also and introduce a config
pratyakshsharma commented on pull request #2424:
URL: https://github.com/apache/hudi/pull/2424#issuecomment-758154222
> @n3nash what is the commit being reverted?
https://github.com/apache/hudi/commit/6d7ca2cf7e441ad19d32d7a25739e454f39ed253
[
https://issues.apache.org/jira/browse/HUDI-1509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17262865#comment-17262865
]
Pratyaksh Sharma commented on HUDI-1509:
[~nishith29] The PR was a generic one where point #2
prashantwason commented on a change in pull request #2424:
URL: https://github.com/apache/hudi/pull/2424#discussion_r555258349
##
File path: hudi-common/src/main/java/org/apache/hudi/avro/HoodieAvroUtils.java
##
@@ -292,53 +284,57 @@ public static GenericRecord
satishkotha commented on a change in pull request #2418:
URL: https://github.com/apache/hudi/pull/2418#discussion_r555258959
##
File path:
hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/table/action/rollback/HoodieClientRollbackTestBase.java
##
@@ -96,4 +99,61 @@
satishkotha commented on a change in pull request #2418:
URL: https://github.com/apache/hudi/pull/2418#discussion_r555258581
##
File path:
hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/table/action/rollback/HoodieClientRollbackTestBase.java
##
@@ -96,4 +99,61 @@
[
https://issues.apache.org/jira/browse/HUDI-1291?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
satish closed HUDI-1291.
Resolution: Fixed
done as part of HUDI-1276
> integration of replace with consolidated metadata
>
prashantwason commented on a change in pull request #2424:
URL: https://github.com/apache/hudi/pull/2424#discussion_r555221595
##
File path: hudi-common/src/main/java/org/apache/hudi/avro/HoodieAvroUtils.java
##
@@ -292,53 +284,57 @@ public static GenericRecord
prashantwason commented on a change in pull request #2424:
URL: https://github.com/apache/hudi/pull/2424#discussion_r555219601
##
File path: hudi-common/src/main/java/org/apache/hudi/avro/HoodieAvroUtils.java
##
@@ -292,53 +284,57 @@ public static GenericRecord
zhedoubushishi commented on a change in pull request #2412:
URL: https://github.com/apache/hudi/pull/2412#discussion_r555214109
##
File path: pom.xml
##
@@ -110,9 +110,10 @@
2.4.4
3.0.0
1.8.2
-2.11.12
+2.11.12
Review comment:
Make sense to me.
vinothchandar commented on pull request #2421:
URL: https://github.com/apache/hudi/pull/2421#issuecomment-758098600
@nsivabalan pushed some small fixes. Please land once CI passes
This is an automated message from the Apache
vinothchandar commented on a change in pull request #2421:
URL: https://github.com/apache/hudi/pull/2421#discussion_r555199075
##
File path:
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieTableMetadataUtil.java
##
@@ -262,18 +264,33 @@ private static void
vinothchandar merged pull request #2428:
URL: https://github.com/apache/hudi/pull/2428
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go
This is an automated email from the ASF dual-hosted git repository.
vinoth pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/hudi.git
The following commit(s) were added to refs/heads/master by this push:
new de42adc [HUDI-1520] add configure for spark sql
vinothchandar commented on a change in pull request #2428:
URL: https://github.com/apache/hudi/pull/2428#discussion_r555203377
##
File path:
hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/hudi/HoodieSparkSqlWriter.scala
##
@@ -340,6 +333,12 @@ private[hudi] object
vinothchandar commented on a change in pull request #2428:
URL: https://github.com/apache/hudi/pull/2428#discussion_r555202292
##
File path:
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/TestMORDataSource.scala
##
@@ -278,7 +278,6 @@ class
vinothchandar commented on a change in pull request #2428:
URL: https://github.com/apache/hudi/pull/2428#discussion_r555202292
##
File path:
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/TestMORDataSource.scala
##
@@ -278,7 +278,6 @@ class
pranotishanbhag removed a comment on issue #2414:
URL: https://github.com/apache/hudi/issues/2414#issuecomment-758042202
Hi,
I tried copy_on_write with insert mode for 4.6 TB dataset which is failing
with lost nodes (previously tried bulk_insert which worked fine). I tried to
tweak
pranotishanbhag commented on issue #2414:
URL: https://github.com/apache/hudi/issues/2414#issuecomment-758042202
Hi,
I tried copy_on_write with insert mode for 4.6 TB dataset which is failing
with lost nodes (previously tried bulk_insert which worked fine). I tried to
tweak the
loukey-lj opened a new pull request #2433:
URL: https://github.com/apache/hudi/pull/2433
InstantGenerateOperator support multiple parallelism.
When InstantGenerateOperator subtask size greater than 1 we can set subtask
0 as a main subtask, only main task create new instant.
The
garyli1019 commented on a change in pull request #2378:
URL: https://github.com/apache/hudi/pull/2378#discussion_r555064333
##
File path:
hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/hudi/MergeOnReadSnapshotRelation.scala
##
@@ -108,7 +111,7 @@ class
yui2010 commented on a change in pull request #2427:
URL: https://github.com/apache/hudi/pull/2427#discussion_r555027450
##
File path:
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/storage/HoodieHFileWriter.java
##
@@ -121,17 +121,10 @@ public void
yui2010 commented on a change in pull request #2427:
URL: https://github.com/apache/hudi/pull/2427#discussion_r555027450
##
File path:
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/storage/HoodieHFileWriter.java
##
@@ -121,17 +121,10 @@ public void
codecov-io commented on pull request #2431:
URL: https://github.com/apache/hudi/pull/2431#issuecomment-757929313
# [Codecov](https://codecov.io/gh/apache/hudi/pull/2431?src=pr=h1) Report
> Merging
[#2431](https://codecov.io/gh/apache/hudi/pull/2431?src=pr=desc) (fa597aa)
into
quitozang opened a new issue #2432:
URL: https://github.com/apache/hudi/issues/2432
When i write hudi data using DeltaStreamer, sometimes will get this error
below
**Environment Description**
* Hudi version : 0.6.0
* Spark version : 2.4.4
* Hive version :
teeyog opened a new pull request #2431:
URL: https://github.com/apache/hudi/pull/2431
…asource.write.partitionpath.field`
## *Tips*
- *Thank you very much for contributing to Apache Hudi.*
- *Please review https://hudi.apache.org/contributing.html before opening a
pull request.*
codecov-io edited a comment on pull request #2424:
URL: https://github.com/apache/hudi/pull/2424#issuecomment-757403445
# [Codecov](https://codecov.io/gh/apache/hudi/pull/2424?src=pr=h1) Report
> Merging
[#2424](https://codecov.io/gh/apache/hudi/pull/2424?src=pr=desc) (37126a3)
into
codecov-io edited a comment on pull request #2430:
URL: https://github.com/apache/hudi/pull/2430#issuecomment-757736411
# [Codecov](https://codecov.io/gh/apache/hudi/pull/2430?src=pr=h1) Report
> Merging
[#2430](https://codecov.io/gh/apache/hudi/pull/2430?src=pr=desc) (7961488)
into
codecov-io edited a comment on pull request #2430:
URL: https://github.com/apache/hudi/pull/2430#issuecomment-757736411
# [Codecov](https://codecov.io/gh/apache/hudi/pull/2430?src=pr=h1) Report
> Merging
[#2430](https://codecov.io/gh/apache/hudi/pull/2430?src=pr=desc) (7961488)
into
danny0405 commented on a change in pull request #2430:
URL: https://github.com/apache/hudi/pull/2430#discussion_r554904669
##
File path: hudi-flink/src/main/java/org/apache/hudi/HoodieFlinkStreamer.java
##
@@ -160,6 +156,19 @@ public static void main(String[] args) throws
codecov-io commented on pull request #2430:
URL: https://github.com/apache/hudi/pull/2430#issuecomment-757736411
# [Codecov](https://codecov.io/gh/apache/hudi/pull/2430?src=pr=h1) Report
> Merging
[#2430](https://codecov.io/gh/apache/hudi/pull/2430?src=pr=desc) (7961488)
into
63 matches
Mail list logo
