tooptoop4 opened a new issue #2393:
URL: https://github.com/apache/hudi/issues/2393
steps taken (3 sequential spark submits):
1. 3 row insert (from a single csv)
2. 3 row update (from a single csv.gz)
3. 19967954 row update (from a single csv.gz)
there are over 23000 parquet files on s3, all around 480kb
```
2020-11-25 09:37:53 0 mytable/.hoodie/.aux/
2020-11-25 11:50:41 0 mytable/.hoodie/.temp/
2020-11-25 09:38:04 2181 mytable/.hoodie/20201125093753.commit
2020-11-25 09:37:54 0
mytable/.hoodie/20201125093753.commit.requested
2020-11-25 09:37:57 350 mytable/.hoodie/20201125093753.inflight
2020-11-25 09:52:49 1159 mytable/.hoodie/20201125095231.clean
2020-11-25 09:52:49 875 mytable/.hoodie/20201125095231.clean.inflight
2020-11-25 09:52:48 875 mytable/.hoodie/20201125095231.clean.requested
2020-11-25 09:52:44 2197 mytable/.hoodie/20201125095231.commit
2020-11-25 09:52:33 0
mytable/.hoodie/20201125095231.commit.requested
2020-11-25 09:52:39 1032 mytable/.hoodie/20201125095231.inflight
2020-11-25 11:50:53 1299 mytable/.hoodie/20201125104200.clean
2020-11-25 11:50:52 945 mytable/.hoodie/20201125104200.clean.inflight
2020-11-25 11:50:51 945 mytable/.hoodie/20201125104200.clean.requested
2020-11-25 11:50:42 26296154 mytable/.hoodie/20201125104200.commit
2020-11-25 10:42:01 0
mytable/.hoodie/20201125104200.commit.requested
2020-11-25 11:02:47 1032 mytable/.hoodie/20201125104200.inflight
2020-11-25 13:14:56 0
mytable/.hoodie/20201125131454.commit.requested
2020-11-25 09:37:53 213 mytable/.hoodie/hoodie.properties
2020-11-25 09:37:59 93 mytable/unknown/.hoodie_partition_metadata
2020-11-25 11:20:01 482269
mytable/unknown/0000c472-f607-408c-8989-66caf3a1f1ff-0_7388-25-7557_20201125104200.parquet
2020-11-25 11:39:11 480273
mytable/unknown/0002d53a-9ab6-4c31-9bd9-507b89d6ac31-0_18623-25-18792_20201125104200.parquet
2020-11-25 11:17:43 481345
mytable/unknown/00031d81-a16e-478c-91b0-fffdb1cc5f3c-0_6128-25-6297_20201125104200.parquet
2020-11-25 11:26:23 481335
mytable/unknown/0003248b-084c-4567-8d47-d20f91af8c29-0_10941-25-11110_20201125104200.parquet
2020-11-25 11:30:49 482571
mytable/unknown/00040167-e102-4777-949d-d61eb834e8d6-0_13618-25-13787_20201125104200.parquet
2020-11-25 11:13:37 483496
mytable/unknown/00050b40-f11e-40ac-a232-36a82af81e38-0_4039-25-4208_20201125104200.parquet
2020-11-25 11:46:12 483891
mytable/unknown/000a8e9c-e54b-4459-b81c-45b8dd82efb9-0_22801-25-22970_20201125104200.parquet
2020-11-25 11:28:10 478902
mytable/unknown/002041f0-d33d-4321-a88a-953f7d2be3f2-0_12040-25-12209_20201125104200.parquet
2020-11-25 11:29:46 481187
mytable/unknown/00255b66-ad0d-4987-8e81-52dd14c17dcc-0_12990-25-13159_20201125104200.parquet
2020-11-25 11:19:04 481080
mytable/unknown/00269076-3986-4014-8540-9e038069b9c0-0_6864-25-7033_20201125104200.parquet
2020-11-25 11:07:09 481899
mytable/unknown/0026d816-001f-4afb-8849-c747fb3c0fa1-0_632-25-801_20201125104200.parquet
2020-11-25 11:08:05 481763
mytable/unknown/002a820e-6e83-44ce-aa72-ae7be1ddc644-0_1116-25-1285_20201125104200.parquet
2020-11-25 11:44:22 482357
mytable/unknown/002abf16-952d-49c3-8de5-09087fff93fd-0_21703-25-21872_20201125104200.parquet
2020-11-25 11:11:20 478808
mytable/unknown/002cb443-5787-42d5-bbcc-a127dad79535-0_2843-25-3012_20201125104200.parquet
2020-11-25 11:44:17 479243
mytable/unknown/00335faa-901e-4567-9478-07b8860be876-0_21662-25-21831_20201125104200.parquet
2020-11-25 11:08:11 481899
mytable/unknown/003bf393-9181-4aa2-b02f-d125931053e0-0_1168-25-1337_20201125104200.parquet
2020-11-25 11:13:52 478852
mytable/unknown/003d948c-b7f7-4737-97a4-6ab688b958f6-0_4183-25-4352_20201125104200.parquet
2020-11-25 11:32:24 480843
mytable/unknown/0040267c-f03f-4558-bd3b-4b010681dfb3-0_14526-25-14695_20201125104200.parquet
2020-11-25 11:12:45 479753
mytable/unknown/004131d9-365d-4a57-b0a0-ec86eca595cf-0_3590-25-3759_20201125104200.parquet
2020-11-25 11:15:21 479938
mytable/unknown/0045ceef-7180-4b7f-a186-2c91a9758b91-0_4886-25-5055_20201125104200.parquet
2020-11-25 11:07:51 481140
mytable/unknown/0048ffa6-4197-48c8-bbdd-95195c5606ad-0_1002-25-1171_20201125104200.parquet
2020-11-25 11:36:49 481792
mytable/unknown/004b470e-0bfb-4122-ac6b-03f5d1333027-0_17224-25-17393_20201125104200.parquet
2020-11-25 11:30:54 480000
mytable/unknown/004c4f54-61be-4baa-9714-fb3a00f10e25-0_13668-25-13837_20201125104200.parquet
2020-11-25 11:26:30 481597
mytable/unknown/004cb4b5-3f8b-4d14-8fd1-042a9c5fa59a-0_11016-25-11185_20201125104200.parquet
2020-11-25 11:40:26 483142
mytable/unknown/004ec33a-7715-4769-8b64-e83c26b0b7e8-0_19330-25-19499_20201125104200.parquet
2020-11-25 11:38:09 481798
mytable/unknown/0050e53f-9972-47b4-96c4-e0087f3f5be0-0_18001-25-18170_20201125104200.parquet
2020-11-25 11:43:13 483521
mytable/unknown/0052dd7c-0e92-4268-8812-a6ed53dc18e6-0_21049-25-21218_20201125104200.parquet
2020-11-25 11:42:58 479095
mytable/unknown/00553bac-c633-4a57-9aec-b96d1e71d819-0_20907-25-21076_20201125104200.parquet
2020-11-25 11:43:17 479115
mytable/unknown/0058b4db-fc13-47f3-948b-67641363f10a-0_21082-25-21251_20201125104200.parquet
2020-11-25 11:07:18 481308
mytable/unknown/0059e0da-ea61-431f-81ec-3d79e981554b-0_721-25-890_20201125104200.parquet
2020-11-25 11:31:57 480673
mytable/unknown/005a4171-0405-4fc1-a336-dee2097574be-0_14272-25-14441_20201125104200.parquet
2020-11-25 11:28:47 481765
mytable/unknown/005aa617-2670-4e2a-9642-bba82e68c23b-0_12395-25-12564_20201125104200.parquet
2020-11-25 11:15:23 482319
mytable/unknown/005d7704-2d8b-4522-a1cf-8d04f9b4be36-0_4901-25-5070_20201125104200.parquet
2020-11-25 11:44:32 480161
mytable/unknown/0060462c-4736-43b9-801e-47d889149d37-0_21792-25-21961_20201125104200.parquet
2020-11-25 11:34:35 481754
mytable/unknown/006131ee-af8d-48b3-aafe-448bacd13fcc-0_15856-25-16025_20201125104200.parquet
2020-11-25 11:06:19 485536
mytable/unknown/0063330a-9682-48de-bc5d-b67fa13ec999-0_167-25-336_20201125104200.parquet
2020-11-25 11:08:17 483132
mytable/unknown/0064d620-68e8-43a6-b473-106275d47154-0_1228-25-1397_20201125104200.parquet
2020-11-25 11:19:00 481348
mytable/unknown/006639d5-5d63-4160-994b-6d966d851aff-0_6823-25-6992_20201125104200.parquet
2020-11-25 11:09:03 480887
mytable/unknown/00668c0c-a6b5-4312-93a4-6f248161b3b4-0_1641-25-1810_20201125104200.parquet
2020-11-25 11:41:54 480986
mytable/unknown/00693591-7e2b-4aef-9cf2-62dd61c414e7-0_20260-25-20429_20201125104200.parquet
2020-11-25 11:11:50 481486
mytable/unknown/0075b246-fa43-48f0-844f-6ccb6893c574-0_3105-25-3274_20201125104200.parquet
2020-11-25 11:06:46 480313
mytable/unknown/0075c608-0a47-4492-9a6f-0a9365d6719f-0_404-25-573_20201125104200.parquet
2020-11-25 11:46:12 482541
mytable/unknown/0078d5fe-340f-4ba9-a154-21caa6d74f81-0_22798-25-22967_20201125104200.parquet
2020-11-25 11:30:01 483351
mytable/unknown/0079fde5-5899-4efa-96a0-ddf1a3f8cd81-0_13166-25-13335_20201125104200.parquet
2020-11-25 11:13:19 482010
mytable/unknown/007c3edb-9d58-4236-9276-aa4afb5ff291-0_3889-25-4058_20201125104200.parquet
2020-11-25 11:15:51 481009
mytable/unknown/007d0204-1ed3-4632-a606-32905b38bcd7-0_5164-25-5333_20201125104200.parquet
2020-11-25 11:24:11 483761
mytable/unknown/007fdf3d-1ccc-42a7-b484-71bbf3bdbfe4-0_9718-25-9887_20201125104200.parquet
2020-11-25 11:34:31 481006
mytable/unknown/00892b02-98e9-4185-9904-9cee3e1ba363-0_15816-25-15985_20201125104200.parquet
2020-11-25 11:23:54 481104
mytable/unknown/008a3f28-dace-4b83-837d-31dd5a5dc752-0_9543-25-9712_20201125104200.parquet
2020-11-25 11:46:31 480697
mytable/unknown/008d05ab-ddea-41d8-9e19-b6c2759aba45-0_22969-25-23138_20201125104200.parquet
2020-11-25 11:38:18 481123
mytable/unknown/009ef3ad-7c36-4cd8-8a6c-b2a8f48f2833-0_18072-25-18241_20201125104200.parquet
```
...the list of parquet files continues (around 23000)
commits show
```
║ CommitTime │ Total Bytes Written │ Total Files Added │ Total Files
Updated │ Total Partitions Written │ Total Records Written │ Total Update
Records Written │ Total Errors ║
╠════════════════╪═════════════════════╪═══════════════════╪═════════════════════╪══════════════════════════╪═══════════════════════╪══════════════════════════════╪══════════════╣
║ 20201125104200 │ 10.4 GB │ 23164 │ 1
│ 1 │ 19967954 │ 3
│ 0 ║
╟────────────────┼─────────────────────┼───────────────────┼─────────────────────┼──────────────────────────┼───────────────────────┼──────────────────────────────┼──────────────╢
║ 20201125095231 │ 427.2 KB │ 0 │ 1
│ 1 │ 3 │ 3
│ 0 ║
╟────────────────┼─────────────────────┼───────────────────┼─────────────────────┼──────────────────────────┼───────────────────────┼──────────────────────────────┼──────────────╢
║ 20201125093753 │ 427.2 KB │ 1 │ 0
│ 1 │ 3 │ 0
│ 0 ║
╚════════════════════════════════╧════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════
```
commit showpartitions --commit 20201125104200 --desc true --limit 10
```
║ Partition Path │ Total Files Added │ Total Files Updated │ Total Records
Inserted │ Total Records Updated │ Total Bytes Written │ Total Errors ║
╠════════════════╪═══════════════════╪═════════════════════╪════════════════════════╪═══════════════════════╪═════════════════════╪══════════════╣
║ unknown │ 23164 │ 1 │ 19967951
│ 3 │ 10.4 GB │ 0 ║
╚════════════════════════════════╧════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════
```
commit showfiles --commit 20201125104200
(there were 46350 lines returned from that cli command, just showing a few
below)
```
║ Partition Path │ File ID │ Previous Commit
│ Total Records Updated │ Total Records Written │ Total Bytes Written │ Total
Errors │ File Size ║
╠════════════════╪════════════════════════════════════════╪═════════════════╪═══════════════════════╪═══════════════════════╪═════════════════════╪══════════════╪═══════════╣
║ unknown │ ee95426b-4708-4804-88a6-515ac04cbf66-0 │ 20201125095231
│ 3 │ 23825 │ 1545005 │ 0
│ 1545005 ║
╟────────────────┼────────────────────────────────────────┼─────────────────┼───────────────────────┼───────────────────────┼─────────────────────┼──────────────┼───────────╢
║ unknown │ 87762d9c-518e-48c7-a717-9f3d12ebf02d-0 │ null
│ 0 │ 891 │ 483220 │ 0
│ 483220 ║
╟────────────────┼────────────────────────────────────────┼─────────────────┼───────────────────────┼───────────────────────┼─────────────────────┼──────────────┼───────────╢
║ unknown │ 74d4703a-f9fc-4c49-96d2-724fc1c225de-0 │ null
│ 0 │ 898 │ 483704 │ 0
│ 483704 ║
╟────────────────┼────────────────────────────────────────┼─────────────────┼───────────────────────┼───────────────────────┼─────────────────────┼──────────────┼───────────╢
║ unknown │ efd84547-0899-4517-b3ce-3832f9acb2da-0 │ null
│ 0 │ 892 │ 483882 │ 0
│ 483882 ║
╟────────────────┼────────────────────────────────────────┼─────────────────┼───────────────────────┼───────────────────────┼─────────────────────┼──────────────┼───────────╢
║ unknown │ 95f4567b-f94d-48b3-921a-5efefc69b42e-0 │ null
│ 0 │ 870 │ 482258 │ 0
│ 482258 ║
╟────────────────┼────────────────────────────────────────┼─────────────────┼───────────────────────┼───────────────────────┼─────────────────────┼──────────────┼───────────╢
║ unknown │ 08b2c30d-a03e-4df8-9202-75a07932e337-0 │ null
│ 0 │ 872 │ 481902 │ 0
│ 481902 ║
╟────────────────┼────────────────────────────────────────┼─────────────────┼───────────────────────┼───────────────────────┼─────────────────────┼──────────────┼───────────╢
║ unknown │ 161b5e44-b676-4b31-9301-ece6645cec46-0 │ null
│ 0 │ 867 │ 482395 │ 0
│ 482395 ║
```
**stats wa**
```
║ CommitTime │ Total Upserted │ Total Written │ Write Amplification
Factor ║
╠════════════════╪════════════════╪═══════════════╪════════════════════════════╣
║ 20201125093753 │ 0 │ 3 │ 0
║
╟────────────────┼────────────────┼───────────────┼────────────────────────────╢
║ 20201125095231 │ 3 │ 3 │ 1.00
║
╟────────────────┼────────────────┼───────────────┼────────────────────────────╢
║ 20201125104200 │ 3 │ 19967954 │ 6655984.50
║
╟────────────────┼────────────────┼───────────────┼────────────────────────────╢
║ Total │ 6 │ 19967960 │ 3327993.25
║
╚════════════════╧════════════════╧═══════════════╧════════════════════════════╝
```
**Expected behavior**
i would expect much less parquet files, each of larger size (ie around 110mb
each)
**Environment Description**
Hudi version : 0.5.3
Spark version : 2.4.6
Hive version : 2.3.4
Hadoop version : 2.8.5
Storage (HDFS/S3/GCS..) : S3
Running on Docker? (yes/no) : no
COW table, spark datasource
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
