Fokko opened a new pull request, #7588: URL: https://github.com/apache/iceberg/pull/7588
This includes the fix https://github.com/apache/arrow/pull/34015 The patch reduces the number of calls from 38 to 33, less is more! Now: ``` [root@fa7688db87e9 /]# mc admin trace --call s3 minio 2023-05-11T20:06:32.268 [206 Partial Content] s3.GetObject warehouse.minio:9000/nyc/taxis/metadata/00004-5eeae12f-6f3b-4c36-9413-2fbef26a94b7.metadata.json 172.22.0.2 2.791ms ↑ 148 B ↓ 9.6 KiB 2023-05-11T20:06:32.558 [200 OK] s3.HeadObject 127.0.0.1:9000/warehouse/nyc/taxis/metadata/snap-3831732930279655857-1-9f7fb22f-eb4a-42d2-9657-87cc8039004d.avro 172.22.0.1 2.379ms ↑ 153 B ↓ 0 B 2023-05-11T20:06:32.566 [206 Partial Content] s3.GetObject 127.0.0.1:9000/warehouse/nyc/taxis/metadata/snap-3831732930279655857-1-9f7fb22f-eb4a-42d2-9657-87cc8039004d.avro 172.22.0.1 1.925ms ↑ 159 B ↓ 3.9 KiB 2023-05-11T20:06:32.583 [200 OK] s3.HeadObject 127.0.0.1:9000/warehouse/nyc/taxis/metadata/8e396fa9-dc45-4998-8d37-38665bf45724-m0.avro 172.22.0.1 528µs ↑ 153 B ↓ 0 B 2023-05-11T20:06:32.584 [200 OK] s3.HeadObject 127.0.0.1:9000/warehouse/nyc/taxis/metadata/9f7fb22f-eb4a-42d2-9657-87cc8039004d-m0.avro 172.22.0.1 347µs ↑ 153 B ↓ 0 B 2023-05-11T20:06:32.584 [200 OK] s3.HeadObject 127.0.0.1:9000/warehouse/nyc/taxis/metadata/273e2d75-5d3a-4c2c-bebd-a6ef37cadbe2-m0.avro 172.22.0.1 684µs ↑ 153 B ↓ 0 B 2023-05-11T20:06:32.584 [200 OK] s3.HeadObject 127.0.0.1:9000/warehouse/nyc/taxis/metadata/5f6429bd-1ae4-4a54-820e-f53387e04e2b-m0.avro 172.22.0.1 502µs ↑ 153 B ↓ 0 B 2023-05-11T20:06:32.584 [200 OK] s3.HeadObject 127.0.0.1:9000/warehouse/nyc/taxis/metadata/8f666929-42ff-4198-a203-af5c85cfb233-m0.avro 172.22.0.1 962µs ↑ 153 B ↓ 0 B 2023-05-11T20:06:32.589 [206 Partial Content] s3.GetObject 127.0.0.1:9000/warehouse/nyc/taxis/metadata/5f6429bd-1ae4-4a54-820e-f53387e04e2b-m0.avro 172.22.0.1 1.805ms ↑ 159 B ↓ 7.1 KiB 2023-05-11T20:06:32.589 [206 Partial Content] s3.GetObject 127.0.0.1:9000/warehouse/nyc/taxis/metadata/8e396fa9-dc45-4998-8d37-38665bf45724-m0.avro 172.22.0.1 1.54ms ↑ 159 B ↓ 7.1 KiB 2023-05-11T20:06:32.588 [206 Partial Content] s3.GetObject 127.0.0.1:9000/warehouse/nyc/taxis/metadata/9f7fb22f-eb4a-42d2-9657-87cc8039004d-m0.avro 172.22.0.1 2.27ms ↑ 159 B ↓ 7.1 KiB 2023-05-11T20:06:32.589 [206 Partial Content] s3.GetObject 127.0.0.1:9000/warehouse/nyc/taxis/metadata/273e2d75-5d3a-4c2c-bebd-a6ef37cadbe2-m0.avro 172.22.0.1 1.239ms ↑ 159 B ↓ 7.1 KiB 2023-05-11T20:06:32.589 [206 Partial Content] s3.GetObject 127.0.0.1:9000/warehouse/nyc/taxis/metadata/8f666929-42ff-4198-a203-af5c85cfb233-m0.avro 172.22.0.1 2.783ms ↑ 159 B ↓ 7.1 KiB 2023-05-11T20:06:32.618 [200 OK] s3.HeadObject 127.0.0.1:9000/warehouse/nyc/taxis/data/00003-1272-519910f9-3e1b-4a64-8745-3c34d879233d-00001.parquet 172.22.0.1 445µs ↑ 153 B ↓ 0 B 2023-05-11T20:06:32.618 [200 OK] s3.HeadObject 127.0.0.1:9000/warehouse/nyc/taxis/data/00003-1290-df4bb385-1e5f-4841-960f-85600d3708e5-00001.parquet 172.22.0.1 319µs ↑ 153 B ↓ 0 B 2023-05-11T20:06:32.618 [200 OK] s3.HeadObject 127.0.0.1:9000/warehouse/nyc/taxis/data/00003-1254-03dd06d8-f2da-4b9b-89ba-a2421e3f60fd-00001.parquet 172.22.0.1 263µs ↑ 153 B ↓ 0 B 2023-05-11T20:06:32.618 [200 OK] s3.HeadObject 127.0.0.1:9000/warehouse/nyc/taxis/data/00003-1263-f568eb3b-a4d6-4a93-bcae-2b22f8baa1e7-00001.parquet 172.22.0.1 929µs ↑ 153 B ↓ 0 B 2023-05-11T20:06:32.618 [200 OK] s3.HeadObject 127.0.0.1:9000/warehouse/nyc/taxis/data/00003-1281-d941a1d6-d041-4cf3-9e44-778191ae68fb-00001.parquet 172.22.0.1 275µs ↑ 153 B ↓ 0 B 2023-05-11T20:06:32.622 [206 Partial Content] s3.GetObject 127.0.0.1:9000/warehouse/nyc/taxis/data/00003-1290-df4bb385-1e5f-4841-960f-85600d3708e5-00001.parquet 172.22.0.1 2.981ms ↑ 159 B ↓ 64 KiB 2023-05-11T20:06:32.622 [206 Partial Content] s3.GetObject 127.0.0.1:9000/warehouse/nyc/taxis/data/00003-1272-519910f9-3e1b-4a64-8745-3c34d879233d-00001.parquet 172.22.0.1 6.991ms ↑ 159 B ↓ 64 KiB 2023-05-11T20:06:32.624 [206 Partial Content] s3.GetObject 127.0.0.1:9000/warehouse/nyc/taxis/data/00003-1263-f568eb3b-a4d6-4a93-bcae-2b22f8baa1e7-00001.parquet 172.22.0.1 4.944ms ↑ 159 B ↓ 64 KiB 2023-05-11T20:06:32.624 [206 Partial Content] s3.GetObject 127.0.0.1:9000/warehouse/nyc/taxis/data/00003-1281-d941a1d6-d041-4cf3-9e44-778191ae68fb-00001.parquet 172.22.0.1 6.287ms ↑ 159 B ↓ 64 KiB 2023-05-11T20:06:32.622 [206 Partial Content] s3.GetObject 127.0.0.1:9000/warehouse/nyc/taxis/data/00003-1254-03dd06d8-f2da-4b9b-89ba-a2421e3f60fd-00001.parquet 172.22.0.1 9.967ms ↑ 159 B ↓ 64 KiB 2023-05-11T20:06:32.641 [206 Partial Content] s3.GetObject 127.0.0.1:9000/warehouse/nyc/taxis/data/00003-1281-d941a1d6-d041-4cf3-9e44-778191ae68fb-00001.parquet 172.22.0.1 95.637ms ↑ 159 B ↓ 6.6 MiB 2023-05-11T20:06:32.638 [206 Partial Content] s3.GetObject 127.0.0.1:9000/warehouse/nyc/taxis/data/00003-1272-519910f9-3e1b-4a64-8745-3c34d879233d-00001.parquet 172.22.0.1 711.24ms ↑ 159 B ↓ 15 MiB 2023-05-11T20:06:32.638 [206 Partial Content] s3.GetObject 127.0.0.1:9000/warehouse/nyc/taxis/data/00003-1290-df4bb385-1e5f-4841-960f-85600d3708e5-00001.parquet 172.22.0.1 767.182ms ↑ 159 B ↓ 16 MiB 2023-05-11T20:06:32.640 [206 Partial Content] s3.GetObject 127.0.0.1:9000/warehouse/nyc/taxis/data/00003-1263-f568eb3b-a4d6-4a93-bcae-2b22f8baa1e7-00001.parquet 172.22.0.1 1.694288s ↑ 159 B ↓ 27 MiB 2023-05-11T20:06:32.640 [206 Partial Content] s3.GetObject 127.0.0.1:9000/warehouse/nyc/taxis/data/00003-1263-f568eb3b-a4d6-4a93-bcae-2b22f8baa1e7-00001.parquet 172.22.0.1 1.748438s ↑ 159 B ↓ 27 MiB 2023-05-11T20:06:32.640 [206 Partial Content] s3.GetObject 127.0.0.1:9000/warehouse/nyc/taxis/data/00003-1272-519910f9-3e1b-4a64-8745-3c34d879233d-00001.parquet 172.22.0.1 1.895823s ↑ 159 B ↓ 29 MiB 2023-05-11T20:06:32.638 [206 Partial Content] s3.GetObject 127.0.0.1:9000/warehouse/nyc/taxis/data/00003-1290-df4bb385-1e5f-4841-960f-85600d3708e5-00001.parquet 172.22.0.1 1.925349s ↑ 159 B ↓ 32 MiB 2023-05-11T20:06:32.644 [206 Partial Content] s3.GetObject 127.0.0.1:9000/warehouse/nyc/taxis/data/00003-1281-d941a1d6-d041-4cf3-9e44-778191ae68fb-00001.parquet 172.22.0.1 1.944678s ↑ 159 B ↓ 30 MiB 2023-05-11T20:06:33.200 [206 Partial Content] s3.GetObject 127.0.0.1:9000/warehouse/nyc/taxis/data/00003-1254-03dd06d8-f2da-4b9b-89ba-a2421e3f60fd-00001.parquet 172.22.0.1 1.510093s ↑ 159 B ↓ 27 MiB 2023-05-11T20:06:33.972 [206 Partial Content] s3.GetObject 127.0.0.1:9000/warehouse/nyc/taxis/data/00003-1254-03dd06d8-f2da-4b9b-89ba-a2421e3f60fd-00001.parquet 172.22.0.1 1.326988s ↑ 159 B ↓ 26 MiB ``` Before: ``` ^C[root@fa7688db87e9 /]# mc admin trace --call s3 minio 2023-05-11T20:19:09.894 [206 Partial Content] s3.GetObject warehouse.minio:9000/nyc/taxis/metadata/00004-5eeae12f-6f3b-4c36-9413-2fbef26a94b7.metadata.json 172.22.0.2 897µs ↑ 148 B ↓ 9.6 KiB 2023-05-11T20:19:11.092 [200 OK] s3.HeadObject 127.0.0.1:9000/warehouse/nyc/taxis/metadata/snap-3831732930279655857-1-9f7fb22f-eb4a-42d2-9657-87cc8039004d.avro 172.22.0.1 2.395ms ↑ 153 B ↓ 0 B 2023-05-11T20:19:11.113 [206 Partial Content] s3.GetObject 127.0.0.1:9000/warehouse/nyc/taxis/metadata/snap-3831732930279655857-1-9f7fb22f-eb4a-42d2-9657-87cc8039004d.avro 172.22.0.1 1.439ms ↑ 159 B ↓ 3.9 KiB 2023-05-11T20:19:11.135 [200 OK] s3.HeadObject 127.0.0.1:9000/warehouse/nyc/taxis/metadata/5f6429bd-1ae4-4a54-820e-f53387e04e2b-m0.avro 172.22.0.1 1.135ms ↑ 153 B ↓ 0 B 2023-05-11T20:19:11.136 [200 OK] s3.HeadObject 127.0.0.1:9000/warehouse/nyc/taxis/metadata/273e2d75-5d3a-4c2c-bebd-a6ef37cadbe2-m0.avro 172.22.0.1 376µs ↑ 153 B ↓ 0 B 2023-05-11T20:19:11.136 [200 OK] s3.HeadObject 127.0.0.1:9000/warehouse/nyc/taxis/metadata/8f666929-42ff-4198-a203-af5c85cfb233-m0.avro 172.22.0.1 465µs ↑ 153 B ↓ 0 B 2023-05-11T20:19:11.136 [200 OK] s3.HeadObject 127.0.0.1:9000/warehouse/nyc/taxis/metadata/8e396fa9-dc45-4998-8d37-38665bf45724-m0.avro 172.22.0.1 841µs ↑ 153 B ↓ 0 B 2023-05-11T20:19:11.135 [200 OK] s3.HeadObject 127.0.0.1:9000/warehouse/nyc/taxis/metadata/9f7fb22f-eb4a-42d2-9657-87cc8039004d-m0.avro 172.22.0.1 2.065ms ↑ 153 B ↓ 0 B 2023-05-11T20:19:11.143 [206 Partial Content] s3.GetObject 127.0.0.1:9000/warehouse/nyc/taxis/metadata/9f7fb22f-eb4a-42d2-9657-87cc8039004d-m0.avro 172.22.0.1 1.153ms ↑ 159 B ↓ 7.1 KiB 2023-05-11T20:19:11.144 [206 Partial Content] s3.GetObject 127.0.0.1:9000/warehouse/nyc/taxis/metadata/8e396fa9-dc45-4998-8d37-38665bf45724-m0.avro 172.22.0.1 1.105ms ↑ 159 B ↓ 7.1 KiB 2023-05-11T20:19:11.143 [206 Partial Content] s3.GetObject 127.0.0.1:9000/warehouse/nyc/taxis/metadata/8f666929-42ff-4198-a203-af5c85cfb233-m0.avro 172.22.0.1 2.29ms ↑ 159 B ↓ 7.1 KiB 2023-05-11T20:19:11.143 [206 Partial Content] s3.GetObject 127.0.0.1:9000/warehouse/nyc/taxis/metadata/5f6429bd-1ae4-4a54-820e-f53387e04e2b-m0.avro 172.22.0.1 2.646ms ↑ 159 B ↓ 7.1 KiB 2023-05-11T20:19:11.144 [206 Partial Content] s3.GetObject 127.0.0.1:9000/warehouse/nyc/taxis/metadata/273e2d75-5d3a-4c2c-bebd-a6ef37cadbe2-m0.avro 172.22.0.1 1.914ms ↑ 159 B ↓ 7.1 KiB 2023-05-11T20:19:11.176 [200 OK] s3.HeadObject 127.0.0.1:9000/warehouse/nyc/taxis/data/00003-1281-d941a1d6-d041-4cf3-9e44-778191ae68fb-00001.parquet 172.22.0.1 368µs ↑ 153 B ↓ 0 B 2023-05-11T20:19:11.176 [200 OK] s3.HeadObject 127.0.0.1:9000/warehouse/nyc/taxis/data/00003-1290-df4bb385-1e5f-4841-960f-85600d3708e5-00001.parquet 172.22.0.1 380µs ↑ 153 B ↓ 0 B 2023-05-11T20:19:11.176 [200 OK] s3.HeadObject 127.0.0.1:9000/warehouse/nyc/taxis/data/00003-1272-519910f9-3e1b-4a64-8745-3c34d879233d-00001.parquet 172.22.0.1 280µs ↑ 153 B ↓ 0 B 2023-05-11T20:19:11.177 [200 OK] s3.HeadObject 127.0.0.1:9000/warehouse/nyc/taxis/data/00003-1254-03dd06d8-f2da-4b9b-89ba-a2421e3f60fd-00001.parquet 172.22.0.1 255µs ↑ 153 B ↓ 0 B 2023-05-11T20:19:11.177 [200 OK] s3.HeadObject 127.0.0.1:9000/warehouse/nyc/taxis/data/00003-1263-f568eb3b-a4d6-4a93-bcae-2b22f8baa1e7-00001.parquet 172.22.0.1 433µs ↑ 153 B ↓ 0 B 2023-05-11T20:19:11.180 [206 Partial Content] s3.GetObject 127.0.0.1:9000/warehouse/nyc/taxis/data/00003-1281-d941a1d6-d041-4cf3-9e44-778191ae68fb-00001.parquet 172.22.0.1 4.36ms ↑ 159 B ↓ 64 KiB 2023-05-11T20:19:11.182 [206 Partial Content] s3.GetObject 127.0.0.1:9000/warehouse/nyc/taxis/data/00003-1263-f568eb3b-a4d6-4a93-bcae-2b22f8baa1e7-00001.parquet 172.22.0.1 3.465ms ↑ 159 B ↓ 64 KiB 2023-05-11T20:19:11.180 [206 Partial Content] s3.GetObject 127.0.0.1:9000/warehouse/nyc/taxis/data/00003-1290-df4bb385-1e5f-4841-960f-85600d3708e5-00001.parquet 172.22.0.1 6.57ms ↑ 159 B ↓ 64 KiB 2023-05-11T20:19:11.180 [206 Partial Content] s3.GetObject 127.0.0.1:9000/warehouse/nyc/taxis/data/00003-1272-519910f9-3e1b-4a64-8745-3c34d879233d-00001.parquet 172.22.0.1 7.434ms ↑ 159 B ↓ 64 KiB 2023-05-11T20:19:11.182 [206 Partial Content] s3.GetObject 127.0.0.1:9000/warehouse/nyc/taxis/data/00003-1254-03dd06d8-f2da-4b9b-89ba-a2421e3f60fd-00001.parquet 172.22.0.1 6.438ms ↑ 159 B ↓ 64 KiB 2023-05-11T20:19:11.196 [206 Partial Content] s3.GetObject 127.0.0.1:9000/warehouse/nyc/taxis/data/00003-1290-df4bb385-1e5f-4841-960f-85600d3708e5-00001.parquet 172.22.0.1 3.827ms ↑ 159 B ↓ 64 KiB 2023-05-11T20:19:11.196 [206 Partial Content] s3.GetObject 127.0.0.1:9000/warehouse/nyc/taxis/data/00003-1263-f568eb3b-a4d6-4a93-bcae-2b22f8baa1e7-00001.parquet 172.22.0.1 3.971ms ↑ 159 B ↓ 64 KiB 2023-05-11T20:19:11.194 [206 Partial Content] s3.GetObject 127.0.0.1:9000/warehouse/nyc/taxis/data/00003-1281-d941a1d6-d041-4cf3-9e44-778191ae68fb-00001.parquet 172.22.0.1 5.977ms ↑ 159 B ↓ 64 KiB 2023-05-11T20:19:11.198 [206 Partial Content] s3.GetObject 127.0.0.1:9000/warehouse/nyc/taxis/data/00003-1254-03dd06d8-f2da-4b9b-89ba-a2421e3f60fd-00001.parquet 172.22.0.1 7.024ms ↑ 159 B ↓ 64 KiB 2023-05-11T20:19:11.196 [206 Partial Content] s3.GetObject 127.0.0.1:9000/warehouse/nyc/taxis/data/00003-1272-519910f9-3e1b-4a64-8745-3c34d879233d-00001.parquet 172.22.0.1 9.302ms ↑ 159 B ↓ 64 KiB 2023-05-11T20:19:11.213 [206 Partial Content] s3.GetObject 127.0.0.1:9000/warehouse/nyc/taxis/data/00003-1281-d941a1d6-d041-4cf3-9e44-778191ae68fb-00001.parquet 172.22.0.1 69.011ms ↑ 159 B ↓ 6.6 MiB 2023-05-11T20:19:11.211 [206 Partial Content] s3.GetObject 127.0.0.1:9000/warehouse/nyc/taxis/data/00003-1272-519910f9-3e1b-4a64-8745-3c34d879233d-00001.parquet 172.22.0.1 685.555ms ↑ 159 B ↓ 15 MiB 2023-05-11T20:19:11.213 [206 Partial Content] s3.GetObject 127.0.0.1:9000/warehouse/nyc/taxis/data/00003-1290-df4bb385-1e5f-4841-960f-85600d3708e5-00001.parquet 172.22.0.1 886.157ms ↑ 159 B ↓ 16 MiB 2023-05-11T20:19:11.211 [206 Partial Content] s3.GetObject 127.0.0.1:9000/warehouse/nyc/taxis/data/00003-1263-f568eb3b-a4d6-4a93-bcae-2b22f8baa1e7-00001.parquet 172.22.0.1 1.711973s ↑ 159 B ↓ 27 MiB 2023-05-11T20:19:11.213 [206 Partial Content] s3.GetObject 127.0.0.1:9000/warehouse/nyc/taxis/data/00003-1263-f568eb3b-a4d6-4a93-bcae-2b22f8baa1e7-00001.parquet 172.22.0.1 1.740533s ↑ 159 B ↓ 27 MiB 2023-05-11T20:19:11.211 [206 Partial Content] s3.GetObject 127.0.0.1:9000/warehouse/nyc/taxis/data/00003-1272-519910f9-3e1b-4a64-8745-3c34d879233d-00001.parquet 172.22.0.1 1.754817s ↑ 159 B ↓ 29 MiB 2023-05-11T20:19:11.209 [206 Partial Content] s3.GetObject 127.0.0.1:9000/warehouse/nyc/taxis/data/00003-1281-d941a1d6-d041-4cf3-9e44-778191ae68fb-00001.parquet 172.22.0.1 1.960224s ↑ 159 B ↓ 30 MiB 2023-05-11T20:19:11.211 [206 Partial Content] s3.GetObject 127.0.0.1:9000/warehouse/nyc/taxis/data/00003-1290-df4bb385-1e5f-4841-960f-85600d3708e5-00001.parquet 172.22.0.1 2.066043s ↑ 159 B ↓ 32 MiB 2023-05-11T20:19:11.784 [206 Partial Content] s3.GetObject 127.0.0.1:9000/warehouse/nyc/taxis/data/00003-1254-03dd06d8-f2da-4b9b-89ba-a2421e3f60fd-00001.parquet 172.22.0.1 1.618623s ↑ 159 B ↓ 27 MiB 2023-05-11T20:19:12.526 [206 Partial Content] s3.GetObject 127.0.0.1:9000/warehouse/nyc/taxis/data/00003-1254-03dd06d8-f2da-4b9b-89ba-a2421e3f60fd-00001.parquet 172.22.0.1 1.407968s ↑ 159 B ↓ 26 MiB ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
