This is an automated email from the ASF dual-hosted git repository.
sivabalan pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/hudi.git
The following commit(s) were added to refs/heads/asf-site by this push:
new 55e5820 [MINOR] Fix wording and table in the marker blog (#3588)
55e5820 is described below
commit 55e58205048f15de65e35e68e0622bfeb48723a5
Author: Y Ethan Guo <[email protected]>
AuthorDate: Tue Sep 7 09:14:07 2021 -0700
[MINOR] Fix wording and table in the marker blog (#3588)
---
.../blog/2021-08-18-improving-marker-mechanism.md | 36 +++++++++++-----------
1 file changed, 18 insertions(+), 18 deletions(-)
diff --git a/website/blog/2021-08-18-improving-marker-mechanism.md
b/website/blog/2021-08-18-improving-marker-mechanism.md
index e9b4021..840deb5 100644
--- a/website/blog/2021-08-18-improving-marker-mechanism.md
+++ b/website/blog/2021-08-18-improving-marker-mechanism.md
@@ -15,30 +15,30 @@ very large writes. We demonstrate how we improve write
performance with introduc
A **marker** in Hudi, such as a marker file with a unique filename, is a label
to indicate that a corresponding data file exists in storage, which then Hudi
uses to automatically clean up uncommitted data during failure and rollback
scenarios. Each marker entry is composed of three parts, the data file name,
the marker extension (`.marker`), and the I/O operation created the file
(`CREATE` - inserts, `MERGE` - updates/deletes, or `APPEND` - either). For
example, the marker
`91245ce3-bb82-4f9f-969e-343364159174-0_140-579-0_20210820173605.parquet.marker.CREATE`
indicates
-that the corresponding data file is
`91245ce3-bb82-4f9f-969e-343364159174-0_140-579-0_20210820173605.parquet` and
the I/O type is `CREATE`. Before writing each data file, the Hudi write client
creates a marker first in storage, which is persistent until they are
explicitly deleted
-by the write client after a commit is successful.
+that the corresponding data file is
`91245ce3-bb82-4f9f-969e-343364159174-0_140-579-0_20210820173605.parquet` and
the I/O type is `CREATE`. Hudi creates a marker before creating the
corresponding data file in the file system and deletes all markers pertaining
to a commit when it succeeds.
-The markers are useful for efficiently carrying out different operations by
the write client. Two important operations use markers to find uncommitted data
files of interest efficiently, instead of scanning the whole Hudi table:
- - **Removing duplicate/partial data files**: in Spark, the Hudi write client
delegates the data file writing to multiple executors. One executor can fail
the task, leaving partial data files written, and Spark retries the task in
this case until it succeeds. When speculative execution is enabled, there can
also be multiple successful attempts at writing out the same data into
different files, only one of which is finally handed to the Spark driver
process for committing. The markers h [...]
- - **Rolling back failed commits**: the write operation can fail in the
middle, leaving some data files written in storage. In this case, the marker
entries stay in storage as the commit is failed. In the next write operation,
the write client first rolls back the failed commits, by identifying the data
files written in these commits through the markers and deleting them.
+The markers are useful for efficiently carrying out different operations by
the write client. Markers serve as a way to track data files of interest
rather than scanning the whole Hudi table by listing all files in the table.
Two important operations use markers which come in handy to find uncommitted
data files of interest efficiently:
+ - **Removing duplicate/partial data files**: in Spark, the Hudi write client
delegates the data file writing to multiple executors. One executor can fail
the task, leaving partial data files written, and Spark retries the task in
this case until it succeeds. When speculative execution is enabled, there can
also be multiple successful attempts at writing out the same data into
different files, only one of which is finally handed to the Spark driver
process for committing. The markers h [...]
+
+ - **Rolling back failed commits**: the write operation can fail in the
middle, leaving some data files written in storage. In this case, the marker
entries stay in storage as the commit is failed. In the next write operation,
the write client rolls back the failed commit before proceeding with the new
write. The rollback is done with the help of markers to identify the data files
written as part of the failed commit.
Next, we dive into the existing marker mechanism, explain its performance
problem, and demonstrate the new timeline-server-based marker mechanism to
address the problem.
## Existing Direct Marker Mechanism and its limitations
-The **existing marker mechanism** simply creates a new marker file
corresponding to each data file, with the marker filename as described above.
Each marker file is written to storage in the same directory hierarchy, i.e.,
commit instant and partition path, under a temporary folder `.hoodie/.temp`
under the base path of the Hudi table. For example, the figure below shows one
example of the marker files created and the corresponding data files when
writing data to the Hudi table. When [...]
+The **existing marker mechanism** simply creates a new marker file
corresponding to each data file, with the marker filename as described above.
The marker file does not have any content, i.e., empty. Each marker file is
written to storage in the same directory hierarchy, i.e., commit instant and
partition path, under a temporary folder `.hoodie/.temp` under the base path of
the Hudi table. For example, the figure below shows one example of the marker
files created and the correspondi [...]
-While it's much efficient over scanning the entire table for uncommitted data
files, as the number of data files to write increases, so does the number of
marker files to create. This can create performance bottlenecks for cloud
storage such as AWS S3. In AWS S3, each file create and delete call triggers
an HTTP request and there is
[rate-limiting](https://docs.aws.amazon.com/AmazonS3/latest/userguide/optimizing-performance.html)
on how many requests can be processed per second per pref [...]
+While it's much efficient over scanning the entire table for uncommitted data
files, as the number of data files to write increases, so does the number of
marker files to create. For large writes which need to write significant
number of data files, e.g., 10K or more, this can create performance
bottlenecks for cloud storage such as AWS S3. In AWS S3, each file create and
delete call triggers an HTTP request and there is
[rate-limiting](https://docs.aws.amazon.com/AmazonS3/latest/userg [...]
## Timeline-server-based marker mechanism improving write performance
-To address the performance bottleneck due to rate-limiting of AWS S3 explained
above, we introduce a **new marker mechanism leveraging the timeline server**,
which optimizes the marker-related latency for storage with non-trivial file
I/O latency. The **timeline server** in Hudi serves as a centralized place for
providing the file system and timeline views. As shown below, the new
timeline-server-based marker mechanism delegates the marker creation and other
marker-related operations fr [...]
+To address the performance bottleneck due to rate-limiting of AWS S3 explained
above, we introduce a **new marker mechanism leveraging the timeline server**,
which optimizes the marker-related latency for storage with non-trivial file
I/O latency. The **timeline server** in Hudi serves as a centralized place for
providing the file system and timeline views. As shown below, the new
timeline-server-based marker mechanism delegates the marker creation and other
marker-related operations fr [...]
-To improve the efficiency of processing marker creation requests, we design
the batched handling of marker requests at the timeline server. Each marker
creation request is handled asynchronously in the Javalin timeline server and
queued before processing. For every batch interval, e.g., 20ms, a dispatching
thread pulls the pending requests from the queue and sends them to the worker
thread for processing. Each worker thread processes the marker creation
requests, sets the responses, and [...]
+To improve the efficiency of processing marker creation requests, we design
the batched handling of marker requests at the timeline server. Each marker
creation request is handled asynchronously in the Javalin timeline server and
queued before processing. For every batch interval, e.g., 20ms, the timeline
server pulls the pending marker creation requests from the queue and writes all
markers to the next file in a round robin fashion. Inside the timeline server,
such batch processing is [...]
@@ -47,26 +47,26 @@ Note that the worker thread always checks whether the
marker has already been cr
## Marker-related write options
-We introduce the following new marker-related write options in `0.9.0`
release, to configure the marker mechanism.
+We introduce the following new marker-related write options in `0.9.0`
release, to configure the marker mechanism. Note that the
timeline-server-based marker mechanism is not yet supported for HDFS in `0.9.0`
release, and we plan to support the timeline-server-based marker mechanism for
HDFS in the future.
| Property Name | Default | Meaning |
| ------------- | ----------- | :-------------:|
-| `hoodie.write.markers.type` | direct | Marker type to use. Two modes
are supported: (1) `direct`: individual marker file corresponding to each data
file is directly created by the writer; (2) `timeline_server_based`: marker
operations are all handled at the timeline service which serves as a proxy.
New marker entries are batch processed and stored in a limited number of
underlying files for efficiency. |
+| `hoodie.write.markers.type` | direct | Marker type to use. Two modes
are supported: (1) `direct`: individual marker file corresponding to each data
file is directly created by the executor; (2) `timeline_server_based`: marker
operations are all handled at the timeline service which serves as a proxy.
New marker entries are batch processed and stored in a limited number of
underlying files for efficiency. |
| `hoodie.markers.timeline_server_based.batch.num_threads` | 20 | Number
of threads to use for batch processing marker creation requests at the timeline
server. |
| `hoodie.markers.timeline_server_based.batch.interval_ms` | 50 | The batch
interval in milliseconds for marker creation batch processing. |
## Performance
-We evaluate the write performance over both direct and timeline-server-based
marker mechanisms by bulk-inserting a large dataset using Amazon EMR with Spark
and S3. The input data is around 100GB. We configure the write operation to
generate a large number of data files concurrently by setting the max parquet
file size to be 1MB and parallelism to be 240. As we noted before, while the
latency of direct marker mechanism is acceptable for incremental writes with
smaller number of data fil [...]
+We evaluate the write performance over both direct and timeline-server-based
marker mechanisms by bulk-inserting a large dataset using Amazon EMR with Spark
and S3. The input data is around 100GB. We configure the write operation to
generate a large number of data files concurrently by setting the max parquet
file size to be 1MB and parallelism to be 240. Note that it is unlikely to set
max parquet file size to 1MB in production and such a setup is only to evaluate
the performance rega [...]
-As shown below, the timeline-server-based marker mechanism generates much
fewer files storing markers because of the batch processing, leading to much
less time on marker-related I/O operations, thus achieving 31% lower write
completion time compared to the direct marker file mechanism.
+As shown below, direct marker mechanism works really well, when a part of the
table is written, e.g., 1K out of 165K data files. However, the time of direct
marker operations is non-trivial when we need to write significant number of
data files. Compared to the direct marker mechanism, the timeline-server-based
marker mechanism generates much fewer files storing markers because of the
batch processing, leading to much less time on marker-related I/O operations,
thus achieving 31% lower [...]
-| Marker Type | Total Files | Num data files written | Files created for
markers | Marker deletion time | Bulk Insert Time (including marker deletion) |
+| Marker Type | Input data size | Num data files written | Files created
for markers | Marker deletion time | Bulk Insert Time (including marker
deletion) |
| ----------- | --------- | :---------: | :---------: | :---------: |
:---------: |
-| Direct | 165K | 1k | 165k | 5.4secs | - |
-| Direct | 165K | 165k | 165k | 15min | 55min |
-| Timeline-server-based | 165K | 165k | 20 | ~3s | 38min |
+| Direct | 600MB | 1k | 1k | 5.4secs | - |
+| Direct | 100GB | 165k | 165k | 15min | 55min |
+| Timeline-server-based | 100GB | 165k | 20 | ~3s | 38min |
## Conclusion
-We identify that the existing direct marker file mechanism incurs performance
bottlenecks due to the rate-limiting of file create and delete calls on cloud
storage like AWS S3. To address this issue, we introduce a new marker
mechanism leveraging the timeline server, which delegates the marker creation
and other marker-related operations from individual executors to the timeline
server and uses batch processing to improve performance. Performance
evaluations on Amazon EMR with Spark an [...]
\ No newline at end of file
+We identify that for large writes which need to write significant number of
data files, the existing direct marker file mechanism can incur performance
bottlenecks due to the rate-limiting of file create and delete calls on cloud
storage like AWS S3. To address this issue, we introduce a new marker
mechanism leveraging the timeline server, which delegates the marker creation
and other marker-related operations from individual executors to the timeline
server and uses batch processing to [...]
\ No newline at end of file
