GitHub user zsxwing opened a pull request:
https://github.com/apache/spark/pull/12435
Add a file sink log to support versioning and compaction
## What changes were proposed in this pull request?
This PR adds a special log for FileStreamSink for two purposes:
- Versioning. A future Spark version should be able to read the metadata of
an old FileStreamSink.
- Compaction. As reading from many small files is usually pretty slow, we
should compact small metadata files into big files.
FileStreamSinkLog has a new log format instead of Java serialization
format. It will write one log file for each batch. The first line of the log
file is the version number, and there are multiple JSON lines following. Each
JSON line is a JSON format of FileLog.
FileStreamSinkLog will compact log files every
"spark.sql.sink.file.log.compactLen" batches into a big file. When doing a
compact, it will read all history logs and merge them with the new batch.
During the compaction, it will also delete the files that are deleted (marked
by FileLog.action). When the reader uses allLogs to list all files, this method
only returns the visible files (drops the deleted files).
## How was this patch tested?
FileStreamSinkLogSuite
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/zsxwing/spark sink-log
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/12435.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #12435
----
commit 29e3088131beff4b36841a3d6544d399b0bfa0c2
Author: Shixiong Zhu <[email protected]>
Date: 2016-04-15T19:28:46Z
Add a file sink log to support versioning and compaction
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
