liwei created HUDI-897:
--------------------------
Summary: hudi support log append scenario with better write and
asynchronous compaction
Key: HUDI-897
URL: https://issues.apache.org/jira/browse/HUDI-897
Project: Apache Hudi (incubating)
Issue Type: Bug
Components: Compaction, Performance
Reporter: liwei
Fix For: 0.6.0
Attachments: image-2020-05-14-19-51-37-938.png
一、scenario
The business scenarios of the data lake mainly include analysis of databases,
logs, and files.
!image-2020-05-14-19-51-37-938.png|width=444,height=286!
Databricks delta lake also aim at these three scenario.
[https://databricks.com/product/delta-lake-on-databricks]
二、Hudi current situation
At present, hudi can better support the scenario where the database cdc is
incrementally written to hudi, and it is also doing bulkload files to hudi.
However, there is no good native support for log scenarios (requiring
high-throughput writes, no updates, deletions, and focusing on small file
scenarios);now can write through inserts without deduplication, but they will
still merge on the write side.
* In copy on write mode when "hoodie.parquet.small.file.limit" is 100MB, but
every batch small will cost some time for merge,it will reduce write
throughput.
* This scene is not suitable for merge on read.
* the actual scenario only needs to write parquet in batches when writing, and
then provide reverse compaction (similar to delta lake )
三、what we can do
1. On the write side just insert the parquet ,and use the hudi snapshot
mechanism
2. alse support asynchronous merge small files like databricks delta lake 's
optimize command .
[https://docs.databricks.com/delta/optimizations/file-mgmt.html]
[however|http://dict.youdao.com/search?q=however&keyfrom=chrome.extension]
[haʊˈevə(r)]
[详细|http://dict.youdao.com/search?q=however&keyfrom=chrome.extension]X
基本翻译
conj. 然而；可是
adv. 无论如何；不管怎样
网络释义
[However:|http://dict.youdao.com/search?q=However&keyfrom=chrome.extension&le=eng]
然而
[however
also:|http://dict.youdao.com/search?q=however%20also&keyfrom=chrome.extension&le=eng]
不过也
[however
often:|http://dict.youdao.com/search?q=however%20often&keyfrom=chrome.extension&le=eng]
但是往往
[do|http://dict.youdao.com/search?q=do&keyfrom=chrome.extension] [duː]
do&type=1[详细|http://dict.youdao.com/search?q=do&keyfrom=chrome.extension]X
基本翻译
n. (Do) 多（人名 Dorothea 和 Dorothy 的昵称）
v.
做，干，办（某事）；行动，表现；顺利进行，进展；从事（工作）；执行（任务）；学习，研究；解答，解决，算出；制作，做出；演出，编排（戏剧等）；仿效，模仿；完成，做好；供给，提供；走过，行过；以……速度行进；达到（一定销售额）；参观，游览，旅行；处理；适合，足够；煮制；欺骗；惩罚；（从某地方）盗窃；护理；（非正式）吸（毒）；（非正式）性交；度过（一段时间）；导致，引起；（非正式）痛打，杀死；生长；整理
aux. 与动词连用构成否定句和疑问句；用于构成疑问尾句；用于避免重复某一动词；用于强调主要动词；用于祈使句中表示礼貌；用于肯定或否定回答；用在倒装句中
n. 聚会，社交活动；要求，规定；C大调音阶中的第一音；（口语）事件；（主英国口语）诈骗；（主英国、新西兰口语）宴会；（口语）必须做到的事情；（女子）发型
网络释义
[DO:|http://dict.youdao.com/search?q=DO&keyfrom=chrome.extension&le=eng] 溶解氧
[do
wrong:|http://dict.youdao.com/search?q=do%20wrong&keyfrom=chrome.extension&le=eng]
做错
[I Do:|http://dict.youdao.com/search?q=I%20Do&keyfrom=chrome.extension&le=eng]
我愿意
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
