[
https://issues.apache.org/jira/browse/HDFS-12543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chen Liang updated HDFS-12543:
------------------------------
Attachment: HDFS-12543-HDFS-7240.007.patch
Thanks [~xyao] for the review!
All comments addressed in v007 patch, with some clarifications below:
bq. Line 126: do we need to check and ensure all the containers of the key is
created before any write? Will this cause problem if someone create a key with
large size like 1TB specified?
bq. Line 168-173: that's a debatable optimization. Preallocate slow the first
request, which cause long delay for large key creation. If we really want this,
we need a limit to the number Of block/container preallocated.
Good point about the first reques! Added a config value for max preallocation
size. If a huge open key size is requested, only the configurable max size will
be allocate at request time. Further writes will need to issue new block
allocation request.
bq. Line 175-177: can we have take a String keyName to avoid converting keyName
from bytes to string and string to baytes.You may use getKeyKeyPrefix() to get
a string version of key name.
{{getKeyKeyPrefix}} was only internal to {{KSMMetadataManagerImpl}}, I think
this is a generally useful method, so I made it part of {{KSMMetadataManager}}
interface. And the name getKeyKeyPrefix looks confusing, so I renamed it to
{{getFullKeyName}}
bq. Line 209-219: why do we need a 10000 loop retry here? Is it to avoid
conflict due to openKey generated based on random id?
Yes, ids are random, and possible to conflict. The loop is to make sure a
different id is picked. 10000 is just some arbitrary number just to avoid using
a while true loop...to make sure no matter what happened, it does not hang
forever and we always guarantee to get out of the loop at some point.
bq. Line 239: How do we handle open keys that never got committed? Please file
follow up JIRAs for this if we plan to handle that later.
then there needs to be some threads that clean up the dead open key entries
from the meta data, and ideally also the allocated blocks of them. Will follow
up on this in other JIRAs.
bq. Can you elaborate why testErrorWrite is removed? Do we expect write over
the specified size allowed after this change? We could disable the test and
open follow up JIRAs if it cannot be easily fixed.
Yes, before the patch, a key must always meet a pre-specified size limit. With
this patch, there is no such restriction. Added back the test but with @Ignore
in v007 patch.
> Ozone : allow create key without specifying size
> ------------------------------------------------
>
> Key: HDFS-12543
> URL: https://issues.apache.org/jira/browse/HDFS-12543
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Reporter: Chen Liang
> Assignee: Chen Liang
> Labels: ozoneMerge
> Attachments: HDFS-12543-HDFS-7240.001.patch,
> HDFS-12543-HDFS-7240.002.patch, HDFS-12543-HDFS-7240.003.patch,
> HDFS-12543-HDFS-7240.004.patch, HDFS-12543-HDFS-7240.005.patch,
> HDFS-12543-HDFS-7240.006.patch, HDFS-12543-HDFS-7240.007.patch
>
>
> Currently when creating a key, it is required to specify the total size of
> the key. This makes it inconvenient for the case where a key is created and
> data keeps coming and being appended. This JIRA is remove the requirement of
> specifying the size on key creation, and allows appending to the key
> indefinitely.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
