[
https://issues.apache.org/jira/browse/HDFS-11926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16038653#comment-16038653
]
Weiwei Yang commented on HDFS-11926:
------------------------------------
Thanks [~anu] for the comments, see my response below
bq. Should we make MAX_NUM_OF_ENTRIES a config setting instead of hardcoding
and make 1024 the default value ?
Well, we could but I don't prefer to. {{LevelDBStore#getRangeKVs}} is an
internal helper function that is not exposed to any end users, expose such
config to end user is a bit odd. And in my patch, {{MAX_NUM_OF_ENTRIES}} is
used only when the given count < 0, that implies caller wants to get all
entries so we need a limit. If caller wants to get more than this count, they
can just set {{count}} to a bigger value. This way it helps to
# A safe limit is set if caller doesn't want to explicitly specify a count
# Provide the flexibility for caller to explicitly set any valid count they
like, it's up to them
# Caller can set their own limits when they call this function, according to
their requirement. E.g to list keys we may want a bigger limit that list
containers. And those can be exposed via configuration if necessary.
bq. There are couple of subtle race conditions here...
Thanks for pointing this out. In latest patch, I used the snapshot approach.
Since this is a bunch of read only steps, it makes sense to take a snapshot to
avoid race conditions. I also tested the performance overhead, I tried to
create 10,000,000 records in levelDB, it takes 180mb disk space. The time it
spent to create a snapshot is 1 million second, that's trivial. Please let me
know if this looks OK to you.
Thanks.
> Ozone: Implement a common helper to return a range of KVs in levelDB
> --------------------------------------------------------------------
>
> Key: HDFS-11926
> URL: https://issues.apache.org/jira/browse/HDFS-11926
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
> Reporter: Weiwei Yang
> Assignee: Weiwei Yang
> Priority: Blocker
> Attachments: HDFS-11926-HDFS-7240.001.patch,
> HDFS-11926-HDFS-7240.002.patch, HDFS-11926-HDFS-7240.003.patch
>
>
> There are quite some *LIST* operations need to get a range of keys or values
> from levelDB, and filter entries with key prefix.
> # HDFS-11782 listKeys
> # HDFS-11779 listBuckets
> # HDFS-11773 listVolumes
> # HDFS-11679 listContainers
> we need to implement a common utility for them.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
