Attila Doroszlai created HDDS-2467:
--------------------------------------
Summary: Allow running Freon validators with limited memory
Key: HDDS-2467
URL: https://issues.apache.org/jira/browse/HDDS-2467
Project: Hadoop Distributed Data Store
Issue Type: Improvement
Components: freon
Reporter: Attila Doroszlai
Assignee: Attila Doroszlai
Freon validators read each item to be validated completely into a {{byte[]}}
buffer. This allows timing only the read (and buffer allocation), but not the
subsequent digest calculation. However, it also means that memory required for
running the validators is proportional to key size.
I propose to add a command-line flag to allow calculating the digest while
reading the input stream. This changes timing results a bit, since values will
include the time required for digest calculation. On the other hand, Freon
will be able to validate huge keys with limited memory.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
