Author: stefan2
Date: Fri May 23 22:56:22 2014
New Revision: 1597212
URL: http://svn.apache.org/r1597212
Log:
Sync'ing FSX with FSFS:
Port the low-level checksum support including verification and tests.
This is not a simple merge but mainly a "guided reimplementation",
taking the FSFS code as a blueprint but applying similar changes in
more places (as we have more item types in FSX) and accounting for
all the low-level differences.
As utility code, add private svn_checksum__* APIs to wrap streams
with a convenient checksum calculator.
Record the following revisions as affectively having been merged
from FSFS to FSX: r1573371,1588812,1588899,1588906.
* subversion/libsvn_fs_x
(): Updated mergeinfo.
* subversion/include/private/svn_subr_private.h
(svn_checksum__wrap_write_stream,
svn_checksum__wrap_write_stream_fnv1a_32x4): Declare new private API;
only the second one is currently in use.
* subversion/libsvn_subr/checksum.c
(stream_baton_t,
write_handler,
close_handler,
wrap_write_stream): Implement generic checksumming stream wrapper.
(svn_checksum__wrap_write_stream,
close_handler_fnv1a_32x4,
svn_checksum__wrap_write_stream_fnv1a_32x4): Implement new private
APIs on top of it.
* subversion/libsvn_fs_x/index.h
(svn_fs_x__p2l_entry_t): Attach a checksum to every item.
* subversion/libsvn_fs_x/index.c
(svn_fs_x__p2l_index_create): Store the checksums.
(read_entry): Read them again.
(p2l_index_lookup): Explicitly init the checksum for the dynamically
created "past-end-of-file" dummy entry.
* subversion/libsvn_fs_x/fs_x.c
(write_revision_zero): Update index P2L file template for r0.
* subversion/libsvn_fs_x/transaction.c
(rep_write_baton): Hold the new low-level checksum.
(rep_write_get_baton): Wrap the raw rep stream with a checksum calculator.
(rep_write_contents_close): Store the checksum in the proto index entry.
(write_container_delta_rep): Wrap the raw rep stream with a checksum
calculator and tell it to write directly
to the proto index entry.
(write_final_rev): Calculate checksums for noderevs as well as ...
(write_final_changed_path_info): ... for changed paths lists.
* subversion/libsvn_fs_x/pack.c
(auto_pad_block): Explicitly initialize checksums for unused ranges.
(write_nodes_container,
write_reps_container,
write_changes_container): Calculate and store the low-level checksums.
* subversion/libsvn_fs_x/verify.c
(STREAM_THRESHOLD,
expect_buffer_nul,
read_all_nul,
expected_checksum,
expected_buffered_checksum,
expected_streamed_checksum,
compare_p2l_to_rev): New low-level checksum checking code taken
over from FSFS.
(verify_index_consistency): Execute the new verification step as well.
* subversion/tests/cmdline/svnadmin_tests.py
(verify_quickly): Enable for FSX as well.
Modified:
subversion/trunk/subversion/include/private/svn_subr_private.h
subversion/trunk/subversion/libsvn_fs_x/ (props changed)
subversion/trunk/subversion/libsvn_fs_x/fs_x.c
subversion/trunk/subversion/libsvn_fs_x/index.c
subversion/trunk/subversion/libsvn_fs_x/index.h
subversion/trunk/subversion/libsvn_fs_x/pack.c
subversion/trunk/subversion/libsvn_fs_x/transaction.c
subversion/trunk/subversion/libsvn_fs_x/verify.c
subversion/trunk/subversion/libsvn_subr/checksum.c
subversion/trunk/subversion/tests/cmdline/svnadmin_tests.py
Modified: subversion/trunk/subversion/include/private/svn_subr_private.h
URL:
http://svn.apache.org/viewvc/subversion/trunk/subversion/include/private/svn_subr_private.h?rev=1597212&r1=1597211&r2=1597212&view=diff
==============================================================================
--- subversion/trunk/subversion/include/private/svn_subr_private.h (original)
+++ subversion/trunk/subversion/include/private/svn_subr_private.h Fri May 23
22:56:22 2014
@@ -270,6 +270,32 @@ svn_checksum__from_digest_fnv1a_32x4(con
/**
+ * Return a stream that calculates a checksum of type @a kind over all
+ * data written to the @a inner_stream. When the returned stream gets
+ * closed, write the checksum to @a *checksum.
+ * Allocate the result in @a pool.
+ *
+ * @note The stream returned only supports #svn_stream_write and
+ * #svn_stream_close.
+ */
+svn_stream_t *
+svn_checksum__wrap_write_stream(svn_checksum_t **checksum,
+ svn_stream_t *inner_stream,
+ svn_checksum_kind_t kind,
+ apr_pool_t *pool);
+
+/**
+ * Return a stream that calculates a 32 bit modified FNV-1a checksum
+ * over all data written to the @a inner_stream and writes the digest
+ * to @a *digest when the returned stream gets closed.
+ * Allocate the stream in @a pool.
+ */
+svn_stream_t *
+svn_checksum__wrap_write_stream_fnv1a_32x4(apr_uint32_t *digest,
+ svn_stream_t *inner_stream,
+ apr_pool_t *pool);
+
+/**
* Return a 32 bit FNV-1a checksum for the first @a len bytes in @a input.
* The representation is in Big Endian.
*
Propchange: subversion/trunk/subversion/libsvn_fs_x/
------------------------------------------------------------------------------
Merged
/subversion/trunk/subversion/libsvn_fs_fs:r1573371,1588812,1588899,1588906
Modified: subversion/trunk/subversion/libsvn_fs_x/fs_x.c
URL:
http://svn.apache.org/viewvc/subversion/trunk/subversion/libsvn_fs_x/fs_x.c?rev=1597212&r1=1597211&r2=1597212&view=diff
==============================================================================
--- subversion/trunk/subversion/libsvn_fs_x/fs_x.c (original)
+++ subversion/trunk/subversion/libsvn_fs_x/fs_x.c Fri May 23 22:56:22 2014
@@ -829,14 +829,16 @@ write_revision_zero(svn_fs_t *fs)
path = svn_fs_x__path_p2l_index(fs, 0, fs->pool);
SVN_ERR(svn_io_file_create_binary
(path,
- "\0" /* start rev */
- "\x80\x80\4\1\x11" /* 64k pages, 1 page using 17 bytes */
+ "\0\x7b" /* start rev, rev file size */
+ "\x80\x80\4\1\x21" /* 64k pages, 1 page using 33 bytes */
"\0" /* offset entry 0 page 1 */
- "\x1d\x11\0\6" /* len, type + 16 * count, (rev, 2*item)* */
- "\x5d\x15\0\4"
- "\1\x16\0\2"
- "\x85\xff\3\0", /* last entry fills up 64k page */
- 23,
+ /* len, type & count, checksum,
+ (rev, 2*item)* */
+ "\x1d\x11\x8e\xef\xf2\xd6\x01\0\6"
+ "\x5d\x15\xb6\xea\x97\xe0\x0f\0\4"
+ "\1\x16\x9d\x9e\xa9\x94\x0f\0\2"
+ "\x85\xff\3\0\0", /* last entry fills up 64k page */
+ 40,
fs->pool));
SVN_ERR(svn_io_set_file_read_only(path, FALSE, fs->pool));
Modified: subversion/trunk/subversion/libsvn_fs_x/index.c
URL:
http://svn.apache.org/viewvc/subversion/trunk/subversion/libsvn_fs_x/index.c?rev=1597212&r1=1597211&r2=1597212&view=diff
==============================================================================
--- subversion/trunk/subversion/libsvn_fs_x/index.c (original)
+++ subversion/trunk/subversion/libsvn_fs_x/index.c Fri May 23 22:56:22 2014
@@ -1925,6 +1925,9 @@ svn_fs_x__p2l_index_create(svn_fs_t *fs,
SVN_ERR(svn_spillbuf__write(buffer, (const char *)encoded,
encode_uint(encoded, entry.type +
entry.item_count * 16),
iter_pool));
+ SVN_ERR(svn_spillbuf__write(buffer, (const char *)encoded,
+ encode_uint(encoded, entry.fnv1_checksum),
+ iter_pool));
/* container contents (only one for non-container items) */
for (sub_item = 0; sub_item < entry.item_count; ++sub_item)
@@ -2227,6 +2230,8 @@ read_entry(packed_number_stream_t *strea
SVN_ERR(packed_stream_get(&value, stream));
entry.type = (int)value % 16;
entry.item_count = (apr_uint32_t)(value / 16);
+ SVN_ERR(packed_stream_get(&value, stream));
+ entry.fnv1_checksum = (apr_uint32_t)value;
if (entry.item_count == 0)
{
@@ -2672,6 +2677,7 @@ p2l_index_lookup(apr_array_header_t *ent
entry->offset = entry_end;
entry->size = block_end - entry_end;
entry->type = SVN_FS_X__ITEM_TYPE_UNUSED;
+ entry->fnv1_checksum = 0;
entry->item_count = 0;
entry->items = NULL;
}
Modified: subversion/trunk/subversion/libsvn_fs_x/index.h
URL:
http://svn.apache.org/viewvc/subversion/trunk/subversion/libsvn_fs_x/index.h?rev=1597212&r1=1597211&r2=1597212&view=diff
==============================================================================
--- subversion/trunk/subversion/libsvn_fs_x/index.h (original)
+++ subversion/trunk/subversion/libsvn_fs_x/index.h Fri May 23 22:56:22 2014
@@ -67,6 +67,9 @@ typedef struct svn_fs_x__p2l_entry_t
/* type of the item (see SVN_FS_X__ITEM_TYPE_*) defines */
unsigned type;
+ /* modified FNV-1a checksum. 0 if unknown checksum */
+ apr_uint32_t fnv1_checksum;
+
/* Number of items in this block / container. Their list can be found
* in *ITEMS. 0 for unused sections. 1 for non-container items,
* > 1 for containers. */
Modified: subversion/trunk/subversion/libsvn_fs_x/pack.c
URL:
http://svn.apache.org/viewvc/subversion/trunk/subversion/libsvn_fs_x/pack.c?rev=1597212&r1=1597211&r2=1597212&view=diff
==============================================================================
--- subversion/trunk/subversion/libsvn_fs_x/pack.c (original)
+++ subversion/trunk/subversion/libsvn_fs_x/pack.c Fri May 23 22:56:22 2014
@@ -914,6 +914,7 @@ auto_pad_block(pack_context_t *context,
null_entry.offset = context->pack_offset;
null_entry.size = padding;
null_entry.type = SVN_FS_X__ITEM_TYPE_UNUSED;
+ null_entry.fnv1_checksum = 0;
null_entry.item_count = 0;
null_entry.items = NULL;
@@ -1082,17 +1083,20 @@ write_nodes_container(pack_context_t *co
return SVN_NO_ERROR;
/* serialize container */
- pack_stream = svn_stream_from_aprfile2(context->pack_file, TRUE,
- scratch_pool);
-
+ container_entry = apr_palloc(context->info_pool, sizeof(*container_entry));
+ pack_stream = svn_checksum__wrap_write_stream_fnv1a_32x4
+ (&container_entry->fnv1_checksum,
+ svn_stream_from_aprfile2(context->pack_file,
+ TRUE, scratch_pool),
+ scratch_pool);
SVN_ERR(svn_fs_x__write_noderevs_container(pack_stream, *container,
- scratch_pool));
+ scratch_pool));
+ SVN_ERR(svn_stream_close(pack_stream));
SVN_ERR(svn_io_file_seek(context->pack_file, APR_CUR, &offset,
scratch_pool));
/* replace first noderev item in ENTRIES with the container
- and set all others to NULL */
- container_entry = apr_palloc(context->info_pool, sizeof(*container_entry));
+ and set all others to NULL */
container_entry->offset = context->pack_offset;
container_entry->size = offset - container_entry->offset;
container_entry->type = SVN_FS_X__ITEM_TYPE_NODEREVS_CONT;
@@ -1233,9 +1237,14 @@ write_reps_container(pack_context_t *con
svn_fs_x__p2l_entry_t container_entry;
svn_stream_t *pack_stream
- = svn_stream_from_aprfile2(context->pack_file, TRUE, pool);
+ = svn_checksum__wrap_write_stream_fnv1a_32x4
+ (&container_entry.fnv1_checksum,
+ svn_stream_from_aprfile2(context->pack_file,
+ TRUE, pool),
+ pool);
SVN_ERR(svn_fs_x__write_reps_container(pack_stream, container, pool));
+ SVN_ERR(svn_stream_close(pack_stream));
SVN_ERR(svn_io_file_seek(context->pack_file, SEEK_CUR, &offset, pool));
container_entry.offset = context->pack_offset;
@@ -1539,11 +1548,16 @@ write_changes_container(pack_context_t *
svn_fs_x__p2l_entry_t container_entry;
svn_stream_t *pack_stream
- = svn_stream_from_aprfile2(context->pack_file, TRUE, pool);
+ = svn_checksum__wrap_write_stream_fnv1a_32x4
+ (&container_entry.fnv1_checksum,
+ svn_stream_from_aprfile2(context->pack_file,
+ TRUE, pool),
+ pool);
SVN_ERR(svn_fs_x__write_changes_container(pack_stream,
container,
pool));
+ SVN_ERR(svn_stream_close(pack_stream));
SVN_ERR(svn_io_file_seek(context->pack_file, SEEK_CUR, &offset, pool));
container_entry.offset = context->pack_offset;
Modified: subversion/trunk/subversion/libsvn_fs_x/transaction.c
URL:
http://svn.apache.org/viewvc/subversion/trunk/subversion/libsvn_fs_x/transaction.c?rev=1597212&r1=1597211&r2=1597212&view=diff
==============================================================================
--- subversion/trunk/subversion/libsvn_fs_x/transaction.c (original)
+++ subversion/trunk/subversion/libsvn_fs_x/transaction.c Fri May 23 22:56:22
2014
@@ -1898,6 +1898,9 @@ struct rep_write_baton
svn_checksum_ctx_t *md5_checksum_ctx;
svn_checksum_ctx_t *sha1_checksum_ctx;
+ /* Receives the low-level checksum when closing REP_STREAM. */
+ apr_uint32_t fnv1a_checksum;
+
apr_pool_t *pool;
apr_pool_t *parent_pool;
@@ -2135,7 +2138,10 @@ rep_write_get_baton(struct rep_write_bat
b->pool));
b->file = file;
- b->rep_stream = svn_stream_from_aprfile2(file, TRUE, b->pool);
+ b->rep_stream = svn_checksum__wrap_write_stream_fnv1a_32x4(
+ &b->fnv1a_checksum,
+ svn_stream_from_aprfile2(file, TRUE, b->pool),
+ b->pool);
SVN_ERR(svn_fs_x__get_file_offset(&b->rep_offset, file, b->pool));
@@ -2368,6 +2374,7 @@ rep_write_contents_close(void *baton)
{
svn_fs_x__p2l_entry_t entry;
svn_fs_x__id_part_t noderev_id;
+ svn_checksum_t *checksum;
noderev_id.change_set = SVN_FS_X__INVALID_CHANGE_SET;
noderev_id.number = rep->id.number;
@@ -2377,6 +2384,7 @@ rep_write_contents_close(void *baton)
entry.type = SVN_FS_X__ITEM_TYPE_FILE_REP;
entry.item_count = 1;
entry.items = &noderev_id;
+ entry.fnv1_checksum = b->fnv1a_checksum;
SVN_ERR(store_sha1_rep_mapping(b->fs, b->noderev, b->pool));
SVN_ERR(store_p2l_index_entry(b->fs, txn_id, &entry, b->pool));
@@ -2595,6 +2603,7 @@ write_container_delta_rep(representation
svn_stream_t *stream;
representation_t *base_rep;
representation_t *old_rep;
+ svn_fs_x__p2l_entry_t entry;
svn_stream_t *source;
svn_fs_x__rep_header_t header = { 0 };
@@ -2626,7 +2635,10 @@ write_container_delta_rep(representation
header.type = svn_fs_x__rep_self_delta;
}
- file_stream = svn_stream_from_aprfile2(file, TRUE, pool);
+ file_stream = svn_checksum__wrap_write_stream_fnv1a_32x4(
+ &entry.fnv1_checksum,
+ svn_stream_from_aprfile2(file, TRUE, pool),
+ pool);
SVN_ERR(svn_fs_x__write_rep_header(&header, file_stream, pool));
SVN_ERR(svn_fs_x__get_file_offset(&delta_start, file, pool));
@@ -2670,7 +2682,6 @@ write_container_delta_rep(representation
}
else
{
- svn_fs_x__p2l_entry_t entry;
svn_fs_x__id_part_t noderev_id;
/* Write out our cosmetic end marker. */
@@ -2820,6 +2831,7 @@ write_final_rev(const svn_fs_id_t **new_
svn_fs_x__txn_id_t txn_id = svn_fs_x__id_txn_id(id);
svn_fs_x__p2l_entry_t entry;
svn_fs_x__change_set_t change_set = svn_fs_x__change_set_by_rev(rev);
+ svn_stream_t *file_stream;
*new_id_p = NULL;
@@ -2959,8 +2971,12 @@ write_final_rev(const svn_fs_id_t **new_
if (at_root)
SVN_ERR(validate_root_noderev(fs, noderev, rev, pool));
- SVN_ERR(svn_fs_x__write_noderev(svn_stream_from_aprfile2(file, TRUE, pool),
- noderev, ffd->format, pool));
+ file_stream = svn_checksum__wrap_write_stream_fnv1a_32x4(
+ &entry.fnv1_checksum,
+ svn_stream_from_aprfile2(file, TRUE, pool),
+ pool);
+ SVN_ERR(svn_fs_x__write_noderev(file_stream, noderev, ffd->format, pool));
+ SVN_ERR(svn_stream_close(file_stream));
/* reference the root noderev from the log-to-phys index */
noderev_id.change_set = SVN_FS_X__INVALID_CHANGE_SET;
@@ -2995,6 +3011,7 @@ write_final_changed_path_info(apr_off_t
apr_pool_t *pool)
{
apr_off_t offset;
+ svn_stream_t *stream;
apr_hash_index_t *hi;
svn_fs_x__p2l_entry_t entry;
svn_fs_x__id_part_t rev_item
@@ -3013,8 +3030,12 @@ write_final_changed_path_info(apr_off_t
change->copyfrom_rev = new_rev - 1;
}
- SVN_ERR(svn_fs_x__write_changes(svn_stream_from_aprfile2(file, TRUE, pool),
- fs, changed_paths, TRUE, pool));
+ /* write to target file & calculate checksum */
+ stream = svn_checksum__wrap_write_stream_fnv1a_32x4(&entry.fnv1_checksum,
+ svn_stream_from_aprfile2(file, TRUE, pool),
+ pool);
+ SVN_ERR(svn_fs_x__write_changes(stream, fs, changed_paths, TRUE, pool));
+ SVN_ERR(svn_stream_close(stream));
*offset_p = offset;
Modified: subversion/trunk/subversion/libsvn_fs_x/verify.c
URL:
http://svn.apache.org/viewvc/subversion/trunk/subversion/libsvn_fs_x/verify.c?rev=1597212&r1=1597211&r2=1597212&view=diff
==============================================================================
--- subversion/trunk/subversion/libsvn_fs_x/verify.c (original)
+++ subversion/trunk/subversion/libsvn_fs_x/verify.c Fri May 23 22:56:22 2014
@@ -22,6 +22,7 @@
#include "verify.h"
#include "fs_x.h"
+#include "private/svn_subr_private.h"
#include "cached_data.h"
#include "rep-cache.h"
@@ -314,8 +315,258 @@ compare_p2l_to_l2p_index(svn_fs_t *fs,
}
}
+ if (cancel_func)
+ SVN_ERR(cancel_func(cancel_baton));
+ }
+
+ svn_pool_destroy(iterpool);
+
+ return SVN_NO_ERROR;
+}
+
+/* Items smaller than this can be read at once into a buffer and directly
+ * be checksummed. Larger items require stream processing.
+ * Must be a multiple of 8. */
+#define STREAM_THRESHOLD 4096
+
+/* Verify that the next SIZE bytes read from FILE are NUL.
+ * SIZE must not exceed STREAM_THRESHOLD. Use POOL for allocations.
+ */
+static svn_error_t *
+expect_buffer_nul(apr_file_t *file,
+ apr_off_t size,
+ apr_pool_t *pool)
+{
+ union
+ {
+ unsigned char buffer[STREAM_THRESHOLD];
+ apr_uint64_t chunks[STREAM_THRESHOLD / sizeof(apr_uint64_t)];
+ } data;
+
+ apr_size_t i;
+ SVN_ERR_ASSERT(size <= STREAM_THRESHOLD);
+
+ /* read the whole data block; error out on failure */
+ data.chunks[(size - 1)/ sizeof(apr_uint64_t)] = 0;
+ SVN_ERR(svn_io_file_read_full2(file, data.buffer, size, NULL, NULL, pool));
+
+ /* chunky check */
+ for (i = 0; i < size / sizeof(apr_uint64_t); ++i)
+ if (data.chunks[i] != 0)
+ break;
+
+ /* byte-wise check upon mismatch or at the end of the block */
+ for (i *= sizeof(apr_uint64_t); i < size; ++i)
+ if (data.buffer[i] != 0)
+ {
+ const char *file_name;
+ apr_off_t offset;
+
+ SVN_ERR(svn_io_file_name_get(&file_name, file, pool));
+ SVN_ERR(svn_fs_x__get_file_offset(&offset, file, pool));
+ offset -= size - i;
+
+ return svn_error_createf(SVN_ERR_FS_CORRUPT, NULL,
+ _("Empty section in file %s contains "
+ "non-NUL data at offset %s"),
+ file_name, apr_off_t_toa(pool, offset));
+ }
+
+ return SVN_NO_ERROR;
+}
+
+/* Verify that the next SIZE bytes read from FILE are NUL.
+ * Use POOL for allocations.
+ */
+static svn_error_t *
+read_all_nul(apr_file_t *file,
+ apr_off_t size,
+ apr_pool_t *pool)
+{
+ for (; size >= STREAM_THRESHOLD; size -= STREAM_THRESHOLD)
+ SVN_ERR(expect_buffer_nul(file, STREAM_THRESHOLD, pool));
+
+ if (size)
+ SVN_ERR(expect_buffer_nul(file, size, pool));
+
+ return SVN_NO_ERROR;
+}
+
+/* Compare the ACTUAL checksum with the one expected by ENTRY.
+ * Return an error in case of mismatch. Use the name of FILE
+ * in error message. Allocate data in POOL.
+ */
+static svn_error_t *
+expected_checksum(apr_file_t *file,
+ svn_fs_x__p2l_entry_t *entry,
+ apr_uint32_t actual,
+ apr_pool_t *pool)
+{
+ if (actual != entry->fnv1_checksum)
+ {
+ const char *file_name;
+
+ SVN_ERR(svn_io_file_name_get(&file_name, file, pool));
+ SVN_ERR(svn_io_file_name_get(&file_name, file, pool));
+ return svn_error_createf(SVN_ERR_FS_CORRUPT, NULL,
+ _("Checksum mismatch item at offset %s of "
+ "length %s bytes in file %s"),
+ apr_off_t_toa(pool, entry->offset),
+ apr_off_t_toa(pool, entry->size), file_name);
+ }
+
+ return SVN_NO_ERROR;
+}
+
+/* Verify that the FNV checksum over the next ENTRY->SIZE bytes read
+ * from FILE will match ENTRY's expected checksum. SIZE must not
+ * exceed STREAM_THRESHOLD. Use POOL for allocations.
+ */
+static svn_error_t *
+expected_buffered_checksum(apr_file_t *file,
+ svn_fs_x__p2l_entry_t *entry,
+ apr_pool_t *pool)
+{
+ unsigned char buffer[STREAM_THRESHOLD];
+ SVN_ERR_ASSERT(entry->size <= STREAM_THRESHOLD);
+
+ SVN_ERR(svn_io_file_read_full2(file, buffer, (apr_size_t)entry->size,
+ NULL, NULL, pool));
+ SVN_ERR(expected_checksum(file, entry,
+ svn__fnv1a_32x4(buffer, (apr_size_t)entry->size),
+ pool));
+
+ return SVN_NO_ERROR;
+}
+
+/* Verify that the FNV checksum over the next ENTRY->SIZE bytes read from
+ * FILE will match ENTRY's expected checksum. Use POOL for allocations.
+ */
+static svn_error_t *
+expected_streamed_checksum(apr_file_t *file,
+ svn_fs_x__p2l_entry_t *entry,
+ apr_pool_t *pool)
+{
+ unsigned char buffer[STREAM_THRESHOLD];
+ svn_checksum_t *checksum;
+ svn_checksum_ctx_t *context
+ = svn_checksum_ctx_create(svn_checksum_fnv1a_32x4, pool);
+ apr_off_t size = entry->size;
+
+ while (size > 0)
+ {
+ apr_size_t to_read = size > sizeof(buffer)
+ ? sizeof(buffer)
+ : (apr_size_t)size;
+ SVN_ERR(svn_io_file_read_full2(file, buffer, to_read, NULL, NULL,
+ pool));
+ SVN_ERR(svn_checksum_update(context, buffer, to_read));
+ size -= to_read;
+ }
+
+ SVN_ERR(svn_checksum_final(&checksum, context, pool));
+ SVN_ERR(expected_checksum(file, entry,
+ ntohl(*(const apr_uint32_t *)checksum->digest),
+ pool));
+
+ return SVN_NO_ERROR;
+}
+
+/* Verify that for all phys-to-log index entries for revisions START to
+ * START + COUNT-1 in FS match the actual pack / rev file contents.
+ * If given, invoke CANCEL_FUNC with CANCEL_BATON at regular intervals.
+ * Use POOL for allocations.
+ *
+ * Please note that we can only check on pack / rev file granularity and
+ * must only be called for a single rev / pack file.
+ */
+static svn_error_t *
+compare_p2l_to_rev(svn_fs_t *fs,
+ svn_revnum_t start,
+ svn_revnum_t count,
+ svn_cancel_func_t cancel_func,
+ void *cancel_baton,
+ apr_pool_t *pool)
+{
+ fs_x_data_t *ffd = fs->fsap_data;
+ apr_pool_t *iterpool = svn_pool_create(pool);
+ apr_off_t max_offset;
+ apr_off_t offset = 0;
+ apr_file_t *rev_file;
+
+ /* open the pack / rev file that is covered by the p2l index */
+ SVN_ERR(svn_fs_x__open_pack_or_rev_file(&rev_file, fs, start, pool));
+
+ /* check file size vs. range covered by index */
+ SVN_ERR(svn_io_file_seek(rev_file, APR_END, &offset, pool));
+ SVN_ERR(svn_fs_x__p2l_get_max_offset(&max_offset, fs, start, pool));
+
+ if (offset != max_offset)
+ return svn_error_createf(SVN_ERR_FS_ITEM_INDEX_INCONSISTENT, NULL,
+ _("File size of %s for revision r%ld does "
+ "not match p2l index size of %s"),
+ apr_off_t_toa(pool, offset), start,
+ apr_off_t_toa(pool, max_offset));
+
+ SVN_ERR(svn_io_file_aligned_seek(rev_file, ffd->block_size, NULL, 0, pool));
+
+ /* for all offsets in the file, get the P2L index entries and check
+ them against the L2P index */
+ for (offset = 0; offset < max_offset; )
+ {
+ apr_array_header_t *entries;
+ int i;
+
svn_pool_clear(iterpool);
+ /* get all entries for the current block */
+ SVN_ERR(svn_fs_x__p2l_index_lookup(&entries, fs, start, offset,
+ ffd->p2l_page_size, iterpool));
+
+ /* process all entries (and later continue with the next block) */
+ for (i = 0; i < entries->nelts; ++i)
+ {
+ svn_fs_x__p2l_entry_t *entry
+ = &APR_ARRAY_IDX(entries, i, svn_fs_x__p2l_entry_t);
+
+ /* skip bits we previously checked */
+ if (i == 0 && entry->offset < offset)
+ continue;
+
+ /* skip zero-sized entries */
+ if (entry->size == 0)
+ continue;
+
+ /* p2l index must cover all rev / pack file offsets exactly once */
+ if (entry->offset != offset)
+ return svn_error_createf(SVN_ERR_FS_ITEM_INDEX_INCONSISTENT,
+ NULL,
+ _("p2l index entry for revision r%ld"
+ " is non-contiguous between offsets "
+ " %s and %s"),
+ start,
+ apr_off_t_toa(pool, offset),
+ apr_off_t_toa(pool, entry->offset));
+
+ /* empty sections must contain NUL bytes only */
+ if (entry->type == SVN_FS_X__ITEM_TYPE_UNUSED)
+ {
+ /* skip filler entry at the end of the p2l index */
+ if (entry->offset != max_offset)
+ SVN_ERR(read_all_nul(rev_file, entry->size, pool));
+ }
+ else if (entry->fnv1_checksum)
+ {
+ if (entry->size < STREAM_THRESHOLD)
+ SVN_ERR(expected_buffered_checksum(rev_file, entry, pool));
+ else
+ SVN_ERR(expected_streamed_checksum(rev_file, entry, pool));
+ }
+
+ /* advance offset */
+ offset += entry->size;
+ }
+
if (cancel_func)
SVN_ERR(cancel_func(cancel_baton));
}
@@ -360,6 +611,10 @@ verify_index_consistency(svn_fs_t *fs,
SVN_ERR(compare_p2l_to_l2p_index(fs, pack_start, pack_end - pack_start,
cancel_func, cancel_baton, iterpool));
+ /* verify in-index checksums and types vs. actual rev / pack files */
+ SVN_ERR(compare_p2l_to_rev(fs, pack_start, pack_end - pack_start,
+ cancel_func, cancel_baton, iterpool));
+
svn_pool_clear(iterpool);
}
Modified: subversion/trunk/subversion/libsvn_subr/checksum.c
URL:
http://svn.apache.org/viewvc/subversion/trunk/subversion/libsvn_subr/checksum.c?rev=1597212&r1=1597211&r2=1597212&view=diff
==============================================================================
--- subversion/trunk/subversion/libsvn_subr/checksum.c (original)
+++ subversion/trunk/subversion/libsvn_subr/checksum.c Fri May 23 22:56:22 2014
@@ -661,3 +661,140 @@ svn_checksum_is_empty_checksum(svn_check
SVN_ERR_MALFUNCTION_NO_RETURN();
}
}
+
+/* Checksum calculating stream wrappers.
+ */
+
+/* Baton used by write_handler and close_handler to calculate the checksum
+ * and return the result to the stream creator. It accommodates the data
+ * needed by svn_checksum__wrap_write_stream_fnv1a_32x4 as well as
+ * svn_checksum__wrap_write_stream.
+ */
+typedef struct stream_baton_t
+{
+ /* Stream we are wrapping. Forward write() and close() operations to it. */
+ svn_stream_t *inner_stream;
+
+ /* Build the checksum data in here. */
+ svn_checksum_ctx_t *context;
+
+ /* Write the final checksum here. May be NULL. */
+ svn_checksum_t **checksum;
+
+ /* Copy the digest of the final checksum. May be NULL. */
+ unsigned char *digest;
+
+ /* Allocate the resulting checksum here. */
+ apr_pool_t *pool;
+} stream_baton_t;
+
+/* Implement svn_write_fn_t.
+ * Update checksum and pass data on to inner stream.
+ */
+static svn_error_t *
+write_handler(void *baton,
+ const char *data,
+ apr_size_t *len)
+{
+ stream_baton_t *b = baton;
+
+ SVN_ERR(svn_checksum_update(b->context, data, *len));
+ SVN_ERR(svn_stream_write(b->inner_stream, data, len));
+
+ return SVN_NO_ERROR;
+}
+
+/* Implement svn_close_fn_t.
+ * Finalize checksum calculation and write results. Close inner stream.
+ */
+static svn_error_t *
+close_handler(void *baton)
+{
+ stream_baton_t *b = baton;
+ svn_checksum_t *local_checksum;
+
+ /* Ensure we can always write to *B->CHECKSUM. */
+ if (!b->checksum)
+ b->checksum = &local_checksum;
+
+ /* Get the final checksum. */
+ SVN_ERR(svn_checksum_final(b->checksum, b->context, b->pool));
+
+ /* Extract digest, if wanted. */
+ if (b->digest)
+ {
+ apr_size_t digest_size = DIGESTSIZE((*b->checksum)->kind);
+ memcpy(b->digest, (*b->checksum)->digest, digest_size);
+ }
+
+ /* Done here. Now, close the underlying stream as well. */
+ return svn_error_trace(svn_stream_close(b->inner_stream));
+}
+
+/* Common constructor function for svn_checksum__wrap_write_stream and
+ * svn_checksum__wrap_write_stream_fnv1a_32x4, taking the superset of their
+ * respecting parameters.
+ *
+ * In the current usage, either CHECKSUM or DIGEST will be NULL but this
+ * function does not enforce any such restriction. Also, the caller must
+ * make sure that DIGEST refers to a buffer of sufficient length.
+ */
+svn_stream_t *
+wrap_write_stream(svn_checksum_t **checksum,
+ unsigned char *digest,
+ svn_stream_t *inner_stream,
+ svn_checksum_kind_t kind,
+ apr_pool_t *pool)
+{
+ svn_stream_t *outer_stream;
+
+ stream_baton_t *baton = apr_pcalloc(pool, sizeof(*baton));
+ baton->inner_stream = inner_stream;
+ baton->context = svn_checksum_ctx_create(kind, pool);
+ baton->checksum = checksum;
+ baton->digest = digest;
+ baton->pool = pool;
+
+ outer_stream = svn_stream_create(baton, pool);
+ svn_stream_set_write(outer_stream, write_handler);
+ svn_stream_set_close(outer_stream, close_handler);
+
+ return outer_stream;
+}
+
+svn_stream_t *
+svn_checksum__wrap_write_stream(svn_checksum_t **checksum,
+ svn_stream_t *inner_stream,
+ svn_checksum_kind_t kind,
+ apr_pool_t *pool)
+{
+ return wrap_write_stream(checksum, NULL, inner_stream, kind, pool);
+}
+
+/* Implement svn_close_fn_t.
+ * For FNV-1a-like checksums, we want the checksum as 32 bit integer instead
+ * of a big endian 4 byte sequence. This simply wraps close_handler adding
+ * the digest conversion.
+ */
+static svn_error_t *
+close_handler_fnv1a_32x4(void *baton)
+{
+ stream_baton_t *b = baton;
+ SVN_ERR(close_handler(baton));
+
+ *(apr_uint32_t *)b->digest = ntohl(*(apr_uint32_t *)b->digest);
+ return SVN_NO_ERROR;
+}
+
+svn_stream_t *
+svn_checksum__wrap_write_stream_fnv1a_32x4(apr_uint32_t *digest,
+ svn_stream_t *inner_stream,
+ apr_pool_t *pool)
+{
+ svn_stream_t *result
+ = wrap_write_stream(NULL, (unsigned char *)digest, inner_stream,
+ svn_checksum_fnv1a_32x4, pool);
+ svn_stream_set_close(result, close_handler_fnv1a_32x4);
+
+ return result;
+}
Modified: subversion/trunk/subversion/tests/cmdline/svnadmin_tests.py
URL:
http://svn.apache.org/viewvc/subversion/trunk/subversion/tests/cmdline/svnadmin_tests.py?rev=1597212&r1=1597211&r2=1597212&view=diff
==============================================================================
--- subversion/trunk/subversion/tests/cmdline/svnadmin_tests.py (original)
+++ subversion/trunk/subversion/tests/cmdline/svnadmin_tests.py Fri May 23
22:56:22 2014
@@ -2478,7 +2478,7 @@ def verify_metadata_only(sbox):
'STDOUT', [], output)
-@SkipUnless(svntest.main.is_fs_type_fsfs)
+@Skip(svntest.main.is_fs_type_bdb)
def verify_quickly(sbox):
"verify quickly using metadata"
