Author: kotkov
Date: Fri Sep 8 22:36:38 2017
New Revision: 1807836
URL: http://svn.apache.org/viewvc?rev=1807836&view=rev
Log:
svnadmin: Introduce the `--normalize-props` option for the load and
load-revprops commands.
Currently, the only performed normalization is the automatic translation
of non-LF line endings in the svn: property values. Apparently, this is
a common issue reported by our users, as Subversion versions prior to 1.6
allowed such values for properties like svn:log and so they can be present
in the dump files. The behavior before this changeset is that a user can
either force loading the dump with such invalid properties using the
--bypass-prop-validation (that affects reading them in the future),
manually repair the dump file or use a tool like svnsync that performs
the property normalization. It would be better if the users had an option
to fix this right in the svnadmin load command.
See also the corresponding FAQ entry and related issues that have been
fixed in svnsync and svnrdump:
https://subversion.apache.org/faq.html#fix-nonLF-log
https://issues.apache.org/jira/browse/SVN-3404
https://issues.apache.org/jira/browse/SVN-4263
* subversion/include/svn_repos.h
(svn_repos_load_fs6): New version of svn_repos_load_fs5() accepting the
new `normalize_props` argument.
(svn_repos_load_fs5): Deprecate.
(svn_repos_load_fs_revprops): Accept the new `normalize_props` argument.
(svn_repos_get_fs_build_parser6): New version of
svn_repos_get_fs_build_parser5() accepting the new `normalize_props`
argument.
* subversion/include/private/svn_repos_private.h
(svn_repos__normalize_prop): New private helper. The `normalized_p`
argument will be currently unused by all calling sites, but it's added
as the groundwork to simplify implementing the notifications about
normalized properties in the future.
* subversion/libsvn_repos/deprecated.c
(svn_repos_load_fs5, svn_repos_get_fs_build_parser5): Implement by
forwarding to the new versions of these functions.
* subversion/libsvn_repos/fs-wrap.c
(): Include svn_subst.h.
(svn_repos__normalize_prop): Implement this new helper function.
The implementation is close to how svn_rdump__normalize_prop() is
written, but this function accepts two pools and handles the new
`normalized_p` output argument.
* subversion/libsvn_repos/load-fs-vtable.c
(struct parse_baton): Add new `normalize_props` field.
(change_rev_prop): Accept new `normalize_props` argument, call the
new helper function if it's set.
(close_revision): Normalize the revision properties if necessary.
(svn_repos_get_fs_build_parser6): New, replacing ...
(svn_repos_get_fs_build_parser5): ...this function.
(svn_repos_load_fs6): New, replacing ...
(svn_repos_load_fs5): ...this function.
(revprops_close_revision): Update call to change_rev_prop().
(build_revprop_parser): Accept new `normalize_props` argument, use it
to initialize the new field in the parse_baton.
(svn_repos_load_fs_revprops): Handle the new `normalize_props` argument
by forwarding it to build_revprop_parser().
* subversion/svnadmin/svnadmin.c
(svnadmin__cmdline_options_t, options_table): Add svnadmin__normalize_props.
(cmd_table): Enable the new option for load and load-revprops commands.
(struct svnadmin_opt_state): Add new `normalize_props` field and ...
(sub_main): ...initialize it here.
(subcommand_load, subcommand_load_revprops): Normalize properties when
running with --normalize-props. Use separate hints when encountering
invalid properties with non-LF line endings and all other kinds of
invalid properties.
* subversion/svnrdump/svnrdump.h
(svn_rdump__normalize_prop): Remove this declaration. This function is
now superseded by the new svn_repos__normalize_prop() helper.
* subversion/svnrdump/util.c
(): Include svn_repos_private.h, drop other includes.
(svn_rdump__normalize_prop): Remove. This function is now superseded
by the new svn_repos__normalize_prop() helper.
(svn_rdump__normalize_props): Use the new svn_repos__normalize_prop()
helper function.
* subversion/svnrdump/load_editor.c
(set_revision_property, set_node_property): Use the new
svn_repos__normalize_prop() helper function.
* subversion/tests/libsvn_repos/dump-load-test.c
(test_load_bad_props): Use the new svn_repos_load_fs6() function, keep
the previous behavior unchanged, i.e., do not normalize the properties.
* subversion/tests/cmdline/svntest/actions.py
(run_and_verify_load, load_repo): Accept new `normalize_props` argument.
* subversion/tests/cmdline/svnadmin_tests.py
(load_bad_props): Test loading with --normalize-props. Check for the error
codes reported during load with non-LF line endings and without the new
option. In cases when the load is successful, check the resulting
svn:log property value in the repository.
* tools/client-side/bash-completion
(_svnadmin): Extend completion info.
Modified:
subversion/trunk/subversion/include/private/svn_repos_private.h
subversion/trunk/subversion/include/svn_repos.h
subversion/trunk/subversion/libsvn_repos/deprecated.c
subversion/trunk/subversion/libsvn_repos/fs-wrap.c
subversion/trunk/subversion/libsvn_repos/load-fs-vtable.c
subversion/trunk/subversion/svnadmin/svnadmin.c
subversion/trunk/subversion/svnrdump/load_editor.c
subversion/trunk/subversion/svnrdump/svnrdump.h
subversion/trunk/subversion/svnrdump/util.c
subversion/trunk/subversion/tests/cmdline/svnadmin_tests.py
subversion/trunk/subversion/tests/cmdline/svntest/actions.py
subversion/trunk/subversion/tests/libsvn_repos/dump-load-test.c
subversion/trunk/tools/client-side/bash_completion
Modified: subversion/trunk/subversion/include/private/svn_repos_private.h
URL:
http://svn.apache.org/viewvc/subversion/trunk/subversion/include/private/svn_repos_private.h?rev=1807836&r1=1807835&r2=1807836&view=diff
==============================================================================
--- subversion/trunk/subversion/include/private/svn_repos_private.h (original)
+++ subversion/trunk/subversion/include/private/svn_repos_private.h Fri Sep 8
22:36:38 2017
@@ -75,6 +75,30 @@ svn_repos__validate_prop(const char *nam
const svn_string_t *value,
apr_pool_t *pool);
+/* Attempt to normalize a Subversion property if it "needs translation"
+ * (according to svn_prop_needs_translation(), currently all svn:* props).
+ *
+ * At this time, the only performed normalization is translation of
+ * the line endings of the property value so that it would only contain
+ * LF (\n) characters. "\r" characters found mid-line are replaced with "\n".
+ * "\r\n" sequences are replaced with "\n".
+ *
+ * NAME is used to check that VALUE should be normalized, and if this
+ * is the case, VALUE is then normalized, allocated from RESULT_POOL.
+ * If no normalization is required, VALUE will be copied to RESULT_POOL
+ * unchanged. If NORMALIZED_P is not NULL, and the normalization
+ * happened, set *NORMALIZED_P to non-zero. If the property is returned
+ * unchanged and NORMALIZED_P is not NULL, then *NORMALIZED_P will be
+ * set to zero. SCRATCH_POOL will be used for temporary allocations.
+ */
+svn_error_t *
+svn_repos__normalize_prop(const svn_string_t **result_p,
+ svn_boolean_t *normalized_p,
+ const char *name,
+ const svn_string_t *value,
+ apr_pool_t *result_pool,
+ apr_pool_t *scratch_pool);
+
/**
* Given the error @a err from svn_repos_fs_commit_txn(), return an
* string containing either or both of the svn_fs_commit_txn() error
Modified: subversion/trunk/subversion/include/svn_repos.h
URL:
http://svn.apache.org/viewvc/subversion/trunk/subversion/include/svn_repos.h?rev=1807836&r1=1807835&r2=1807836&view=diff
==============================================================================
--- subversion/trunk/subversion/include/svn_repos.h (original)
+++ subversion/trunk/subversion/include/svn_repos.h Fri Sep 8 22:36:38 2017
@@ -3436,6 +3436,15 @@ svn_repos_dump_fs(svn_repos_t *repos,
* to be stamped as if they were newly created via the normal commit
* process.
*
+ * If @a normalize_props is set, attempt to normalize invalid Subversion
+ * revision and node properties (those in the svn: namespace) so that
+ * their values would follow the established rules for them. For example,
+ * for such properties, typically the value must be in UTF-8 with LF
+ * line endings.
+ *
+ * @note The details or the performed normalizations are deliberately
+ * left unspecified and may change in the future.
+ *
* If non-NULL, use @a notify_func and @a notify_baton to send notification
* of events to the caller.
*
@@ -3443,8 +3452,34 @@ svn_repos_dump_fs(svn_repos_t *repos,
* @a cancel_baton as argument to see if the client wishes to cancel
* the load.
*
+ * @since New in 1.10.
+ */
+svn_error_t *
+svn_repos_load_fs6(svn_repos_t *repos,
+ svn_stream_t *dumpstream,
+ svn_revnum_t start_rev,
+ svn_revnum_t end_rev,
+ enum svn_repos_load_uuid uuid_action,
+ const char *parent_dir,
+ svn_boolean_t use_pre_commit_hook,
+ svn_boolean_t use_post_commit_hook,
+ svn_boolean_t validate_props,
+ svn_boolean_t ignore_dates,
+ svn_boolean_t normalize_props,
+ svn_repos_notify_func_t notify_func,
+ void *notify_baton,
+ svn_cancel_func_t cancel_func,
+ void *cancel_baton,
+ apr_pool_t *pool);
+
+/**
+ * Similar to svn_repos_load_fs6(), but with the @a normalize_props
+ * parameter always set to @c FALSE.
+ *
* @since New in 1.9.
+ * @deprecated Provided for backward compatibility with the 1.9 API.
*/
+SVN_DEPRECATED
svn_error_t *
svn_repos_load_fs5(svn_repos_t *repos,
svn_stream_t *dumpstream,
@@ -3567,6 +3602,15 @@ svn_repos_load_fs(svn_repos_t *repos,
* @a dumpstream, keeping whatever timestamps the revisions currently
* have.
*
+ * If @a normalize_props is set, attempt to normalize invalid Subversion
+ * revision and node properties (those in the svn: namespace) so that
+ * their values would follow the established rules for them. For example,
+ * for such properties, typically the value must be in UTF-8 with LF
+ * line endings.
+ *
+ * @note The details or the performed normalizations are deliberately
+ * left unspecified and may change in the future.
+ *
* If non-NULL, use @a notify_func and @a notify_baton to send notification
* of events to the caller.
*
@@ -3585,6 +3629,7 @@ svn_repos_load_fs_revprops(svn_repos_t *
svn_revnum_t end_rev,
svn_boolean_t validate_props,
svn_boolean_t ignore_dates,
+ svn_boolean_t normalize_props,
svn_repos_notify_func_t notify_func,
void *notify_baton,
svn_cancel_func_t cancel_func,
@@ -3788,12 +3833,47 @@ svn_repos_parse_dumpstream3(svn_stream_t
* to be stamped as if they were newly created via the normal commit
* process.
*
+ * If @a normalize_props is set, attempt to normalize invalid Subversion
+ * revision and node properties (those in the svn: namespace) so that
+ * their values would follow the established rules for them. For example,
+ * for such properties, typically the value must be in UTF-8 with LF
+ * line endings.
+ *
+ * @note The details or the performed normalizations are deliberately
+ * left unspecified and may change in the future.
+ *
* If @a parent_dir is not NULL, then the parser will reparent all the
* loaded nodes, from root to @a parent_dir. The directory @a parent_dir
* must be an existing directory in the repository.
*
+ * @since New in 1.10.
+ */
+svn_error_t *
+svn_repos_get_fs_build_parser6(const svn_repos_parse_fns3_t **parser,
+ void **parse_baton,
+ svn_repos_t *repos,
+ svn_revnum_t start_rev,
+ svn_revnum_t end_rev,
+ svn_boolean_t use_history,
+ svn_boolean_t validate_props,
+ enum svn_repos_load_uuid uuid_action,
+ const char *parent_dir,
+ svn_boolean_t use_pre_commit_hook,
+ svn_boolean_t use_post_commit_hook,
+ svn_boolean_t ignore_dates,
+ svn_boolean_t normalize_props,
+ svn_repos_notify_func_t notify_func,
+ void *notify_baton,
+ apr_pool_t *pool);
+
+/**
+ * Similar to svn_repos_get_fs_build_parser6(), but with the
+ * @a normalize_props parameter always set to @c FALSE.
+ *
* @since New in 1.9.
+ * @deprecated Provided for backward compatibility with the 1.9 API.
*/
+SVN_DEPRECATED
svn_error_t *
svn_repos_get_fs_build_parser5(const svn_repos_parse_fns3_t **parser,
void **parse_baton,
Modified: subversion/trunk/subversion/libsvn_repos/deprecated.c
URL:
http://svn.apache.org/viewvc/subversion/trunk/subversion/libsvn_repos/deprecated.c?rev=1807836&r1=1807835&r2=1807836&view=diff
==============================================================================
--- subversion/trunk/subversion/libsvn_repos/deprecated.c (original)
+++ subversion/trunk/subversion/libsvn_repos/deprecated.c Fri Sep 8 22:36:38
2017
@@ -901,6 +901,31 @@ svn_repos_verify_fs(svn_repos_t *repos,
/*** From load.c ***/
svn_error_t *
+svn_repos_load_fs5(svn_repos_t *repos,
+ svn_stream_t *dumpstream,
+ svn_revnum_t start_rev,
+ svn_revnum_t end_rev,
+ enum svn_repos_load_uuid uuid_action,
+ const char *parent_dir,
+ svn_boolean_t use_pre_commit_hook,
+ svn_boolean_t use_post_commit_hook,
+ svn_boolean_t validate_props,
+ svn_boolean_t ignore_dates,
+ svn_repos_notify_func_t notify_func,
+ void *notify_baton,
+ svn_cancel_func_t cancel_func,
+ void *cancel_baton,
+ apr_pool_t *pool)
+{
+ return svn_repos_load_fs6(repos, dumpstream, start_rev, end_rev,
+ uuid_action, parent_dir,
+ use_post_commit_hook, use_post_commit_hook,
+ validate_props, ignore_dates, FALSE,
+ notify_func, notify_baton,
+ cancel_func, cancel_baton, pool);
+}
+
+svn_error_t *
svn_repos_load_fs4(svn_repos_t *repos,
svn_stream_t *dumpstream,
svn_revnum_t start_rev,
@@ -1092,6 +1117,40 @@ svn_repos_load_fs(svn_repos_t *repos,
}
svn_error_t *
+svn_repos_get_fs_build_parser5(const svn_repos_parse_fns3_t **parser,
+ void **parse_baton,
+ svn_repos_t *repos,
+ svn_revnum_t start_rev,
+ svn_revnum_t end_rev,
+ svn_boolean_t use_history,
+ svn_boolean_t validate_props,
+ enum svn_repos_load_uuid uuid_action,
+ const char *parent_dir,
+ svn_boolean_t use_pre_commit_hook,
+ svn_boolean_t use_post_commit_hook,
+ svn_boolean_t ignore_dates,
+ svn_repos_notify_func_t notify_func,
+ void *notify_baton,
+ apr_pool_t *pool)
+{
+ SVN_ERR(svn_repos_get_fs_build_parser6(parser, parse_baton,
+ repos,
+ start_rev, end_rev,
+ use_history,
+ validate_props,
+ uuid_action,
+ parent_dir,
+ use_pre_commit_hook,
+ use_post_commit_hook,
+ ignore_dates,
+ FALSE /* normalize_props */,
+ notify_func,
+ notify_baton,
+ pool));
+ return SVN_NO_ERROR;
+}
+
+svn_error_t *
svn_repos_get_fs_build_parser4(const svn_repos_parse_fns3_t **callbacks,
void **parse_baton,
svn_repos_t *repos,
Modified: subversion/trunk/subversion/libsvn_repos/fs-wrap.c
URL:
http://svn.apache.org/viewvc/subversion/trunk/subversion/libsvn_repos/fs-wrap.c?rev=1807836&r1=1807835&r2=1807836&view=diff
==============================================================================
--- subversion/trunk/subversion/libsvn_repos/fs-wrap.c (original)
+++ subversion/trunk/subversion/libsvn_repos/fs-wrap.c Fri Sep 8 22:36:38 2017
@@ -33,6 +33,7 @@
#include "svn_repos.h"
#include "svn_time.h"
#include "svn_sorts.h"
+#include "svn_subst.h"
#include "repos.h"
#include "svn_private_config.h"
#include "private/svn_repos_private.h"
@@ -259,6 +260,34 @@ svn_repos__validate_prop(const char *nam
}
return SVN_NO_ERROR;
+}
+
+
+svn_error_t *
+svn_repos__normalize_prop(const svn_string_t **result_p,
+ svn_boolean_t *normalized_p,
+ const char *name,
+ const svn_string_t *value,
+ apr_pool_t *result_pool,
+ apr_pool_t *scratch_pool)
+{
+ if (svn_prop_needs_translation(name) && value)
+ {
+ svn_string_t *new_value;
+
+ SVN_ERR(svn_subst_translate_string2(&new_value, NULL, normalized_p,
+ value, "UTF-8", TRUE,
+ result_pool, scratch_pool));
+ *result_p = new_value;
+ }
+ else
+ {
+ *result_p = svn_string_dup(value, result_pool);
+ if (normalized_p)
+ *normalized_p = FALSE;
+ }
+
+ return SVN_NO_ERROR;
}
Modified: subversion/trunk/subversion/libsvn_repos/load-fs-vtable.c
URL:
http://svn.apache.org/viewvc/subversion/trunk/subversion/libsvn_repos/load-fs-vtable.c?rev=1807836&r1=1807835&r2=1807836&view=diff
==============================================================================
--- subversion/trunk/subversion/libsvn_repos/load-fs-vtable.c (original)
+++ subversion/trunk/subversion/libsvn_repos/load-fs-vtable.c Fri Sep 8
22:36:38 2017
@@ -55,6 +55,7 @@ struct parse_baton
svn_boolean_t use_history;
svn_boolean_t validate_props;
svn_boolean_t ignore_dates;
+ svn_boolean_t normalize_props;
svn_boolean_t use_pre_commit_hook;
svn_boolean_t use_post_commit_hook;
enum svn_repos_load_uuid uuid_action;
@@ -163,8 +164,12 @@ change_rev_prop(svn_repos_t *repos,
const char *name,
const svn_string_t *value,
svn_boolean_t validate_props,
+ svn_boolean_t normalize_props,
apr_pool_t *pool)
{
+ if (normalize_props)
+ SVN_ERR(svn_repos__normalize_prop(&value, NULL, name, value, pool, pool));
+
if (validate_props)
return svn_repos_fs_change_rev_prop4(repos, revision, NULL, name,
NULL, value, FALSE, FALSE,
@@ -1024,7 +1029,8 @@ close_revision(void *baton)
const svn_prop_t *prop = &APR_ARRAY_IDX(diff, i, svn_prop_t);
SVN_ERR(change_rev_prop(pb->repos, 0, prop->name, prop->value,
- pb->validate_props, rb->pool));
+ pb->validate_props, pb->normalize_props,
+ rb->pool));
}
}
@@ -1042,6 +1048,23 @@ close_revision(void *baton)
prop->value = NULL;
}
+ if (rb->pb->normalize_props)
+ {
+ apr_pool_t *iterpool;
+ int i;
+
+ iterpool = svn_pool_create(rb->pool);
+ for (i = 0; i < rb->revprops->nelts; i++)
+ {
+ svn_prop_t *prop = &APR_ARRAY_IDX(rb->revprops, i, svn_prop_t);
+
+ svn_pool_clear(iterpool);
+ SVN_ERR(svn_repos__normalize_prop(&prop->value, NULL, prop->name,
+ prop->value, rb->pool, iterpool));
+ }
+ svn_pool_destroy(iterpool);
+ }
+
/* Apply revision property changes. */
if (rb->pb->validate_props)
SVN_ERR(svn_repos_fs_change_txn_props(rb->txn, rb->revprops, rb->pool));
@@ -1158,7 +1181,7 @@ close_revision(void *baton)
svn_error_t *
-svn_repos_get_fs_build_parser5(const svn_repos_parse_fns3_t **callbacks,
+svn_repos_get_fs_build_parser6(const svn_repos_parse_fns3_t **callbacks,
void **parse_baton,
svn_repos_t *repos,
svn_revnum_t start_rev,
@@ -1170,6 +1193,7 @@ svn_repos_get_fs_build_parser5(const svn
svn_boolean_t use_pre_commit_hook,
svn_boolean_t use_post_commit_hook,
svn_boolean_t ignore_dates,
+ svn_boolean_t normalize_props,
svn_repos_notify_func_t notify_func,
void *notify_baton,
apr_pool_t *pool)
@@ -1218,6 +1242,7 @@ svn_repos_get_fs_build_parser5(const svn
pb->use_pre_commit_hook = use_pre_commit_hook;
pb->use_post_commit_hook = use_post_commit_hook;
pb->ignore_dates = ignore_dates;
+ pb->normalize_props = normalize_props;
*callbacks = parser;
*parse_baton = pb;
@@ -1226,7 +1251,7 @@ svn_repos_get_fs_build_parser5(const svn
svn_error_t *
-svn_repos_load_fs5(svn_repos_t *repos,
+svn_repos_load_fs6(svn_repos_t *repos,
svn_stream_t *dumpstream,
svn_revnum_t start_rev,
svn_revnum_t end_rev,
@@ -1235,6 +1260,7 @@ svn_repos_load_fs5(svn_repos_t *repos,
svn_boolean_t use_pre_commit_hook,
svn_boolean_t use_post_commit_hook,
svn_boolean_t validate_props,
+ svn_boolean_t normalize_props,
svn_boolean_t ignore_dates,
svn_repos_notify_func_t notify_func,
void *notify_baton,
@@ -1247,7 +1273,7 @@ svn_repos_load_fs5(svn_repos_t *repos,
/* This is really simple. */
- SVN_ERR(svn_repos_get_fs_build_parser5(&parser, &parse_baton,
+ SVN_ERR(svn_repos_get_fs_build_parser6(&parser, &parse_baton,
repos,
start_rev, end_rev,
TRUE, /* look for copyfrom revs */
@@ -1257,6 +1283,7 @@ svn_repos_load_fs5(svn_repos_t *repos,
use_pre_commit_hook,
use_post_commit_hook,
ignore_dates,
+ normalize_props,
notify_func,
notify_baton,
pool));
@@ -1345,7 +1372,8 @@ revprops_close_revision(void *baton)
const svn_prop_t *prop = &APR_ARRAY_IDX(diff, i, svn_prop_t);
SVN_ERR(change_rev_prop(pb->repos, rb->rev, prop->name, prop->value,
- pb->validate_props, rb->pool));
+ pb->validate_props, pb->normalize_props,
+ rb->pool));
}
if (pb->notify_func)
@@ -1386,6 +1414,11 @@ revprops_close_revision(void *baton)
*
* If IGNORE_DATES is set, ignore any revision datestamps found in
* DUMPSTREAM, keeping whatever timestamps the revisions currently have.
+ *
+ * If NORMALIZE_PROPS is set, attempt to normalize invalid Subversion
+ * revision and node properties (those in the svn: namespace) so that
+ * their values would follow the established rules for them. Currently,
+ * this means translating non-LF line endings in the property values to LF.
*/
static svn_error_t *
build_revprop_parser(const svn_repos_parse_fns3_t **callbacks,
@@ -1395,6 +1428,7 @@ build_revprop_parser(const svn_repos_par
svn_revnum_t end_rev,
svn_boolean_t validate_props,
svn_boolean_t ignore_dates,
+ svn_boolean_t normalize_props,
svn_repos_notify_func_t notify_func,
void *notify_baton,
apr_pool_t *result_pool)
@@ -1440,6 +1474,7 @@ build_revprop_parser(const svn_repos_par
pb->use_pre_commit_hook = FALSE;
pb->use_post_commit_hook = FALSE;
pb->ignore_dates = ignore_dates;
+ pb->normalize_props = normalize_props;
*callbacks = parser;
*parse_baton = pb;
@@ -1454,6 +1489,7 @@ svn_repos_load_fs_revprops(svn_repos_t *
svn_revnum_t end_rev,
svn_boolean_t validate_props,
svn_boolean_t ignore_dates,
+ svn_boolean_t normalize_props,
svn_repos_notify_func_t notify_func,
void *notify_baton,
svn_cancel_func_t cancel_func,
@@ -1470,6 +1506,7 @@ svn_repos_load_fs_revprops(svn_repos_t *
start_rev, end_rev,
validate_props,
ignore_dates,
+ normalize_props,
notify_func,
notify_baton,
scratch_pool));
Modified: subversion/trunk/subversion/svnadmin/svnadmin.c
URL:
http://svn.apache.org/viewvc/subversion/trunk/subversion/svnadmin/svnadmin.c?rev=1807836&r1=1807835&r2=1807836&view=diff
==============================================================================
--- subversion/trunk/subversion/svnadmin/svnadmin.c (original)
+++ subversion/trunk/subversion/svnadmin/svnadmin.c Fri Sep 8 22:36:38 2017
@@ -151,7 +151,8 @@ enum svnadmin__cmdline_options_t
svnadmin__compatible_version,
svnadmin__check_normalization,
svnadmin__metadata_only,
- svnadmin__no_flush_to_disk
+ svnadmin__no_flush_to_disk,
+ svnadmin__normalize_props
};
/* Option codes and descriptions.
@@ -274,6 +275,10 @@ static const apr_getopt_option_t options
N_("disable flushing to disk during the operation\n"
" (faster, but unsafe on power off)")},
+ {"normalize-props", svnadmin__normalize_props, 0,
+ N_("normalize property values found in the dumpstream\n"
+ " (currently, only translates non-LF line
endings)")},
+
{NULL}
};
@@ -396,7 +401,8 @@ static const svn_opt_subcommand_desc2_t
{'q', 'r', svnadmin__ignore_uuid, svnadmin__force_uuid,
svnadmin__ignore_dates,
svnadmin__use_pre_commit_hook, svnadmin__use_post_commit_hook,
- svnadmin__parent_dir, svnadmin__bypass_prop_validation, 'M',
+ svnadmin__parent_dir, svnadmin__normalize_props,
+ svnadmin__bypass_prop_validation, 'M',
svnadmin__no_flush_to_disk, 'F'},
{{'F', N_("read from file ARG instead of stdin")}} },
@@ -407,8 +413,8 @@ static const svn_opt_subcommand_desc2_t
"repository will cause an error. Progress feedback is sent to stdout.\n"
"If --revision is specified, limit the loaded revisions to only those\n"
"in the dump stream whose revision numbers match the specified range.\n"),
- {'q', 'r', svnadmin__force_uuid, svnadmin__bypass_prop_validation,
- svnadmin__no_flush_to_disk, 'F'},
+ {'q', 'r', svnadmin__force_uuid, svnadmin__normalize_props,
+ svnadmin__bypass_prop_validation, svnadmin__no_flush_to_disk, 'F'},
{{'F', N_("read from file ARG instead of stdin")}} },
{"lock", subcommand_lock, {0}, N_
@@ -547,6 +553,7 @@ struct svnadmin_opt_state
svn_boolean_t bypass_prop_validation; /*
--bypass-prop-validation */
svn_boolean_t ignore_dates; /* --ignore-dates */
svn_boolean_t no_flush_to_disk; /* --no-flush-to-disk */
+ svn_boolean_t normalize_props; /* --normalize_props */
enum svn_repos_load_uuid uuid_action; /* --ignore-uuid,
--force-uuid */
apr_uint64_t memory_cache_size; /* --memory-cache-size M */
@@ -1534,20 +1541,32 @@ subcommand_load(apr_getopt_t *os, void *
if (! opt_state->quiet)
feedback_stream = recode_stream_create(stdout, pool);
- err = svn_repos_load_fs5(repos, in_stream, lower, upper,
+ err = svn_repos_load_fs6(repos, in_stream, lower, upper,
opt_state->uuid_action, opt_state->parent_dir,
opt_state->use_pre_commit_hook,
opt_state->use_post_commit_hook,
!opt_state->bypass_prop_validation,
+ opt_state->normalize_props,
opt_state->ignore_dates,
opt_state->quiet ? NULL : repos_notify_handler,
feedback_stream, check_cancel, NULL, pool);
- if (err && err->apr_err == SVN_ERR_BAD_PROPERTY_VALUE)
- return svn_error_quick_wrap(err,
- _("Invalid property value found in "
- "dumpstream; consider repairing the source "
- "or using --bypass-prop-validation while "
- "loading."));
+
+ if (svn_error_find_cause(err, SVN_ERR_BAD_PROPERTY_VALUE_EOL))
+ {
+ return svn_error_quick_wrap(err,
+ _("A property with invalid line ending "
+ "found in dumpstream; consider using "
+ "--normalize-props while loading."));
+ }
+ else if (err && err->apr_err == SVN_ERR_BAD_PROPERTY_VALUE)
+ {
+ return svn_error_quick_wrap(err,
+ _("Invalid property value found in "
+ "dumpstream; consider repairing the "
+ "source or using --bypass-prop-validation "
+ "while loading."));
+ }
+
return err;
}
@@ -1584,16 +1603,28 @@ subcommand_load_revprops(apr_getopt_t *o
err = svn_repos_load_fs_revprops(repos, in_stream, lower, upper,
!opt_state->bypass_prop_validation,
+ opt_state->normalize_props,
opt_state->ignore_dates,
opt_state->quiet ? NULL
: repos_notify_handler,
feedback_stream, check_cancel, NULL, pool);
- if (err && err->apr_err == SVN_ERR_BAD_PROPERTY_VALUE)
- return svn_error_quick_wrap(err,
- _("Invalid property value found in "
- "dumpstream; consider repairing the source "
- "or using --bypass-prop-validation while "
- "loading."));
+
+ if (svn_error_find_cause(err, SVN_ERR_BAD_PROPERTY_VALUE_EOL))
+ {
+ return svn_error_quick_wrap(err,
+ _("A property with invalid line ending "
+ "found in dumpstream; consider using "
+ "--normalize-props while loading."));
+ }
+ else if (err && err->apr_err == SVN_ERR_BAD_PROPERTY_VALUE)
+ {
+ return svn_error_quick_wrap(err,
+ _("Invalid property value found in "
+ "dumpstream; consider repairing the "
+ "source or using --bypass-prop-validation "
+ "while loading."));
+ }
+
return err;
}
@@ -2884,6 +2915,9 @@ sub_main(int *exit_code, int argc, const
case svnadmin__no_flush_to_disk:
opt_state.no_flush_to_disk = TRUE;
break;
+ case svnadmin__normalize_props:
+ opt_state.normalize_props = TRUE;
+ break;
default:
{
SVN_ERR(subcommand_help(NULL, NULL, pool));
Modified: subversion/trunk/subversion/svnrdump/load_editor.c
URL:
http://svn.apache.org/viewvc/subversion/trunk/subversion/svnrdump/load_editor.c?rev=1807836&r1=1807835&r2=1807836&view=diff
==============================================================================
--- subversion/trunk/subversion/svnrdump/load_editor.c (original)
+++ subversion/trunk/subversion/svnrdump/load_editor.c Fri Sep 8 22:36:38 2017
@@ -713,8 +713,8 @@ set_revision_property(void *baton,
{
struct revision_baton *rb = baton;
- SVN_ERR(svn_rdump__normalize_prop(&value, name, value, rb->pool));
-
+ SVN_ERR(svn_repos__normalize_prop(&value, NULL, name, value,
+ rb->pool, rb->pool));
SVN_ERR(svn_repos__validate_prop(name, value, rb->pool));
if (rb->rev > 0)
@@ -775,7 +775,7 @@ set_node_property(void *baton,
value = new_value;
}
- SVN_ERR(svn_rdump__normalize_prop(&value, name, value, pool));
+ SVN_ERR(svn_repos__normalize_prop(&value, NULL, name, value, pool, pool));
SVN_ERR(svn_repos__validate_prop(name, value, pool));
Modified: subversion/trunk/subversion/svnrdump/svnrdump.h
URL:
http://svn.apache.org/viewvc/subversion/trunk/subversion/svnrdump/svnrdump.h?rev=1807836&r1=1807835&r2=1807836&view=diff
==============================================================================
--- subversion/trunk/subversion/svnrdump/svnrdump.h (original)
+++ subversion/trunk/subversion/svnrdump/svnrdump.h Fri Sep 8 22:36:38 2017
@@ -106,23 +106,6 @@ svn_rdump__normalize_props(apr_hash_t **
apr_hash_t *props,
apr_pool_t *result_pool);
-/* Normalize the line ending style of a single property that "needs
- * translation" (according to svn_prop_needs_translation(),
- * currently all svn:* props) so that they contain only LF (\n) line endings.
- * "\r" characters found mid-line are replaced with "\n".
- * "\r\n" sequences are replaced with "\n"
- *
- * NAME is used to check that VALUE should be normalized, and if this is the
- * case, *RESULT_P is then set to the normalized property value. If no
- * normalization is required, VALUE will be copied to RESULT_POOL
- * unchanged.
- */
-svn_error_t *
-svn_rdump__normalize_prop(const svn_string_t **result_p,
- const char *name,
- const svn_string_t *value,
- apr_pool_t *result_pool);
-
#ifdef __cplusplus
}
#endif /* __cplusplus */
Modified: subversion/trunk/subversion/svnrdump/util.c
URL:
http://svn.apache.org/viewvc/subversion/trunk/subversion/svnrdump/util.c?rev=1807836&r1=1807835&r2=1807836&view=diff
==============================================================================
--- subversion/trunk/subversion/svnrdump/util.c (original)
+++ subversion/trunk/subversion/svnrdump/util.c Fri Sep 8 22:36:38 2017
@@ -21,58 +21,35 @@
* ====================================================================
*/
-#include "svn_error.h"
-#include "svn_pools.h"
-#include "svn_string.h"
-#include "svn_props.h"
-#include "svn_subst.h"
+#include "private/svn_repos_private.h"
#include "svnrdump.h"
svn_error_t *
-svn_rdump__normalize_prop(const svn_string_t **result_p,
- const char *name,
- const svn_string_t *value,
- apr_pool_t *result_pool)
-{
- if (svn_prop_needs_translation(name) && value)
- {
- const char *cstring;
-
- SVN_ERR(svn_subst_translate_cstring2(value->data, &cstring,
- "\n", TRUE,
- NULL, FALSE,
- result_pool));
-
- *result_p = svn_string_create(cstring, result_pool);
- }
- else
- {
- *result_p = svn_string_dup(value, result_pool);
- }
-
- return SVN_NO_ERROR;
-}
-
-svn_error_t *
svn_rdump__normalize_props(apr_hash_t **normal_props,
apr_hash_t *props,
apr_pool_t *result_pool)
{
apr_hash_index_t *hi;
+ apr_pool_t *iterpool;
*normal_props = apr_hash_make(result_pool);
+ iterpool = svn_pool_create(result_pool);
for (hi = apr_hash_first(result_pool, props); hi;
hi = apr_hash_next(hi))
{
const char *key = apr_hash_this_key(hi);
const svn_string_t *value = apr_hash_this_val(hi);
- SVN_ERR(svn_rdump__normalize_prop(&value, key, value, result_pool));
+ svn_pool_clear(iterpool);
+ SVN_ERR(svn_repos__normalize_prop(&value, NULL, key, value,
+ result_pool, iterpool));
svn_hash_sets(*normal_props, key, value);
}
+ svn_pool_destroy(iterpool);
+
return SVN_NO_ERROR;
}
Modified: subversion/trunk/subversion/tests/cmdline/svnadmin_tests.py
URL:
http://svn.apache.org/viewvc/subversion/trunk/subversion/tests/cmdline/svnadmin_tests.py?rev=1807836&r1=1807835&r2=1807836&view=diff
==============================================================================
--- subversion/trunk/subversion/tests/cmdline/svnadmin_tests.py (original)
+++ subversion/trunk/subversion/tests/cmdline/svnadmin_tests.py Fri Sep 8
22:36:38 2017
@@ -1616,14 +1616,34 @@ text
sbox.build(empty=True)
# Try to load the dumpstream, expecting a failure (because of mixed EOLs).
- load_and_verify_dumpstream(sbox, [], svntest.verify.AnyOutput,
- dumpfile_revisions, False, dump_str,
- '--ignore-uuid')
+ exp_err = svntest.verify.RegexListOutput(['svnadmin: E125005',
+ 'svnadmin: E125005',
+ 'svnadmin: E125017'],
+ match_all=False)
+ load_and_verify_dumpstream(sbox, [], exp_err, dumpfile_revisions,
+ False, dump_str, '--ignore-uuid')
# Now try it again bypassing prop validation. (This interface takes
# care of the removal and recreation of the original repository.)
svntest.actions.load_repo(sbox, dump_str=dump_str,
bypass_prop_validation=True)
+ # Getting the property should fail.
+ svntest.actions.run_and_verify_svn(None, 'svn: E135000: ',
+ 'pg', 'svn:log', '--revprop', '-r1',
+ sbox.repo_url)
+
+ # Now try it again with prop normalization.
+ svntest.actions.load_repo(sbox, dump_str=dump_str,
+ bypass_prop_validation=False,
+ normalize_props=True)
+ # We should get the expected property value.
+ exit_code, output, _ = svntest.main.run_svn(None, 'pg', 'svn:log',
+ '--revprop', '-r1',
+ '--no-newline',
+ sbox.repo_url)
+ svntest.verify.verify_exit_code(None, exit_code, 0)
+ if output != ['\n', '\n']:
+ raise svntest.Failure("Unexpected property value %s" % output)
# This test intentionally corrupts a revision and assumes an FSFS
# repository. If you can make it work with BDB please do so.
Modified: subversion/trunk/subversion/tests/cmdline/svntest/actions.py
URL:
http://svn.apache.org/viewvc/subversion/trunk/subversion/tests/cmdline/svntest/actions.py?rev=1807836&r1=1807835&r2=1807836&view=diff
==============================================================================
--- subversion/trunk/subversion/tests/cmdline/svntest/actions.py (original)
+++ subversion/trunk/subversion/tests/cmdline/svntest/actions.py Fri Sep 8
22:36:38 2017
@@ -355,7 +355,8 @@ def run_and_verify_svn2(expected_stdout,
return exit_code, out, err
def run_and_verify_load(repo_dir, dump_file_content,
- bypass_prop_validation = False):
+ bypass_prop_validation = False,
+ normalize_props = False):
"Runs 'svnadmin load' and reports any errors."
if not isinstance(dump_file_content, list):
raise TypeError("dump_file_content argument should have list type")
@@ -363,6 +364,8 @@ def run_and_verify_load(repo_dir, dump_f
args = ()
if bypass_prop_validation:
args += ('--bypass-prop-validation',)
+ if normalize_props:
+ args += ('--normalize-props',)
main.run_command_stdin(
main.svnadmin_binary, expected_stderr, 0, True, dump_file_content,
'load', '--force-uuid', '--quiet', repo_dir, *args)
@@ -473,7 +476,8 @@ def run_and_verify_svnsync2(expected_std
def load_repo(sbox, dumpfile_path = None, dump_str = None,
- bypass_prop_validation = False,create_wc=True):
+ bypass_prop_validation = False, create_wc=True,
+ normalize_props = False):
"Loads the dumpfile into sbox"
if not dump_str:
dump_str = open(dumpfile_path, "rb").read()
@@ -485,7 +489,7 @@ def load_repo(sbox, dumpfile_path = None
# Load the mergetracking dumpfile into the repos, and check it out the repo
run_and_verify_load(sbox.repo_dir, dump_str.splitlines(True),
- bypass_prop_validation)
+ bypass_prop_validation, normalize_props)
if create_wc:
run_and_verify_svn(None, [], "co", sbox.repo_url, sbox.wc_dir)
Modified: subversion/trunk/subversion/tests/libsvn_repos/dump-load-test.c
URL:
http://svn.apache.org/viewvc/subversion/trunk/subversion/tests/libsvn_repos/dump-load-test.c?rev=1807836&r1=1807835&r2=1807836&view=diff
==============================================================================
--- subversion/trunk/subversion/tests/libsvn_repos/dump-load-test.c (original)
+++ subversion/trunk/subversion/tests/libsvn_repos/dump-load-test.c Fri Sep 8
22:36:38 2017
@@ -120,13 +120,14 @@ test_load_bad_props(svn_stringbuf_t *dum
svn_revnum_t youngest_rev;
svn_string_t *loaded_prop_val;
- SVN_ERR(svn_repos_load_fs5(repos, stream,
+ SVN_ERR(svn_repos_load_fs6(repos, stream,
SVN_INVALID_REVNUM, SVN_INVALID_REVNUM,
svn_repos_load_uuid_default,
parent_fspath,
FALSE, FALSE, /*use_*_commit_hook*/
validate_props,
FALSE /*ignore_dates*/,
+ FALSE /*normalize_props*/,
notify_func, notify_baton,
NULL, NULL, /*cancellation*/
pool));
Modified: subversion/trunk/tools/client-side/bash_completion
URL:
http://svn.apache.org/viewvc/subversion/trunk/tools/client-side/bash_completion?rev=1807836&r1=1807835&r2=1807836&view=diff
==============================================================================
--- subversion/trunk/tools/client-side/bash_completion (original)
+++ subversion/trunk/tools/client-side/bash_completion Fri Sep 8 22:36:38 2017
@@ -1153,7 +1153,7 @@ _svnadmin ()
cmdOpts="--ignore-uuid --force-uuid --parent-dir -q --quiet \
--use-pre-commit-hook --use-post-commit-hook \
--bypass-prop-validation -M --memory-cache-size \
- --no-flush-to-disk -F --file"
+ --no-flush-to-disk --normalize-props -F --file"
;;
lstxns)
cmdOpts="-r --revision"
