Both are parts of the game.
The order actually was significant as shown by the test case mainly because
of the order difference between the java util Properties object and the
felix Properties object. The first one is relies on a Hashtable while the
second relies on a LinkedHashMap.
This is significant because of the way the substitution was done.
if you start from a = $\{var}, ab = ${a}b, abc = ${ab}c
you had the following steps:
a = $\{var}, ab = ${a}b, abc = ${ab}c
a = $\{var}, ab = ${var}b, abc = ${ab}c
a = $\{var}, ab = ${var}b, abc = bc
The reason is that substitution were done using already substituted
variables, so when computing ${ab}c, it was using
${ab}c
${var}bc
bc
instead of
${ab}c
${a}bc
$\{var}bc
${var}bc
So the problem wan't really the order of the values, but the fact that the
substitution was done using already substituted values, which then made the
order significant.
Note that the result is now (and irrespective of the order of the lines):
a = ${var}, ab = ${var}b, abc = ${var}bc
So I think the escaping is now more deterministic. Please give it some
testing and let me know if you still have problems in this area.
2013/12/4 Bengt Rodehav <[email protected]>
> BTW. I did some experimenting with declaring the properties in different
> order in the configuration file. It did not seem to matter. I was under the
> impression that the recursive variable substitution is what makes it
> non-deterministic.
>
> If a property has been evaluated already it should not be evaluated again
> because another layer of the escape characters will then be removed.
>
> /Bengt
>
>
> 2013/12/4 Bengt Rodehav <[email protected]>
>
> > OK.
> >
> >
> > 2013/12/4 Guillaume Nodet <[email protected]>
> >
> >> Unfortunately, it does not seem to be sufficient, I'm investigating
> >> further
> >>
> >>
> >> 2013/12/4 Bengt Rodehav <[email protected]>
> >>
> >> > I noticed that you seem to have fixed the issues I had reported
> >> Guillaume.
> >> > Thanks a lot! Looking forward to the next release.
> >> >
> >> > /Bengt
> >> >
> >> >
> >> > 2013/12/2 Bengt Rodehav <[email protected]>
> >> >
> >> > > Thanks Guillaume!
> >> > >
> >> > >
> >> > > 2013/12/2 Guillaume Nodet <[email protected]>
> >> > >
> >> > >> I'll try to have a look at those today or tomorrow.
> >> > >>
> >> > >>
> >> > >> 2013/12/2 Bengt Rodehav <[email protected]>
> >> > >>
> >> > >> > I've replaced FELIX-4332 with FELIX-4338 and FELIX-4339.
> >> > >> >
> >> > >> > I have attached a patch for FELIX-4338 and hope that someone can
> >> have
> >> > a
> >> > >> > look at it and possibly commit it.
> >> > >> >
> >> > >> > FELIX-4339 is trickier but I would appreciate a discussion about
> >> how
> >> > >> this
> >> > >> > should be handled.
> >> > >> >
> >> > >> > /Bengt
> >> > >> >
> >> > >> >
> >> > >> > 2013/11/29 Bengt Rodehav <[email protected]>
> >> > >> >
> >> > >> > > I've tested more with the proposed change in order to stop
> >> > >> FileInstall to
> >> > >> > > incorrectly change the contents of the configuration file
> >> (problem
> >> > b)
> >> > >> > from
> >> > >> > > my previous post). It seems to work fine. I would really like
> >> that
> >> > to
> >> > >> be
> >> > >> > > fixed. Would you like me to create a patch atttached to the
> JIRA?
> >> > >> > >
> >> > >> > > Problem a) is probably not trivial to fix. I've experimented a
> >> lot
> >> > and
> >> > >> > > it's very hard for me to foresee how many escape characters I
> >> need
> >> > in
> >> > >> > > different circumstances. One real life example for me is how I
> >> > >> configure
> >> > >> > an
> >> > >> > > integration service that uses a Camel route underneath. If I
> put
> >> the
> >> > >> > > followiing contents in a test.cfg file:
> >> > >> > >
> >> > >> > > *mydir=C:/temp*
> >> > >> > >
> >> > >> > >
> >> > >> >
> >> > >>
> >> >
> >>
> *timestampedfile=$\\\\{file:onlyname\\\\}-$\\\\{date:now:yyyyMMddHHmmssSSS\\\\}.$\\\\{file:ext\\\\}*
> >> > >> > >
> >> *move=${mydir}/archive/$\\{date:now:yyyyMMdd\\}/${timestampedfile}*
> >> > >> > > *moveFailed=${mydir}/failed/${timestampedfile}*
> >> > >> > > *fromUri=file:${mydir}?move=${move}&moveFailed=${moveFailed}*
> >> > >> > >
> >> > >> > > And execute the following command:
> >> > >> > >
> >> > >> > >
> >> > >> > > *config:list "(service.pid=test)"*
> >> > >> > >
> >> > >> > > I get the following output:
> >> > >> > >
> >> > >> > >
> >> *----------------------------------------------------------------*
> >> > >> > > *Pid: test*
> >> > >> > > *BundleLocation: null*
> >> > >> > > *Properties:*
> >> > >> > > * moveFailed =
> >> > >> > >
> >> > >> >
> >> > >>
> >> >
> >>
> C:/temp/failed/${file:onlyname}-${date:now:yyyyMMddHHmmssSSS}.${file:ext}*
> >> > >> > > * mydir = C:/temp*
> >> > >> > > * timestampedfile =
> >> > >> > >
> $\{file:onlyname\}-$\{date:now:yyyyMMddHHmmssSSS\}.$\{file:ext\}*
> >> > >> > > * service.pid = test*
> >> > >> > > * fromUri =
> >> > >> > >
> >> file:C:/temp?move=C:/temp/archive//-.&moveFailed=C:/temp/failed/-.*
> >> > >> > > * move =
> >> > >> > >
> >> > >> >
> >> > >>
> >> >
> >>
> C:/temp/archive/${date:now:yyyyMMdd}/${file:onlyname}-${date:now:yyyyMMddHHmmssSSS}.${file:ext}*
> >> > >> > > * felix.fileinstall.filename =
> >> > >> > > file:/C:/dev/karaf/connect/common/etc/test.cfg*
> >> > >> > >
> >> > >> > > Thus, the variables "move" and "moveFailed" looks the way I
> want
> >> but
> >> > >> the
> >> > >> > > final variable "fromUri" is messed up because of an extra
> >> variable
> >> > >> > > substitution.
> >> > >> > >
> >> > >> > > I haven't managed to come up with any number of backslashes
> that
> >> > will
> >> > >> > > produce the correct result for me.
> >> > >> > >
> >> > >> > > The only workaround I have right now is to not use variables at
> >> all.
> >> > >> It
> >> > >> > > does, however, make the configuration files extremely verbose
> and
> >> > it's
> >> > >> > easy
> >> > >> > > to introduce errors that way.
> >> > >> > >
> >> > >> > > Presently, variable substitution is very unpredictable since
> it's
> >> > >> being
> >> > >> > > done in a recursive way. I would prefer doing it in an
> iterative
> >> > >> manner
> >> > >> > to
> >> > >> > > make it predictable. E g "${a}" should always evaluate to the
> >> same
> >> > >> value
> >> > >> > no
> >> > >> > > matter where in the configuration file it is referenced.
> >> > >> > >
> >> > >> > > /Bengt
> >> > >> > >
> >> > >> > >
> >> > >> > >
> >> > >> > >
> >> > >> > >
> >> > >> > >
> >> > >> > >
> >> > >> > >
> >> > >> > > 2013/11/28 Bengt Rodehav <[email protected]>
> >> > >> > >
> >> > >> > >> I've investigated this a bit more. There are actually two
> >> different
> >> > >> > >> problems:
> >> > >> > >>
> >> > >> > >> a) The number of escape characters I need depends on from
> where
> >> I
> >> > >> > >> reference the variable. For every indirection I need to double
> >> the
> >> > >> > number
> >> > >> > >> of backslashes. This also means that all uses of a variable
> >> > >> containing
> >> > >> > >> escape characters must be used from the same level of
> >> indirection.
> >> > A
> >> > >> bit
> >> > >> > >> complicated but it's due to the fact that all variables are
> >> > evaluated
> >> > >> > >> dynamically. This means that unescaping can occur several
> times.
> >> > >> > >>
> >> > >> > >> b) FileInstall incorrectly thinks that a configuration
> property
> >> is
> >> > >> > >> changed and therefore overwrites the property with the
> evaluated
> >> > >> value.
> >> > >> > >>
> >> > >> > >> I think I've found the reason (and possibly a solution) to b).
> >> > >> > >>
> >> > >> > >> In the ConfigInstaller.setConfig() method the properties are
> >> read
> >> > >> from a
> >> > >> > >> configuration file and propagated as a configuration. Here is
> an
> >> > >> excerpt
> >> > >> > >> from that method:
> >> > >> > >>
> >> > >> > >> * final Properties p = new Properties();*
> >> > >> > >> * in.mark(1);*
> >> > >> > >> * boolean isXml = in.read() == '<';*
> >> > >> > >> * in.reset();*
> >> > >> > >> * if (isXml) {*
> >> > >> > >> * p.loadFromXML(in);*
> >> > >> > >> * } else {*
> >> > >> > >> * p.load(in);*
> >> > >> > >> * }*
> >> > >> > >> * InterpolationHelper.performSubstitution((Map)
> >> p,
> >> > >> > >> context);*
> >> > >> > >> * ht.putAll(p);*
> >> > >> > >>
> >> > >> > >> Note that the file is read using Java's standard Properties
> >> class.
> >> > >> The
> >> > >> > >> unescaping is also done by that class. Then, at the end, the
> >> > variable
> >> > >> > >> substitution is done as a separate call.
> >> > >> > >>
> >> > >> > >> Then look at the ConfigInstaller.configurationEvent() method:
> >> > >> > >>
> >> > >> > >> * if (configurationEvent.getType() ==
> >> > >> > >> ConfigurationEvent.CM_UPDATED)*
> >> > >> > >> * {*
> >> > >> > >> * try*
> >> > >> > >> * {*
> >> > >> > >> * Configuration config =
> >> > >> > >> getConfigurationAdmin().getConfiguration(*
> >> > >> > >> *
> >> > >> > configurationEvent.getPid(),*
> >> > >> > >> *
> >> > >> > >> configurationEvent.getFactoryPid());*
> >> > >> > >> * Dictionary dict = config.getProperties();*
> >> > >> > >> * String fileName = (String) dict.get(
> >> > >> > >> DirectoryWatcher.FILENAME );*
> >> > >> > >> * File file = fileName != null ?
> >> > >> fromConfigKey(fileName)
> >> > >> > :
> >> > >> > >> null;*
> >> > >> > >> * if( file != null && file.isFile() ) {*
> >> > >> > >> * if( fileName.endsWith( ".cfg" ) )*
> >> > >> > >> * {*
> >> > >> > >> *
> >> > org.apache.felix.utils.properties.Properties
> >> > >> > >> props = new org.apache.felix.utils.properties.Properties(
> file,
> >> > >> context
> >> > >> > );*
> >> > >> > >>
> >> > >> > >> Note that now the configuration file is read using
> >> > >> > >> org.apache.felix.utils.properties.Properties class. It turns
> out
> >> > that
> >> > >> > they
> >> > >> > >> don't produce identical results. I haven't investigated
> exactly
> >> how
> >> > >> they
> >> > >> > >> differ but they do.
> >> > >> > >>
> >> > >> > >> A simple test:
> >> > >> > >>
> >> > >> > >> 1. Create a configuration file with the following content:
> >> > >> > >>
> >> > >> > >> a=$\\\\{var}
> >> > >> > >> ab=${a}b
> >> > >> > >> abc=${ab}c
> >> > >> > >>
> >> > >> > >> 2. Add the following line at the end:
> >> > >> > >>
> >> > >> > >> d=foo
> >> > >> > >>
> >> > >> > >> 3. FileInstall will now incorrectly change the contents of the
> >> > >> > >> configuration file to:
> >> > >> > >>
> >> > >> > >> a=$\\\\{var}
> >> > >> > >> ab=${a}b
> >> > >> > >> abc = ${var}bc
> >> > >> > >> d=foo
> >> > >> > >>
> >> > >> > >> Now if I change the ConfigInstaller.setConfig() method to the
> >> > >> following:
> >> > >> > >>
> >> > >> > >> *org.apache.felix.utils.properties.Properties p = new
> >> > >> > >> org.apache.felix.utils.properties.Properties( f, context );*
> >> > >> > >> *InterpolationHelper.performSubstitution((Map) p, context);*
> >> > >> > >>
> >> > >> > >> Then FileInstall will not incorrectly change the contents of
> the
> >> > >> > >> configuration file.
> >> > >> > >>
> >> > >> > >> I propose to do this change in order to solve problem b)
> above.
> >> I
> >> > >> > >> appreciate if you have any thoughts on this.
> >> > >> > >>
> >> > >> > >> I realize that problem a) is trickier due to the dynamic
> nature
> >> of
> >> > >> > >> variable substitution. I haven't yet determined how I think
> the
> >> > >> escape
> >> > >> > >> characters should be handled but the current situation is not
> >> > ideal.
> >> > >> > >>
> >> > >> > >> /Bengt
> >> > >> > >>
> >> > >> > >>
> >> > >> > >>
> >> > >> > >>
> >> > >> > >> 2013/11/28 Bengt Rodehav <[email protected]>
> >> > >> > >>
> >> > >> > >>> JIRA created:
> >> > >> > >>>
> >> > >> > >>> https://issues.apache.org/jira/browse/FELIX-4332
> >> > >> > >>>
> >> > >> > >>> /Bengt
> >> > >> > >>>
> >> > >> > >>>
> >> > >> > >>> 2013/11/28 Bengt Rodehav <[email protected]>
> >> > >> > >>>
> >> > >> > >>>> I've come up with easily reproducable errors using Karaf
> >> 2.3.3:
> >> > >> > >>>>
> >> > >> > >>>> - Install a fresh Karaf 2.3.3
> >> > >> > >>>> - Add the following line to etc/custom.properties:
> >> > >> > >>>> felix.fileinstall.enableConfigSave = true
> >> > >> > >>>>
> >> > >> > >>>> Create a file etc/test.cfg with the following contents:
> >> > >> > >>>>
> >> > >> > >>>> a=$\\{var}
> >> > >> > >>>> ab=${a}b
> >> > >> > >>>> abc=${ab}c
> >> > >> > >>>>
> >> > >> > >>>> I expect this to be evaluated to:
> >> > >> > >>>> a=$\{var}
> >> > >> > >>>> ab=$\{var}b
> >> > >> > >>>> abc=$\{var}bc
> >> > >> > >>>>
> >> > >> > >>>> But if I execute the Karaf command:
> >> > >> > >>>>
> >> > >> > >>>> config:list "(service.pid=test)"
> >> > >> > >>>>
> >> > >> > >>>> I get:
> >> > >> > >>>>
> >> > >> > >>>>
> >> ----------------------------------------------------------------
> >> > >> > >>>> Pid: test
> >> > >> > >>>> BundleLocation: null
> >> > >> > >>>> Properties:
> >> > >> > >>>> service.pid = test
> >> > >> > >>>> a = ${var}
> >> > >> > >>>> abc = bc
> >> > >> > >>>> felix.fileinstall.filename =
> >> > >> > >>>> file:/C:/dev/Karaf/apache-karaf-2.3.3/etc/test.cfg
> >> > >> > >>>> ab = b
> >> > >> > >>>>
> >> > >> > >>>> My interpretation of this is that the variable "a" has been
> >> > >> correctly
> >> > >> > >>>> evaluated. But, when evalutating the variable "ab" it seems
> >> that
> >> > >> the
> >> > >> > >>>> variable "a" is evaluated again despite the fact that it has
> >> > >> already
> >> > >> > been
> >> > >> > >>>> evaluated. FileInstall now looks for the value of a variable
> >> > called
> >> > >> > "var"
> >> > >> > >>>> which evalutes to an empty string because there is no such
> >> > >> variable.
> >> > >> > >>>>
> >> > >> > >>>> The variable "abc" consequently evaluates to "bc" since the
> >> > >> variable
> >> > >> > >>>> "ab" has been evaluated to "b".
> >> > >> > >>>>
> >> > >> > >>>> To make it even worse, now change the first row in test.cfg
> >> to:
> >> > >> > >>>>
> >> > >> > >>>> a=$\\\\{var}
> >> > >> > >>>>
> >> > >> > >>>> We now get:
> >> > >> > >>>>
> >> > >> > >>>>
> >> ----------------------------------------------------------------
> >> > >> > >>>> Pid: test
> >> > >> > >>>> BundleLocation: null
> >> > >> > >>>> Properties:
> >> > >> > >>>> service.pid = test
> >> > >> > >>>> a = $\{var}
> >> > >> > >>>> abc = ${var}bc
> >> > >> > >>>> felix.fileinstall.filename =
> >> > >> > >>>> file:/C:/dev/Karaf/apache-karaf-2.3.3/etc/test.cfg
> >> > >> > >>>> ab = ${var}b
> >> > >> > >>>>
> >> > >> > >>>> Thus we get the same phenomenom. The variable "a" is
> evaluated
> >> > >> > >>>> differently if it is evaluated on its own or as part of
> >> another
> >> > >> > expression.
> >> > >> > >>>> But, due to having configured FileInstall to write back
> >> changes,
> >> > >> the
> >> > >> > >>>> contents of the test.cfg is now changed by FileInstall
> despite
> >> > the
> >> > >> > fact
> >> > >> > >>>> that the configuration has not changed at all. The contents
> of
> >> > >> > test.cfg is
> >> > >> > >>>> now:
> >> > >> > >>>>
> >> > >> > >>>> a=$\\\\{var}
> >> > >> > >>>> ab=${a}b
> >> > >> > >>>> abc = ${var}bc
> >> > >> > >>>>
> >> > >> > >>>> The "abc" variable has been altered. FileInstall has
> >> incorrectly
> >> > >> > >>>> determined that its value has changed.
> >> > >> > >>>>
> >> > >> > >>>> This is clearly a bug. I will create a JIRA.
> >> > >> > >>>>
> >> > >> > >>>> /Bengt
> >> > >> > >>>>
> >> > >> > >>>>
> >> > >> > >>>>
> >> > >> > >>>> 2013/11/26 Bengt Rodehav <[email protected]>
> >> > >> > >>>>
> >> > >> > >>>>> I'm using Apache Karaf 2.3.3 which comes with FileInstall
> >> > 3.2.6. I
> >> > >> > >>>>> have set the felix.fileinstall.enableConfigSave property to
> >> true
> >> > >> in
> >> > >> > order
> >> > >> > >>>>> to have FileInstall write back configuration changes to the
> >> > file.
> >> > >> > Normally
> >> > >> > >>>>> all configuration changes are done by editing the
> >> configuration
> >> > >> file
> >> > >> > but
> >> > >> > >>>>> there is one property that I change programmatically using
> >> > >> > ConfigAdmin (an
> >> > >> > >>>>> "enable" property to start/stop my service). I am dependent
> >> on
> >> > >> that
> >> > >> > >>>>> property being persisted in the configuration file which is
> >> why
> >> > I
> >> > >> > set the
> >> > >> > >>>>> enableConfigSave property to true.
> >> > >> > >>>>>
> >> > >> > >>>>> When configuring FileInstall to write back configuration
> >> changes
> >> > >> to
> >> > >> > >>>>> the configuration file, it is important that variables are
> >> not
> >> > >> > substituted
> >> > >> > >>>>> for the evaluated value. This normally works since
> >> FileInstall
> >> > >> > evalutates
> >> > >> > >>>>> the property in the configuration file and compares it with
> >> the
> >> > >> > >>>>> configuration admin's value. If they are the same, the
> value
> >> in
> >> > >> the
> >> > >> > >>>>> configuration file is kept unchanged.
> >> > >> > >>>>>
> >> > >> > >>>>> However, when using the escape character this is broken. In
> >> my
> >> > >> case
> >> > >> > >>>>> I'm using Apache Camel underneath. When configuring routes
> >> via
> >> > the
> >> > >> > config
> >> > >> > >>>>> admin, I sometimes need to set a value to
> >> > >> > >>>>> "${expression-to-be-evaluated-by-camel}". I therefore
> escape
> >> the
> >> > >> "{"
> >> > >> > and
> >> > >> > >>>>> "}" to stop FileInstall from trying to evaluate the
> >> expression.
> >> > >> Like
> >> > >> > this:
> >> > >> > >>>>>
> >> > >> > >>>>> $\\{expression-to-be-evaluated-by-camel\\}
> >> > >> > >>>>>
> >> > >> > >>>>> This also normally works but not when I have an
> indirection.
> >> E g
> >> > >> when
> >> > >> > >>>>> specifying the following:
> >> > >> > >>>>>
> >> > >> > >>>>> a=$\\{var}
> >> > >> > >>>>> ab=${a}b
> >> > >> > >>>>>
> >> > >> > >>>>> FileInstall will change the configuration file to:
> >> > >> > >>>>>
> >> > >> > >>>>> a=$\\{var}
> >> > >> > >>>>> ab = ${var}b
> >> > >> > >>>>>
> >> > >> > >>>>> Note that the variable "ab" has now been expanded and
> written
> >> > >> back to
> >> > >> > >>>>> the configuration file even if neither of the variables "a"
> >> and
> >> > >> "ab"
> >> > >> > have
> >> > >> > >>>>> been changed.
> >> > >> > >>>>>
> >> > >> > >>>>> I think this is because FileInstall does the following:
> >> > >> > >>>>>
> >> > >> > >>>>> 1. Calculates the value of "a" to "$\{var}
> >> > >> > >>>>> 2. Calculates the value of "b" to "${var}b
> >> > >> > >>>>>
> >> > >> > >>>>> Note that every evaluation will perform "unescaping". This
> >> means
> >> > >> that
> >> > >> > >>>>> an extra "unescaping" will be done for every indirection
> >> which
> >> > >> fools
> >> > >> > >>>>> FileInstall into thinking that the property has been
> changed.
> >> > >> > >>>>>
> >> > >> > >>>>> I'm not exactly sure how this should be fixed in
> FileInstall.
> >> > One
> >> > >> > idea
> >> > >> > >>>>> is to never "unescape" already evaluated variables.
> Actually
> >> I
> >> > >> think
> >> > >> > this
> >> > >> > >>>>> is probably what would fix this...
> >> > >> > >>>>>
> >> > >> > >>>>> Does anybody have any ideas about this? Should I create a
> >> JIRA?
> >> > >> > >>>>>
> >> > >> > >>>>> /Bengt
> >> > >> > >>>>>
> >> > >> > >>>>>
> >> > >> > >>>>>
> >> > >> > >>>>>
> >> > >> > >>>>>
> >> > >> > >>>>>
> >> > >> > >>>>>
> >> > >> > >>>>
> >> > >> > >>>
> >> > >> > >>
> >> > >> > >
> >> > >> >
> >> > >>
> >> > >>
> >> > >>
> >> > >> --
> >> > >> -----------------------
> >> > >> Guillaume Nodet
> >> > >> ------------------------
> >> > >> Red Hat, Open Source Integration
> >> > >>
> >> > >> Email: [email protected]
> >> > >> Web: http://fusesource.com
> >> > >> Blog: http://gnodet.blogspot.com/
> >> > >>
> >> > >
> >> > >
> >> >
> >>
> >
> >
>
