Just committed the initial support for the soft hyphen. As we had two in favour of having the SHY always produce a break opportunity and only one against that's the route I took.
I had no luck with giving the SHY a reduced penalty and have the Knuth algorithm favour them before normal hyphenation breaks. Even with a penalty value of 1 fop still chooses the hyphenation break with a penalty of 50. Either I do something wrong or I misunderstand how the Knuth breaking calculation is suppose to work. May be one of the Knuth experts can have a look at this PLEASE. Also not correctly working (yet) is ipd calculation when kerning and a SHY break is involved. But may be that's a more general issue. For those looking closer at the commit the area handling within the text layout manager has changed a bit. Before this patch the assumption was made that the sequence of characters given to the LM will be fully output to the area tree. Now we have for the first time the case that characters (the SHY) can be dropped. This led to changes with respect to certain indexing loops. Manuel