Jon, After I get Solaris building again, I'm going to take a stab at upgrading the docs system to modern docbook. In the past I've resisted removing in-house components in favor of external dependencies, but I recognize that there's a difference between maintaining a quality codebase and stomping blind through a minefield of memory mismanagement... This code is brittle, and it's not serving our needs. It needs to go.
I know you've been poking around this area some. Do you have anything to share to get me up to speed quicker, or am I going at this from scratch? Chase, It's up to you, but maybe you want to hold off on beating your head against this wall for a while --- at least until I've done some preliminary work and have a better idea of how this will play out. If we're going to rip out the system and replace it, you might be doing duplicate / unnecessary work. -mrt Jon Trulson <j...@radscan.com> writes: > On 10/24/18 4:13 PM, Chase wrote: >> So the C docs work, the international docs don't, they all fail with >> a multibyte character error. How would I go about debugging this >> (and the regex patch earlier)? Also, when I noted that there were >> translation warnings being thrown, this was only for the man pages, >> the other docs didn't throw this. >> > > Yeah, thats what I was afraid of... instant is *very* delicate and > seems to have a variety of memory and other problems. I fixed several > during the tcl conversion. > > During my recent attempt at converting to utf8, I ran into many more > problems with it crashing or corrupting memory. Some of them I could > fix by using the appropriate Tcl_Alloc/Free routines instead of > malloc/free (discovered via valgrind). > > In the end, I'm thinking we cannot convert the docs to utf8 until we > revamp the whole doc generation system (using XML and a modern > docbook). Locale stuff (catalogs and the like) work fine with utf8 in > all the locales. > > Debugging these issues will be difficult. You can run the dtdocbook > script with the -x and -v parameters. This outputs commands before > executing them, and leaves some of the artifacts around afterwards > that you can use for investigation. > > I also used valgrind a couple of times to find issues in instant, > though the crashes were often deep in the Tcl libs - so, more memory > corruption somewhere that I could often fix with appropriate uses of > Tcl_Free/Alloc, especially when dealing with strings that are fed to > the interpreter. > > Modern Tcl can handle utf8 fine, as long as they are proper utf8 > sequences. It cannot handle other 8-bit encodings, like the upper > chars (128+) present in iso8859* locales. Instant has this really > stupid encode/decode functionality to get around that problem. > > It is very poorly written software, and debugging it is hard due to > it's use in a pipeline and it's dependences on various ENV vars being > set. > > As for the man pages, yes those are all C so locale translation isn't > needed. Perhaps that warning/error could be disabled if a locale > translation DB wasn't specified as an option. If a translation db is > specified, it should definitely fail if it can't be opened (I noticed > you changed that from a fatal error to a warning). > > I am seriously toying with the idea of making document building > optional. We would just ship pre-generated files and install those by > default. > > Then some major rework could go on with the docs subsystem without > affecting mainline cde. Or, they could stay frozen in time until > someone with the urge decides they want to work on it :) > > There has got to be a cleaner and better way to do this than the way > it's being done now. > > Sorry I can't give you specifics on how to proceed with debugging. > One way is to start at the beginning (say with your regexp changes) > and do small commits/changes, one at a time until you find a change > where it starts to blow up. Then start debugging at that point. > > I don't know your debugging skill level, but the CDE doc system will > no doubt test it well :) > > -jon > > >> >> Thank you for your time, >> -Chase >> >> ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐ >> On Tuesday, October 23, 2018 9:22 PM, Jon Trulson <j...@radscan.com> wrote: >> >>> I don't have time to test this tonight, but in addition to man pages, >>> have you verified that all of the help text for all locales is also >>> generated properly? Ie: when building this stuff on linux, you should >>> definitely build all of the locales not just the C locale. >>> >>> I'm thinking that building for all locales should be the default on >>> Linux again... >>> >>> To make sure all of the locales are built, use: >>> >>> make World IMAKE_DEFINES='-DDtLocalesToBuild="de_DE.ISO8859-1 >>> es_ES.ISO8859-1 fr_FR.ISO8859-1 it_IT.ISO8859-1 en_US.UTF-8"' >>> >>> Instead of just a 'make World'. >>> >>> You need to make sure all of these build and install properly. You >>> should also make sure they actually work - login with a de_DE locale for >>> example. >>> >>> The locale DB stuff is interesting and might be a problem. It's >>> probably not used for generating manpages, which are all C locale, but >>> it's definitely needed for help text. >>> >>> -jon >>> >>> On 10/23/18 7:45 PM, Chase via cdesktopenv-devel wrote: >>> >>>> This is a big patch, I have thoroughly tested it and man pages generate >>>> perfectly. The only problem that I've encountered, is that it writes a >>>> lot of nonsense to stderr, mostly warnings that it could not find a >>>> translation database. I am leaving the (disabled) extra copy of instant >>>> in dbtoman just in case anyone wants to see if there is anything else >>>> that can be taken from it, but most of the remaining code is seemingly >>>> redundant or useless, so if there are no qualms about it, I will remove >>>> it in another patch. >>>> Thank you for your time, >>>> -Chase >>>> >>>> cdesktopenv-devel mailing list >>>> cdesktopenv-devel@lists.sourceforge.net >>>> https://lists.sourceforge.net/lists/listinfo/cdesktopenv-devel >>> >>> -- >>> >>> Jon Trulson >>> >>> "In the game of chess, you can never let your adversary see your pieces." >>> >>> - Zapp Brannigan >>> >>> cdesktopenv-devel mailing list >>> cdesktopenv-devel@lists.sourceforge.net >>> https://lists.sourceforge.net/lists/listinfo/cdesktopenv-devel >> _______________________________________________ cdesktopenv-devel mailing list cdesktopenv-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/cdesktopenv-devel
