On 10/24/18 6:36 PM, Matthew R. Trower wrote:
Jon,
After I get Solaris building again, I'm going to take a stab at
upgrading the docs system to modern docbook. In the past I've resisted
removing in-house components in favor of external dependencies, but I
recognize that there's a difference between maintaining a quality
codebase and stomping blind through a minefield of memory
mismanagement... This code is brittle, and it's not serving our needs.
It needs to go.
I know you've been poking around this area some. Do you have anything
to share to get me up to speed quicker, or am I going at this from
scratch?
Not really, it is not very well documented (well SGML and docbook sure
are), but there is a lot to learn there - I was hoping someone on this
list had way more experience with it and would be willing to work on it. :)
Essentially, we have an old SGML docbook dtd (2.1 IIRC) , and then these
"helpers" like dthelp_tag*, instant, and the like that pull pieces of
text from the docbook formatted SGML files to produce SDL (compressed)
format files.
SDL 1.2, according to the docbook manpage is actually a standard. Try
googling for SDL 1.2, to see if you can find it's specification :). It
is supposedly a real specification, but my googling didn't turn up
anything useful, though I admit not spending a lot of time on it.
As it is supposedly a standardized format, in theory we should just be
able to ship pre-generated .sdl files with CDE and they should work
across the board.
The man pages are also generated using another version of instant
(non-tcl) from the main docbook .sgml source.
I am hoping that pre-generated man pages are compatible, but that may be
another problem.
But as I know little (still) about docbook and the like, it's an uphill
climb. That's what I got hung up on doing the utf8 conversion - the
help system and especially instant.
I managed to hack around and/or fix many bugs, but then Calander help
for the es_ES locale caused a crash I no longer had time (or the desire,
to be honest) to dig into.
So, I will end up redoing the utf8 conversion, leaving the help and
dtinfo and manpages alone as much as possible.
I may still remove nsgmls in favor of onsgmls, since that still seems to
work, and would clearly be the way forward, but other than that I'm
going to leave it alone, and if possible, not even build it - just ship
pre-generated sdl, dtinfo and man pages :)
-jon
Chase,
It's up to you, but maybe you want to hold off on beating your head
against this wall for a while --- at least until I've done some
preliminary work and have a better idea of how this will play out. If
we're going to rip out the system and replace it, you might be doing
duplicate / unnecessary work.
-mrt
Jon Trulson <j...@radscan.com> writes:
On 10/24/18 4:13 PM, Chase wrote:
So the C docs work, the international docs don't, they all fail with
a multibyte character error. How would I go about debugging this
(and the regex patch earlier)? Also, when I noted that there were
translation warnings being thrown, this was only for the man pages,
the other docs didn't throw this.
Yeah, thats what I was afraid of... instant is *very* delicate and
seems to have a variety of memory and other problems. I fixed several
during the tcl conversion.
During my recent attempt at converting to utf8, I ran into many more
problems with it crashing or corrupting memory. Some of them I could
fix by using the appropriate Tcl_Alloc/Free routines instead of
malloc/free (discovered via valgrind).
In the end, I'm thinking we cannot convert the docs to utf8 until we
revamp the whole doc generation system (using XML and a modern
docbook). Locale stuff (catalogs and the like) work fine with utf8 in
all the locales.
Debugging these issues will be difficult. You can run the dtdocbook
script with the -x and -v parameters. This outputs commands before
executing them, and leaves some of the artifacts around afterwards
that you can use for investigation.
I also used valgrind a couple of times to find issues in instant,
though the crashes were often deep in the Tcl libs - so, more memory
corruption somewhere that I could often fix with appropriate uses of
Tcl_Free/Alloc, especially when dealing with strings that are fed to
the interpreter.
Modern Tcl can handle utf8 fine, as long as they are proper utf8
sequences. It cannot handle other 8-bit encodings, like the upper
chars (128+) present in iso8859* locales. Instant has this really
stupid encode/decode functionality to get around that problem.
It is very poorly written software, and debugging it is hard due to
it's use in a pipeline and it's dependences on various ENV vars being
set.
As for the man pages, yes those are all C so locale translation isn't
needed. Perhaps that warning/error could be disabled if a locale
translation DB wasn't specified as an option. If a translation db is
specified, it should definitely fail if it can't be opened (I noticed
you changed that from a fatal error to a warning).
I am seriously toying with the idea of making document building
optional. We would just ship pre-generated files and install those by
default.
Then some major rework could go on with the docs subsystem without
affecting mainline cde. Or, they could stay frozen in time until
someone with the urge decides they want to work on it :)
There has got to be a cleaner and better way to do this than the way
it's being done now.
Sorry I can't give you specifics on how to proceed with debugging.
One way is to start at the beginning (say with your regexp changes)
and do small commits/changes, one at a time until you find a change
where it starts to blow up. Then start debugging at that point.
I don't know your debugging skill level, but the CDE doc system will
no doubt test it well :)
-jon
Thank you for your time,
-Chase
‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
On Tuesday, October 23, 2018 9:22 PM, Jon Trulson <j...@radscan.com> wrote:
I don't have time to test this tonight, but in addition to man pages,
have you verified that all of the help text for all locales is also
generated properly? Ie: when building this stuff on linux, you should
definitely build all of the locales not just the C locale.
I'm thinking that building for all locales should be the default on
Linux again...
To make sure all of the locales are built, use:
make World IMAKE_DEFINES='-DDtLocalesToBuild="de_DE.ISO8859-1
es_ES.ISO8859-1 fr_FR.ISO8859-1 it_IT.ISO8859-1 en_US.UTF-8"'
Instead of just a 'make World'.
You need to make sure all of these build and install properly. You
should also make sure they actually work - login with a de_DE locale for
example.
The locale DB stuff is interesting and might be a problem. It's
probably not used for generating manpages, which are all C locale, but
it's definitely needed for help text.
-jon
On 10/23/18 7:45 PM, Chase via cdesktopenv-devel wrote:
This is a big patch, I have thoroughly tested it and man pages generate
perfectly. The only problem that I've encountered, is that it writes a
lot of nonsense to stderr, mostly warnings that it could not find a
translation database. I am leaving the (disabled) extra copy of instant
in dbtoman just in case anyone wants to see if there is anything else
that can be taken from it, but most of the remaining code is seemingly
redundant or useless, so if there are no qualms about it, I will remove
it in another patch.
Thank you for your time,
-Chase
cdesktopenv-devel mailing list
cdesktopenv-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/cdesktopenv-devel
--
Jon Trulson
"In the game of chess, you can never let your adversary see your pieces."
- Zapp Brannigan
cdesktopenv-devel mailing list
cdesktopenv-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/cdesktopenv-devel
_______________________________________________
cdesktopenv-devel mailing list
cdesktopenv-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/cdesktopenv-devel
--
Jon Trulson
"In the game of chess, you can never let your adversary see your pieces."
- Zapp Brannigan
_______________________________________________
cdesktopenv-devel mailing list
cdesktopenv-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/cdesktopenv-devel
