Using LyX for writing Internet RFCs

[Nico Williams]

I love LyX.  I want two things it doesn't have/do today:

 - a vi editing mode (I looked at the list archives, I know this has
come up before)

 - a way to export to xml2rfc's XML schema

For the latter I'm playing with TeX4ht and htlatex, followed by
applying an XSLT stylesheet.  First I export to LaTeX (plain), then I
convert to XML, then I apply the sylesheet, then invoke xml2rfc.
It's... not perfect.  In particular LyX doesn't have a rich enough
representation of authors.  In Internet-Drafts and RFCs authors have:
full names, initials, organization name, e-mail address, and mailing
address.  I suspect I'll run into similar issues with respect to
references, but there it will be easier to just use a convention.
Other issues (e.g., how to specify the name of the document, as
opposed to its title, and how to specify the abbreviation of the
title) I've solved by using XSLT parameters.

I don't know enough about TeX, but I'm guessing custom insets would be
able to represent the additional author meta-data that LyX is
currently missing.  But would that meta-data be preserved somehow when
exporting to TeX?  And would the TeX->XML conversion tools preserve
that meta-data?  If not, then I think I'll stop here and stick to
editing xml2rfc documents in their XML form in VIM.

As for a vi editing mode, I'm guessing the only way it'd be added to
LyX is if someone who wants it goes and does the dev work.  Sadly I
don't have time to do that.  Someday, when I go write a book, I may do
it, but not anytime soon :(

Nico
--

Re: Using LyX for writing Internet RFCs

[Nico Williams]

On Mon, May 7, 2012 at 11:04 AM, Pavel Sanda  wrote:
> Nico Williams wrote:
>>  - a vi editing mode (I looked at the list archives, I know this has
>> come up before)
>>
>>  - a way to export to xml2rfc's XML schema
>
> asking for the obvious - have you tried Docbook XML output or
> xhtml output via LyXHTML?

Is there an option for Docbook XML output?  I didn't see it.  So I
export to TeX then convert using TeX4ht.  Now, TeX4ht doesn't support
more than one author, which is.. lame, but even so LyX doesn't have a
way to represent all the author metadata I need.  Latexml does a much
better job of preserving multiple author information, and has a
simpler schema too, but still...

One possibility is to store author data in xml2rfc format and emit
directives from the XSLT stylesheet to include xml files named after
the authors.

Nico
--

Straghtforward XML export?

[Nico Williams]

The LaTeX->XML tools I've tried leave me... sad.  They tend to drop
some things.  For example: vertical space, which becomes a simple
newline in a paragraph's text.  It would be better to translate
vertical space into  elements -- that'd be much, much more
useful in XSLT than embedded newlines!

So I'm wondering: why couldn't LyX export to XML using a native schema
that preserves as much LyX markup as possible, indeed, if not all of
it?

How does LyX represent documents internally?  If it does it in an
objectified form then it should be fairly straightforward to walk the
document tree and emit XML, no?  Or, looking at .lyx files, maybe it
should be possible to script a simple LyX->XML conversion has
anyone tried this before?

Nico
--

Re: Straghtforward XML export?

[Nico Williams]

On Mon, May 7, 2012 at 12:07 PM, Pavel Sanda  wrote:
> Nico Williams wrote:
>> The LaTeX->XML tools I've tried leave me... sad.  They tend to drop
>> some things.  For example: vertical space, which becomes a simple
>> newline in a paragraph's text.  It would be better to translate
>> vertical space into  elements -- that'd be much, much more
>> useful in XSLT than embedded newlines!
>>
>> So I'm wondering: why couldn't LyX export to XML using a native schema
>> that preserves as much LyX markup as possible, indeed, if not all of
>> it?
>
> google says:
> http://bgu.perso.libertysurf.fr/doc/db4lyx/

I did see that link when I was researching this.  It's very out of date.

> http://www.neomantic.com/tutorials/lyx-and-docbookXML/

This I hadn't seen.  One thing to note is that the LyX I'm running (on
Ubuntu) has no option to save as or export to SGML or DocBook.  I
gather from the link you gave me that SGML and Docbook are natively
supported export formats, so I guess Ubuntu's build must be lacking
that feature.  Is that correct?

Nico
--

Re: Straghtforward XML export?

[Nico Williams]

On Mon, May 7, 2012 at 12:07 PM, Pavel Sanda  wrote:
> Nico Williams wrote:
>> How does LyX represent documents internally?  If it does it in an
>> objectified form then it should be fairly straightforward to walk the
>> document tree and emit XML, no?  Or, looking at .lyx files, maybe it
>> should be possible to script a simple LyX->XML conversion has
>> anyone tried this before?
>
> we miss someone who knows docbook/sgml/xml rather well and would like to help
> to bring lyx output more up-to-date or at least clearly state what needs to 
> be done.
> http://article.gmane.org/gmane.editors.lyx.devel/119220

Lookingat LyX's format, it seems like translating to XML using a
LyX-specific schema should be utterly straightforward.  For example,
something like this:


\lyxformat 413
\begin_document
\begin_header
\textclass article
...
\end_header

\begin_body

\begin_layout Title
Some Doc
\end_layout

\begin_layout Author
Joe Sixpack
\begin_inset VSpace defskip
\end_inset

Sixpack Corp.
\end_layout

\begin_layout Abstract
Foo bar baz blah blah.
\end_layout

\begin_layout Abstract
Two paragrap abstract, eh?
\end_layout

...

should translate into:


Some Doc
Joe SixpackSixpack Corp.
Foo bar baz blah blah.
Two paragrap abstract, eh?
...


Translating insets and layouts into XML elements and attributes seems
relatively straightforward.  Translating directives seems
straightforward also.  Now, note that the two paragraph abstract would
be translated into two  elements, but an XSLT stylesheet
could easily translate that into:

..

A straightforward LyX->XML translation seems like the best approach to
LyX->XML translation because translation to any other schemas can then
be done via XSLT.

Nico
--

Re: Straghtforward XML export?

[Nico Williams]

On Mon, May 7, 2012 at 12:41 PM, Pavel Sanda  wrote:
> Nico Williams wrote:
>> This I hadn't seen.  One thing to note is that the LyX I'm running (on
>> Ubuntu) has no option to save as or export to SGML or DocBook.  I
>> gather from the link you gave me that SGML and Docbook are natively
>> supported export formats, so I guess Ubuntu's build must be lacking
>> that feature.  Is that correct?
>
> export items depend on software you have installed, in case of docbook
> sgml-tools are needed. not using it i can't say much more, but it seems
> that your question are answered in the older link.

Ah, that works.  Thanks!  I'll take a look and see if the native
DocBook export works for me.

Re: Straghtforward XML export?

[Nico Williams]

No, i got that. I don't actually care for docbook. I want a straightforward
translation to XML that preserves all data and metadata. If I need a
specific schema I can always use XSLT to get output in that form.

Nico
--

Re: Straghtforward XML export?

[Nico Williams]

On Mon, May 7, 2012 at 12:56 PM, Nico Williams  wrote:
> Ah, that works.  Thanks!  I'll take a look and see if the native
> DocBook export works for me.

Nope, it still doesn't allow more than one author in docbook, though
it does merge all the authors listed in the LyX document source.

Re: Straghtforward XML export?

[Nico Williams]

Is there canonical documentation of the LyX file format?  I can't find
it...  I did find this: http://wiki.lyx.org/Devel/LyXFileFormat , but
that's just a changelog.  There's nothing else obvious in
http://wiki.lyx.org/Devel/ ...  The development/FORMAT file in the
source tree is also a changelog.

Nico
--

Re: Straghtforward XML export?

[Nico Williams]

On Tue, May 8, 2012 at 12:40 AM, Guenter Milde  wrote:
> So how about XHTML as starting point for your XSLT transformations?
>
> Otherwise, you could use the native XHTML formatter as a model for adding
> "native XML" output.
>
> Another starting point would be the external "elyxer" tool: a Python
> package that takes a LyX file and converts it to XHTML.
> http://elyxer.nongnu.org/

Ah, those are good ideas.  I'll take a look.  Thanks!

Re: Straghtforward XML export?

[Nico Williams]

On Tue, May 8, 2012 at 12:40 AM, Guenter Milde  wrote:
> So how about XHTML as starting point for your XSLT transformations?

LyXHTML looks very promising.  It certainly preserves everything I
have in my [admittedly small] test file.  If it preserves custom inset
names then I could probably use custom insets to provide the
additional metadata I need (I still haven't quite figured out how to
create custom insets, but give me time).  XSLT can do the rest.

> Otherwise, you could use the native XHTML formatter as a model for adding
> "native XML" output.

Indeed, I think that would be a good last resort.

Ideally there'd be a terribly straightforward LyXML, but LyXHTML looks
manageable.

> Another starting point would be the external "elyxer" tool: a Python
> package that takes a LyX file and converts it to XHTML.
> http://elyxer.nongnu.org/

That looks pretty good too.  That's a lot of realistic options.  Thanks again,

Nico
--

Re: Straghtforward XML export?

[Nico Williams]

On Tue, May 8, 2012 at 10:58 PM, Richard Heck  wrote:
> On 05/08/2012 07:30 PM, Nico Williams wrote:
>> LyXHTML looks very promising.  It certainly preserves everything I
>> have in my [admittedly small] test file.  If it preserves custom inset
>> names then I could probably use custom insets to provide the
>> additional metadata I need (I still haven't quite figured out how to
>> create custom insets, but give me time).  XSLT can do the rest.
>>
> It will do with custom insets whatever you ask it to do. If I remember
> correctly, it defaults to something like:
> 
> or an equivalent span, depending upon whether its a charstyle or a
> flex inset.

Excellent.  I've got an XSLT stylesheet in the works that does what I want.

I don't know how to create a custom inset that does.. nothing much
except have a custom inset name.  Specifically I need variants of the
Author inset to represent the metadata I need (author organization,
e-mail address, and postal address).  With that I'd be set.

> In principle, you can also tell the LyXHTML output to use some other
> tag than div or span. This is all customized in the layout files, as is
> explained in the bits on XHTML in the customization manual. So I'm
> guessing that you could get quite a long way towards XML simply in
> that sort of way.

The divs are fine.  I can address them just fine with XPath, so I'm
quite happy.  If the LyXHTML schema changes radically I'll just have
to re-write the XSLT stylesheet I'm writing now, but as long as no
metadata is lost I'll be fine.

Eventually I'll probably want to develop a layout and class for
actually dealing with RFCs directly in LyX.  The typesetting rules for
RFCs are... trivial in comparison to most other layouts.  But I
confess knowing nothing about LaTeX, so it will be sometime before I
get there.  For now I'm just happy -ecstatic even- to just consume
LyXHTML with XSLT.

[Actually, I'm noticing one problem with LyXHTML: it doesn't preserve
vertical spacing in any way, not even as horizontal spacing!  I'm
talking about Insert->Formatting->Vertical Space.  I suspect that
there are other such things that aren't preserved.  For now I'll live.
 Vertical space is useful for multi-paragraph list items, which are
very common in RFCs and Internet-Drafts.  If need be I suspect I can
write a patch and submit it.]

Thanks for your help.  Sorry to need so much handholding, I'm out of
my element here,

Nico
--

Re: Straghtforward XML export?

[Nico Williams]

> [Actually, I'm noticing one problem with LyXHTML: it doesn't preserve
> vertical spacing in any way, not even as horizontal spacing!  I'm
> talking about Insert->Formatting->Vertical Space.  I suspect that
> there are other such things that aren't preserved.  For now I'll live.
>  Vertical space is useful for multi-paragraph list items, which are
> very common in RFCs and Internet-Drafts.  If need be I suspect I can
> write a patch and submit it.]

Found a solution to that: a nest list with no bulleting/numbering is
rendered as a single  with s for the nested list elements,
which works out perfectly.  No doubt the vspace loss will come up
elsewhere, but for now it's fine.

Re: Straghtforward XML export?

[Nico Williams]

On Thu, May 10, 2012 at 10:02 AM, Richard Heck  wrote:
> On 05/09/2012 02:29 AM, Nico Williams wrote:
>>> [Actually, I'm noticing one problem with LyXHTML: it doesn't preserve
>>> vertical spacing in any way, not even as horizontal spacing!  I'm
>>> talking about Insert->Formatting->Vertical Space.  I suspect that
>>> there are other such things that aren't preserved.  For now I'll live.
>>>  Vertical space is useful for multi-paragraph list items, which are
>>> very common in RFCs and Internet-Drafts.  If need be I suspect I can
>>> write a patch and submit it.]
>
> I basically didn't know what to do with the vspace stuff, the issue being
> that
> HTML in a way just doesn't have that kind of concept. But if you have an
> idea,
> please let me know, and I'll be happy to put it in.

Ah, good point.  Hmmm, could you use ?  Or
maybe an XML entity that gets defined into a newline but with a
processor could replace with an element?

Nico
--

Re: Straghtforward XML export?

[Nico Williams]

On Thu, May 10, 2012 at 3:31 PM, Richard Heck  wrote:
> Actually, it looks like this got fixed a while ago. In a simple text
> document I get:

I'm running LyX 2.0.0.  The vspace I had was in an author inset, FWIW.
 The output you show is certainly fine.

> If you want to post a simple example file that does the wrong thing, please
> do.

Here's a LyX snippet:

\begin_layout Standard
A paragraph.
\begin_inset VSpace defskip
\end_inset

 Text after a vspace.
\end_layout

FYI, right now I'm struggling with how to transform h2, h3, h4
elements into nested section elements; this seems very difficult to do
in XSLT 1.0, but I'm still exploring ideas, including XSLT 2.0.  (This
actually seems like a common problem, some recipes for which I do find
online and in books, but no solutions general enough.)  Of course, the
way LyX represents sections/subsections/subsubsections internally is
exactly the same as in its XHTML output, and it'd be asking a lot to
ask for LyX to wrap section contents in a div -- if I can do this with
XSLT you might be able to incorporate that solution as an option in
LyX, say.

[I'm guessing that LyX's XHTML output is not stable, but I can cope,
provided I find a way to transform those h elements into nested
sections.]

Nico
--

Re: Straghtforward XML export?

[Nico Williams]

On Thu, May 10, 2012 at 8:27 PM, Richard Heck  wrote:
> On 05/10/2012 04:52 PM, Nico Williams wrote:
>> On Thu, May 10, 2012 at 3:31 PM, Richard Heck  wrote:
>> Here's a LyX snippet:
>
> OK, I see the problem. The vertical space gets moved, for reasons
> that probably aren't very interesting. Can you file a bug about this on
> trac? I can fix it, but it will take a little thought about how best to do
> it.

Filed http://www.lyx.org/trac/ticket/8154

Thanks.

>> FYI, right now I'm struggling with how to transform h2, h3, h4
>> elements into nested section elements; [...]
>>
> It could be done in LyX, but I guess I'd suggest pre-processing the
> whole thing with some kind of script. It shouldn't be too hard to do.
> Find h1, write a start tag; when you see another h1, write the end tag
> for the first one; etc.

I've figured out how to handle this with XSLT 2.0.  Here's a snippet:





















The key is the << operator (here encoded, so <<).  The right
operand had to be stored in a variable because there's no other way
(that I could find!) to refer to the node I wanted to there.

That took a lot of effort to work out.  Much more than I'd wanted to.
And it requires XSLT 2.0.  But it works and it's not terribly
inelegant -- more elegant than any robust script to do the same, most
likely.

>> [I'm guessing that LyX's XHTML output is not stable, but I can cope,
>> provided I find a way to transform those h elements into nested
>> sections.]
>>
> It's generally stable, but of course under development. Mostly, I want
> it to be as modular and customizable as possible, in which case we can
> all make it do what we want.

Great.  Thanks so much for your work and your help!

Nico
--

Re: Straghtforward XML export?

[Nico Williams]

On Thu, May 10, 2012 at 10:09 AM, Richard Heck  wrote:
>> I don't know how to create a custom inset that does.. [...]
>
> Try putting this into Local Layout, under Document>Settings:

Excellent, that worked great.

> I guess if you want these as metadata, you should also add:
>    InTitle 1
> to each of them.

I want them as metadata in my XSLT stylesheet's output, so it sufficed
without the InTitle bit.

Thanks!

Nico
--

Re: Straghtforward XML export?

[Nico Williams]

Well, thanks lots for your help.   I have something that's very close.
 Close enough that I can now author I-Ds in LyX.  I've found one more
bug in the LyX XHTML output, and I filed a bug for it (bibitem anchor
generation is not working properly), and I can work around it.
Cheers!

Nico
--

Re: HTML and RTF: Very basic import and export strategy

[Nico Williams]

Richard,

Does LyX XHTML output preserve enough LyX metadata to be suitable as
an import format?

Nico
--

Re: HTML and RTF: Very basic import and export strategy

[Nico Williams]

On Mon, May 14, 2012 at 9:30 AM, Richard Heck  wrote:
> On 05/14/2012 10:26 AM, Nico Williams wrote:
>>
>> Richard,
>>
>> Does LyX XHTML output preserve enough LyX metadata to be suitable as
>> an import format?
>
> You mean back into LyX?

Yes.  With XML formats becoming ubiquitous that seems like it'd be useful.

Nico
--

Re: Straghtforward XML export?

[Nico Williams]

FWIW, I've put up a github repo with my LyX->xml2rfc tool, though it's
still a work in progress: https://github.com/nicowilliams/lyx2rfc

BTW, I can't get "lyx -e lyxhtml ..." to work.  lyx -e xhtml does
work, but then there are some differences from the LyXHTML option in
the File->Export menu.  The differences appear to be confined to the
magicparlabel numbering, so I don't think I care; I mention this only
because it seems odd.

Thanks for all the help!

Nico
--

Re: LyX SEGSEGV crash just after GUI window appears

[Nico Williams]

strace lyx running as root and not, then look to see what system calls
are failing in the latter case leading up to the segfault.  That might
give you a clue as to where else to look.

Nico
--

Re: program to use Lyx as a code-evaluating notebook

[Nico Williams]

On Mon, May 28, 2012 at 8:34 AM, Richard Heck  wrote:
> If you're at all serious about this program, then I'd suggest
> you create a git repo for it somewhere like gitorious, and
> then people can have easy access, as well as follow the
> changes you're making. List it as pre-alpha or whatever,
> and people know what they're getting.

+1

Git has become my preferred downloader for these sorts of things.
There's a number of git service providers (github, gitorious, ...),
and you can always host your own.

Nico
--

Re: program to use Lyx as a code-evaluating notebook

[Nico Williams]

On Wed, May 30, 2012 at 11:24 AM, Allen Barker  wrote:
> I've set up a repo on github for the program.
>
> The code:
> https://github.com/abarker/lyxNotebook
>
> For a quick overview, see the first section of the documentation:
> https://github.com/abarker/lyxNotebook/blob/master/lyxNotebookDocs.pdf

I looked at the README and I get it.  This is very much like the
weaver tool, that lets you compile statistics using R right in the LyX
document.  But yours is more generic.  The idea is very cool.  I
second the sentiment that this should be added to the tools wiki page.

Nico

PS: I have no use for this right now, but I'll keep it in mind.  One
use that comes to mind is using this approach to automatically test
code samples in a book, but that would need a way to request that code
be tested but output not included in the document.

Re: Working in LyX from the get-go---more or less

[Nico Williams]

Using XHTMP export has the benefit that you can then use XSLT.  I've
used this successfully to export LyX format to an unrelated XML
schema.

Nico
--

Re: Mostly successful experiment: LyX -> LaTeX -> Word using pandoc

[Nico Williams]

On Wed, Aug 8, 2012 at 1:13 AM, Jerry  wrote:
> I just tried a quick experiment with pandoc, saving a 3.5 page LyX document 
> (article class) of mostly lorem ipsum as LaTeX and then using pandoc to 
> convert that to .docx, Word's XML format. I then viewed the .docx in 
> Microsoft Word 2011 for OS X (Macintosh).

Sounds very cool, but I would think that anytime you want to convert
to some XML schema it's probably easier to export to LyXHTML and then
write and apply an XSLT to convert to the desired schema.  This has
worked for me before.  It's very possible that pandoc uses XSLT under
the covers, in which case you may be able to hack on one of those
stylesheets till you get it to do what you want.

It would be very cool if LyX could add export formats as the
composition of LyXHTML and XSLs!

Nico
--

LyX document diff/merge tools for cooperative editing?

[Nico Williams]

Hi,

I'd like to use LyX + git (or some such VCS) to cooperatively edit
documents.  This requires a diff/merge tool that is LyX-aware and can
result in LyX documents that contain diffs that users can merge
manually.  Are there any such tools?  Thanks,

Nico
--

Re: LyX document diff/merge tools for cooperative editing?

[Nico Williams]

What would the exact command-line invocation be?

Nico
--

Re: LyX document diff/merge tools for cooperative editing?

[Nico Williams]

Ideally the syntax should be something like: lyx --merge --out=doc3
doc1 doc2, and the result should be a merged document with change
tracking so that the user can do all semantic merging.  In this mode
LyX should not open a window nor require user interaction, as the idea
is to use this as a merge strategy for git and the like.

This would be extremely useful for distributed authoring/editing.  I'd
go farther and ask for a merge tool that can work like traditional
version control merge tools, only at the LyX layer, but this seems
likely to be very difficult, as opposed to existing change tracking
feature.

If you wish I can open the feature request.

Thanks,

Nico
--

Any way to have custom inset labels always displayed?

[Nico Williams]

I'm using custom insets to represent various metadata.  The problem I
run into is that there's no way in the LyX display to see what each
inset is.  If I click inside such an inset the layout is shown as
plain -- nothing in the UI shows the inset type.

In some cases I'm resorting to using param=value in the inset contents
to make the user experience better, but this is hardly a great
solution.

Now, I notice that there's a LabelString parameter for custom insets,
but I don't see how to have that label string appear in the UI.  Oh,
well, I notice that there's right click menu item for showing the
label -- I just want the labels for my custom insets displayed at all
times.

Nico
--

Custom metadata

[Nico Williams]

It'd be nice to have a custom inset type just for metadata, that is,
one that doesn't get rendered.

It'd also be nice to be able to define a filter that must be applied
before rendering.  Such a filter might convert metadata into LyX
directives, thus pre-rendering said metadata.

I'll file a request if there's no way to do these things.  Thanks,

Nico
--

Re: Any way to have custom inset labels always displayed?

[Nico Williams]

On Tue, Dec 4, 2012 at 12:56 PM, Jean-Marc Lasgouttes
 wrote:
> Le 04/12/2012 19:53, Jürgen Spitzmüller a écrit :
>> You are talking about "conglomerate"-style flex insets, right?
>> Currently, they are always inserted "collapsed" and you need to
>> manually uncollapse them. We could introduce a layout param to
>> customize this behaviour (we already had a similar discussion wrt to
>> "classic" custom insets).
>
> Or we could always insert them opened.

This, yes.

Re: Any way to have custom inset labels always displayed?

[Nico Williams]

Never mind.  I really had to define LabelStrings for all my insets,
and then View->Open All Insets was enough.  And apparently they are
inserted open anyways.  I've closed the ticket I opened for this.

Re: Custom metadata

[Nico Williams]

On Wed, Dec 5, 2012 at 1:13 AM, Liviu Andronic  wrote:
> On Tue, Dec 4, 2012 at 8:21 PM, Nico Williams  wrote:
>> It'd also be nice to be able to define a filter that must be applied
>> before rendering.  Such a filter might convert metadata into LyX
>> directives, thus pre-rendering said metadata.
>>
> Did you look into the possibility of adding a new file format (say,
> .xml2rfc) and a converter from .lyx to .xml2rfc and a 2nd one from
> .xml2rfc to .lyx? I think current LyX infrastructure allows you to do
> what you want.

I already convert from LyX to xml2rfc (see my github repo, linked
earlier), using export to XHTML + an XSL.  What I'm after is the
ability to render RFCs in PDF and HTML directly from Lyx, instead of
having to take a side-trip through xml2rfc.  There may be many ways to
do this, but I don't know where to start.  And even my lyx2rfc tool,
if that's the only way to do this, I'd like to be able to invoke from
the File->Export menu item.  And I'd like to be able to record in the
.lyx file that this really can only be exported/printed that way, at
least so lyx2rfc comes up first in that menu.

Nico
--

Re: about the .lyx formt

[Nico Williams]

On Wed, Dec 5, 2012 at 8:15 AM, Alex Vergara Gil  wrote:
> The question is why to have a .lyx format in plain text? Isn't xml a better
> one? I know plain text is easy to edit but we really doesn't work with it
> since LyX do this for us. The advantages would be obvious:
> Insertion/Deletion of metadata, a better way of handling everything, etc.
> This is my point of view, so I might be wrong but I doesn't feel like that.

There's a wiki where this is discussed.  I agree, XML is better, not
because XML is great, but because of tools like XSLT.  If there was a
loss-free, bi-directional conversion between LyX and some XML schema
(XHTML, say), then that'd be good enough for now.  There is an export
to XHTML function, but I don't think there's an import from XHTML

Nico
--

Re: Custom metadata

[Nico Williams]

On Wed, Dec 5, 2012 at 2:20 PM, Liviu Andronic  wrote:
> On Wed, Dec 5, 2012 at 9:08 PM, Nico Williams  wrote:
>> do this, but I don't know where to start.  And even my lyx2rfc tool,
>> if that's the only way to do this, I'd like to be able to invoke from
>> the File->Export menu item.  And I'd like to be able to record in the
>>
> This can be done. In File Format check 'document format' and 'show in
> export menu'.

I don't see a File->Format menu.  What's the correct path to this?

Re: LyX document diff/merge tools for cooperative editing?

[Nico Williams]

On Wed, Dec 5, 2012 at 12:44 PM, Pavel Sanda  wrote:
> Jean-Marc Lasgouttes wrote:
>> Le 04/12/2012 18:10, Nico Williams a écrit :
>>> What would the exact command-line invocation be?
>>
>> Unfortunately, it seems that there is no lfun to do that. The only one is
>> vc-compare, which compares documents already in version control.
>
> Not part of the official API (so no guarantee for future) but at this
> moment "dialog-show compare run file1 file2" LFUN should work.
>
> Apart from the problems how to cram it into commandline, the bug #6889
> is in my experience quite limiting for real usage of LyX comparison tool.

I see.  Also, I will really need a 3-way diff/merge...  Thanks.

Re: Custom metadata

[Nico Williams]

Ah, I see it, Tools->Preferences->File Handling->File Formats.

Re: about the .lyx formt

[Nico Williams]

On Wed, Dec 5, 2012 at 3:37 PM, Alex Vergara Gil  wrote:
> Can you point me the address to the wiki please.
> I agree that having a bidirectional conversion is nice at first glance (or
> first step approximation lets say) but what I really expect in the future is
> to have the .lyx format natively use the xml scheme. So then there would be
> no need for an importer-exporter tool (except for compatibility with
> previous versions) and with all the benefits of xml then LyX will overwhelm
> any other word processing software.

http://wiki.lyx.org/Devel/XML

is all I can find now.

Re: Custom metadata

[Nico Williams]

On Wed, Dec 5, 2012 at 3:20 PM, Nico Williams  wrote:
> Ah, I see it, Tools->Preferences->File Handling->File Formats.

OK, this is getting there.  How can I deliver these things with a package?

Re: Custom metadata

[Nico Williams]

OK, thanks.

Re: about the .lyx formt

[Nico Williams]

On Thu, Dec 6, 2012 at 4:29 AM, Oleg Parashchenko  wrote:
> Some time ago I experimented with LyX<->XML roundtrip. In my idea, XML
> was not one-to-one mapping of .lyx structure, but an XML format
>
> * which convenient for processing and generation, and
> * with enough hints to restore one-to-one correspondence.
>
> Unfortunately, I had to switch to other projects, and left the work
> abandoned. Only a part of functionality was implemented:
>
> * http://wiki.lyx.org/Devel/LyXFileFormatReverse
> * https://github.com/olpa/tex/tree/master/lyxml
>
> Use "lyxml.py" for testing. In any case, "lyxparser.py" is a good
> callback-style Python parser of .lyx files.

Interesting.  I may give it a try, but I was hoping that XML->LyX
might be a matter of applying an XSLT that outputs text.  Also, the
schema matters, and since LyX has LyXHTML (XHTML), and since I'm using
that, I was hoping I could find an importer from XHTML in particular.
I'd still have to write an XSL for translating from the schema I use,
so it's not like I was looking for a turn-key solution :) but still,
I'm glad someone's done something in this space!  Thanks,

Nico
--

Re: LyX document diff/merge tools for cooperative editing?

[Nico Williams]

On Fri, Dec 7, 2012 at 6:02 AM, Rainer M Krug  wrote:
> On 07/12/12 12:45, Gregory Jefferis wrote:
>> Has anyone tried using latexdiff for change comparison in not merging? 
>> Presumably a small script to export both versions to latex, invoke latexdiff 
>> and then generate the marked-up PDF (which is what you get with latexdiff) 
>> would work. For latexdiff there are already scripts to handle git revisions 
>> which should allow one to diff a pair of arbitrary revisions in a git 
>> repository.
>
> This should work, but it would be really nice if these changes were on LyX 
> level and that one could
> save a file LyX file containing the differences as tracked changes.

Right, the goal is to merge LyX documents.  Unless converstion to/from
LaTeX is loss-less this is not a solution.

Nico
--

No bibliography environment in DocBook document class

[Nico Williams]

Why is there no bibliography environment in the DocBook document
classes?  There must be a reason, of course, but then, bibtex is still
supported, so that's strange...

Nico
--

Re: No bibliography environment in DocBook document class

[Nico Williams]

On Mon, Dec 10, 2012 at 5:22 PM, José Matos  wrote:
> On 12/10/2012 10:46 PM, Nico Williams wrote:
>> Why is there no bibliography environment in the DocBook document
>> classes?  There must be a reason, of course, but then, bibtex is still
>> supported, so that's strange...
>
> The reason why it was never implemented is/was lake of time, no more no
> less. :-)

OK, thanks :) that's the answer I was hoping for.  Not that I have
time to contribute code here, but just for my edification: where would
I look in the source code?  In particular I'm using the docbook class
but exporting to LyXHTML.  I'm not sure why I'm doing that -- only
that I've found that to work best for my purposes (namely: that more
metadata gets preserved in the resulting XHTML).

Thanks!

Re: LyX document diff/merge tools for cooperative editing?

[Nico Williams]

On Tue, Dec 11, 2012 at 10:26 AM, Pavel Sanda  wrote:
> We currently use integrated SVN support of LyX for this.
> The merge conflicts are avoided by locking the whole document or childern 
> (parts of the text).

It is precisely because locking doesn't scale that we have branching
and merging.  Locking simply does not scale.  This is true even of
documents (as opposed to source code).

> While we are at possible LyX support of git, can you explicitly name the 
> commands used for your workflow?

Like any team workflow:

 - each team member must be able to work on their changes independent
of the rest of the team, even when the changes overlap

This requires support for merging.

I've been thinking about this a fair bit and I think there's a number
of possible strategies to computing a 3-way diff suitable for merging
with manual conflict resolution.  Here's my stab at low-disruption (to
LyX source, that is) strategy:

1. export each of the two or three LyX document versions as XML, using
a schema that is as direct a representation of LyX document structure
as possible
2. apply an XSL to change the schema to a document structure that is
deeper (nested sections, ...)
3. apply an XML diff algorithm that looks for additions/deletions of
nodes, node *moves*, and node changes
4. apply an automatic merge algorithm to the base version using the
diffs from (3)
5. apply an XSL to flatten the document structure
6. convert the result back to .lyx format
7. present the result to the user for manual conflict resolution

Of these the only difficult (magical) step is (4), and that's probably
only because I've not researched it well enough.

> I'm also interested for criteria for commit/push/merge/pull (e.g. do you push 
> immediatelly after commit?)

I have several different workflows.  The most common one is to create
a branch as a copy of master, do a bunch of work there, then pull and
update master, rebase my branch, repeat until done, then send a pull
request.  Another workflow is to maintain a branch that is a merger of
several feature branches, which I use until all those feature branches
are merged upstream.

> To me, git concepts are unnecessarily complicated and tools like GNU RCS or 
> SVN are better suited for document tracking.

I've used a large number of VCSes (Teamware, CVS, SVN, Clearcase,
PRCS, fossil, git, and others), and I find none as powerful or awesome
as git.  I understand why many find git hard to understand, but I
don't think this is as much git's fault as it is that distributed,
lock-less version control is inherently hard, but also the only game
in town (see above comment about locking not scaling).

Nico
--

Re: LyX document diff/merge tools for cooperative editing?

[Nico Williams]

On Tue, Dec 11, 2012 at 12:25 PM, Pavel Sanda  wrote:
> Nico Williams wrote:
>> It is precisely because locking doesn't scale that we have branching
>> and merging.  Locking simply does not scale.  This is true even of
>> documents (as opposed to source code).
>
> Conditions I had in mind were:
> a)
> - small team working on e.g. scientific paper

I believe merging is easier than locking, even for small teams.  The
alternative is dealing with someone locking a file then going on
vacation -- you can break the lock, but either they lose their work
or... they have to merge.

Merging is *not* hard provided there's a way to display the conflicts.
 Displaying conflicts usefully can be really hard for some data, such
as LyX documents, but only due to the lack of suitable tools.

> - avoid endless email exchanges of manuscript (locking exists by definition)

This is true whether you use git or SVN.

> - only subset of people are "IT-aware" while the others are capable at
>   most of "push the red button" to commit the change and unlock the
>   given chapter.
> b)

Again, merging is not hard.

> In such world even knowing what "merging" or "branching" means qualifies
> you as someone who really does not need LyX to manipulate version tracking
> and who will stay happily with specialized software or command line :)

You don't need to know what branching is.  You do need to know what
merging is, but it's not hard.

Re: LyX document diff/merge tools for cooperative editing?

[Nico Williams]

On Tuesday, December 11, 2012, Pavel Sanda wrote:

> Nico Williams wrote:
> > I believe merging is easier than locking, even for small teams.
>
> That's because you probably work with people with computer science
> background.
> The moment you step out of these waters the most you can expect is "push
> red
> button, push green button" and I know what I'm talking about :)
>

Anyone who would use track changes can merge.  In any case, a merge
capability would be huge.

Re: LyX document diff/merge tools for cooperative editing?

[Nico Williams]

On Wed, Dec 12, 2012 at 7:49 AM, Pavel Sanda  wrote:
> Gregory Jefferis wrote:

> One solution would be to detect merge conflicts and call our diff algorithm 
> for the different versions.
> This would be beautiful but an order of magnitude harder to implement (surely 
> not me:)

That's what I want.  I think it's quite doable if we have a consistent
XML representation of LyX that can be converted in both directions.
That's because there are nice XML diff algorithms.  LyX already
supports conversions to XHTML that are reasonably faithful (more on
that some other time), but there's no XML->LyX converter.

I can't contribute much C++ code to LyX, but I could contribute XSLs
to do XML->.lyx conversion, if you point me at documentation on how
the .lyx format and the LyXHTML and XHTML export formats (specifically
what conventions are used to represent LyX elements in XHTML).

>> if you wanted to have bare bones support for git in lyx you would need
>> * commit (saving your changes)
>> * push (share your work)
>> * pull (fetch changes and merge with your work)
>
> I know what I need, but the question was about your workflow.
> In particular, does it make sense in your case to automatically push after 
> commit?

I don't want LyX to be in the business of being a git GUI, but if it
were going to be so we'd need either to always do git commit -a
(ouch), or git add the files LyX knows were touched (risky) then
commit those, or have a git add UI then git commit.  That's either
unsatisfying or a lot of work, so I'd rather LyX stay out of the git
UI business.

What I want is that if git (or whatever) has merged a .lyx file *with*
conflicts, then LyX should be able to display those such that a user
could fix them by normal editing operations in LyX.

>> After testing out the existing svn implementation I think this is actually 
>> simpler to automatically merge with git than to lock files with svn.
>
> Maybe more powerful, but simpler? I have hard time to understand how 
> resolving merge conflicts is _simpler_ than automatical locking part of the 
> document you work on.
> But Ok, we don't agree here, let is sleep.

Locking is simpler until you realize that it causes serious problems
(already discussed).

I don't want to stop you from locking.  I want the option to merge.

Nico
--

Re: Using LyX to edit & organise bibliographies

[Nico Williams]

On Thu, Dec 13, 2012 at 5:58 AM, Eric Weir  wrote:
> On Apr 20, 2012, at 11:23 AM, stefano franchi wrote:
>> Moreover, it is not even clear that BibTeX will remain the only
>> database format for TeX users. Biber, a much more flexible and heavily
>> developed BibTeX replacement, is starting to integrate other formats
>> (such as EndNote). One more reason, in my opinion, to keep LyX away
>> from these complexities.
>
> Late coming to this topic, but I've been wondering about exploiting Zotero 
> for reference, citation, and bibliography management. I'm familiar with some 
> work that's been done on this 
>  but 
> perhaps there have been other developments elsewhere.

While we're on the subject of bibliography...

 - Please see bibutils, a tool that uses XML as an intermediate format
so it can convert between a any two of a large number of biblio
formats.

 - It'd be nice if LyX supported remote (HTTP) bib repos.  This makes
picking references harder, but it's worthwhile.

 - It'd be nice if LyX supported using bibutils so as to consume any
of the formats supported by bibutils.

Nico
--

Would a native LyX XML schema be accepted?

[Nico Williams]

I've written a Python script for converting .lyx files to XML.
There's no DTD or schema, but it's a straightforward mapping of .lyx
format to XML.  Things like \being_layout ... \end_layout become
... and so on.  This is much more faithful a
mapping to XML than LyXHTML, or any of the other XML export formats,
therefore it's much more amenable to transformations via XSLT.  I
think it should be easy to produce an XSL for mapping back to .lyx,
which should also help produce import from various XML schemas
functionality.

Would this be accepted?  Mind if I use a BSD license?

Nico
--

Rendering metadata

[Nico Williams]

I've been using custom insets to represent metadata that LyX doesn't
natively handle.  I'd like this to get rendered when printing or
exporting to, say, PDF.  The rendering would first take the form of lyx
code, so that LyX needn't have too much change.  How would I go about
making this happen?  Could I define a copier to do this?  How might one do
this more generally?

Nico
--

Re: Would a native LyX XML schema be accepted?

[Nico Williams]

On Mon, Dec 17, 2012 at 12:21 PM, Richard Heck  wrote:
> On 12/17/2012 11:45 AM, Alex Vergara Gil wrote:
>>> There still are long-term plans to convert the native LyX format to XML.
>>> But it turns out to be more work than one might expect.
>>>
>>> rh
>>
>> Really nice to hear about this, is there any schedule???
>>
> No, we've been discussing it for years, but no-one has ever been
> sufficiently motivated actually to do it. As I said, it's harder than it
> sounds.

I've filed some bugs last night about things that make it hard... :)

Meanwhile, the script I'm working on will effectively get us there,
though in a hackish way: externally, rather than by changing the C++
code.  But I think that's OK.  We can make a lot of progress by first
making sure there's a faithful XML representation of .lyx, as that
will allow all sorts of progress in terms of import/export
conversions, and having such an XML format to play with may help the
team to think more about how to structure the C++ code.

Nico
--

Re: Would a native LyX XML schema be accepted?

[Nico Williams]

On Mon, Dec 17, 2012 at 2:02 PM, stefano franchi
 wrote:
> Sounds like a great idea. One question: would having an external but
> faithful XML Lyx file allow XLS-translations to XML-based word-processing
> formats?
> That would solve once and for all (hopefully) the problem of the Lyx-->Word
> conversion that still plagues those of us working in the Humanities (At
> least, it still affects me).

Yes, that's precisely why I started this.  My particular goal is to
map to he xml2rfc schema.  I already have an XSL for doing that, but I
depend on LyXHTML output and there's much about LyXHTML that leaves me
unsatisfied -- a more faithful schema would make my task much simpler.
 Also, with a decent XML->.lyx XSL I could also map back from xml2rfc
schema, which is also a goal of mine.

I actually want to produce and contribute the following:

 - lyx2xml -- a script for converting .lyx to XML
 - xml2lyx -- an XSL for converting XML back to .lyx

 - an XSL for turning LyX XML into a more structure schema, from this:



 - an XSL to flatten the same and return to the native LyX XML schema

Nico
--

Re: Would a native LyX XML schema be accepted?

[Nico Williams]

On Mon, Dec 17, 2012 at 2:25 PM, Richard Heck  wrote:
> I'm not so sure. If someone were to write a converter that mapped LyX's XML
> onto ODT's, or whatever, then yes. But that would still need doing.

Yes, of course.  However, I think it should be easier to write such
XSLs to map from a straightforward LyX XML schema than from LyXHTML
output.  The latter depends too much on the document class and has
issues, while the former should preserve everything about the original
.lyx as much as possible.

Nico
--

Re: Would a native LyX XML schema be accepted?

[Nico Williams]

On Mon, Dec 17, 2012 at 3:15 PM, Alex Vergara Gil  wrote:
>
>> On Mon, Dec 17, 2012 at 2:25 PM, Richard Heck  wrote:
>>>
>>> I'm not so sure. If someone were to write a converter that mapped LyX's
>>> XML
>>> onto ODT's, or whatever, then yes. But that would still need doing.
>>
>>
>> Yes, of course.  However, I think it should be easier to write such
>> XSLs to map from a straightforward LyX XML schema than from LyXHTML
>> output.  The latter depends too much on the document class and has
>> issues, while the former should preserve everything about the original
>> .lyx as much as possible.
>>
>> Nico
>> --
>>
> And it would be easier for the final user the fact that all converters can
> operate automatically inside the LyX, instead of calling them, so you would
> see that copying a MathML equation in LOWriter and then pasting in LyX would
> result in a LyX equation where you insert them instead of having to convert
> it to latex using whatever script you have and the copy paste inside an
> equation box (is there an easier way right now? I think my way is too much
> omplicated), the same for figures (you don't need to save the image and then
> load it from LyX, just paste it where you want it) and tables (I think this
> is a lot more complicated example, but there is XLST converter).

Conversion at the File->Import... and File->Export... level would be
just a matter of writing the necessary XSLs.

Conversion at the cut-n-paste level is not something I can speak to
with confidence at this time.  I'll note only that if the content
being pasted is known to be XML in some schema then LyX could invoke
an XSL to convert it to .lyx.

Nico
--

Re: Would a native LyX XML schema be accepted?

[Nico Williams]

On Tue, Dec 18, 2012 at 10:43 PM, Steve Litt  wrote:
> On Tue, 18 Dec 2012 10:22:17 +0100
> Rainer M Krug  wrote:
>> This all sounds very exciting and extremely useful for import /
>> export / collaboration, but there is one aspect which I would be
>> missing in an XML file: At the moment, I can open a .lyx file with
>> emacs and do change / replace in the .lyx file, when e.g. I have
>> moved my images around. Or changing anything formating consistently
>> throughout the text - this is much more time consuming in LyX itself.
>> So my question: would this new XML format mean the "good bye" to the
>> plain text format of the .lyx file, or would the XML be a new
>> parallel, fully (and I mean fully!) equivalent and exchangeable
>> format in LyX? I know that an XML is also a text file, but at least
>> the ones I looked into were not nearly as editable as the .lyx plain
>> text?
>
> My impression was that Nico was making a converter to convert LyX
> native format to XML, *for export*. If anybody is making LyX format any
> more XML than it already is, I object strenuously for the exact reason
> you stated --- I like working on and diagnosing LyX files in Vim. In
> the twelve years I've used LyX, its native format has constantly become
> harder for a human to deal with.

That is what I'm doing, plus an XSL to produce .lyx from the XML.

As for the editability of .lyx vs. XML... well, both are editable in
$EDITOR.  And XML benefits from the ability to use XPath, XSL, ..., so
that if you have such XML tools at hand (and you would have to if LyX
were to switch to a native XML format) then you'd have a much easier
time doing programmatic transformations outside LyX than you do with
the current .lyx format.

Also, if LyX were to switch to a native XML format then my script and
XSLs could still be used to produce old-style .lyx for editing the way
you want to.  Such tools would still need to be supported for a long
time for migration purposes.

> XML itself is incredibly human-hostile, but its misuse by developers is
> astounding. Look at the XML for an OpenOffice file as an example.
> Probably six different files, with all sorts of redundant information
> scattered within those files. If you change something in one file
> without changing its count in another, it simply breaks the file.

It's not XML that demands "six files".

> Then there's the fact that some of us tweak our Lyx files with a Perl,
> Python, Ruby or Lua script before actually compiling it. This was easy
> [...]

See above!

Nico
--

Re: Would a native LyX XML schema be accepted?

[Nico Williams]

FWIW, I edit XML in vim plenty.  It's not awesome, but it's not
particularly painful either.  XML is not unfriendly to $EDITOR, but it
is more verbose.  The great value that XML provides is in technologies
like XPath, XQuery, and XSLT.  I ache for the power of XSLT in
association with LyX.

But also, even if LyX adopts a native XML format, the old .lyx format
will necessarily stick arou d for a long time due to the need to
support migration from one format to the other, so it should always be
possible to edit .lyx in $EDITOR for those who want it.  The .lyx
format can't be going away forever anytime soon!

Regarding collaborations...  there's an online, web-based,
collaborative LaTeX editor out there that is clearly superior to LyX
as far as collaborative editing goes.  It'd be nice if LyX could grow
some awesome collaborative functionality, and I think that XML
technologies can help there.

Nico
--

Multiple references sections?

[Nico Williams]

When using the bibliography environment a section is created called
'References'.  It's possible to add bibliography all over a document,
each set of which will be in its own section, but they're all called
"References".  Is there any way to change this other than editing the
.lyx to change the \labelstring?

Nico
--

Re: Multiple references sections?

[Nico Williams]

Never mind, found it: paragraph settings.

Re: Multiple references sections?

[Nico Williams]

On Fri, Dec 28, 2012 at 2:17 PM, Nico Williams  wrote:
> Never mind, found it: paragraph settings.

No, \labelwidthstring is something else.  It'd be nice to be able to
title References sections something else.  In RFC typesetting we need
to be able to have "Normative References" and "Informative References"
sections (and numbered ones at that, so, regular sections).

Re: The pLyX system

[Nico Williams]

On Jan 2, 2013 8:13 AM, "Richard Heck"  wrote:
>
> On 01/02/2013 01:40 AM, Liviu Andronic wrote:
>>
>> On Tue, Jan 1, 2013 at 4:32 PM, Richard Heck  wrote:
>>>
>>> all of this explanation, and these files, there. Better yet, for the
files,
>>> create a git repo on github (or somewhere) and point at it.
>>>
>> Could we have a 'wiki' branch on our GIT server? So that contributed
>> files (modules, scripts, etc.) remain in-house, and contributors can
>> commit their files there?
>>
> I assume we could set up a separate repo, but I'm not that expert with
git.

Yes, a separate repo (on the same server, of curse) is the best way.

(The Fossil VCS is git-like and has billy-in versioned wiki and ticketing.
Not that you should switch, just saying...  Mind you, i love git.)

Nico
--

Re: Word won't open simplest LyXHTML file

[Nico Williams]

I've been using LyXHTML and writing an XSL to convert from that.
That's left me unsatisfied.

I think the right approach is to have a LyX XML schema, export to
that, then apply an XSL to convert to whatever format you want.  I've
written a script to convert .lyx to XML using a very simple-minded
approach; I don't have a schema defined for this, just well-formed XML
output.  I'm still writing an XSL to replace the one I have that uses
LyXHTML input.

There was a thread about this a while back and the conclusion was that
I'm on my own with this, that for now my lyx2xml is not to be part of
LyX.  Partly there's a desire to consider making XML a native thing in
LyX, and partly there's a desire to not add more things into the LyX
source tree that might break when new features are added elsewhere in
the tree.

Dear LyX dev team: eventually you *really* do need to go the XML
route.  It's getting painful to not have a LyX XML format.  I know
this is a volunteer organization, so it's not like this can be made a
priority, but if you want LyX to be more relevant, then you need to do
this.

Nico
--

Re: Word won't open simplest LyXHTML file

[Nico Williams]

Hmm, the thought occurs that my lyx2xml script wouldn't be broken by any
changes to.lyx, not if it was always chained with lyx2lyx so that it's
input were always in a version supported by lyx2xml...

Re: Word won't open simplest LyXHTML file

[Nico Williams]

On Tue, Jan 22, 2013 at 8:52 AM, Richard Heck  wrote:
> On 01/22/2013 02:38 AM, Nico Williams wrote:
>> Hmm, the thought occurs that my lyx2xml script wouldn't be broken by any
>> changes to.lyx, not if it was always chained with lyx2lyx so that it's input
>> were always in a version supported by lyx2xml...
>
> It wouldn't be broken, but it wouldn't support new features that way, since
> those are typically reverted to ERT. This was one of the main considerations
> around the XHTML issue.

That's true.  Although lyx2xml is so generic that you'd have to do
something fairly radical to break it.  For example, merely adding a
new \begin_something wouldn't do it, but adding one that does not have
a corresponding \end_something would.  It has some things hard-coded,
line that \index is a begin-like token, or that \color, \emph, ... are
text styling tokens, and that \color appears in the header as
not-a-text-style token, which means that anything that would have to
be added to the sets of hard-coded tokens would indeed break lyx2xml,
but it's trivial to add those.  Perhaps you might like to review the
script and see for yourself?  If nothing else the script shows the few
odd things in .lyx that tripped me up.

(The script is still missing a post of the JavaScript LaTeX math ->
MathML code, which I should do at some point.  And it's only really
working for the features I use or am aware of.  There's probably other
things missing besides math, though to be fair, the LaTeX math does
get preserved in the XML output, but as the original LaTeX.)

Nico
--

Re: Word won't open simplest LyXHTML file

[Nico Williams]

On Wed, Jan 23, 2013 at 9:15 AM, Guenter Milde  wrote:
> On 2013-01-22, Nico Williams wrote:
>> There was a thread about this a while back and the conclusion was that
>> I'm on my own with this, that for now my lyx2xml is not to be part of
>> LyX.  Partly there's a desire to consider making XML a native thing in
>> LyX, and partly there's a desire to not add more things into the LyX
>> source tree that might break when new features are added elsewhere in
>> the tree.
>
> Please don't be discouraged to continue the work on and publication of
> the lyx2xml script. I think it is a useful addition to LyX.

I'm not.  I soldier one because I want to use LyX and I need XML.  In
fact, for *my* use cases the script is done.  It's not complete
however, and I was hoping others would be interested in contributing.
Instead it seems I will have to finish my XSLs (I'm doing this in my
spare time) and write a nice blog post about all this (with some
exposition of XSL code and explanations) to get people interested.

I've been surprised at how close .lyx is to being a form of XML, and
how irritating some of the ways in which it's not are.  (For example,
when mixing text styles there's no well-formed-XML-like closing of
"tags".  The most irritating thing by far is the fact that description
lists items' title/description are separated by the first breaking
space.)

> I don't know whether it will become possible to have "painless" LyX<->Word
> conversions at all, because the document models differ widely, but XSL may
> still be the best route for this conversion.

Right, there may be loss of metadata, and it may be that multiple
XSLs, or one with many options, will be needed to provide a useful
experience.  That's to be expected in schema transformations.

Nico
--

Re: LyX2YAML: was Re: Word won't open simplest LyXHTML file

[Nico Williams]

On Wed, Jan 23, 2013 at 1:57 PM, Steve Litt  wrote:
> On Wed, 23 Jan 2013 15:15:15 + (UTC), Guenter Milde said:
>> Please don't be discouraged to continue the work on and publication of
>> the lyx2xml script. I think it is a useful addition to LyX.
>
> I have a question for you...
>
> When is somebody going to write a lyx2yaml script?
>
> Another question: Instead of having LyX's native format being
> human-opaque XML, why not make it YAML, the ultimate in human
> readability and writeability?

I want XML because of XSLT (and XPath, and XQuery): I can write XSLs
to convert from one schema to another, and this then gets to be
completely external to LyX.  If the LyX XML schema changes, of course,
the XSLs need updating, but it's XSLT code, not C++ or what have you
-- it's a simpler update.  It's more than that.  You can serve XML and
XSLs from some web server and let the browser apply the XSLs to create
XHTML.  And more.  There's lots of databases that specialize in XML
documents and make it easy to search them in regular ways (i.e., with
XPath or XQuery).  I'm probably only scratching the surface here.

Also, properly formatted XML is not human-opaque, but merely annoying.

Don't get me wrong: I'm *not* a fan of XML.  I'm a fan of the DSLs and
tooling that has been built around XML, which -for me- overcome the
downsides to XML itself.

Of course, if our world had been built on programming languages with
hygienic macro facilities with all the power expect of a Scheme, then
we could write these DSLs as needed.  XML is so much wheel
re-invention, but there's a ton of value in standardizing these DSLs:
you can write portable code in them, and you need only know those
standard DSLs instead of having to learn and hack on ad-hoc DSLs.

In a sense YAML saddens me: it's even more wheel re-invention, but
without the DSLs to go with it that XML has.  Why would I want this
for any purpose other than UI?  And why would I want it even for that
purpose when LyX *is* the UI?

> http://en.wikipedia.org/wiki/Yaml

I don't really care what it is as long as there's a trivial,
*lossless* mapping onto XML.  (My script's translation of .lyx to XML
is lossless, though the return trip will not obtain the same original
.lyx because some things in .lyx content need to get normalized in the
conversion to XML.  But no data and metadata get lost in the
conversion.)

> As I understand, the LyX project's preferred scripting language is
> Python, and PyYaml is an excellent YAML parser and emitter.
>
> I don't have time or knowledge to write a lyx2yaml script, but if
> somebody else takes the captain's chair, I'll for sure help.

Well, there's http://yaxml.rubyforge.org/ , which is a tool that
converts between (in both directions) YAXML and YAML.  Pair this with
my lyx2xml script and you have a way to convert to YAML.

I don't think converting directly from .lyx to YAML is going to be any
easier than converting directly to XML is, but if it's *as* easy then
FYI you'll still run into pretty much the same issues I did in
lyx2xml.

Nico
--

Re: LyX2YAML: was Re: Word won't open simplest LyXHTML file

[Nico Williams]

On Wed, Jan 23, 2013 at 5:01 PM, Steve Litt  wrote:
> On Wed, 23 Jan 2013 14:20:57 -0600, Nico Williams said:
>
>> Well, there's http://yaxml.rubyforge.org/ , which is a tool that
>> converts between (in both directions) YAXML and YAML.  Pair this with
>> my lyx2xml script and you have a way to convert to YAML.
>
> Well cool, I guess that's one less program I have to write. How close
> to finished is your converter?

It... works.  For that subset of LyX that I use.  This includes
tables, but not math (someone needs to port to Python the
LaTeX->MathML code that can be found all over), and who knows what
else.

You can find it at
https://github.com/nicowilliams/lyx/tree/lyxml/lib/lyx2lyx  -- look
for lyx2xml*.  Forgive the sloppy code.

If this is of use to you and you need more features and want to
contribute them, or maybe tell me what they'd be...  we can make this
better.

Nico
--

Re: Share LaTeX

[Nico Williams]

Mind you, merging LyX contents via a VCS is not easy...

What I've been doing so far with my colleagues is to pass around a
"your turn" e-mail, then we each make our edits in turn with change
tracking, and one of us (or the next one to get a turn) merges edits.

It'd be nice to have a LyX diff/merge feature.  (We've had a thread about this.)

Nico
--

Re: Share LaTeX

[Nico Williams]

On Tue, Feb 26, 2013 at 2:14 AM, Rainer M Krug  wrote:
> On 26/02/13 03:08, Nico Williams wrote:
>> It'd be nice to have a LyX diff/merge feature.  (We've had a thread about 
>> this.)
>
> Yes - and I think it would be *really* useful.
>
> Maybe a Google Sumer Of Code project, or would it be to complex?

Maybe.  I wonder whether an XML diff/merge tool could produce
semantically-correct output given lyxml (the output of my lyx2xml
script).  I'm thinking "close, but no" -- if nothing else because of
conflicts, but even in the case that there are no conflicts there
might be some issues.  The next question is whether it might be
possible to write a stylesheet (in XSLT) to produce
semantically-correct output.

There's a at least one open source XML diff/merge tool ("3DM") that
might serve as the basis for this, but it looks like it needs help to
fit the bill.  There's lots of proprietary XML diff/merge tools that
might work, but we'd need an open source one.

Nico
--

Re: feature request: ribbon menus

[Nico Williams]

Since we're piling on...

I don't mind the sort of ribbon menu as they've evolved to be at MSFT,
but I do prefer pull-down menus with *text* instead of icons.  I'm a
textual user, I prefer everything as textual as possible.  I'd even
like a search feature for menus/functions, and in general I'd like
search to be as powerful as possible.  I'm so text-oriented that I'd
rather you add vi keybindings than ribbon menus.

Still, that said, if you organize the implementation cleverly, then
you could easily give the user the option of traditional vs. "ribbon"
menus.  I don't care if you do implement the option for ribbon menus
but it strikes me as a good idea to organize the implementation such
that you *can* add the option with minimal effort later.

Nico
--

Re: feature request: ribbon menus

[Nico Williams]

On Tue, Apr 9, 2013 at 1:36 PM, Liviu Andronic  wrote:
> On Tue, Apr 9, 2013 at 8:20 PM, Nico Williams  wrote:
>> textual user, I prefer everything as textual as possible.  I'd even
>> like a search feature for menus/functions, and in general I'd like
>>
> That's a good idea and easy to pull-off: Put in a search box that will
> filter UI-elements given a keyword. Similar to what we do in Settings
> and in outline.

OK, well, while we're at it, here's another thing I'd like: to be able
to create my own menus and toolbars, and this probably (oddly enough,
for me) graphically by dragging things into custom menus/toolbars.  In
particular I use custom insets a fair bit, so I'd like to make use of
custom insets much easier than now.

Nico
--

Re: Converting lyx to odt

[Nico Williams]

On Wed, May 1, 2013 at 7:21 AM, ehud.kaplan  wrote:
> I had tried to convert from a Lyx document (a Ph.D. thesis, ~140 pages) to
> LibreOffice (File/Export/HTML).  Much of it worked, but there were many
> problems:
>
> Equation numbers moved from right to left
> Figures were totally distorted (size scaled up),
> Some equations and algorithms were mangled
> Several sections appeared centered instead of being left justified as they
> were originally.
>
> Using File/Export/LYXHTML produced similar results, although the equation
> numbers were not mangled.
>
> In short, such conversions do a lot, but they also leave a lot for manual
> fixing.  I suspect that if such a path were available, many more people
> would use Lyx.

If you understand the target XML schemata then just write an XSL (in
XSLT) for the conversion from LyXHTML to that target.  I've done this.
 It's a small project, and it does depend on the stability of the
LyXHTML "schema" (XHTML documents don't have a schema so much as
conventions), which is probably not really stable.  You could also use
my lyx2xml converter and write XSLs that consume that.  The latter
depends on the .lyx "schema", which too is not stable, but working
with raw LyX metadata is easier than an XHTML rendering of a LyX
document, and you can always use lyx2lyx to workaround lack of
stability in .lyx, whereas nothing really helps w.r.t. LyXHTML
instability.

If the OpenOffice XML schemata are too complex but there's some other
intermediate XML schema for which there's a suitably faithful
converter out there, then you could use it as a step in the conversion
process.

Nico
--

Re: Converting lyx to odt

[Nico Williams]

A quick search for "odt xslt" finds some possibly useful tools and other things:

http://stackoverflow.com/questions/13031514/html-to-odt-xslt
http://opendocumentfellowship.com/projects/odftools
http://wiki.openoffice.org/wiki/ODF_Toolkit/Efforts/ODTransform
http://open.comsultia.com/docbook2odf/
http://swik.net/OpenOffice+XSLT
http://xhtml2odt.org/

LOTS of converters here:

http://wiki.openoffice.org/wiki/Xml#Filters_and_Conversions_based_on_OpenDocument.2FOpenOffice.org_XML

http://www.tei-c.org/Tools/Stylesheets/

Reverse direction:

http://www.openoffice.org/xml/sx2ml/
http://books.evc-cit.info/odf_utils/odt_to_xhtml.html
http://writer2latex.sourceforge.net/

Misc:

http://www.jejik.com/odf-xslt/manual/

Re: LyXHTML, design intent?

[Nico Williams]

On Mon, Jul 8, 2013 at 8:52 PM, Steve Litt  wrote:
> [...]

Let me do a broken disk impression...

I've used XHTML + XSLT successfully to convert to other XML schemas.
XHTML doesn't quite preserve enough metadata for me though, so I've
also written a LyX -> XML converter[0] that preserves as much metadata
as possible (if I had written an XML -> .lyx XSLT I could test that it
round-trips, which it should, with only minute differences[1]).

I know there's been some efforts recently to add native XML support to
LyX, so soon you might not need to use either LyXHTML nor my lyx2xml.

[0] https://github.com/nicowilliams/lyx2rfc/blob/master/src/lyx2xml
(and associated *.py files in the same directory).

[1] LyX allows you to mix styles in such a way that doesn't involve
proper containership.  lyx2xml works around this by closing and
re-opening styles to restore proper containership.  This is the only
thing that wouldn't round-trip exactly, but you wouldn't notice this
difference in the GUI.

Nico
--

Re: HTML2LyX???

[Nico Williams]

On Wed, Jul 31, 2013 at 3:00 PM, Steve Litt  wrote:
> Oh, I should have made it clear that style preservation would be my top
> priority. But you remind me of something: Assuming one writes
> completely styles-based HTML, the conversion to either LyX or LaTeX
> would be trivial -- everything would be implemented by a style which
> could then be defined (better defined) in LyX or LaTeX. It probably
> wouldn't be too difficult to turn  into a character style, and
> , ,  into environments, and just pass the styles straight
> through.

If it's XHTML then you could write an XSL to convert to LyX or LaTeX...

Re: LyX XMPP-Enhanced Chat

[Nico Williams]

Hi, this is pretty awesome indeed!

Would it be possible to do somethin OTR-like (in the sense of hiding
extra data) for exchanging cursor movement operations / cursor
location, and changes (typing, ...)?  That would make it possible to
collaboratively edit LyX docs.