Re: [GSoC 2014]Interested in Round trip conversion between LyX and .docx formats

[Rob Oakes]

Dear Developers,

I've been following this conversation from the shadows, but I had one 
thought that they be relevant.

I think everyone agrees that we want to avoid reimplementing the 
internal logic of a LyX document. While thinking about what might be 
the best way to implement a converter, it occurred to me that it might 
be worthwhile to implement the converters as clients that are able to 
use a LyX server session.

For export, the server could load a document, provide data about inset 
and paragraph contents, and the client could then convert the logical 
representation to docx or odf. For import, the process might work in 
reverse. The importer reads in the data from the document, translates 
it to a logical format that could be consumed by the server, and the 
server creates a new document. Document internals remain internal to 
LyX, while the server-client communicate over a well-defined protocol 
to interactively build the document.

If I understand the LyX server pipe, it is already possible to retrieve 
information about insets, citations, cross-references, and other 
document components. Moreover, there are only a few properties for a 
particular type of inset (ID, text, meta, etc.), which provides a 
manageable way to handle output data. Might it even be possible to 
export the data for a particular inset in a serialized key/value 
format? This would allow for insets without a good output format to 
main their data as DOCX/ODF metadata which could be faithfully 
translated back during import.

As I've followed the pandoc conversation, I've been concerned about the 
introduction of another dependency. If the import/export tools can be 
kept to LyX and some processing script, that seems easier to maintain 
than a complex chain of tools and external dependencies which we don't 
control.

Just my 0.02.

Cheers,

Rob

Re: [GSoC 2014]Interested in Round trip conversion between LyX and .docx formats

[Rob Oakes]

Dear Developers,

I've been following this conversation from the shadows, but I had one 
thought that they be relevant.

I think everyone agrees that we want to avoid reimplementing the 
internal logic of a LyX document. While thinking about what might be 
the best way to implement a converter, it occurred to me that it might 
be worthwhile to implement the converters as clients that are able to 
use a LyX server session.

For export, the server could load a document, provide data about inset 
and paragraph contents, and the client could then convert the logical 
representation to docx or odf. For import, the process might work in 
reverse. The importer reads in the data from the document, translates 
it to a logical format that could be consumed by the server, and the 
server creates a new document. Document internals remain internal to 
LyX, while the server-client communicate over a well-defined protocol 
to interactively build the document.

If I understand the LyX server pipe, it is already possible to retrieve 
information about insets, citations, cross-references, and other 
document components. Moreover, there are only a few properties for a 
particular type of inset (ID, text, meta, etc.), which provides a 
manageable way to handle output data. Might it even be possible to 
export the data for a particular inset in a serialized key/value 
format? This would allow for insets without a good output format to 
main their data as DOCX/ODF metadata which could be faithfully 
translated back during import.

As I've followed the pandoc conversation, I've been concerned about the 
introduction of another dependency. If the import/export tools can be 
kept to LyX and some processing script, that seems easier to maintain 
than a complex chain of tools and external dependencies which we don't 
control.

Just my 0.02.

Cheers,

Rob

Re: Corkboard code

[Rob Oakes]

On 10/26/2013 03:00 AM, Peter Kümmel wrote:
 Hi Rob,

 did you make any progress? What is the current status of the code,
 and what is your plan? Is someone still working on it?

 If current state is not too hopeless I would also work on it.

 Peter

Hi Peter,

Unfortunately, I have not been able to make much progress. I've got a
couple of projects at work that are sapping nearly all of my time. Free
or otherwise. (We are several weeks over a deadline.)

The current state is actually pretty good. There is some refactoring
that needs to happen so that the code exists inside of a single cohesive
class, and that then needs to be integrated in to the LyX GUI.

On the side of the model, there is some additional work that needs to be
done so that we can support features in the corkboard, but I've got some
good ideas on how it needs to be implemented.

If you're interested, let's set up a Google chat to discuss the current
state. I can describe most everything in a half hour or so. Right now,
that is an easier commitment for me to make than trying to discuss over
email. (Which has been somewhat hard to monitor.)

Cheers,

Rob

Re: Corkboard code

[Rob Oakes]

On 10/26/2013 03:00 AM, Peter Kümmel wrote:
> Hi Rob,
>
> did you make any progress? What is the current status of the code,
> and what is your plan? Is someone still working on it?
>
> If current state is not too "hopeless" I would also work on it.
>
> Peter

Hi Peter,

Unfortunately, I have not been able to make much progress. I've got a
couple of projects at work that are sapping nearly all of my time. Free
or otherwise. (We are several weeks over a deadline.)

The current state is actually pretty good. There is some refactoring
that needs to happen so that the code exists inside of a single cohesive
class, and that then needs to be integrated in to the LyX GUI.

On the side of the model, there is some additional work that needs to be
done so that we can support features in the corkboard, but I've got some
good ideas on how it needs to be implemented.

If you're interested, let's set up a Google chat to discuss the current
state. I can describe most everything in a half hour or so. Right now,
that is an easier commitment for me to make than trying to discuss over
email. (Which has been somewhat hard to monitor.)

Cheers,

Rob

Re: Corkboard code

[Rob Oakes]

Hey all,

I'm in the process of merging this code into the LyX codebase.
I'll push what I have as soon as I have a chance.
(Probably tomorrow or Saturday.)

Rob

Re: Corkboard code

[Rob Oakes]

Hey all,

I'm in the process of merging this code into the LyX codebase.
I'll push what I have as soon as I have a chance.
(Probably tomorrow or Saturday.)

Rob

Corkboard Development and Bug Squashing

[Rob Oakes]

Hi Ashley,

On Thu, 2013-07-18 at 16:23 -0500, Ashley Shan wrote:

 Sorry for my mistake! I worked on another local branch, and I believe
 it didn't get pushed. Now I pushed again, and please let me know if
 you still can't find them.


Thanks for pushing the most recent changes.

I will look at them and try and get back to you. I'm also about halfway
through implementations of some of the more problematic methods. I seem
to have fixed most of the random crashes, and want to complete the
implementation.

How is this for a plan?

Over the rest of the weekend, I will get through the remainder of the
changes and push them back to the developer repository on oak-tree.us.
I'm going to be traveling and will have only intermittent access to
Internet and email.

In the meantime, I think it would be good for you to start learning the
main LyX source code. We've talked about starting to work on bugs to do
this. This weekend, can you focus on bug squashing? Please visit the LyX
bug tracker, find two or three of the bugs marked as easy fixes, and
begin working on patches. As a goal, let's try and have two or three
bugs fixed in the master LyX source code.

The developers on lyx-devel should be able to help suggest bugs and
offer suggestions if you get stuck. On Monday of next week and I'll walk
you through the code changes I've made.

I'll also merge in the changes you've just sent me and try and have a
fully functioning corkboard so that we can move to the next phase of the
project.

Hope your classes on going well. 

Cheers,

Rob



signature.asc
Description: This is a digitally signed message part

Corkboard Development and Bug Squashing

[Rob Oakes]

Hi Ashley,

On Thu, 2013-07-18 at 16:23 -0500, Ashley Shan wrote:

> Sorry for my mistake! I worked on another local branch, and I believe
> it didn't get pushed. Now I pushed again, and please let me know if
> you still can't find them.


Thanks for pushing the most recent changes.

I will look at them and try and get back to you. I'm also about halfway
through implementations of some of the more problematic methods. I seem
to have fixed most of the random crashes, and want to complete the
implementation.

How is this for a plan?

Over the rest of the weekend, I will get through the remainder of the
changes and push them back to the developer repository on oak-tree.us.
I'm going to be traveling and will have only intermittent access to
Internet and email.

In the meantime, I think it would be good for you to start learning the
main LyX source code. We've talked about starting to work on bugs to do
this. This weekend, can you focus on bug squashing? Please visit the LyX
bug tracker, find two or three of the bugs marked as easy fixes, and
begin working on patches. As a goal, let's try and have two or three
bugs fixed in the master LyX source code.

The developers on lyx-devel should be able to help suggest bugs and
offer suggestions if you get stuck. On Monday of next week and I'll walk
you through the code changes I've made.

I'll also merge in the changes you've just sent me and try and have a
fully functioning corkboard so that we can move to the next phase of the
project.

Hope your classes on going well. 

Cheers,

Rob



signature.asc
Description: This is a digitally signed message part

Re: XML Parsing Library [was Re: XML For LyX]

[Rob Oakes]

On Thu, May 9, 2013 at 10:52 AM, Richard Heck rgh...@lyx.org wrote:


I just had a look at those. He had an XML parser here:
http://www.lyx.org/trac/browser/lyxsvn/lyx-devel/branches/personal/larsbj/xml/src/support/xmlparser.h?rev=19478
but it appears to be based upon xmlpp, which I cannot get to compile 
on my machine. It's a very old library. An older version uses expat, 
which is pretty heavy duty.


I did some googling and found this page:
http://lars.ruoff.free.fr/xmlcpp/
which describes a bunch of free XML libraries and was updated 2/2012. 
Most of what's there is either (a) very large, like Xerces and 
libxml2, or else (b) a DOM-style parser, which is not what we wantm, 
I think. The best of the options appears to be:

http://www.fxtech.com/xmlio/
which is a very lightweight (53KB source) and simple, SAX-like 
parser. LGPL. It is also quite old, but it compiles just fine here. 
Of course, it also writes XML.


It could probably use some updating if we were going to use it, but 
the code is very simple, so this would be easy to do.


Is there a reason we would want to avoid libxml? I've found it to offer 
the best feature set and ease of use. It also ships with a set of 
excellent Python bindings, which we could incorporate into the Python 
we ship. Between the two, there is very little that wouldn't be 
possible from an XML processing standpoint.


We might even be able to incorporate some of the XSL processing that 
some of the users have been salivating over.

Re: XML Parsing Library [was Re: XML For LyX]

[Rob Oakes]

On Thu, May 9, 2013 at 10:52 AM, Richard Heck  wrote:


I just had a look at those. He had an XML parser here:
http://www.lyx.org/trac/browser/lyxsvn/lyx-devel/branches/personal/larsbj/xml/src/support/xmlparser.h?rev=19478
but it appears to be based upon xmlpp, which I cannot get to compile 
on my machine. It's a very old library. An older version uses expat, 
which is pretty heavy duty.


I did some googling and found this page:
http://lars.ruoff.free.fr/xmlcpp/
which describes a bunch of free XML libraries and was updated 2/2012. 
Most of what's there is either (a) very large, like Xerces and 
libxml2, or else (b) a DOM-style parser, which is not what we wantm, 
I think. The best of the options appears to be:

http://www.fxtech.com/xmlio/
which is a very lightweight (53KB source) and simple, SAX-like 
parser. LGPL. It is also quite old, but it compiles just fine here. 
Of course, it also writes XML.


It could probably use some updating if we were going to use it, but 
the code is very simple, so this would be easy to do.


Is there a reason we would want to avoid libxml? I've found it to offer 
the best feature set and ease of use. It also ships with a set of 
excellent Python bindings, which we could incorporate into the Python 
we ship. Between the two, there is very little that wouldn't be 
possible from an XML processing standpoint.


We might even be able to incorporate some of the XSL processing that 
some of the users have been salivating over.

Re: Proposal updated

[Rob Oakes]

Hi Richard,

On Fri, 2013-05-03 at 00:35 -0500, Vanderbilt wrote:

 There is only one point in your comment that I'm not clear about; if
 I'm understanding you correctly, do you mean the community is not sure
 about whether to use signals/slots so I probably shouldn't include
 that part in my proposal? But Rob said we are using signals/slots;
 it's just the type of signals that is not clear (if I'm understanding
 him correctly). Could you clarify a little bit?

I apologize for the radio silence, I'm still struggling to get my
project wrapped up and delivered. It absolutely must be done today,
though. I will have time this afternoon/over the weekend to respond to
the signal/slot questions on the list.

I'll continue that conversation in the other thread.

Cheers,

Rob

Re: Signals/Slots in Core (branch from UI improvements and non-linear enhancements)

[Rob Oakes]

On Mon, 2013-04-29 at 14:46 +0200, Vincent van Ravesteijn wrote:

 There have been ideas to remove the ugly connections using the 
 WorkAreaManager and GuiBufferDelegate and to replace them by Qt signals.
 
 I'm not sure it is a good idea to have a connection between each inset 
 and the gui. I'm afraid it will become a mess. What's wrong with telling 
 the Buffer I (inset 0x3234234) have changed, and let the buffer 
 forward this to the gui.
 
 That is, if we decide to use this kind of connection.

I'm not wedded to using Qt signals/slots in the core, but I think we
need a better way to communicate changes to the UI. It's not that the
scanning is terribly costly, it's everything that happens on top of it
(the UI repainting, etc) that I'm worried about.

What I would really like to see if a method of updating the UI that
components of the document have changed, without having to rebuild the
entire view every time a user hits enter or backspace. This would solve
many problems. 

For example, one issue I've had reported is that there is a noticeable
flicker in the Corkboard each time there is a document update (I've had
complaints on every keystroke, other say only on backspace and enter.
The problem doesn't appear on OS X, since Apple seems to use double
buffering to repaint UI flicker.) I've tried very hard to double buffer
and only repaint once on all platforms, but catching all of the events
which trigger the repaint event has proved ... difficult. Tying
individual items on the corkboard to some underlying structure and
updating on change would be much better. It seemed that signals/slots
were a natural way to do this, as they are already used within the UI.

The salient question, though, is where the update should happen. Right
now we recreate the toc model and related classes on every update. Is
the toc model the best place, or would it be better to try and handle it
in WorkAreaManager, or GuiBufferDelegate, ...

Thoughts?

Cheers,

Rob

Re: Proposal updated

[Rob Oakes]

Hi Richard,

On Fri, 2013-05-03 at 00:35 -0500, Vanderbilt wrote:

> There is only one point in your comment that I'm not clear about; if
> I'm understanding you correctly, do you mean the community is not sure
> about whether to use signals/slots so I probably shouldn't include
> that part in my proposal? But Rob said we are using signals/slots;
> it's just the type of signals that is not clear (if I'm understanding
> him correctly). Could you clarify a little bit?

I apologize for the radio silence, I'm still struggling to get my
project wrapped up and delivered. It absolutely must be done today,
though. I will have time this afternoon/over the weekend to respond to
the signal/slot questions on the list.

I'll continue that conversation in the other thread.

Cheers,

Rob

Re: Signals/Slots in Core (branch from UI improvements and non-linear enhancements)

[Rob Oakes]

On Mon, 2013-04-29 at 14:46 +0200, Vincent van Ravesteijn wrote:

> There have been ideas to remove the ugly connections using the 
> WorkAreaManager and GuiBufferDelegate and to replace them by Qt signals.
> 
> I'm not sure it is a good idea to have a connection between each inset 
> and the gui. I'm afraid it will become a mess. What's wrong with telling 
> the Buffer "I (inset 0x3234234) have changed", and let the buffer 
> forward this to the gui.
> 
> That is, if we decide to use this kind of connection.

I'm not wedded to using Qt signals/slots in the core, but I think we
need a better way to communicate changes to the UI. It's not that the
scanning is terribly costly, it's everything that happens on top of it
(the UI repainting, etc) that I'm worried about.

What I would really like to see if a method of updating the UI that
components of the document have changed, without having to rebuild the
entire view every time a user hits enter or backspace. This would solve
many problems. 

For example, one issue I've had reported is that there is a noticeable
flicker in the Corkboard each time there is a document update (I've had
complaints on every keystroke, other say only on backspace and enter.
The problem doesn't appear on OS X, since Apple seems to use double
buffering to repaint UI flicker.) I've tried very hard to double buffer
and only repaint once on all platforms, but catching all of the events
which trigger the repaint event has proved ... difficult. Tying
individual items on the corkboard to some underlying structure and
updating on change would be much better. It seemed that signals/slots
were a natural way to do this, as they are already used within the UI.

The salient question, though, is where the update should happen. Right
now we recreate the toc model and related classes on every update. Is
the toc model the best place, or would it be better to try and handle it
in WorkAreaManager, or GuiBufferDelegate, ...

Thoughts?

Cheers,

Rob

Re: Proposal submitted; suggestions wanted

[Rob Oakes]

Hi Xueqing,

I apologize for not responding earlier. I saw your draft proposal, and I
had several things I wanted to point out. I am still desperately trying
to finish my work project (our deadline has been extended), and will
respond in more detail when able.

With that said:

I agree with Cyrille about the notepad application. In general, I would
devote no more than a day or two to this, as it is the equivalent of a
Qt hello world. It's value mostly relies in its ability to help build
confidence.

Because Qt is a full-featured UI toolkit, a notepad can be created very
easily with Qt Designer and a few lines of C++ code. Additionally, the
classes you'll be exploring are only tangentially related to LyX.

Instead, I would look at creating an example that uses the Model/View
classes. Those are directly related to the work you would be doing in
LyX. This link provides an overview of the model/view/delegate system in
Qt:
http://qt-project.org/doc/qt-4.8/model-view-programming.html

There are several good examples distributed with the Qt demo programs
that can serve as a basis for exploration:
http://qt-project.org/doc/qt-4.8/all-examples.html

In particular, I would look at the itemviews:
http://qt-project.org/doc/qt-4.8/examples-itemviews.html

I think most of the things you will need to learn can be picked up in a
day or two of exploration. Qt is a very nice toolkit and, as far as
these things go, comes with a fairly low learning curve. (Learning
standard C++ is much more difficult, in my opinion.)

In addition to the model/view system, I think you would also be well
served by looking into the Graphics View framework:
http://qt-project.org/doc/qt-4.8/graphicsview.html

Indeed, I think a great way to learn Qt would be to build a simple
notecard app which uses the model for data and QGraphicsScene for
drawing. The link below provides some suggestions on how to start:
http://stackoverflow.com/questions/3188584/how-to-use-qt-model-view-framework-with-the-graphics-view-framework

It might be good for us to visit via Google Talk. Would it be possible
for you to email me (off-list) a set of times when you might be
available?

Cheers,

Rob

Branch for UI Improvements/Outliner

[Rob Oakes]

Dear Developers,

Would it be possible for someone to create a branch on git.lyx.org for
the UI improvement code? Since there has been some recent interest in
it, I thought it would be good to have it in a format more easily merged
with lyx-master. To that end, I created a git repo with all of my code
from Launchpad in it. I'm now trying to figure out where the best place
to stash it might be.

Cheers,

Rob

Re: Proposal submitted; suggestions wanted

[Rob Oakes]

Hi Xueqing,

I apologize for not responding earlier. I saw your draft proposal, and I
had several things I wanted to point out. I am still desperately trying
to finish my work project (our deadline has been extended), and will
respond in more detail when able.

With that said:

I agree with Cyrille about the notepad application. In general, I would
devote no more than a day or two to this, as it is the equivalent of a
Qt hello world. It's value mostly relies in its ability to help build
confidence.

Because Qt is a full-featured UI toolkit, a notepad can be created very
easily with Qt Designer and a few lines of C++ code. Additionally, the
classes you'll be exploring are only tangentially related to LyX.

Instead, I would look at creating an example that uses the Model/View
classes. Those are directly related to the work you would be doing in
LyX. This link provides an overview of the model/view/delegate system in
Qt:
http://qt-project.org/doc/qt-4.8/model-view-programming.html

There are several good examples distributed with the Qt demo programs
that can serve as a basis for exploration:
http://qt-project.org/doc/qt-4.8/all-examples.html

In particular, I would look at the itemviews:
http://qt-project.org/doc/qt-4.8/examples-itemviews.html

I think most of the things you will need to learn can be picked up in a
day or two of exploration. Qt is a very nice toolkit and, as far as
these things go, comes with a fairly low learning curve. (Learning
standard C++ is much more difficult, in my opinion.)

In addition to the model/view system, I think you would also be well
served by looking into the Graphics View framework:
http://qt-project.org/doc/qt-4.8/graphicsview.html

Indeed, I think a great way to learn Qt would be to build a simple
notecard app which uses the model for data and QGraphicsScene for
drawing. The link below provides some suggestions on how to start:
http://stackoverflow.com/questions/3188584/how-to-use-qt-model-view-framework-with-the-graphics-view-framework

It might be good for us to visit via Google Talk. Would it be possible
for you to email me (off-list) a set of times when you might be
available?

Cheers,

Rob

Branch for UI Improvements/Outliner

[Rob Oakes]

Dear Developers,

Would it be possible for someone to create a branch on git.lyx.org for
the UI improvement code? Since there has been some recent interest in
it, I thought it would be good to have it in a format more easily merged
with lyx-master. To that end, I created a git repo with all of my code
from Launchpad in it. I'm now trying to figure out where the best place
to stash it might be.

Cheers,

Rob

Re: GSoC idea: UI improvements and non-linear writing enhancements (Technical)

[Rob Oakes]

Hi Xueqing,

Again, this is a brief reply. I've got a massive deadline looming on
Monday and I will not have much time to respond until after then.
Hopefully the notes below will be enough to get you started. If not, let
me know and we can set up a time to visit via Skype/Google Chat.

A lot of this would be much faster to explain than to write out.

On Tue, 2013-04-23 at 01:18 -0500, Ashley Shan wrote

 Thank you for all the information and materials. It took me a while to
 read all your emails and linked blogs/infos, so please correct me if
 I'm being unclear or if I omit anything. Again, thank you for your
 patience and help.

Glad to hear that they were helpful.


# Proposal Contents #
 
 Non-linear writing process and the goal of this project
 There are several ways for a writer to write creatively, and for this
 project, if I'm understanding you correctly, there are two major
 functionalities you want to achieve from a non-linear supportive
 computer program: 1) to integrate the entire writing process
 (collecting information, playing with ideas, and actually writing)
 into one program, and 2) specifically, to provide a more intuitive way
 for the user to play with ideas. 

This is correct. The tools in my original proposal are geared toward
playing with ideas and actually writing. The first, collecting
information, is something I would also like to add support for, but
should probably remain outside of the scope of the project aims. It is a
project unto itself.

 Hence, if we are to finish up the existing work, the project would
 focus on one of the following: 
 1. Modifying the LyX model (make QObject the base class of text/note
 insets so that the program doesn't generate a new model each time the
 text was changed. 
 2. Modifying the corkboard view to use QGraphicsScene, for which we
 can include functionalities such as clipping texts into cards, more
 flexible arrangement of cards (the hierarchical view), and more
 flexible notecard views.

Right.
 
 Regarding my GSoC application, I wish to get your suggestions for 1)
 whether I should focus on both or only one of the goals mentioned
 above and 2) whether there are other functionalities that I've raised
 in previous emails and will be clarified in the rest of this email
 which may be incorporated into the main goals.

This depends upon how comfortable you feel with your programming
abilities. If I were doing the work, I would probably need about a month
to modify the LyX model so that it can utilize signals/slots. From
there, it would be the work of a few weeks to modify the corkboard to
use QGraphicsScene and to provide for the additional features. In total,
this would mean about six weeks of part time (15 to 20 hours) work.

As a rough estimate, I frequently assume it will take someone less
familiar with the code between twice and three times as long to finish,
or between 12 to 18 weeks. The first half could be spent on the LyX
model, the second on the corkboard improvements.

The model is by far the more delicate of the two, as it involves
re-working several core classes in LyX. (This could result in breakages
that would need to be addressed.)

If you wished to be more conservative, you could focus your proposal on
improving the model to use QObject.

As you are trying to plan, I would contact Abdel Younes
(you...@lyx.org), I have cc:ed him on this email. He is familiar with
how LyX's internals  work and could provide guidance as you think about
the design of your code. He's also in a good position to know about how
much time it would take to implement the model changes, and the
subsequent implications that would have for your proposal.

I also have some ideas on this might be done, which I will try and flesh
out in an email early next week. As you visit with Abdel, it would be
good to have milestones (or specific tasks with deliverables) that you
can include in your proposal. For the first one or two weeks, it might
be good to break these down into daily milestones. Later, it's not
necessary to be quite so detailed.


# Proposal Suggestions #

I've now seen a few proposals that students have submitted for LyX, and
in general, I've found the description of milestones and deliverables to
be vague. The more specific you can be about your work product
(especially right up front), the better it looks.

Instead of using Study LyX model internals for your first week, it
would be much better to use that as a description for the week, and the
provide daily milestones and deliverables:

Week 1: Draft QObject Based Design for Insets and LyX TocModel

Day 1: Document existing inset and toc model structure, create class
diagram showing relevant interactions

Day 2: Create proposed design spec reworking insets and toc model to use
QObject as a base class, determine signals/slots needed for corkboard,
draft, and outline enhancements

Day 3: Meet with mentor to discuss design, adjust based on feedback.
Begin Qobject implementation in core inset classes. 

GSOC Proposal Thoughts

[Rob Oakes]

Dear GSOC Students,

What follows is an excerpt from another email, but I thought that it
might be useful for others than the intended recipient.

_

I've now seen a few proposals that students have submitted for LyX, and
in general, I've found the description of milestones and deliverables to
be vague. The more specific you can be about your work product
(especially right up front), the better it looks.

Instead of using Study LyX model internals for your first week, it
would be much better to use that as a description for the week, and the
provide daily milestones and deliverables:

Week 1: Draft QObject Based Design for Insets and LyX TocModel

Day 1: Document existing inset and toc model structure, create class
diagram showing relevant interactions

Day 2: Create proposed design spec reworking insets and toc model to use
QObject as a base class, determine signals/slots needed for corkboard,
draft, and outline enhancements

Day 3: Meet with mentor to discuss design, adjust based on feedback.
Begin Qobject implementation in core inset classes. Submit patch for
review to lyx-devel.
...

(I would consider the timeline above to be fairly credible and you might
use it as a jumping off point for your project.)

For each day, I would recommend that you create something concrete, even
if it is very simple (such as a diagram). This makes it much easier for
me and the other mentors to help you and gives us something concrete to
critique. It also creates documentation that might be useful in the
future as we attempt to explain to others how LyX works. 

(After the first week or two, it's more appropriate to have larger
milestones.)
_

As the LyX devs read through your proposal, there are a few questions we
will be asking ourselves:

1.) How much effort has the student spent planning his time?
2.) Does she have a roadmap to begin the work?
3.) In addition to code, what other benefits will the project receive
from the student's time (e.g., documentation of existing code/design, as
the student tries to understand a particular problem)?
4.) Does the proposal show that the student has effectively planned
their time and made good use of available resources?

As a mentor, I want for students to be successful; I want them to have a
good experience; and I want for them to contribute to LyX in a
meaningful way. A student who spends on the project timeline and
description of deliverables is more likely to devote that same kind of
care to their overall work.

Which brings me to the point. The existing proposals I have seen are:

1.) Too vague
2.) Incomplete
3.) Lack sufficient description of deliverables

This does not mean you have to be verbose, but the more concrete you can
be in timeline and deliverables the better. It's a hallmark of a good
proposal, and it does not matter if it is a GSOC application for five
thousand dollars or an NSF/NIH grant for five million.

Perhaps the other mentors have suggestions as well?

Cheers,

Rob

Re: GSoC idea: UI improvements and non-linear writing enhancements (Technical)

[Rob Oakes]

Hi Xueqing,

Again, this is a brief reply. I've got a massive deadline looming on
Monday and I will not have much time to respond until after then.
Hopefully the notes below will be enough to get you started. If not, let
me know and we can set up a time to visit via Skype/Google Chat.

A lot of this would be much faster to explain than to write out.

On Tue, 2013-04-23 at 01:18 -0500, Ashley Shan wrote

> Thank you for all the information and materials. It took me a while to
> read all your emails and linked blogs/infos, so please correct me if
> I'm being unclear or if I omit anything. Again, thank you for your
> patience and help.

Glad to hear that they were helpful.


# Proposal Contents #
> 
> Non-linear writing process and the goal of this project
> There are several ways for a writer to write creatively, and for this
> project, if I'm understanding you correctly, there are two major
> functionalities you want to achieve from a non-linear supportive
> computer program: 1) to integrate the entire writing process
> (collecting information, playing with ideas, and actually writing)
> into one program, and 2) specifically, to provide a more intuitive way
> for the user to "play with ideas". 

This is correct. The tools in my original proposal are geared toward
"playing with ideas" and "actually writing." The first, "collecting
information," is something I would also like to add support for, but
should probably remain outside of the scope of the project aims. It is a
project unto itself.

> Hence, if we are to finish up the existing work, the project would
> focus on one of the following: 
> 1. Modifying the LyX model (make QObject the base class of text/note
> insets so that the program doesn't generate a new model each time the
> text was changed. 
> 2. Modifying the corkboard view to use QGraphicsScene, for which we
> can include functionalities such as "clipping texts into cards", "more
> flexible arrangement of cards ( hierarchical view)", and "more
> flexible notecard views".

Right.
> 
> Regarding my GSoC application, I wish to get your suggestions for 1)
> whether I should focus on both or only one of the goals mentioned
> above and 2) whether there are other functionalities that I've raised
> in previous emails and will be clarified in the rest of this email
> which may be incorporated into the main goals.

This depends upon how comfortable you feel with your programming
abilities. If I were doing the work, I would probably need about a month
to modify the LyX model so that it can utilize signals/slots. From
there, it would be the work of a few weeks to modify the corkboard to
use QGraphicsScene and to provide for the additional features. In total,
this would mean about six weeks of part time (15 to 20 hours) work.

As a rough estimate, I frequently assume it will take someone less
familiar with the code between twice and three times as long to finish,
or between 12 to 18 weeks. The first half could be spent on the LyX
model, the second on the corkboard improvements.

The model is by far the more delicate of the two, as it involves
re-working several core classes in LyX. (This could result in breakages
that would need to be addressed.)

If you wished to be more conservative, you could focus your proposal on
improving the model to use QObject.

As you are trying to plan, I would contact Abdel Younes
(you...@lyx.org), I have cc:ed him on this email. He is familiar with
how LyX's internals  work and could provide guidance as you think about
the design of your code. He's also in a good position to know about how
much time it would take to implement the model changes, and the
subsequent implications that would have for your proposal.

I also have some ideas on this might be done, which I will try and flesh
out in an email early next week. As you visit with Abdel, it would be
good to have milestones (or specific tasks with "deliverables") that you
can include in your proposal. For the first one or two weeks, it might
be good to break these down into daily milestones. Later, it's not
necessary to be quite so detailed.


# Proposal Suggestions #

I've now seen a few proposals that students have submitted for LyX, and
in general, I've found the description of milestones and deliverables to
be vague. The more specific you can be about your work product
(especially right up front), the better it looks.

Instead of using "Study LyX model internals" for your first week, it
would be much better to use that as a description for the week, and the
provide daily milestones and deliverables:

Week 1: Draft QObject Based Design for Insets and LyX TocModel

Day 1: Document existing inset and toc model structure, create class
diagram showing relevant interactions

Day 2: Create proposed design spec reworking insets and toc model to use
QObject as a base class, determine signals/slots needed for corkboard,
draft, and outline enhancements

Day 3: Meet with mentor to discuss design, adjust based on feedback.
Begin 

GSOC Proposal Thoughts

[Rob Oakes]

Dear GSOC Students,

What follows is an excerpt from another email, but I thought that it
might be useful for others than the intended recipient.

_

I've now seen a few proposals that students have submitted for LyX, and
in general, I've found the description of milestones and deliverables to
be vague. The more specific you can be about your work product
(especially right up front), the better it looks.

Instead of using "Study LyX model internals" for your first week, it
would be much better to use that as a description for the week, and the
provide daily milestones and deliverables:

Week 1: Draft QObject Based Design for Insets and LyX TocModel

Day 1: Document existing inset and toc model structure, create class
diagram showing relevant interactions

Day 2: Create proposed design spec reworking insets and toc model to use
QObject as a base class, determine signals/slots needed for corkboard,
draft, and outline enhancements

Day 3: Meet with mentor to discuss design, adjust based on feedback.
Begin Qobject implementation in core inset classes. Submit patch for
review to lyx-devel.
...

(I would consider the timeline above to be fairly credible and you might
use it as a jumping off point for your project.)

For each day, I would recommend that you create something concrete, even
if it is very simple (such as a diagram). This makes it much easier for
me and the other mentors to help you and gives us something concrete to
critique. It also creates documentation that might be useful in the
future as we attempt to explain to others how LyX works. 

(After the first week or two, it's more appropriate to have larger
milestones.)
_

As the LyX devs read through your proposal, there are a few questions we
will be asking ourselves:

1.) How much effort has the student spent planning his time?
2.) Does she have a roadmap to begin the work?
3.) In addition to code, what other benefits will the project receive
from the student's time (e.g., documentation of existing code/design, as
the student tries to understand a particular problem)?
4.) Does the proposal show that the student has effectively planned
their time and made good use of available resources?

As a mentor, I want for students to be successful; I want them to have a
good experience; and I want for them to contribute to LyX in a
meaningful way. A student who spends on the project timeline and
description of deliverables is more likely to devote that same kind of
care to their overall work.

Which brings me to the point. The existing proposals I have seen are:

1.) Too vague
2.) Incomplete
3.) Lack sufficient description of deliverables

This does not mean you have to be verbose, but the more concrete you can
be in timeline and deliverables the better. It's a hallmark of a good
proposal, and it does not matter if it is a GSOC application for five
thousand dollars or an NSF/NIH grant for five million.

Perhaps the other mentors have suggestions as well?

Cheers,

Rob

Re: Asking IRC Channel

[Rob Oakes]

Hi Sanjeev,

On Tue, 2013-04-23 at 00:57 +0530, sanjeevkumar shetty wrote:
 I searched for your IRC Channel, I couldnt get it, Please help me to
 get it, so that i can specify myself and about my project(advanced
 find and replace). I want  to utilize my knowing knowledge for it and
 also i love learning so . plz.

The IRC is #lyx-devel on Freenode.

Re: Asking IRC Channel

[Rob Oakes]

Hi Sanjeev,

On Tue, 2013-04-23 at 00:57 +0530, sanjeevkumar shetty wrote:
> I searched for your IRC Channel, I couldnt get it, Please help me to
> get it, so that i can specify myself and about my project(advanced
> find and replace). I want  to utilize my knowing knowledge for it and
> also i love learning so . plz.

The IRC is #lyx-devel on Freenode.

Signals/Slots in Core (branch from UI improvements and non-linear enhancements)

[Rob Oakes]

Dear Developers,

If I remember correctly, as of the last developer meeting, QtCore
classes were being tolerated in the core of LyX.
http://www.mail-archive.com/lyx-devel@lists.lyx.org/msg172533.html

Is this correct? Are there any examples of QtCore and Signals/Slots
being integrated into the core of LyX?

I've been corresponding with one of the GSOC students about using the
outline enhancements for a GSOC project, and it seems that moving
certain text insets to use QObject (or related) as their base class
might be an interesting goal. This would allow for insets to make use of
the signal/slot mechanism for communicating updates to the view.

In the case of the outline view and underlying TOC model, it wouldn't be
necessary to scan for changes, and would provide a nice speed boost to
the editing experience.

Would such a project be too ambitious?

Cheers,

Rob

Re: GSoC idea: UI improvements and non-linear writing enhancements (Technical)

[Rob Oakes]

 similar to my
 idea. Maybe the closest idea to dragging blocks of sentences is how
 you drag files in the list (either on a Mac or a PC) and rearrange
 them, and in terms of dragging single words or phrases, think of how
 you drag bookmarks on a bookmark bar in Safari. I personally would be
 totally impressed if my text editor could do that.

I think this sounds like a very nice enhancement to LyX. As of right
now, the outline branch has a limited version of this feature, based
upon defined sections (similar to the way the outline view in Word
works). Each section may then have a note associated with it, and it's
parent text. The notes, title, and text are kept in a model, and can be
manipulated: re-ordered through drag and drop, indented/unindented, etc.

If I understand your thinking correctly, you would like to extend this
behavior to work on arbitrary snippets of text? Is that correct?

There is a writing program called Scrivener which allows for a type of
this where text snippets do not need to be tied to specific sections.
See the link below for an overview of how it works:
http://literatureandlatte.com/scrivener.php

Is this similar to what you were thinking? If not, could you expand upon
your original idea? How did you imagine splitting the writing into
different chunks? What meta-data did you imagine being associated with
each component?

How do you imagine this information being rendered on screen? Would it
follow similar metaphors as used by Scrivener, or a more abstract series
of metaphors used by another writing program (such as Information
Architects writer: http://www.iawriter.com/ipad/).

# Next Steps #

I understand that I've covered quite a bit of ground in this email and
provided a deluge of links. Here are a few next steps that I think will
get you closer to a project proposal.

1. You need to select a one or two *well defined* aspects of the
non-linear writing enhancements that sounds of interest. One example
might be to move the LyX core inset classes to QObject and adjust the
TOC model to work on the basis of signals and slots.

or, you might adjust the corkboard view to use QGraphicsScene:
http://qt-project.org/doc/qt-4.8/qgraphicsscene.html

The QGraphicsScene classes provide for a more performant base than
QWidget (which we use now), and would allow for more flexible notecard
views. Indeed, it might even be possible to create specialized views for
more than just sections of text; figures, for example, might have one
type of notecard (including images); and references, a third.

Or you might decide to pursue something else entirely, such as what your
draft view would require.

2. Once you settle on one or two major goals, we can start to talk about
the best way to implement them, pick milestones, and discuss a timeline
for deliverables. The more concrete you can be about the above, the
better chance your proposal will have of being accepted. (See
http://www.di.ens.fr/~baghdadi/TXT_blog/5_advices_to_get_your_proposal_accepted.lyx.html
 which has several pieces of advice for writing a strong proposal.)

# Conclusion #

As I said before, I'm looking forward to working with you and sincerely
hope that your project will be approved. Discussing the non-linear
enhancements has gotten me excited to pick up on several of my own LyX
projects and to produce a killer writing environment.

Cheers,

Rob Oakes

Re: GSoC idea: UI improvements and non-linear writing enhancements

[Rob Oakes]

Hi Xueqing,

For a more thorough technical overview, please see my earlier response.

On Mon, 2013-04-22 at 00:45 -0500, Ashley Shan wrote:

 Thank you so much for replying and encouragement. I was a little bit
 discouraged because everyone else in the mailing list trying to apply
 to GSoC seems to know everything. But I now have the confidence to
 work on this project! However, since I am a very inexperienced
 programmer, I plan to stick to your plan and focus on refining and
 making changes to the outline-corkboard idea. 
 
I wouldn't let that worry you. Computer science is a broad and rich
field, and there is always something to learn. From personal adventures
in seeking funding (science related endeavors), I've found that what
matters most is the quality of your proposal. The more specific you can
be about your goals and your plan to get there, the better.
 
 Another thing, since my school is on semester, so we have our summer
 break from May 3 to August 21. I asked in the mailing list and I think
 Cyrille said it's okay for me to shift my schedule a little bit
 earlier as long as my mentor is available at that time. I wonder,
 since you have been working on this project for a long time and will
 very likely be my mentor if I get accepted, whether you will have time
 for this project earlier than scheduled.
 
I am currently working toward a deadline a week from today (April 29).
After that, I will be more available to help you. Keep in mind that
student applications are due by the 3rd of May. You should definitely
get started on your application(s) now. As you develop your ideas and
ask questions, I am sure that others on the list will be happy to
provide feedback.
 
 I'm on the process of putting up an initial proposal, and here are
 some of my ideas I think we can incorporate in the outline-corkboard
 project for which I wish I could get some suggestions from you.

For specific item-by-item feedback, see below. In general, though: pick
two or three of the following items (or the examples in my other email).
Many of these items would require a pretty heft amount of time to
implement or extend. Additionally, LyX already provides support for many
of them.

 - Put the entire non-linear writing into a separate draft mode.
 - Track changes of draft and allow the user to undo/redo 100 times
 (+100/-100).

LyX already provides an undo system. Additionally, it has the ability
to merge with version control systems, which are capable of tracking the
entire history of a writing project. The following articles provide for
an overview of how this works:
http://blog.oak-tree.us/index.php/2009/02/13/subversion1
http://blog.oak-tree.us/index.php/2012/01/20/subversion2
http://blog.oak-tree.us/index.php/2012/01/30/subversion34
http://blog.oak-tree.us/index.php/2012/02/01/subversion4

My recommendation would be to take advantage of the external systems
that LyX already supports, rather than re-inventing the wheel.

It might be nice to save the undo history to the LyX-file, though, which
would allow for it to remain persistent between sessions. Along the same
vein, you might discuss ways to improve LyX's handling of version
control branches.

 - Save changes in some temporary repertoire. Hence if the thing crash,
 the user can retrieve his or her effort.

Again, LyX already does something similar. It keeps a separate copy of
the buffer, and if there is a crash, provides the ability to open the
temp copy of the document. It might be worthwhile to see if this
capability could be extended, perhaps integrating with LyX's current

 - Increase time and space efficiency by changing the way how text and
 outline are synchronized.

Not sure what you mean by this. The ideal would be to use the document
object model (DOM) for both outline and text, with outline (and
associated metadata) reflecting what is currently in the text.

 - Adjust LyX to non-linear writing by re-designing tags. For example,
 in our draft mode, we can have brainstorm tags, outline tags,
 need refinement tags to organize the process of writing.

This sounds like a very nice enhancement. There is some core work which
would need to be done in order to accomodate it, such as moving some
insets (LyX's internal representation of text) to a signal/slot type of
architecture.

 - Arrangement of yellow sticky notes: instead of let them be squares
 of the same size and lined up in order, we can allow the user to drag
 and place them to wherever he wants, such as nodes in other mind
 mapping applications.
 - Allow the user to clip texts into cards.
 - Allow the user to define his own outline independent of what is on
 the text. (I personally find myself limited once I feel I have to
 stick to the outline. I'm not sure about this idea.)
 
Again, an interesting idea. I think we need to explore more what you
have in mind, though. Clipping texts into cards is pretty
self-explanatory, but how would you see the outline developing? When you
say independent of what is in the 

Signals/Slots in Core (branch from UI improvements and non-linear enhancements)

[Rob Oakes]

Dear Developers,

If I remember correctly, as of the last developer meeting, QtCore
classes were being tolerated in the core of LyX.
http://www.mail-archive.com/lyx-devel@lists.lyx.org/msg172533.html

Is this correct? Are there any examples of QtCore and Signals/Slots
being integrated into the core of LyX?

I've been corresponding with one of the GSOC students about using the
outline enhancements for a GSOC project, and it seems that moving
certain text insets to use QObject (or related) as their base class
might be an interesting goal. This would allow for insets to make use of
the signal/slot mechanism for communicating updates to the view.

In the case of the outline view and underlying TOC model, it wouldn't be
necessary to scan for changes, and would provide a nice speed boost to
the editing experience.

Would such a project be too ambitious?

Cheers,

Rob

Re: GSoC idea: UI improvements and non-linear writing enhancements (Technical)

[Rob Oakes]

 to drag and rearrange
> elements of the text, I haven't seen any application similar to my
> idea. Maybe the closest idea to dragging blocks of sentences is how
> you drag files in the list (either on a Mac or a PC) and rearrange
> them, and in terms of dragging single words or phrases, think of how
> you drag bookmarks on a bookmark bar in Safari. I personally would be
> totally impressed if my text editor could do that.

I think this sounds like a very nice enhancement to LyX. As of right
now, the outline branch has a limited version of this feature, based
upon defined sections (similar to the way the outline view in Word
works). Each section may then have a note associated with it, and it's
parent text. The notes, title, and text are kept in a model, and can be
manipulated: re-ordered through drag and drop, indented/unindented, etc.

If I understand your thinking correctly, you would like to extend this
behavior to work on arbitrary snippets of text? Is that correct?

There is a writing program called Scrivener which allows for a type of
this where text snippets do not need to be tied to specific sections.
See the link below for an overview of how it works:
http://literatureandlatte.com/scrivener.php

Is this similar to what you were thinking? If not, could you expand upon
your original idea? How did you imagine splitting the writing into
different chunks? What meta-data did you imagine being associated with
each component?

How do you imagine this information being rendered on screen? Would it
follow similar metaphors as used by Scrivener, or a more abstract series
of metaphors used by another writing program (such as Information
Architects writer: http://www.iawriter.com/ipad/).

# Next Steps #

I understand that I've covered quite a bit of ground in this email and
provided a deluge of links. Here are a few next steps that I think will
get you closer to a project proposal.

1. You need to select a one or two *well defined* aspects of the
non-linear writing enhancements that sounds of interest. One example
might be to move the LyX core inset classes to QObject and adjust the
TOC model to work on the basis of signals and slots.

or, you might adjust the corkboard view to use QGraphicsScene:
http://qt-project.org/doc/qt-4.8/qgraphicsscene.html

The QGraphicsScene classes provide for a more performant base than
QWidget (which we use now), and would allow for more flexible notecard
views. Indeed, it might even be possible to create specialized views for
more than just sections of text; figures, for example, might have one
type of notecard (including images); and references, a third.

Or you might decide to pursue something else entirely, such as what your
draft view would require.

2. Once you settle on one or two major goals, we can start to talk about
the best way to implement them, pick milestones, and discuss a timeline
for deliverables. The more concrete you can be about the above, the
better chance your proposal will have of being accepted. (See
http://www.di.ens.fr/~baghdadi/TXT_blog/5_advices_to_get_your_proposal_accepted.lyx.html
 which has several pieces of advice for writing a strong proposal.)

# Conclusion #

As I said before, I'm looking forward to working with you and sincerely
hope that your project will be approved. Discussing the non-linear
enhancements has gotten me excited to pick up on several of my own LyX
projects and to produce a killer writing environment.

Cheers,

Rob Oakes

Re: GSoC idea: UI improvements and non-linear writing enhancements

[Rob Oakes]

Hi Xueqing,

For a more thorough technical overview, please see my earlier response.

On Mon, 2013-04-22 at 00:45 -0500, Ashley Shan wrote:

> Thank you so much for replying and encouragement. I was a little bit
> discouraged because everyone else in the mailing list trying to apply
> to GSoC seems to know everything. But I now have the confidence to
> work on this project! However, since I am a very inexperienced
> programmer, I plan to stick to your plan and focus on refining and
> making changes to the outline-corkboard idea. 
> 
I wouldn't let that worry you. Computer science is a broad and rich
field, and there is always something to learn. From personal adventures
in seeking funding (science related endeavors), I've found that what
matters most is the quality of your proposal. The more specific you can
be about your goals and your plan to get there, the better.
> 
> Another thing, since my school is on semester, so we have our summer
> break from May 3 to August 21. I asked in the mailing list and I think
> Cyrille said it's okay for me to shift my schedule a little bit
> earlier as long as my mentor is available at that time. I wonder,
> since you have been working on this project for a long time and will
> very likely be my mentor if I get accepted, whether you will have time
> for this project earlier than scheduled.
> 
I am currently working toward a deadline a week from today (April 29).
After that, I will be more available to help you. Keep in mind that
student applications are due by the 3rd of May. You should definitely
get started on your application(s) now. As you develop your ideas and
ask questions, I am sure that others on the list will be happy to
provide feedback.
> 
> I'm on the process of putting up an initial proposal, and here are
> some of my ideas I think we can incorporate in the outline-corkboard
> project for which I wish I could get some suggestions from you.

For specific item-by-item feedback, see below. In general, though: pick
two or three of the following items (or the examples in my other email).
Many of these items would require a pretty heft amount of time to
implement or extend. Additionally, LyX already provides support for many
of them.

> - Put the entire non-linear writing into a separate "draft" mode.
> - Track changes of draft and allow the user to undo/redo 100 times
> (+100/-100).

LyX already provides an "undo" system. Additionally, it has the ability
to merge with version control systems, which are capable of tracking the
entire history of a writing project. The following articles provide for
an overview of how this works:
http://blog.oak-tree.us/index.php/2009/02/13/subversion1
http://blog.oak-tree.us/index.php/2012/01/20/subversion2
http://blog.oak-tree.us/index.php/2012/01/30/subversion34
http://blog.oak-tree.us/index.php/2012/02/01/subversion4

My recommendation would be to take advantage of the external systems
that LyX already supports, rather than re-inventing the wheel.

It might be nice to save the undo history to the LyX-file, though, which
would allow for it to remain persistent between sessions. Along the same
vein, you might discuss ways to improve LyX's handling of version
control branches.

> - Save changes in some temporary repertoire. Hence if the thing crash,
> the user can retrieve his or her effort.

Again, LyX already does something similar. It keeps a separate copy of
the buffer, and if there is a crash, provides the ability to open the
temp copy of the document. It might be worthwhile to see if this
capability could be extended, perhaps integrating with LyX's current

> - Increase time and space efficiency by changing the way how text and
> outline are synchronized.

Not sure what you mean by this. The ideal would be to use the document
object model (DOM) for both outline and text, with outline (and
associated metadata) reflecting what is currently in the text.

> - Adjust LyX to non-linear writing by re-designing tags. For example,
> in our "draft" mode, we can have "brainstorm" tags, "outline" tags,
> "need refinement" tags to organize the process of writing.

This sounds like a very nice enhancement. There is some core work which
would need to be done in order to accomodate it, such as moving some
insets (LyX's internal representation of text) to a signal/slot type of
architecture.

> - Arrangement of yellow sticky notes: instead of let them be squares
> of the same size and lined up in order, we can allow the user to drag
> and place them to wherever he wants, such as nodes in other mind
> mapping applications.
> - Allow the user to clip texts into cards.
> - Allow the user to define his own outline independent of what is on
> the text. (I personally find myself limited once I feel I have to
> stick to the outline. I'm not sure about this idea.)
> 
Again, an interesting idea. I think we need to explore more what you
have in mind, though. Clipping texts into cards is pretty
self-explanatory, but how would you see the outline 

Re: GSoC idea: UI improvements and non-linear writing enhancements

[Rob Oakes]

Hi Xueqing,

On Thu, 2013-04-18 at 00:30 -0500, Ashley Shan wrote:

 Thank you for replying. Before you have time to more fully respond, I
 want to briefly explain my concerns and ideas.

I appreciate your forthrightness. I will respond with a technical
outline in a separate email sent to the mailing list. (All email on the
list is archived, and it is a tremendously useful thing to have it
there. It also allows others to comment. There have been many times when
I've needed to go back and refer to points in a discussion.)

I wanted to respond to a couple of points in this email separately,
though.
 
 1. My major concern is my qualifications. As I mentioned earlier, I am
 a first-year student planning to major in computer science. I took
 CS101 (in Java) and CS201 (in C++). In CS201, we learned data types,
 ADTs, and some simple algorithms, and we built some interesting
 projects. I believe (and my professor confirmed) that CS201 qualifies
 me to work on tasks that are a little bit above my current abilities. 

 My biggest problem is, I have never worked with others on a project
 and I'm not familiar with source code directories. The main reasons I
 am so interested in this project are 1) I really want to build my
 programming skills and 2) my personal experience with intensive
 academic writing allow me to really think what writers need, not what
 programmers think writers need. I'm more than prepared to devote a lot
 of time and energy into this project and work extensively. Hence, I
 wonder whether you can give me some evaluations and/or suggestions.
 Also, if you'd like to know more about my coding skills, I can send
 the course description of CS201 to you or my code for its assignments.

I appreciate this concern. When I began programming, I felt much the
same way. It's one thing to hack away on your own, and quite another to
contribute code to an active project. I frequently still feel
self-conscious when I submit large code patches for review.

With that said, the Google Code program is designed to provide
experience to students of *all* skill levels. The secret will be to
define project that you feel you will be able to tackle in the available
time. As long as you are comfortable with basic data structures and
algorithms, I think you will be fine in working on the UI enhancements.

There are many people invested in LyX and who wish for prospective
students to succeed. This includes the official LyX mentors, the other
project developers, and the community at large. For your success, it is
most important you feel comfortable engaging. This means asking
questions, submitting code for review, and experimenting. Everyone knows
that you are still finding your way, but with that said, we all are.

Your contribution is not going to hurt an active project. Many eyes will
look it through before it ends up in the master repository. Code with
confidence.

Cheers,

Rob

Re: GSoC idea: UI improvements and non-linear writing enhancements

[Rob Oakes]

Hi Xueqing,

On Thu, 2013-04-18 at 00:30 -0500, Ashley Shan wrote:

> Thank you for replying. Before you have time to more fully respond, I
> want to briefly explain my concerns and ideas.

I appreciate your forthrightness. I will respond with a technical
outline in a separate email sent to the mailing list. (All email on the
list is archived, and it is a tremendously useful thing to have it
there. It also allows others to comment. There have been many times when
I've needed to go back and refer to points in a discussion.)

I wanted to respond to a couple of points in this email separately,
though.
> 
> 1. My major concern is my qualifications. As I mentioned earlier, I am
> a first-year student planning to major in computer science. I took
> CS101 (in Java) and CS201 (in C++). In CS201, we learned data types,
> ADTs, and some simple algorithms, and we built some interesting
> projects. I believe (and my professor confirmed) that CS201 qualifies
> me to work on tasks that are a little bit above my current abilities. 

> My biggest problem is, I have never worked with others on a project
> and I'm not familiar with source code directories. The main reasons I
> am so interested in this project are 1) I really want to build my
> programming skills and 2) my personal experience with intensive
> academic writing allow me to really think what writers need, not what
> programmers think writers need. I'm more than prepared to devote a lot
> of time and energy into this project and work extensively. Hence, I
> wonder whether you can give me some evaluations and/or suggestions.
> Also, if you'd like to know more about my coding skills, I can send
> the course description of CS201 to you or my code for its assignments.

I appreciate this concern. When I began programming, I felt much the
same way. It's one thing to hack away on your own, and quite another to
contribute code to an active project. I frequently still feel
self-conscious when I submit large code patches for review.

With that said, the Google Code program is designed to provide
experience to students of *all* skill levels. The secret will be to
define project that you feel you will be able to tackle in the available
time. As long as you are comfortable with basic data structures and
algorithms, I think you will be fine in working on the UI enhancements.

There are many people invested in LyX and who wish for prospective
students to succeed. This includes the official LyX mentors, the other
project developers, and the community at large. For your success, it is
most important you feel comfortable engaging. This means asking
questions, submitting code for review, and experimenting. Everyone knows
that you are still finding your way, but with that said, we all are.

Your contribution is not going to hurt an active project. Many eyes will
look it through before it ends up in the master repository. Code with
confidence.

Cheers,

Rob

Re: GSoC idea: UI improvements and non-linear writing enhancements

[Rob Oakes]

Hi Xueqing,

On Tue, 2013-04-16 at 21:42 -0500, Ashley Shan wrote

 My name is Xueqing Shan, and I am a freshman in Vanderbilt University
 in Tennessee planning to participate in Google Summer of Code. I am
 particularly interested in this project of LyX: UI improvements and
 non-linear writing enhancements. I looked at your blog posts and have
 some ideas. I have sent my ideas to LyX developers' mailing list, but
 I think it's a better idea to contact you directly since you have done
 a lot of preliminary work. I would greatly appreciate if you can share
 your thoughts with me or give me some suggestions.

It is wonderful to hear from you and I am glad that you are excited
about this project. I've read over your thoughts and will respond to
them fully.

However, I am currently buried in a project deadline and will not have
time to respond till this weekend. What follows are my brief thoughts.
As I said, I will respond more fully when I am able.

 So here's what I'm thinking. There are two options for a student
 developer working on this project:

 1. Continue to work on the existing project. I saw the source code for
 the outliner. So if I am about to work on the existing project and
 completing the outliner and the corkboard, I wish I could know more
 details about the progress and design of them. By the way, I haven't
 found any code about the corkboard you mentioned in your blog.

The code for both the corkboard and the outliner are available from the
Luanchpad repository:

https://code.launchpad.net/~lyx-outline-devel/lyx-outline/lyx-outline.devel

The outliner and corkboard can be found in the /src/frontends/qt4
folder. The code for the corkboard is in Corkboard.h and Corkboard.cpp.
The code for the outliner is in the OutlineView.h and OutlineView.cpp.

There is also a supporting class called DataManager (.h, .cpp). The UI
is provided by widgets, which have class names similar to Corkboard.h
and Corkboard.cpp.
 
 2. Start a brand new idea or make significant changes to the outliner
 and the corkboard. I have been thinking maybe we can combine
 functionalities of the outliner and the corkboard, making elements in
 the outliner moveable, so the user can drag and insert and edit
 directly in the outline view (and in the actual document at the same
 time). 

At present, this is planned. They are linked through the underlying
model, and drag/drop has been implemented (though it needs to be cleaned
up). It would be good if we could move it to a QGraphicsScene linked to
the underlying document model. Additionally, the way which we process
changes in the document and link the text to the model also needs to be
updated. Right now, it scans the whole text and creates the model each
time a major update is completed.

There has been some talk about allowing signals and slots into the core
of LyX, that might be one option.
 
 Another brand new idea of mine is to have a draft interface for
 writing, and this interface must allows the user to drag and rearrange
 sentences and phrases. Sentences are separated by newline characters
 and dots, while the separation of phrases requires a little bit more
 work to implement an algorithm to smartly determine pieces of
 thoughts. From my personal writing experience and my conversations
 with my humanities-major friends, such an interface would be much
 better than the traditional white blank Microsoft Word window.
 
This sounds very interesting, could you expand more of what you have in
mind. Is there something similar you might be able to point me at so I
can get a better grip on the functionality?
 
 Would you prefer a student developer to work on existing code or start
 a new idea?
 
Either would be wonderful. I've had full intentions for refining the
existing code and releasing an update, but have not had time to do so.
If you are interested in exploring your own ideas, though, I am happy to
discuss those further as well.

Cheers,

Rob

PS, more to follow.

Re: GSoC: Question about UI improvement and non-linear writing

[Rob Oakes]

On Wed, 2013-04-17 at 07:32 +0200, Liviu Andronic wrote:
 On Wed, Apr 17, 2013 at 4:30 AM, Ashley Shan
 xueqing.s...@vanderbilt.edu wrote:

 Rob correct me if I'm wrong, but I think this was part of the original intent.

Yes, and has been implemented. It needs some cleanup, but that should be
relatively minor. I'll respond with more detail as soon as I am able.

Cheers,

Rob

Re: GSoC idea: UI improvements and non-linear writing enhancements

[Rob Oakes]

Hi Xueqing,

On Tue, 2013-04-16 at 21:42 -0500, Ashley Shan wrote

> My name is Xueqing Shan, and I am a freshman in Vanderbilt University
> in Tennessee planning to participate in Google Summer of Code. I am
> particularly interested in this project of LyX: "UI improvements and
> non-linear writing enhancements". I looked at your blog posts and have
> some ideas. I have sent my ideas to LyX developers' mailing list, but
> I think it's a better idea to contact you directly since you have done
> a lot of preliminary work. I would greatly appreciate if you can share
> your thoughts with me or give me some suggestions.

It is wonderful to hear from you and I am glad that you are excited
about this project. I've read over your thoughts and will respond to
them fully.

However, I am currently buried in a project deadline and will not have
time to respond till this weekend. What follows are my brief thoughts.
As I said, I will respond more fully when I am able.

> So here's what I'm thinking. There are two options for a student
> developer working on this project:

> 1. Continue to work on the existing project. I saw the source code for
> the outliner. So if I am about to work on the existing project and
> completing the outliner and the corkboard, I wish I could know more
> details about the progress and design of them. By the way, I haven't
> found any code about the corkboard you mentioned in your blog.

The code for both the corkboard and the outliner are available from the
Luanchpad repository:

https://code.launchpad.net/~lyx-outline-devel/lyx-outline/lyx-outline.devel

The outliner and corkboard can be found in the /src/frontends/qt4
folder. The code for the corkboard is in Corkboard.h and Corkboard.cpp.
The code for the outliner is in the OutlineView.h and OutlineView.cpp.

There is also a supporting class called DataManager (.h, .cpp). The UI
is provided by widgets, which have class names similar to Corkboard.h
and Corkboard.cpp.
> 
> 2. Start a brand new idea or make significant changes to the outliner
> and the corkboard. I have been thinking maybe we can combine
> functionalities of the outliner and the corkboard, making elements in
> the outliner moveable, so the user can drag and insert and edit
> directly in the outline view (and in the actual document at the same
> time). 

At present, this is planned. They are linked through the underlying
model, and drag/drop has been implemented (though it needs to be cleaned
up). It would be good if we could move it to a QGraphicsScene linked to
the underlying document model. Additionally, the way which we process
changes in the document and link the text to the model also needs to be
updated. Right now, it scans the whole text and creates the model each
time a major update is completed.

There has been some talk about allowing signals and slots into the core
of LyX, that might be one option.
> 
> Another brand new idea of mine is to have a "draft" interface for
> writing, and this interface must allows the user to drag and rearrange
> sentences and phrases. Sentences are separated by newline characters
> and dots, while the separation of phrases requires a little bit more
> work to implement an algorithm to smartly determine pieces of
> thoughts. From my personal writing experience and my conversations
> with my humanities-major friends, such an interface would be much
> better than the traditional white blank Microsoft Word window.
> 
This sounds very interesting, could you expand more of what you have in
mind. Is there something similar you might be able to point me at so I
can get a better grip on the functionality?
> 
> Would you prefer a student developer to work on existing code or start
> a new idea?
> 
Either would be wonderful. I've had full intentions for refining the
existing code and releasing an update, but have not had time to do so.
If you are interested in exploring your own ideas, though, I am happy to
discuss those further as well.

Cheers,

Rob

PS, more to follow.

Re: GSoC: Question about UI improvement and non-linear writing

[Rob Oakes]

On Wed, 2013-04-17 at 07:32 +0200, Liviu Andronic wrote:
> On Wed, Apr 17, 2013 at 4:30 AM, Ashley Shan
>  wrote:

> Rob correct me if I'm wrong, but I think this was part of the original intent.

Yes, and has been implemented. It needs some cleanup, but that should be
relatively minor. I'll respond with more detail as soon as I am able.

Cheers,

Rob

Re: GSoC: Improved XHTML export and ePub support

[Rob Oakes]

On Wed, 2013-04-10 at 17:46 -0700, Pavel Sanda wrote:
 Adrián Pereira wrote:
  I'll appreciate more information about this project, so i can start reading
  the documentation and maybe the code.
 
 More people already asked about XHTML/ePUB stuff, so few bits of info below.
 
 Current status of related native LyX export functionality is:

 3) LyX-ePUB
Not done at all. 2. needs to be tweaked to get 3.
Contact not completely clear to me, several people have been around
this (Rob/Liviu/perhaps Richard?).

I apologize for being so quiet in the past few months. I've been
absolutely swamped with work and family commitments (my wife and I were
lucky enough to have a son born last August, and it's amazing how
children change your life).

About a year ago, I started work on ePub support. Based on that
experience, I would strongly recommend that we tighten up our XHTML and
CSS export and use that output for packaging ePub.

As others have already said, the LyX format is a moving target, and I'm
not sure why we would want to maintain two separate implementations. Our
existing XHTML device already provides:

1.) CSS customization
2.) MathML
3.) References and Bibliographies

This covers everything we need to create robust markup for ePub
packaging. As I see it, most of the development tasks fall into the
following general categories:

1.) Developing CSS markup that can be applied to make it more ePub
friendly. This includes a few semantic tag changes which make sense in a
browser, but are less well adapted for devices.

Consider, for example, the CSS property page-break-inside, which
should be used inside figures to prevent them from being split
willy-nilly across several pages, unless absolutely necessary.

It seems that most of this work should be done inside of a module that
can be added to a document. This would maintain the integrity of our
existing customization system.

2.) Creating a system to split documents into multiple XHTML files. ePub
readers perform better if book content is split into multiple files
(especially if the book contains audio/video material.) It speeds up
load time and improves performance.

The last time I checked (which, unfortunately, was a while ago), though,
there wasn't an easy way to specify break points in a document. This
would mean some work on the backend and providing a UI mechanism to
specify breakpoints. Perhaps through a markup tag or through a break
inset, similar to chapter-break (or both).

3.) Add image processing options. Many people who use LyX will likely
want to target multiple output formats (PDF, XHTML, and ePub, for
example).

Electronic devices do not need the super high resolution images that
print requires. There should be some way to provide separate resolution
and processing specs which can be fed to ImageMagick.

4.) Create a system to package ePub, generate supporting files, and
package into a zip archive with an epub extension. This could easily be
done via a Python script.

5.) Providing for a UI that can be used to edit styles and create valid
custom CSS and LaTeX.

There are obviously other approaches, but I think that the outline above
would provide for a robust and customizable system that we can extend to
support additional features (such as audio/video). It also helps us to
avoid the sorts of maintenance headaches that a separate XML - XSLT
route would create.

Other thoughts (Richard, Pavel)?

Cheers,

Rob

Re: GSoC: Improved XHTML export and ePub support

[Rob Oakes]

On Wed, 2013-04-10 at 17:46 -0700, Pavel Sanda wrote:
> Adrián Pereira wrote:
> > I'll appreciate more information about this project, so i can start reading
> > the documentation and maybe the code.
> 
> More people already asked about XHTML/ePUB stuff, so few bits of info below.
> 
> Current status of related native LyX export functionality is:

> 3) LyX->ePUB
>Not done at all. 2. needs to be tweaked to get 3.
>Contact not completely clear to me, several people have been around
>this (Rob/Liviu/perhaps Richard?).

I apologize for being so quiet in the past few months. I've been
absolutely swamped with work and family commitments (my wife and I were
lucky enough to have a son born last August, and it's amazing how
children change your life).

About a year ago, I started work on ePub support. Based on that
experience, I would strongly recommend that we tighten up our XHTML and
CSS export and use that output for packaging ePub.

As others have already said, the LyX format is a moving target, and I'm
not sure why we would want to maintain two separate implementations. Our
existing XHTML device already provides:

1.) CSS customization
2.) MathML
3.) References and Bibliographies

This covers everything we need to create robust markup for ePub
packaging. As I see it, most of the development tasks fall into the
following general categories:

1.) Developing CSS markup that can be applied to make it more ePub
friendly. This includes a few semantic tag changes which make sense in a
browser, but are less well adapted for devices.

Consider, for example, the CSS property "page-break-inside", which
should be used inside figures to prevent them from being split
willy-nilly across several pages, unless absolutely necessary.

It seems that most of this work should be done inside of a module that
can be added to a document. This would maintain the integrity of our
existing customization system.

2.) Creating a system to split documents into multiple XHTML files. ePub
readers perform better if book content is split into multiple files
(especially if the book contains audio/video material.) It speeds up
load time and improves performance.

The last time I checked (which, unfortunately, was a while ago), though,
there wasn't an easy way to specify break points in a document. This
would mean some work on the backend and providing a UI mechanism to
specify breakpoints. Perhaps through a markup tag or through a break
inset, similar to chapter-break (or both).

3.) Add image processing options. Many people who use LyX will likely
want to target multiple output formats (PDF, XHTML, and ePub, for
example).

Electronic devices do not need the super high resolution images that
print requires. There should be some way to provide separate resolution
and processing specs which can be fed to ImageMagick.

4.) Create a system to package ePub, generate supporting files, and
package into a zip archive with an epub extension. This could easily be
done via a Python script.

5.) Providing for a UI that can be used to edit styles and create valid
custom CSS and LaTeX.

There are obviously other approaches, but I think that the outline above
would provide for a robust and customizable system that we can extend to
support additional features (such as audio/video). It also helps us to
avoid the sorts of maintenance headaches that a separate XML -> XSLT
route would create.

Other thoughts (Richard, Pavel)?

Cheers,

Rob

Re: Commercial support for LyX?

[Rob Oakes]

On Mon, 2012-10-22 at 02:32 +0200, Lars Gullik Bjønnes wrote:
 Is that what this is?
 
 http://aprenderlyx.com/
 

Fascinating link.

It looks like a course about how to use LyX for writing a thesis. They
have five or six modules which covers creating a document, typesetting a
document, and entering mathematics. As part of their advanced plan, they
have support via phone and email. From the video, and site, though, it
looks to be editing support rather than technical support.

Very interesting idea. Out of curiosity, do we have any idea how big
LyX's user base really is? The other day I saw someone using it to take
notes in a class, and I was elated. It's the first time I've seen
someone using it in the wild where I hadn't introduced them to the
program.

Cheers,

Rob

Re: Commercial support for LyX?

[Rob Oakes]

On Mon, 2012-10-22 at 02:32 +0200, Lars Gullik Bjønnes wrote:
> Is that what this is?
> 
> http://aprenderlyx.com/
> 

Fascinating link.

It looks like a course about how to use LyX for writing a thesis. They
have five or six modules which covers creating a document, typesetting a
document, and entering mathematics. As part of their advanced plan, they
have support via phone and email. From the video, and site, though, it
looks to be editing support rather than "technical support."

Very interesting idea. Out of curiosity, do we have any idea how big
LyX's user base really is? The other day I saw someone using it to take
notes in a class, and I was elated. It's the first time I've seen
someone using it "in the wild" where I hadn't introduced them to the
program.

Cheers,

Rob

Disable Aspell Support from Compiling

[Rob Oakes]

Dear LyX Developers,

In the past few days, I've finally had some time to work on LyX ebook stuff. 
However, I'm having a very strange problem.

On Mac OS X, I'm getting compiler errors for aspell. Of itself, this isn't so 
strange (since I don't have Aspell installed on the system). What is strange, 
though, is that I can't find a way to tell LyX to skip trying to compile ASpell 
support. I've tried the standard method of 

-DLYX_ASPELL=OFF

but that doesn't appear to do anything. It just goes ahead and tries to compile 
it anyway. I tried to remove ASpellChecker.h and .cpp, but that caused the 
system to complain that they weren't there.

How can I tell the build system to ignore ASpell support?

(I'm using cmake to create the make files.)

Cheers,

Rob

Disable Aspell Support from Compiling

[Rob Oakes]

Dear LyX Developers,

In the past few days, I've finally had some time to work on LyX ebook stuff. 
However, I'm having a very strange problem.

On Mac OS X, I'm getting compiler errors for aspell. Of itself, this isn't so 
strange (since I don't have Aspell installed on the system). What is strange, 
though, is that I can't find a way to tell LyX to skip trying to compile ASpell 
support. I've tried the standard method of 

-DLYX_ASPELL=OFF

but that doesn't appear to do anything. It just goes ahead and tries to compile 
it anyway. I tried to remove ASpellChecker.h and .cpp, but that caused the 
system to complain that they weren't there.

How can I tell the build system to ignore ASpell support?

(I'm using cmake to create the make files.)

Cheers,

Rob

Re: Multiple Body Tags in XHTML Export

[Rob Oakes]

On 4/30/2012 4:27 PM, Richard Heck wrote:
 I don't know if this is possible. The svn repo still exists, but it's
 been closed to commits for a while, and I don't myself know how to
 reactivate it locally.
Okay. I think I may have worked out a way to merge all of my work using
git, but haven't had time to work on it. I'm going to be at a conference
later this week and might try and figure it out then.

In other news, though, I'm happy with how the LyXHTML - ePub translator
is working. I've tested it against both my local builds and the main git
version, and it's been handling XHTML output from both very well.

I still need to work on using HTML from elyxer. Not all of the metadata
that LyXHTML uses is present, which results in an incomplete Table of
Contents and contents.opf file.

Cheers,

Rob

Re: Multiple Body Tags in XHTML Export

[Rob Oakes]

On 4/30/2012 4:27 PM, Richard Heck wrote:
> I don't know if this is possible. The svn repo still exists, but it's
> been closed to commits for a while, and I don't myself know how to
> reactivate it locally.
Okay. I think I may have worked out a way to merge all of my work using
git, but haven't had time to work on it. I'm going to be at a conference
later this week and might try and figure it out then.

In other news, though, I'm happy with how the LyXHTML -> ePub translator
is working. I've tested it against both my local builds and the main git
version, and it's been handling XHTML output from both very well.

I still need to work on using HTML from elyxer. Not all of the metadata
that LyXHTML uses is present, which results in an incomplete Table of
Contents and contents.opf file.

Cheers,

Rob

Multiple Body Tags in XHTML Export

[Rob Oakes]

Dear Developers,

This weekend I've been working on my ePub packaging module for LyX. I've
made good progress. The script will correctly package XHTML output (from
both LyX and eLyXer). It has support for defining points at which you
would like the file to split the file, rewriting links, moving images,
and a couple of other cleanup options. In short, it's getting pretty
close to a point where I can release it for testing.

As I've started to test it on more complicated documents, I've found
something strange. In Ly SVN (2.1), I've started seeing multiple body
tags. These tags are inside of the main body for the document, so,
technically, the XHTML is well formed.

Is there any particular reason for this? (My SVN version is pretty old.
I haven't yet figured out how to merge my working copy, which is kept in
bzr with git.) Did someone start to implement support for splitting
files at chapters or parts and I just didn't get all of the updates?

If this behavior isn't intended, we might want to look at changing it.
In general, there should only be a single body tag in XHTML documents.
Some ePub validators won't validate the document when it has more.

Cheers,

Rob

Re: Multiple Body Tags in XHTML Export

[Rob Oakes]

Hi Richard,

On 4/23/2012 9:20 AM, Richard Heck wrote:
 If you could post a file that caused this, I would be happy to have a
 look. But
 I'm puzzled what might cause it. Are these files with includes?
So far, yes. They've all had includes and child documents. Attached is
one example (I've only included the parent. Let me know if you want me
to send you a child.)

Cheers,

Rob


OpenSource - Complete Text.lyx
Description: application/lyx

Re: Multiple Body Tags in XHTML Export

[Rob Oakes]

On 4/23/2012 12:14 PM, Richard Heck wrote:
 OK, found the problem. Silly error. Should be fixed.
Awesome! Thanks for taking a look at it.

On another point, might it be possible to have master branch of lyx.git
pushed to SVN on a semi-regular basis?

There are a number of branches set up over on Launchpad (using bzr)
which rely on importing from the SVN. I'm trying to get them switched
over to Git, but haven't had much luck. Because of that, the most recent
changes in trunk aren't getting incorporated.

Using git-svn, pushing to SVN is just like pushing to another git
branch. It might even be set to happen via cron. I'd be happy to look
after it, if someone were to give me commit access.

I know that we're trying to move away from SVN, but keeping a mirror of
the trunk equivalent would be very helpful for a number of reasons
(including packaging and the development of some extensions I've been
dong on Launchpad).

Cheers,

Rob

Multiple Body Tags in XHTML Export

[Rob Oakes]

Dear Developers,

This weekend I've been working on my ePub packaging module for LyX. I've
made good progress. The script will correctly package XHTML output (from
both LyX and eLyXer). It has support for defining points at which you
would like the file to split the file, rewriting links, moving images,
and a couple of other cleanup options. In short, it's getting pretty
close to a point where I can release it for testing.

As I've started to test it on more complicated documents, I've found
something strange. In Ly SVN (2.1), I've started seeing multiple 
tags. These tags are inside of the main  for the document, so,
technically, the XHTML is well formed.

Is there any particular reason for this? (My SVN version is pretty old.
I haven't yet figured out how to merge my working copy, which is kept in
bzr with git.) Did someone start to implement support for splitting
files at chapters or parts and I just didn't get all of the updates?

If this behavior isn't intended, we might want to look at changing it.
In general, there should only be a single "body" tag in XHTML documents.
Some ePub validators won't validate the document when it has more.

Cheers,

Rob

Re: Multiple Body Tags in XHTML Export

[Rob Oakes]

Hi Richard,

On 4/23/2012 9:20 AM, Richard Heck wrote:
> If you could post a file that caused this, I would be happy to have a
> look. But
> I'm puzzled what might cause it. Are these files with includes?
So far, yes. They've all had includes and child documents. Attached is
one example (I've only included the parent. Let me know if you want me
to send you a child.)

Cheers,

Rob


OpenSource - Complete Text.lyx
Description: application/lyx

Re: Multiple Body Tags in XHTML Export

[Rob Oakes]

On 4/23/2012 12:14 PM, Richard Heck wrote:
> OK, found the problem. Silly error. Should be fixed.
Awesome! Thanks for taking a look at it.

On another point, might it be possible to have master branch of lyx.git
pushed to SVN on a semi-regular basis?

There are a number of branches set up over on Launchpad (using bzr)
which rely on importing from the SVN. I'm trying to get them switched
over to Git, but haven't had much luck. Because of that, the most recent
changes in trunk aren't getting incorporated.

Using git-svn, pushing to SVN is just like pushing to another git
branch. It might even be set to happen via cron. I'd be happy to look
after it, if someone were to give me commit access.

I know that we're trying to move away from SVN, but keeping a mirror of
the "trunk" equivalent would be very helpful for a number of reasons
(including packaging and the development of some extensions I've been
dong on Launchpad).

Cheers,

Rob

Re: Updates to gitolite progress

[Rob Oakes]

On Mar 12, 2012, at 10:59 AM, Abdelrazak Younes wrote:

 The main repo would be automatically synchronized (only the 2.0 and
 2.1 branches); either via a git hook or a cron script at server side.
 The cron idea is better if only we had some automatic regression
 testing in place. Then only those commits that passes regression on
 the cooking repo would be pushed to the main repo.

Would it be possible to have the main repo also synchronized with the existing 
SVN? git-svn can transparently push right to an SVN repository. That would 
prevent a lot of things from breaking. For example, I've been keeping a mirror 
of SVN on Launchpad that I use for development, testing, and (hopefully) daily 
builds. Retooling this mirror to pull from git would require re-importing the 
whole thing (a multi-day affair).

If there is any way to keep our current SVN branch active with the canonical 
developer sources, that would be a wonderful thing. (Even if most of the 
development is happening in git.)

Cheers,

Rob

Re: Updates to gitolite progress

[Rob Oakes]

On Mar 12, 2012, at 10:59 AM, Abdelrazak Younes wrote:

> The main repo would be automatically synchronized (only the 2.0 and
> 2.1 branches); either via a git hook or a cron script at server side.
> The cron idea is better if only we had some automatic regression
> testing in place. Then only those commits that passes regression on
> the cooking repo would be pushed to the main repo.

Would it be possible to have the main repo also synchronized with the existing 
SVN? git-svn can transparently push right to an SVN repository. That would 
prevent a lot of things from breaking. For example, I've been keeping a mirror 
of SVN on Launchpad that I use for development, testing, and (hopefully) daily 
builds. Retooling this mirror to pull from git would require re-importing the 
whole thing (a multi-day affair).

If there is any way to keep our current SVN branch active with the "canonical" 
developer sources, that would be a wonderful thing. (Even if most of the 
development is happening in git.)

Cheers,

Rob

Re: word2lyx (Word to LyX Translator): Status Update

[Rob Oakes]

On Mar 8, 2012, at 2:21 AM, Rainer M Krug wrote:

 -BEGIN PGP SIGNED MESSAGE-
 Hash: SHA1

 I would love to agree, but round-trip is what is needed most of the
 time. An import word2lyx is perfect, but in most cases only half the
 story. I would use it extensively if the round trip is possible.
 Obviously, we can not deal with the word-editing side (whatever
 program is used for that).

I'm sympathetic to this point. I understand that having a way to go from one to 
the other is important. I've deliberately avoided creating an export to Word 
option, though, because it would essentially require that I recode large 
portions of LyX in Python. I'm resistant to doing that because it's a a lot of 
extra code to maintain. There are already two implementations of LyX document 
parsing libraries: eLyXer and that found in LyX itself. Adding a third and 
trying to keep it in some sort of synchronization would be a huge pain. I'm 
looking into using eLyXer for Word conversion from LyX, but that is lesser 
priority than making Word import work correctly. (At least at the moment.) If 
there is someone (maybe Alex or another eLyXer dev) who would be interest in 
collaborating and handling the export part, I'd be happy to coordinate with 
them so that we're able to round-trip.

 People will take this as a promise and complain that it does not
 work well enough.
 
 Well -  one could state that the round-trip works for MS word version
 abcd, and other versions can / will / might cause problems which are
 not in our hands.

I've already taken that position. I'm willing to work with Word versions 2007 
and 2010, and only files saved in docx. I'm not going to even try and parse doc 
binary files. word2lyx is about a 1000 lines of code. The doc parsing libraries 
I've looked at are easily 10 times that long. Python has excellent libraries 
for parsing XML that do nearly all the heavy lifting. I would have to write my 
own parsing library for doc.

 The difference of structure between word and lyx are too important
 to be able to work in a word-LyX collaboration IMO.
 
 There are obviously basic difference in how LyX and word are viewing
 documents, and these lead to principal differences how the files are
 saved.
 
 But I am thinking that if one can import a docx file into LyX, one
 should be able to do the reverse. And one should be able to define a
 robust subset of features which are maintained when doing a round-trip.
 In the same way that certain features are not converted in word2lyx,
 lyx2word would also only support a subset of features which are
 exported. But if these subsets include the most important features
 used in the editing process on both sides, a round trip should be
 possible.

I agree that it is possible, but there's a lot of code needed to make it work 
correctly. It's also a larger problem set that I want to right now. Once I've 
got the Word import working, including track changes and notes (and probably 
maths, too), I'll be more willing to come back and take a look at it.

But as I said earlier, if there's someone who would like to jump on board and 
work with Word export (lyx2word), I'll be happy to coordinate and work with 
them, too.

Cheers,

Rob

Re: word2lyx (Word to LyX Translator): Status Update

[Rob Oakes]

On Mar 8, 2012, at 2:21 AM, Rainer M Krug wrote:

> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA1

> I would love to agree, but round-trip is what is needed most of the
> time. An import word2lyx is perfect, but in most cases only half the
> story. I would use it extensively if the round trip is possible.
> Obviously, we can not deal with the word-editing side (whatever
> program is used for that).

I'm sympathetic to this point. I understand that having a way to go from one to 
the other is important. I've deliberately avoided creating an export to Word 
option, though, because it would essentially require that I recode large 
portions of LyX in Python. I'm resistant to doing that because it's a a lot of 
extra code to maintain. There are already two implementations of LyX document 
parsing libraries: eLyXer and that found in LyX itself. Adding a third and 
trying to keep it in some sort of synchronization would be a huge pain. I'm 
looking into using eLyXer for Word conversion from LyX, but that is lesser 
priority than making Word import work correctly. (At least at the moment.) If 
there is someone (maybe Alex or another eLyXer dev) who would be interest in 
collaborating and handling the export part, I'd be happy to coordinate with 
them so that we're able to round-trip.

>> People will take this as a promise and complain that it does not
>> work well enough.
> 
> Well -  one could state that the round-trip works for MS word version
> abcd, and other versions can / will / might cause problems which are
> not in our hands.

I've already taken that position. I'm willing to work with Word versions 2007 
and 2010, and only files saved in docx. I'm not going to even try and parse doc 
binary files. word2lyx is about a 1000 lines of code. The doc parsing libraries 
I've looked at are easily 10 times that long. Python has excellent libraries 
for parsing XML that do nearly all the heavy lifting. I would have to write my 
own parsing library for doc.

>> The difference of structure between word and lyx are too important
>> to be able to work in a word<->LyX collaboration IMO.
> 
> There are obviously basic difference in how LyX and word are viewing
> documents, and these lead to principal differences how the files are
> saved.
> 
> But I am thinking that if one can import a docx file into LyX, one
> should be able to do the reverse. And one should be able to define a
> robust subset of features which are maintained when doing a round-trip.
> In the same way that certain features are not converted in word2lyx,
> lyx2word would also only support a subset of features which are
> exported. But if these subsets include the most important features
> used in the editing process on both sides, a round trip should be
> possible.

I agree that it is possible, but there's a lot of code needed to make it work 
correctly. It's also a larger problem set that I want to right now. Once I've 
got the Word import working, including track changes and notes (and probably 
maths, too), I'll be more willing to come back and take a look at it.

But as I said earlier, if there's someone who would like to jump on board and 
work with Word export (lyx2word), I'll be happy to coordinate and work with 
them, too.

Cheers,

Rob

word2lyx: Word to LyX Document Converter

[Rob Oakes]

Dear Users and Developers,

First off, thank you to everyone who sent me documents over the weekend.
I was able to make a lot of tweaks to word2lyx based on what you sent me.

With that hurdle out of the way, I think it's ready to release it into
the wild. If you'd like to download a copy of it, you can download the
code from:

http://oak-tree.us/stuff/LyX/word2lyx-01.zip

A brief write-up of the features and usage can be found at:

http://www.oak-tree.us/2012/03/07/word2lyx01-2/

If you download it and find it useful, please let me know. If you
download it and have problems, also please let me know. (Mostly so I can
fix the problems.)

Cheers,

Rob

word2lyx: Word to LyX Document Converter

[Rob Oakes]

Dear Users and Developers,

First off, thank you to everyone who sent me documents over the weekend.
I was able to make a lot of tweaks to word2lyx based on what you sent me.

With that hurdle out of the way, I think it's ready to release it into
the wild. If you'd like to download a copy of it, you can download the
code from:

http://oak-tree.us/stuff/LyX/word2lyx-01.zip

A brief write-up of the features and usage can be found at:

http://www.oak-tree.us/2012/03/07/word2lyx01-2/

If you download it and find it useful, please let me know. If you
download it and have problems, also please let me know. (Mostly so I can
fix the problems.)

Cheers,

Rob

Import/Export Error Information (word2lyx)

[Rob Oakes]

Dear Developers,

I'm getting ready to publish the source for Word2LyX, but ran into one
last problem. I created an input filter/filetype to completely automate
the conversion of Word documents.

However, when run, I'm getting an error:

An error occurred while running:

python inputfile.docx outputfile.lyx

Is there any way to look at the debug output to try and track what might
be causing the error.

Cheers,

Rob

Import/Export Error Information (word2lyx)

[Rob Oakes]

Dear Developers,

I'm getting ready to publish the source for Word2LyX, but ran into one
last problem. I created an input filter/filetype to completely automate
the conversion of Word documents.

However, when run, I'm getting an error:

An error occurred while running:

python "inputfile.docx" "outputfile.lyx"

Is there any way to look at the debug output to try and track what might
be causing the error.

Cheers,

Rob

eLyXer for Document Parsing

[Rob Oakes]

Dear eLyXer Users and Developers,

I'm still at work on the import/export module for Microsoft Word documents. I'm 
making pretty good progress. I've got a rough prototype that works pretty well 
and I'm now starting to refine it.

My approach up to now has been to use regular expressions to match portions of 
the document and then use a library to translate those to the corresponding 
Word XML structures. It's working pretty well with my simple test documents.

Before going too far with this approach, though, I wanted to post (another 
general query).

In the eLyXer library, there is already a robust set of tools used for 
converting LyX documents to HTML. Does anyone know if the library is written in 
such as way that getting a generic in-memory representation of the document 
would be possible? It would be awesome to re-use as much existing code for the 
Word document export as possible. That would allow me to support a broader 
number of features, and gives me a framework for working with maths.

Any thoughts Alex (and others)? I've downloaded the sources and have begun to 
work through them, but before spending hours to days trying to wrap my head 
around them, I thought I would ask.

Cheers,

Rob

eLyXer for Document Parsing

[Rob Oakes]

Dear eLyXer Users and Developers,

I'm still at work on the import/export module for Microsoft Word documents. I'm 
making pretty good progress. I've got a rough prototype that works pretty well 
and I'm now starting to refine it.

My approach up to now has been to use regular expressions to match portions of 
the document and then use a library to translate those to the corresponding 
Word XML structures. It's working pretty well with my simple test documents.

Before going too far with this approach, though, I wanted to post (another 
general query).

In the eLyXer library, there is already a robust set of tools used for 
converting LyX documents to HTML. Does anyone know if the library is written in 
such as way that getting a generic in-memory representation of the document 
would be possible? It would be awesome to re-use as much existing code for the 
Word document export as possible. That would allow me to support a broader 
number of features, and gives me a framework for working with maths.

Any thoughts Alex (and others)? I've downloaded the sources and have begun to 
work through them, but before spending hours to days trying to wrap my head 
around them, I thought I would ask.

Cheers,

Rob

Re: Import into LyX

[Rob Oakes]

On Feb 2, 2012, at 8:55 AM, Richard Heck wrote:

 On 02/01/2012 03:11 PM, Xu Wang wrote:
 Hey Rob, that sounds like quite a nice project you have in mind!
 
 My two cents: it's not worth carrying it out if you can't get the math to 
 import somewhat well. That seems to be the biggest problem with most ways of 
 converting doc to lyx. I understand it's very difficult, but I think it's 
 also the most important.
 
 As far as I remember, the main complaint about writer2latex has been that it 
 produces ugly LaTeX. In the latest versions of LibreOffice, however, there is 
 an option to export Ultra-clean LaTeX, which works pretty well. Of course, 
 this relies upon Libre's importing Word's math well. So for math heavy 
 documents, that seems a good way to go at the moment.

That tracks with my experience, as well. You can set it for Ultra-clean, but 
even that has a lot of miscellaneous stuff that isn't really necessary and 
complicates import/export.

I'm currently researching solutions for Math and I may have found one using 
MathML. We currently support MathML creation inside of LyX, do we have a way to 
import MathML? If so, how is that done? Is a native library or something that 
we handle with Python?

Cheers,

Rob

Re: Import into LyX

[Rob Oakes]

Thank you everyone for the comments so far. I really appreciate hearing from 
others as it helps me to build out a more detailed use-case. In addition to the 
earlier questions, I have one more:

How important is support of .doc?

I know that it is the standard upon which the publishing industry is built, but 
… It's a real pain to parse. In contrast, docx (the default file format in Word 
2007 and 2010) is very parse. It's basically an XML document in a zipped folder 
with assets.

I've already got a working prototype that can take a very simple LyX document 
and converts it to docx. Here's what supported:

1.) Syles
2.) Images/Figures

Expanding this prototype is pretty easy. Trying to support doc is hard 
(painfully hard). There are pretty good import filters for OpenOffice and 
AbiWord for docx. docx is supported in Microsoft Word 2007 and 2010, and users 
of 2003 can download a plugin which is capable of reading it.

If I go ahead with support for docx, I think I can write a full featured 
import/export plugin, including:

1.) Bibliographies using Word's native format and (maybe) Endnote (I've found a 
python library that can parse BibTeX and building export for these two formats 
is do-able)
2.) Cross-references (I still need to figure out how this is done in Word, but 
so-far, the docx standard is pretty easy to follow)
3.) Comments and Change Tracking

How to deal with maths is still up in the air. LyX offers the ability to 
typeset nearly anything mathematical, which means there's a very large set of 
markup to support. Exporting to MathML might be one option, but that would 
require Word users to install a plugin. Exporting to Office Math Language (the 
new math language in Office 2007 and 2010) is another, but proprietary. 
Exporting to MathType is a third, which is both proprietary and requires that 
users install an add-in (which they have to pay for). I'm not particularly 
thrilled about any of the above. I'll continue to research and report what I 
find.

In the meantime, hearing about what features should be supported would be very 
nice. Hearing your opinions about doc support (versus only docx support) would 
also be very helpful.

Cheers,

Rob

Re: Import into LyX

[Rob Oakes]

On Feb 2, 2012, at 8:55 AM, Richard Heck wrote:

> On 02/01/2012 03:11 PM, Xu Wang wrote:
>> Hey Rob, that sounds like quite a nice project you have in mind!
>> 
>> My two cents: it's not worth carrying it out if you can't get the math to 
>> import somewhat well. That seems to be the biggest problem with most ways of 
>> converting doc to lyx. I understand it's very difficult, but I think it's 
>> also the most important.
>> 
> As far as I remember, the main complaint about writer2latex has been that it 
> produces ugly LaTeX. In the latest versions of LibreOffice, however, there is 
> an option to export "Ultra-clean" LaTeX, which works pretty well. Of course, 
> this relies upon Libre's importing Word's math well. So for math heavy 
> documents, that seems a good way to go at the moment.

That tracks with my experience, as well. You can set it for Ultra-clean, but 
even that has a lot of miscellaneous stuff that isn't really necessary and 
complicates import/export.

I'm currently researching solutions for Math and I may have found one using 
MathML. We currently support MathML creation inside of LyX, do we have a way to 
import MathML? If so, how is that done? Is a native library or something that 
we handle with Python?

Cheers,

Rob

Re: Import into LyX

[Rob Oakes]

Thank you everyone for the comments so far. I really appreciate hearing from 
others as it helps me to build out a more detailed use-case. In addition to the 
earlier questions, I have one more:

How important is support of .doc?

I know that it is the standard upon which the publishing industry is built, but 
… It's a real pain to parse. In contrast, docx (the default file format in Word 
2007 and 2010) is very parse. It's basically an XML document in a zipped folder 
with assets.

I've already got a working prototype that can take a very simple LyX document 
and converts it to docx. Here's what supported:

1.) Syles
2.) Images/Figures

Expanding this prototype is pretty easy. Trying to support doc is hard 
(painfully hard). There are pretty good import filters for OpenOffice and 
AbiWord for docx. docx is supported in Microsoft Word 2007 and 2010, and users 
of 2003 can download a plugin which is capable of reading it.

If I go ahead with support for docx, I think I can write a full featured 
import/export plugin, including:

1.) Bibliographies using Word's native format and (maybe) Endnote (I've found a 
python library that can parse BibTeX and building export for these two formats 
is do-able)
2.) Cross-references (I still need to figure out how this is done in Word, but 
so-far, the docx standard is pretty easy to follow)
3.) Comments and Change Tracking

How to deal with maths is still up in the air. LyX offers the ability to 
typeset nearly anything mathematical, which means there's a very large set of 
markup to support. Exporting to MathML might be one option, but that would 
require Word users to install a plugin. Exporting to Office Math Language (the 
new math language in Office 2007 and 2010) is another, but proprietary. 
Exporting to MathType is a third, which is both proprietary and requires that 
users install an add-in (which they have to pay for). I'm not particularly 
thrilled about any of the above. I'll continue to research and report what I 
find.

In the meantime, hearing about what features should be supported would be very 
nice. Hearing your opinions about doc support (versus only docx support) would 
also be very helpful.

Cheers,

Rob

Import into LyX

[Rob Oakes]

Dear Users and Developers,

Some time ago, I was experimenting with importing documents into LyX
(specifically about how to crack the import MS Word to LyX nut). In the
process, I got really excited about using OpenOffice to convert the word
document to HTML, running tidy on the HTML and then importing that way.
(The original blog article about this can be found at
http://blog.oak-tree.us/index.php/2010/05/14/msword-lyx-import.)

Since I'm (re)writing a book chapter about this topic, I thought that I
would look at alternative strategies for importing Word (and other file
formats) into LyX. While doing research, I came across a (potentially)
much better solution.

Somewhat recently (in 2010), a group of Python libraries were written
that handle document conversions. They are part of the epub-tools
library (http://code.google.com/p/epub-tools/). (I've been experimenting
with ePub document creation from LyX, which is how I found them.)

One of the tools in the library is able to parse Microsoft word
documents and convert them to XHTML in preparation for generating an
ePub file. I think that the tool can be adapted for directly converting
Word docs to LyX. Not to LaTeX and then to LyX, but /directly to LyX/.

I'm putting together a library to experiment with direct conversions
(this is ostensibly being done for the never-ending book project, but
will be released as open code), but before getting too deep into
development, I wanted to poll:

 1. Is this a tool that would prove useful to yourselves, your
collaborators, and others?
 2. What features would you consider essential?

(Right now, styles based conversion looks pretty easy -- going from
Heading 1 in Word to Chapter, for example. But I'm not sure how well
it would convert maths. This is something I'll still need to look
at, and may require writing an additional module.)

 3. What is the best tool to look at for guidance in creating a new
script for word2lyx? tex2lyx?
 4. Does the script need to support special cases, such as importing
Word track changes?
 5. Just how important do you consider round-tripping a document,
e.g., going from LyX to Word and back to LyX.
 6. Is there anyone who might be interested in collaborating on this?

Any thoughts would be greatly appreciated.

Cheers,

Rob Oakes

Import into LyX

[Rob Oakes]

Dear Users and Developers,

Some time ago, I was experimenting with importing documents into LyX
(specifically about how to crack the import MS Word to LyX nut). In the
process, I got really excited about using OpenOffice to convert the word
document to HTML, running tidy on the HTML and then importing that way.
(The original blog article about this can be found at
http://blog.oak-tree.us/index.php/2010/05/14/msword-lyx-import.)

Since I'm (re)writing a book chapter about this topic, I thought that I
would look at alternative strategies for importing Word (and other file
formats) into LyX. While doing research, I came across a (potentially)
much better solution.

Somewhat recently (in 2010), a group of Python libraries were written
that handle document conversions. They are part of the epub-tools
library (http://code.google.com/p/epub-tools/). (I've been experimenting
with ePub document creation from LyX, which is how I found them.)

One of the tools in the library is able to parse Microsoft word
documents and convert them to XHTML in preparation for generating an
ePub file. I think that the tool can be adapted for directly converting
Word docs to LyX. Not to LaTeX and then to LyX, but /directly to LyX/.

I'm putting together a library to experiment with direct conversions
(this is ostensibly being done for the never-ending book project, but
will be released as open code), but before getting too deep into
development, I wanted to poll:

 1. Is this a tool that would prove useful to yourselves, your
collaborators, and others?
 2. What features would you consider essential?

(Right now, styles based conversion looks pretty easy -- going from
Heading 1 in Word to Chapter, for example. But I'm not sure how well
it would convert maths. This is something I'll still need to look
at, and may require writing an additional module.)

 3. What is the best tool to look at for guidance in creating a new
script for word2lyx? tex2lyx?
 4. Does the script need to support special cases, such as importing
Word "track changes"?
 5. Just how important do you consider "round-tripping" a document,
e.g., going from LyX to Word and back to LyX.
 6. Is there anyone who might be interested in collaborating on this?

Any thoughts would be greatly appreciated.

Cheers,

Rob Oakes

Updated Kindle Tools

[Rob Oakes]

Dear LyX Users,

Given the recent discussion about publishing to the Kindle format, I
thought that this announcement might be of interest to you. Amazon just
recently updated its suite of KindleGen tools to support HTML5 and CSS.

http://www.amazon.com/gp/feature.html/ref=amb_link_357613402_1?ie=UTF8docId=1000765211pf_rd_m=ATVPDKIKX0DERpf_rd_s=center-3pf_rd_r=0VB74HFFHHNP48JT0NZ3pf_rd_t=1401pf_rd_p=1343256902pf_rd_i=1000729511

The main tool, KindleGen, is both cross platform and command line, which
means that it might be possible to add converters to LyX. One of the
input formats is HTML/XHTML. It would probably be pretty easy to add a
converter for it.

Cheers,

Rob

Updated Kindle Tools

[Rob Oakes]

Dear LyX Users,

Given the recent discussion about publishing to the Kindle format, I
thought that this announcement might be of interest to you. Amazon just
recently updated its suite of KindleGen tools to support HTML5 and CSS.

http://www.amazon.com/gp/feature.html/ref=amb_link_357613402_1?ie=UTF8=1000765211_rd_m=ATVPDKIKX0DER_rd_s=center-3_rd_r=0VB74HFFHHNP48JT0NZ3_rd_t=1401_rd_p=1343256902_rd_i=1000729511

The main tool, KindleGen, is both cross platform and command line, which
means that it might be possible to add converters to LyX. One of the
input formats is HTML/XHTML. It would probably be pretty easy to add a
converter for it.

Cheers,

Rob

LyX Dependencies - Microsoft Windows

[Rob Oakes]

Dear Developers,

Do you know if anyone has compiled the dependencies for mingw64 bit? I
finally realized that I'm getting linker errors because the dependencies
were compiled with an older, 32 bit version of the mingw. Do you know if
there is a set of dependencies that have been compiled for mingw64?
Alternatively, are there any instructions for how to compile them from
source?

I looked briefly at a few of the source packages (libiconv and libintl),
and it looked like they required autotools (which I suppose would also
need msys and all that). Any ideas would be appreciated.

Cheers,

Rob

LyX Dependencies - Microsoft Windows

[Rob Oakes]

Dear Developers,

Do you know if anyone has compiled the dependencies for mingw64 bit? I
finally realized that I'm getting linker errors because the dependencies
were compiled with an older, 32 bit version of the mingw. Do you know if
there is a set of dependencies that have been compiled for mingw64?
Alternatively, are there any instructions for how to compile them from
source?

I looked briefly at a few of the source packages (libiconv and libintl),
and it looked like they required autotools (which I suppose would also
need msys and all that). Any ideas would be appreciated.

Cheers,

Rob

Compile Errors with MinGW-w64

[Rob Oakes]

Dear Developers,

I had a few minutes this morning so I thought that I would try and fix
some bugs related to XHTML output. However, I don't have access to my
normal laptop and was working on a Windows box with a 64 build of Qt
4.8, QtCreator (x64) and the mingw-w64 compiler.

In the process of trying to get the source to compile, I've run into a
number of problems. Some of these were fairly simple to resolve, such as
substituting intptr_t for int in a couple of places, but I've run into
one error that I can't seem to figure out. Here's the compiler output:

...src\support\ForkedCalls.cpp:-1: In member function 'void
lyx::support::unnamed::Murder::kill()':
...src\support\ForkedCalls.cpp:78: error: ambiguous overload for
'operator' in 'lyx::operator(((lyx::LyXErr)( lyx::lyxerr)),
((const char*)Killed )) 
((lyx::support::unnamed::Murder*)this)-lyx::support::unnamed::Murder::pid_'
...src\support\debug.h:190: note: lyx::LyXErr
lyx::operator(lyx::LyXErr, const char*) near match
...support\debug.h:192: note: lyx::LyXErr
lyx::operator(lyx::LyXErr, char*) near match

in reference to this code (src/support/ForkedCalls.cpp, line 74):

voidkill()

{

if (pid_ != 0)

support::kill(pid_, SIGKILL);

lyxerr  Killed   pid_  endl;

delete this;

}


It then gives a bunch of templates (defined in debug.h) that are near
matches:

LyXErroperator(LyXErr,voidconst*);

LyXErr  operator(LyXErr , const char *);

LyXErr  operator(LyXErr , char);

LyXErr  operator(LyXErr , char *);

LyXErr  operator(LyXErr , int);

LyXErr  operator(LyXErr , unsigned int);

LyXErr  operator(LyXErr , long);

LyXErr  operator(LyXErr , unsigned long);

LyXErr  operator(LyXErr , double);

LyXErr  operator(LyXErr , std::string const );

LyXErr  operator(LyXErr , docstring const );

I've tried to Google the source of this error, but can't figure it. The
code compiles fine on Mac OS X, Linux, and 32 bit Windows mingw.
Moreover, there are templates for const char *, and char * so I don't
understand why it's not happy. The version of gmake is 3.82, the version
of the compiler is 4.5.4.

Is there something obvious that I'm missing?

Cheers,

Rob

Re: Compile Errors with MinGW-w64

[Rob Oakes]

On 1/9/2012 3:03 PM, Jean-Marc Lasgouttes wrote:
 Le 09/01/12 19:19, Rob Oakes a écrit :
 I've tried to Google the source of this error, but can't figure it. The
 code compiles fine on Mac OS X, Linux, and 32 bit Windows mingw.
 Moreover, there are templates for const char *, and char * so I don't
 understand why it's not happy. The version of gmake is 3.82, the version
 of the compiler is 4.5.4.

 What is the type of pid_?

 JMarc

Hi JMarc,

Thanks for the hint! pid_ uses pid_t as its type. I was able to resolve
the compiler errors by adding:

LyXErroperator(LyXErr,pid_t);

to debug.h and:

LyXErroperator(LyXErr,size_t);

to resolve the compiler issues with menus.

I've run into some other problems while linking, though. Some of the
.obj files look like they're not getting generated. For example:
AspellChecker.cpp.obj, Buffer.cpp.obj, BufferView.cpp.obj,
Cursor.cpp.obj, DocIterator.cpp.obj, KeyMap.cpp.obj, lyxfind.cpp.obj,
Paragraph.cpp.obj, and about a dozen others. This is leading to a lot of
undefined reference to ..., which makes me wonder if there's a problem
with the makefiles ... or something.

I'll try clearing out my build directory and reconfiguring, to see if
that clears things up. Any other thoughts for things that I might try?

Cheers,

Rob

Compile Errors with MinGW-w64

[Rob Oakes]

Dear Developers,

I had a few minutes this morning so I thought that I would try and fix
some bugs related to XHTML output. However, I don't have access to my
normal laptop and was working on a Windows box with a 64 build of Qt
4.8, QtCreator (x64) and the mingw-w64 compiler.

In the process of trying to get the source to compile, I've run into a
number of problems. Some of these were fairly simple to resolve, such as
substituting intptr_t for int in a couple of places, but I've run into
one error that I can't seem to figure out. Here's the compiler output:

...src\support\ForkedCalls.cpp:-1: In member function 'void
lyx::supportMurder::kill()':
...src\support\ForkedCalls.cpp:78: error: ambiguous overload for
'operator<<' in 'lyx::operator<<(((lyx::LyXErr&)(& lyx::lyxerr)),
((const char*)"Killed ")) <<
((lyx::supportMurder*)this)->lyx::supportMurder::pid_'
...src\support\debug.h:190: note: lyx::LyXErr&
lyx::operator<<(lyx::LyXErr&, const char*) 
...support\debug.h:192: note: lyx::LyXErr&
lyx::operator<<(lyx::LyXErr&, char*) 

in reference to this code (src/support/ForkedCalls.cpp, line 74):

voidkill()

{

if (pid_ != 0)

support::kill(pid_, SIGKILL);

lyxerr << "Killed " << pid_ << endl;

delete this;

}


It then gives a bunch of templates (defined in debug.h) that are near
matches:

LyXErr<<(LyXErr&,voidconst*);

LyXErr & operator<<(LyXErr &, const char *);

LyXErr & operator<<(LyXErr &, char);

LyXErr & operator<<(LyXErr &, char *);

LyXErr & operator<<(LyXErr &, int);

LyXErr & operator<<(LyXErr &, unsigned int);

LyXErr & operator<<(LyXErr &, long);

LyXErr & operator<<(LyXErr &, unsigned long);

LyXErr & operator<<(LyXErr &, double);

LyXErr & operator<<(LyXErr &, std::string const &);

LyXErr & operator<<(LyXErr &, docstring const &);

I've tried to Google the source of this error, but can't figure it. The
code compiles fine on Mac OS X, Linux, and 32 bit Windows mingw.
Moreover, there are templates for const char *, and char * so I don't
understand why it's not happy. The version of gmake is 3.82, the version
of the compiler is 4.5.4.

Is there something obvious that I'm missing?

Cheers,

Rob

Re: Compile Errors with MinGW-w64

[Rob Oakes]

On 1/9/2012 3:03 PM, Jean-Marc Lasgouttes wrote:
> Le 09/01/12 19:19, Rob Oakes a écrit :
>> I've tried to Google the source of this error, but can't figure it. The
>> code compiles fine on Mac OS X, Linux, and 32 bit Windows mingw.
>> Moreover, there are templates for const char *, and char * so I don't
>> understand why it's not happy. The version of gmake is 3.82, the version
>> of the compiler is 4.5.4.
>
> What is the type of pid_?
>
> JMarc
>
Hi JMarc,

Thanks for the hint! pid_ uses pid_t as its type. I was able to resolve
the compiler errors by adding:

LyXErr<<(LyXErr&,pid_t);

to debug.h and:

LyXErr<<(LyXErr&,size_t);

to resolve the compiler issues with menus.

I've run into some other problems while linking, though. Some of the
.obj files look like they're not getting generated. For example:
AspellChecker.cpp.obj, Buffer.cpp.obj, BufferView.cpp.obj,
Cursor.cpp.obj, DocIterator.cpp.obj, KeyMap.cpp.obj, lyxfind.cpp.obj,
Paragraph.cpp.obj, and about a dozen others. This is leading to a lot of
"undefined reference to ...", which makes me wonder if there's a problem
with the makefiles ... or something.

I'll try clearing out my build directory and reconfiguring, to see if
that clears things up. Any other thoughts for things that I might try?

Cheers,

Rob

Re: LyX Chat

[Rob Oakes]

On 1/2/2012 9:00 AM, Abdelrazak Younes wrote:
 On 02/01/2012 16:53, Tommaso Cucinotta wrote:
 Unless we buy some server I don't think this will be possible. Maybe
 someone wants to deliver this kind of services for free or not... it
 does not have to be a LyX.org service I guess.

If we're just looking for some hardware, there's lots of options we can
pursue. This might include the purchase of a cheap server to looking for
a university or company to donate resources. Alternatively, we might
look at setting up a Google apps instance or an AWS account. In the era
of the cloud, the cost of computing resources is near zero and I don't
think we need to let that stop us from pursuing development.

In the case of donating services, I've got access to several servers
that might be used for development/testing. Depending on what the
dependencies are, I might even be able to donate bandwidth and hosting
(though I don't think we're quite ready for that conversation yet). I
would need to talk with the primary developer about the planned
implementation.

Before that, though, we should talk about what the goals for such a
system are. Is this sort of collaboration server the sort of thing that
we want to develop as a LyX specific resource, or are there similar
systems that already exist? Would it be possible to integrate with
Jabber or another messaging system to connect users and pass messages?
What about OpenFire?

For that matter, is this a premium service that we would want to develop
to create a revenue model for LyX? Would we want a generic
communications client that would enable users to connect to any
collaboration server and interactively work on documents? (Using an
existing solution would seem to me a preferable option than trying to
develop a server side app on our own. Less code to maintain.)

As I said earlier, I don't think that hardware resources should derail
this project. There are lots of ways we could provide them. If there is
someone who is very serious about starting development, I would be happy
to get the basics in place (testing server, shell account, Jabber
instance, etc.) But I'm more interested in hearing how we think the
system should function.

To that end, I'd prefer any collaborative writing features to:

1.) Use an existing open source messaging protocol like Jabber (which
means that any XMPP server could be used for collaboration). Passing
information about geometry, actions, etc. seems like it would be pretty
straightforward.
2.) Reuse external tools and libraries as much as possible (e.g.
libpurple), limiting the LyX-specific code we would need to develop and
maintain.
3.) Remain as open as possible, meaning that we don't tie users to a
specific service. Using a generic solution, like Jabber would be ideal.
(Though running a server through LyX.org might be a good way to raise
funds for LyX development.)

Cheers,

Rob

Re: LyX Chat

[Rob Oakes]

On 1/2/2012 9:00 AM, Abdelrazak Younes wrote:
> On 02/01/2012 16:53, Tommaso Cucinotta wrote:
> Unless we buy some server I don't think this will be possible. Maybe
> someone wants to deliver this kind of services for free or not... it
> does not have to be a LyX.org service I guess.

If we're just looking for some hardware, there's lots of options we can
pursue. This might include the purchase of a cheap server to looking for
a university or company to donate resources. Alternatively, we might
look at setting up a Google apps instance or an AWS account. In the era
of the cloud, the cost of computing resources is near zero and I don't
think we need to let that stop us from pursuing development.

In the case of donating services, I've got access to several servers
that might be used for development/testing. Depending on what the
dependencies are, I might even be able to donate bandwidth and hosting
(though I don't think we're quite ready for that conversation yet). I
would need to talk with the primary developer about the planned
implementation.

Before that, though, we should talk about what the goals for such a
system are. Is this sort of collaboration server the sort of thing that
we want to develop as a LyX specific resource, or are there similar
systems that already exist? Would it be possible to integrate with
Jabber or another messaging system to connect users and pass messages?
What about OpenFire?

For that matter, is this a premium service that we would want to develop
to create a revenue model for LyX? Would we want a generic
communications client that would enable users to connect to any
collaboration server and interactively work on documents? (Using an
existing solution would seem to me a preferable option than trying to
develop a server side app on our own. Less code to maintain.)

As I said earlier, I don't think that hardware resources should derail
this project. There are lots of ways we could provide them. If there is
someone who is very serious about starting development, I would be happy
to get the basics in place (testing server, shell account, Jabber
instance, etc.) But I'm more interested in hearing how we think the
system should function.

To that end, I'd prefer any collaborative writing features to:

1.) Use an existing open source messaging protocol like Jabber (which
means that any XMPP server could be used for collaboration). Passing
information about geometry, actions, etc. seems like it would be pretty
straightforward.
2.) Reuse external tools and libraries as much as possible (e.g.
libpurple), limiting the LyX-specific code we would need to develop and
maintain.
3.) Remain as open as possible, meaning that we don't tie users to a
specific service. Using a generic solution, like Jabber would be ideal.
(Though running a server through LyX.org might be a good way to raise
funds for LyX development.)

Cheers,

Rob

Modifying HTML for Figure Labels, Wraps

[Rob Oakes]

Dear Developers,

I've been working on modifying the XHTML created by LyX to be more compliant 
with HTML5. In the process, though, I've run into a couple of snags, 
particularly with wrap floats and figure labels.

In way of experimentation, I would like to use the figure tag for figures and 
tables and the figcaption tag for figure captions. However, I'm not able to 
get LyX to respect the HTMLTag argument I'm providing. 

It should looks like this:

figure class='wrap'
...
/figure

Instead, it still uses the default:

div class='wrap'
...
/div

Any idea on what I might be doing wrong? Here are the layout commands I'm using:

InsetLayout Wrap
HTMLTag figure
HTMLAttrclass='wrap'
HTMLInnerTagp
HTMLInnerAttr   class='figure-text'
HTMLStyle
.wrap {
float: right;
page-break-inside: avoid;
padding: 1ex;
margin: 1ex;
}
EndHTMLStyle
End

There is a similar story for figcaption:

InsetLayout Caption
HTMLTag figcaption
HTMLInnerTagp
HTMLInnerAttr   class='figure-text'
HTMLStyle
figcaption {
page-break-before: avoid;
}
EndHTMLStyle
End

It ignores the HTMLTag I'm feeding it and defaults to div.

Strangely, though, it uses the HTMLInnerTag that I specify. Any thoughts would 
be greatly appreciated.

Cheers,

Rob

Suppressing Inner Tags

[Rob Oakes]

Dear Developers,

While working on the HTML5 definitions I mentioned in the previous message, I 
had another question about the XHTML output. Is there a way to suppress div 
tags within InsetLayout insets? Frequently, I'm finding that the LyX XHTML will 
open and close a bunch of random, unnecessary tags, whether it be p or div. 
They will sometimes be nested three or four deep.

For example, if you have a standard paragraph inside of a float, LyX will open 
one div because of the inner tag and then a second div for the paragraph 
environment. This seems slightly redundant. Any thoughts on how to make the 
output a little cleaner?

Cheers,

Rob

PS, I'm working with trunk from SVN.

Re: Suppressing Inner Tags

[Rob Oakes]

On Dec 15, 2011, at 10:48 AM, Richard Heck wrote:

 While working on the HTML5 definitions I mentioned in the previous message, 
 I had another question about the XHTML output. Is there a way to suppress 
 div tags within InsetLayout insets? Frequently, I'm finding that the LyX 
 XHTML will open and close a bunch of random, unnecessary tags, whether it 
 bep  ordiv. They will sometimes be nested three or four deep.
 
 For example, if you have a standard paragraph inside of a float, LyX will 
 open onediv  because of the inner tag and then a seconddiv  for the 
 paragraph environment. This seems slightly redundant. Any thoughts on how to 
 make the output a little cleaner?

 If you can post an example file, we can figure out what's happening.

Here's a relatively simple example from a table wrap:

div class='wrap' style='width: 45%;'
p class='figure-text'
div class=plain_layout
a id='magicparlabel-308' /
div class='float-caption float-caption-table'Table 2.1:p 
class='figure-text'div class=plain_layouta id='magicparlabel-312' /
a id=table_Landing_Conditions /
...
/div
/p
/div
/div

div class=plain_layout style='text-align: center;'a id='magicparlabel-317' 
/
table
...
/table
/div
div style='height:3ex'/div
/p
/div
/p

I've done some modification of the tags (which is why you see p elements, if 
these weren't present, you would see div tags). There seems to be several 
levels of unnecessary HTML.

div for the table wrap, p for the first paragraph. div for the structure 
enclosing the label and caption. div for the label itself, etc.

Any ideas on how I might clean this up, what I'd really like to see in the 
output is:

figure
captiondiv class='float-caption-tableTable 2.1:/div Table 
Caption/caption
table
...
/table
/figure

plus any necessary anchors required to link to the material. 

I think some of the unnecessary HTML here might be due to the outer tag, and 
inner-tag mechanism. Is there a way to suppress inner tags in cases where they 
aren't desired or wanted?

Cheers,

Rob

Modifying HTML for Figure Labels, Wraps

[Rob Oakes]

Dear Developers,

I've been working on modifying the XHTML created by LyX to be more compliant 
with HTML5. In the process, though, I've run into a couple of snags, 
particularly with wrap floats and figure labels.

In way of experimentation, I would like to use the  tag for figures and 
tables and the  tag for figure captions. However, I'm not able to 
get LyX to respect the HTMLTag argument I'm providing. 

It should looks like this:


...


Instead, it still uses the default:


...


Any idea on what I might be doing wrong? Here are the layout commands I'm using:

InsetLayout Wrap
HTMLTag figure
HTMLAttrclass='wrap'
HTMLInnerTagp
HTMLInnerAttr   class='figure-text'
HTMLStyle
.wrap {
float: right;
page-break-inside: avoid;
padding: 1ex;
margin: 1ex;
}
EndHTMLStyle
End

There is a similar story for figcaption:

InsetLayout Caption
HTMLTag figcaption
HTMLInnerTagp
HTMLInnerAttr   class='figure-text'
HTMLStyle
figcaption {
page-break-before: avoid;
}
EndHTMLStyle
End

It ignores the HTMLTag I'm feeding it and defaults to .

Strangely, though, it uses the HTMLInnerTag that I specify. Any thoughts would 
be greatly appreciated.

Cheers,

Rob

Suppressing Inner Tags

[Rob Oakes]

Dear Developers,

While working on the HTML5 definitions I mentioned in the previous message, I 
had another question about the XHTML output. Is there a way to suppress div 
tags within InsetLayout insets? Frequently, I'm finding that the LyX XHTML will 
open and close a bunch of random, unnecessary tags, whether it be  or . 
They will sometimes be nested three or four deep.

For example, if you have a standard paragraph inside of a float, LyX will open 
one  because of the inner tag and then a second  for the paragraph 
environment. This seems slightly redundant. Any thoughts on how to make the 
output a little cleaner?

Cheers,

Rob

PS, I'm working with trunk from SVN.

Re: Suppressing Inner Tags

[Rob Oakes]

On Dec 15, 2011, at 10:48 AM, Richard Heck wrote:

>> While working on the HTML5 definitions I mentioned in the previous message, 
>> I had another question about the XHTML output. Is there a way to suppress 
>> div tags within InsetLayout insets? Frequently, I'm finding that the LyX 
>> XHTML will open and close a bunch of random, unnecessary tags, whether it 
>> be  or. They will sometimes be nested three or four deep.
>> 
>> For example, if you have a standard paragraph inside of a float, LyX will 
>> open one  because of the inner tag and then a second  for the 
>> paragraph environment. This seems slightly redundant. Any thoughts on how to 
>> make the output a little cleaner?

> If you can post an example file, we can figure out what's happening.

Here's a relatively simple example from a table wrap:





Table 2.1:

...







...







I've done some modification of the tags (which is why you see  elements, if 
these weren't present, you would see  tags). There seems to be several 
levels of unnecessary HTML.

 for the table wrap,  for the first paragraph.  for the structure 
enclosing the label and caption.  for the label itself, etc.

Any ideas on how I might clean this up, what I'd really like to see in the 
output is:

Re: HTML Footnotes as Endnotes

[Rob Oakes]

On 12/10/2011 1:42 PM, Guenter Milde wrote:
 However, we should also think about graceful degradation, i.e. the
 document should be rendered sensibly in a browser that does not
 understand the newest standards.
 http://www.cs.tut.fi/~jkorpela/html/augm.html Günter 
I agree, most users shouldn't know whether the book is ePub 1, 2, or 3.
For most readers, as long as we provide CSS for the elements, they will
render correctly. (That is the beauty of XHTML/HTML5.) They won't get
all of the benefits of an HTML5 reader, but neither will it destroy the
experience.

Cheers,

Rob

Re: HTML Footnotes as Endnotes

[Rob Oakes]

On 12/10/2011 1:42 PM, Guenter Milde wrote:
> However, we should also think about "graceful degradation", i.e. the
> document should be rendered "sensibly" in a browser that does not
> understand the newest standards.
> http://www.cs.tut.fi/~jkorpela/html/augm.html Günter 
I agree, most users shouldn't know whether the book is ePub 1, 2, or 3.
For most readers, as long as we provide CSS for the elements, they will
render correctly. (That is the beauty of XHTML/HTML5.) They won't get
all of the benefits of an HTML5 reader, but neither will it destroy the
experience.

Cheers,

Rob

Re: HTML Footnotes as Endnotes

[Rob Oakes]

On Dec 9, 2011, at 8:03 AM, rgheck wrote:

 
 I've been thinking a fair bit about this and about the split by chapter 
 issue. I'm pretty sure now that what I'll need to do for labels, etc, to get 
 cross-file links working, is about the same as what I'd need to do for 
 footnotes to get them to print at the end of each chapter. So you can 
 probably go back to using the TOC for now.
 
 The downside to this is that it will give us a bit less flexibility than 
 collecting them as we go, but that is starting to look messy.

I've got  a tentative patch that allows me to do this, which is attached. I 
decided to use the exportdata class to collect the notes. It needs some love 
and attention, but it's working correctly. I'm in the process of testing 
endnote collection in files with multiple chapters. My first test file ran 
through without problems.

Any comments appreciated. Especially if he causes problems with what you're 
planning.

Cheers,

Rob

PS, sorry for the deluge of emails. I found myself confused, which means I was 
overlooking a relatively obvious solution.



htmlendnotes.diff
Description: Binary data

Re: HTML Footnotes as Endnotes

[Rob Oakes]

On Dec 8, 2011, at 10:06 AM, Richard Heck wrote:

 LyX already provides for two and three. With some tweaking, the XHTML that 
 comes out can be very clean and the LyXHTML can be customized via the 
 document or Internal layout file.
 
 Let me know where things need improving re (1) ... For 2.1, perhaps we should 
 be thinking in terms of HTML5 rather than
 XHTML altogether?

I've been putting together a list of HTML5 elements that I think we might 
consider using as default. (Mostly using the ePub 3 spec and a guide to HTML5 
best practices. )

Here is a list of the HTML semantic elements that I think we should use as 
defaults. (The nice thing about XHTML/HTML5 is that it degrades pretty 
gracefully to XHTML.)

Inset-Name  Element Comments


Paragraph*  p Use p 
by default, currently use div
Quote   blockquoteAlready used
Caption*caption   Table Captions, 
currently use div
EmphasisemAlready used
Figure* figure
Currently use div
Figure Caption* figcaptionCurrently use div
Part Title  h1
Chapter Titles  h2
Section Titles  h3
Subsection Titles   h4
Subsubsec   h5
Paragraph   h6
Maps*   map   Might be added 
for interactive electronic content
LyX Codecode  
Code Character Stylecode
Video   video
Audio   audio
Title   title Already 
Use
Footnotefootnote  Currently use 
div
 aside
Author  author
Contact Information address

There are probably others as well. I think in general, that we should use first 
level semantic tags wherever possible and rely less on div with id/class 
attributes.

 I have also been thinking that the inset for printing endnotes might
 well be useful for LaTeX, as a replacement of sorts for the Foot To End
 module. If it's there, we write footnotes as endnotes.

I agree I'll look into implementing it. How does the current module work? Is it 
possible to have per chapter end notes? Does that require a completely separate 
approach to a single list of EndNotes?

 PS, on another note, I'm pretty excited about the export of splitting the 
 HTML file at the chapter or section level. Is there anything that I can do 
 to help facilitate it?
 
 I've been thinking about this and actually think it isn't that hard to
 do. I'll get to it once the semester is over.

Really looking forward to this. The ability to split HTML output over multiple 
files will be a huge step forward when creating eBooks. It's not hard to split 
the files, per se, but creating them with logical breaks will be a huge 
improvement over doing it manually.

 5.) Allow for images, equations and other assets to be moved to a logical 
 subfolder hierarchy (unknown)
 
 The only reason I didn't do this originally was because I was lazy. It
 should be hard, I wouldn't think. The main question is what counts as a
 logical subfolder hierarchy.

I think we might follow the recommended ePub guidelines for the resources 
directory. In the top level directory, you place the XHTML files. Under that, 
there are separate directories for each major class of assets: images, scripts, 
fonts, etc. For the native LyX output, it seems like we might want to create 
two folders: images and equations (if not outputting via MathML).

So, maybe something like this:

root - html files
| images
| equations
| ...

It would also be really nice if we had the ability to specify where the root 
html folder should be.

 6.) Create a specialized inset which is able to insert audio and video using 
 the audio/video tags (unknown)
 
 This ought to be do-able via InsetExternal, though at the moment there
 is no HTML support for that inset. But this was just because I didn't
 see what it would be good for. I guess we would need to extend the
 template description language to include some way of specifying the HTML
 tags used with this type of file. One could more generally allow for
 conversion, though maybe we do not need to do that at the beginning.

Also really excited about and that everything seems to be coming together so 
well. Really cool!

Cheers,

Rob

Re: HTML Footnotes as Endnotes

[Rob Oakes]

On Dec 9, 2011, at 8:03 AM, rgheck wrote:

> 
> I've been thinking a fair bit about this and about the "split by chapter" 
> issue. I'm pretty sure now that what I'll need to do for labels, etc, to get 
> cross-file links working, is about the same as what I'd need to do for 
> footnotes to get them to print at the end of each chapter. So you can 
> probably go back to using the TOC for now.
> 
> The downside to this is that it will give us a bit less flexibility than 
> collecting them as we go, but that is starting to look messy.

I've got  a tentative patch that allows me to do this, which is attached. I 
decided to use the exportdata class to collect the notes. It needs some love 
and attention, but it's working correctly. I'm in the process of testing 
endnote collection in files with multiple chapters. My first test file ran 
through without problems.

Any comments appreciated. Especially if he causes problems with what you're 
planning.

Cheers,

Rob

PS, sorry for the deluge of emails. I found myself confused, which means I was 
overlooking a relatively obvious solution.



htmlendnotes.diff
Description: Binary data

Re: HTML Footnotes as Endnotes

[Rob Oakes]

On Dec 8, 2011, at 10:06 AM, Richard Heck wrote:

>> LyX already provides for two and three. With some tweaking, the XHTML that 
>> comes out can be very clean and the LyXHTML can be customized via the 
>> document or Internal layout file.
>> 
> Let me know where things need improving re (1) ... For 2.1, perhaps we should 
> be thinking in terms of HTML5 rather than
> XHTML altogether?

I've been putting together a list of HTML5 elements that I think we might 
consider using as default. (Mostly using the ePub 3 spec and a guide to HTML5 
best practices. )

Here is a list of the HTML semantic elements that I think we should use as 
defaults. (The nice thing about XHTML/HTML5 is that it degrades pretty 
gracefully to XHTML.)

Inset-Name  Element Comments


Paragraph*   Use  
by default, currently use 
Quote   Already used
Caption*   Table Captions, 
currently use 
EmphasisAlready used
Figure* 
Currently use 
Figure Caption* Currently use 
Part Title  
Chapter Titles  
Section Titles  
Subsection Titles   
Subsubsec   
Paragraph   
Maps*  Might be added 
for interactive electronic content
LyX Code  
Code Character Style
Video   
Audio   
TitleAlready 
Use
Footnote  Currently use 

 
Author  
Contact Information 

There are probably others as well. I think in general, that we should use first 
level semantic tags wherever possible and rely less on  with id/class 
attributes.

> I have also been thinking that the inset for printing endnotes might
> well be useful for LaTeX, as a replacement of sorts for the Foot To End
> module. If it's there, we write footnotes as endnotes.

I agree I'll look into implementing it. How does the current module work? Is it 
possible to have per chapter end notes? Does that require a completely separate 
approach to a single list of EndNotes?

>> PS, on another note, I'm pretty excited about the export of splitting the 
>> HTML file at the chapter or section level. Is there anything that I can do 
>> to help facilitate it?
>> 
> I've been thinking about this and actually think it isn't that hard to
> do. I'll get to it once the semester is over.

Really looking forward to this. The ability to split HTML output over multiple 
files will be a huge step forward when creating eBooks. It's not hard to split 
the files, per se, but creating them with logical breaks will be a huge 
improvement over doing it manually.

>> 5.) Allow for images, equations and other assets to be moved to a logical 
>> subfolder hierarchy (unknown)
>> 
> The only reason I didn't do this originally was because I was lazy. It
> should be hard, I wouldn't think. The main question is what counts as a
> "logical subfolder hierarchy".

I think we might follow the recommended ePub guidelines for the resources 
directory. In the top level directory, you place the XHTML files. Under that, 
there are separate directories for each major class of assets: images, scripts, 
fonts, etc. For the native LyX output, it seems like we might want to create 
two folders: images and equations (if not outputting via MathML).

So, maybe something like this:

root - html files
| images
| equations
| ...

It would also be really nice if we had the ability to specify where the root 
html folder should be.

>> 6.) Create a specialized inset which is able to insert audio and video using 
>> the audio/video tags (unknown)
>> 
> This ought to be do-able via InsetExternal, though at the moment there
> is no HTML support for that inset. But this was just because I didn't
> see what it would be good for. I guess we would need to extend the
> template description language to include some way of specifying the HTML
> tags used with this type of file. One could more generally allow for
> conversion, though maybe we do not need to do that at the beginning.

Also really excited about and that everything seems to be coming together so 
well. Really cool!

Cheers,

Rob

Re: HTML Footnotes as Endnotes

[Rob Oakes]

Hi Guenter, 

On Dec 7, 2011, at 1:14 AM, Guenter Milde wrote:

 IMV, we should make the placement of footnotes in HTML configurable. 
 
 Sensible options include
 
 a) inline (as tooltip via CSS :mouseover:)
 b) in the footer for page-based output (e.g. print from HTML or ePub)
 c) endnotes (before/after other concluding sections and listings?)
 d) per chapter/section/subsection
 e) author-specified position via an inset

I'd really like this too, but I'm not sure what the best way to implement it 
is. Putting HTML footnotes at the end (or collecting them as Richard as 
suggested and I'm currently working on a patch for) is pretty straightforward 
to code. (And I'm a pretty modest C++ coder.)

The reason I like is that I can easily customize and control the output using 
the CSS styles in the layouts. Yesterday, I spent all day at a publishing 
conference talking to in-house programmers/designers and they identified a 
couple of things they would want to see:

1.) the XHTML should be as clean as possible, following XHTML5 and ePub3 
conventions
2.) CSS styling should be as transparent as possible, with intelligent defaults
3.) It's very important that it be easy to customize

LyX already provides for two and three. With some tweaking, the XHTML that 
comes out can be very clean and the LyXHTML can be customized via the document 
or Internal layout file.

I can foresee that a designer might want to tweak the CSS regardless of which 
export option they are using, but what is the best way to do this? In the 
layout, we can only specify one category of CSS that will get applied, 
regardless of which export option is used. It seems like the other approaches 
would require hard coding it, which seems like a less effective approach.

Anyway ... I'll continue work on collecting footnotes and send a patch a little 
bit later today.

Cheers,

Rob

PS, on another note, I'm pretty excited about the export of splitting the HTML 
file at the chapter or section level. Is there anything that I can do to help 
facilitate it?

There are six big challenges I see to getting really excellent HTML/eBook 
output from LyX :

1.) Clean up the HTML tags to follow HTML5 conventions (easy, configurable 
through layouts, in progress)
2.) Export CSS styles as a separate file (done)
3.) Allow for more flexible handling of footnotes (in progress, moderately 
difficult)
4.) Split large HTML documents into more manageable chunks, perhaps at the 
chapter level (unknown)
5.) Allow for images, equations and other assets to be moved to a logical 
subfolder hierarchy (unknown)
6.) Create a specialized inset which is able to insert audio and video using 
the audio/video tags (unknown)

And that's really it. The rest of the ePub processing can be using external 
python modules.

Re: HTML Footnotes as Endnotes

[Rob Oakes]

Hi Richard, 

Just trying to implement the gathering of footnotes via std::list. I placed the 
structure in the LatexFeatures header, however, I didn't see any way to access 
LatexFeatures via OutputParams (e.g. op.features ...). Am I missing something 
obvious?

Cheers,

Rob

Re: HTML Footnotes as Endnotes

[Rob Oakes]

Hi Jean-Marc,

Could you point me at an example of how this might be done or provide a few 
lines of code? I'm trying to add a std::list object to OutputParams and getting 
a wide variety of strange errors. Apparently, including it doesn't like it when 
I include InsetFoot.h. I was able to create the list in LaTeX features 
without trouble, though.

I know very little about how LyX's validate methods work, though.

Cheers,

Rob

Compile Errors in OutputParams

[Rob Oakes]

Dear LyX Developers,

I'm trying to add a simple list objet to the OutputParams header file, but I 
keep getting strange errors and I can't figure out what's causing them.

Here's what I would like to do:

#include list
#include insets/InsetFoot.h
...
std::listInsetFoot const * footlist;

However, I can't get that far. When I include InsetFoot in OutputParams.h, I 
get the following errors:

invalid use of incomplete type 'struct lyx::InsetCommand'
forward declaration of 'struct lyx::InsetCommand'
'InsetCommandParams' has not been declared
'DisplayType' does not name a type
ISO C++ forbids declaration of Paraminfo with no type
expected ';' before 'const
...

I haven't actually done anything yet, the #include statement is sufficient to 
cause the errors. Any ideas on what might be causing this? I've tried to figure 
it out, but I'm at a loss.

Cheers,

Rob

Re: HTML Footnotes as Endnotes

[Rob Oakes]

Okay, so I've got the compilation error taken sorted out. (It was due to 
modifications that I had made to the class.) I've added a list object to the 
output parameters class.

I'm now trying to figure out how to successfully collect the footnotes into the 
list. Using:

In InsetFoot::xhtml():

op.footlist.push_back(this);

doesn't work because op has been declared as const. The compiler doesn't like 
it, a lot.  I tried removing the const declarations and directly modifying the 
object, but that changes how the inset behaves and it no longer correctly 
outputs the inset paragraphs.

I've also looked into using pointers to the footlist object, but I'm not doing 
something right because it keeps crashing LyX (probably due to access an object 
that isn't available in memory). 

Any ideas on where I might go from here?

Cheers,

Rob

Re: HTML Footnotes as Endnotes

[Rob Oakes]

Hi Guenter, 

On Dec 7, 2011, at 1:14 AM, Guenter Milde wrote:

> IMV, we should make the placement of footnotes in HTML configurable. 
> 
> Sensible options include
> 
> a) inline (as "tooltip" via CSS :mouseover:)
> b) in the footer for page-based output (e.g. print from HTML or ePub)
> c) endnotes (before/after other concluding sections and listings?)
> d) per chapter/section/subsection
> e) author-specified position via an inset

I'd really like this too, but I'm not sure what the best way to implement it 
is. Putting HTML footnotes at the end (or collecting them as Richard as 
suggested and I'm currently working on a patch for) is pretty straightforward 
to code. (And I'm a pretty modest C++ coder.)

The reason I like is that I can easily customize and control the output using 
the CSS styles in the layouts. Yesterday, I spent all day at a publishing 
conference talking to in-house programmers/designers and they identified a 
couple of things they would want to see:

1.) the XHTML should be as clean as possible, following XHTML5 and ePub3 
conventions
2.) CSS styling should be as transparent as possible, with intelligent defaults
3.) It's very important that it be easy to customize

LyX already provides for two and three. With some tweaking, the XHTML that 
comes out can be very clean and the LyXHTML can be customized via the document 
or Internal layout file.

I can foresee that a designer might want to tweak the CSS regardless of which 
export option they are using, but what is the best way to do this? In the 
layout, we can only specify one category of CSS that will get applied, 
regardless of which export option is used. It seems like the other approaches 
would require hard coding it, which seems like a less effective approach.

Anyway ... I'll continue work on collecting footnotes and send a patch a little 
bit later today.

Cheers,

Rob

PS, on another note, I'm pretty excited about the export of splitting the HTML 
file at the chapter or section level. Is there anything that I can do to help 
facilitate it?

There are six big challenges I see to getting really excellent HTML/eBook 
output from LyX :

1.) Clean up the HTML tags to follow HTML5 conventions (easy, configurable 
through layouts, in progress)
2.) Export CSS styles as a separate file (done)
3.) Allow for more flexible handling of footnotes (in progress, moderately 
difficult)
4.) Split large HTML documents into more manageable chunks, perhaps at the 
chapter level (unknown)
5.) Allow for images, equations and other assets to be moved to a logical 
subfolder hierarchy (unknown)
6.) Create a specialized inset which is able to insert audio and video using 
the audio/video tags (unknown)

And that's really it. The rest of the ePub processing can be using external 
python modules.

Re: HTML Footnotes as Endnotes

[Rob Oakes]

Hi Richard, 

Just trying to implement the gathering of footnotes via std::list. I placed the 
structure in the LatexFeatures header, however, I didn't see any way to access 
LatexFeatures via OutputParams (e.g. op.features ...). Am I missing something 
obvious?

Cheers,

Rob

Re: HTML Footnotes as Endnotes

[Rob Oakes]

Hi Jean-Marc,

Could you point me at an example of how this might be done or provide a few 
lines of code? I'm trying to add a std::list object to OutputParams and getting 
a wide variety of strange errors. Apparently, including it doesn't like it when 
I include "InsetFoot.h". I was able to create the list in LaTeX features 
without trouble, though.

I know very little about how LyX's validate methods work, though.

Cheers,

Rob

Compile Errors in OutputParams

[Rob Oakes]

Dear LyX Developers,

I'm trying to add a simple list objet to the OutputParams header file, but I 
keep getting strange errors and I can't figure out what's causing them.

Here's what I would like to do:

#include 
#include 
...
std::list footlist;

However, I can't get that far. When I include InsetFoot in OutputParams.h, I 
get the following errors:

invalid use of incomplete type 'struct lyx::InsetCommand'
forward declaration of 'struct lyx::InsetCommand'
'InsetCommandParams' has not been declared
'DisplayType' does not name a type
ISO C++ forbids declaration of Paraminfo with no type
expected ';' before 'const
...

I haven't actually done anything yet, the #include statement is sufficient to 
cause the errors. Any ideas on what might be causing this? I've tried to figure 
it out, but I'm at a loss.

Cheers,

Rob

Re: HTML Footnotes as Endnotes

[Rob Oakes]

Okay, so I've got the compilation error taken sorted out. (It was due to 
modifications that I had made to the class.) I've added a list object to the 
output parameters class.

I'm now trying to figure out how to successfully collect the footnotes into the 
list. Using:

In InsetFoot::xhtml():

op.footlist.push_back(this);

doesn't work because op has been declared as const. The compiler doesn't like 
it, a lot.  I tried removing the const declarations and directly modifying the 
object, but that changes how the inset behaves and it no longer correctly 
outputs the inset paragraphs.

I've also looked into using pointers to the footlist object, but I'm not doing 
something right because it keeps crashing LyX (probably due to access an object 
that isn't available in memory). 

Any ideas on where I might go from here?

Cheers,

Rob

HTML Footnotes as Endnotes

[Rob Oakes]

Dear LyX Developers, 

I've continued working on some of the challenges to getting clean ePub from LyX 
and have finished an inset that tentatively allows you to move footnotes to 
endnotes when exporting to HTML. Attached is a patch implementing the change 
(or the logic of it, at least).

I'd appreciate any comments.

Cheers,

Rob Oakes




htmlendnotelist.diff
Description: Binary data

Re: HTML Footnotes as Endnotes

[Rob Oakes]

Hi Richard,

Thanks for the feedback, I really appreciate it.

On Dec 6, 2011, at 4:39 PM, Richard Heck wrote:

 I wonder if we'd be better just outputting footnotes as endnotes
 all the time. The inline version we now use is cool, but maybe it's too
 cool for it's own good.

I actually think that would be a really good thing. It makes everything much 
easier to work with. (Or at least, that's what I think.)

 Second, I've been thinking recently about introducing some sort of
 chapter splitting capability. Not so important for e-books probably, but
 useful for the good old web.

And very useful for eBooks as well. Due to the way that ePub works, at least, 
smaller HTML files load faster.

 In that case, one would want to be able to output footnotes per chapter.
 There might be other cases where people
 wanted to print endnotes per chapter, even without the splitting. That
 suggests the idea of collecting the footnotes along the way in some
 kind of structure, and then emptying it when it comes time to print
 them, which could then be at any time. Very roughly:
 
 In LaTeXFeatures or some such place:
std::listInsetFoot const * footlist;
 In InsetFoot::xhtml():
op.features.footlist.push_back(this);
 and then in InsetPrintEndnotes::xhtml():
listInsetFoot *  footlist = op.features.footlist;
while (!footlist.empty()) {
InsetFoot const * foot = footlist.front();
footlist.pop_front();
...
// Something like this must be legal
// I think this trick should simplify much of your code...
xs  foot-InsetFootlike::xhtml();

I'll look into implementing this tonight. Having this stuff working for the 
demo I'm doing tomorrow would be great

Cheers,

Rob