Re: LyX 1.6.1 on OS X unable to handle diacritic marks in .bib files

[Konrad Hofbauer]

Grant Jacobs wrote:
I have a suspicion that the software I used to gather the references 
doesn't know" the LaTeX encodings for the umlauts, etc. and just left 
them in their original UniCode.


Ah ... I assumed your bibtex files initially came from BibDesk. Maybe 
it's time to switch to a reference manager with better Latex-support. 
(Do you hear me shouting 'BibDesk' ? ;) )



For developers:


I'm not, nevertheless I throw in my 2 cents... ;)

Is there any sense in having bibtex "convert" files in the same way as I 
have (i.e. from bibtex to bibtex) to "standardise" the files before LyX 
takes them up? While redundant for many users, it might "catch" 
non-standard things and may make this "just work", at least for the 
situation I have.


This is probably something that bibtex should take care of, not LyX.
If you know of an existing script or library that already does that, it 
might be considered in LyX.


This may also provide a means of testing that the bibliography is in a 
standard (or understandable) format and report a meaningful error if 
it's not. (The current error messages are a tad too geeky for 
non-programmers, etc.)


Unfortunately, one could say that about almost every Latex error message. :(


Actually I do understand it ;-) No offence, but if you knew the effort I 
have put into this, telling me to "Try to read up on this" is a bit 
rich! :-)


No offense here either, I just did not get why you would want to 
_export_ from BibDesk, if BibDesk's native file format is exactly what 
LyX&bibtex want (and works).



P.S. Forget the path via RTF!


OK. Bit surprised though: there is an exporter for it in BibDesk and an 
importer in LyX, so in principle it should work, right? 


The LyX-importer (if you have the right external tool installed) is for 
text, only, and not for references/citations.


I've put literally hours 
into this. 


This happens ... (same here on other issues, usually LaTeX-limitations).

I hope you can my perspective: I really expected this to 
"just work" as BibDesk must be used by most LyX users and UniCode is so 
universal now... and all the rest. 


At least, now you know that simply saving in BibDesk solves the problem. 
Fixed for once and forever. :)


/Konrad

Re: LyX 1.6.1 on OS X unable to handle diacritic marks in .bib files

[Grant Jacobs]

Since writing below I have found a work-around, which I hope will 
continue working on the biblio files:


In BibDesk, convert the bibtex database into bibtex. This seems a 
little daft at first, but I picked this up on reading that on 
importing and exporting BibDesk converts umlauts, etc. into the LaTeX 
equivalent. My guess was that by "converting" to the same format, 
would have the effect of "standardising" the umlauts, etc.


I have a suspicion that the software I used to gather the references 
doesn't know" the LaTeX encodings for the umlauts, etc. and just left 
them in their original UniCode.



For developers:

Is there any sense in having bibtex "convert" files in the same way 
as I have (i.e. from bibtex to bibtex) to "standardise" the files 
before LyX takes them up? While redundant for many users, it might 
"catch" non-standard things and may make this "just work", at least 
for the situation I have.


This may also provide a means of testing that the bibliography is in 
a standard (or understandable) format and report a meaningful error 
if it's not. (The current error messages are a tad too geeky for 
non-programmers, etc.)


I haven't time to do let them know right now, but BibDesk ideally 
should let users know that on loading a database, it had to convert 
some stuff, i.e. that the internal version of it is "tainted" with 
respect to the disk version, so that users are warned/asked to save 
the conversions, etc.--? Perhaps even insist that they do it 
immediately after loading it.




You're welcome to read the rest, I might as well let it stand in case 
it's of use to anyone, but what's above it the nutshell take.


Grant



Grant Jacobs wrote:
One thing that bothers me about this being a BibTeX issue, is that 
BibDesk writes these characters to RTF just fine. (It would tempt 
me to think that there might a work-around that gets this directly 
from BibDesk, avoiding BibTex.)


0) You have not understood the concept behind BibDesk - it's only a 
sophisticated frontend to the database (bib) file that bibtex uses. 
Try to read up on this.


Actually I do understand it ;-) No offence, but if you knew the 
effort I have put into this, telling me to "Try to read up on this" 
is a bit rich! :-)



1) You can enter the TeX-replacements for the special characters in 
the bib-file (or in Bibdesk) instead of the unicode characters.

See 
Since you say it is mostly for European author names, this should be enough.


Doing this manually is not a practical option. [Resolved above, though]


2) Make sure you use a recent version of BibDesk. Also, TeXLive-2008 
and LyX 1.6.1 won't hurt.


LyX is 1.6.1 as the subject line says, BibDesk is the latest version. 
Tex will be a little older (I have TexLive-2008 on disk, but I wanted 
to leave installation of that until later, as I didn't wanted a 
messed-up installation delaying me even more!)




HTH,


In an indirect way: it lead to seeing the comment about BiBDesk 
converting on importing and exporting, etc. in a doc on the web. Long 
story.




Konrad

P.S. Forget the path via RTF!


OK. Bit surprised though: there is an exporter for it in BibDesk and 
an importer in LyX, so in principle it should work, right? (But 
obviously not for some reason...!)



At 3:32 PM +0100 26/1/09, Konrad Hofbauer wrote:
Make sure you have 'Unicode to TeX Conversion' enabled in BibDesk's 
'Files' Preferences !!!


Obviously :-)  (I've already been all over the Preferences of both 
programs looking for "likely things", googling, etc. I make a point 
of trying hard myself before resorting to forums. Among others 
things, the back-and-forth in forums takes a lot of time. I've put 
literally hours into this. I hope you can my perspective: I really 
expected this to "just work" as BibDesk must be used by most LyX 
users and UniCode is so universal now... and all the rest. Fair point 
about bibtex's age, though. [See my comment about the file format 
though, para. 3 above.])



Grant

--
---
Grant Jacobs Ph.D. BioinfoTools
ph. +64 3 478 0095  (office, after 10am)   PO Box 6129,
or  +64 27 601 5917 (mobile)   Dunedin,
gjac...@bioinfotools.com   NEW ZEALAND.
   Bioinformatics tools: deriving knowledge from biological data  
Bioinformatics tools - software development - consulting - training

 15 years experience in bioinformatics ready to solve your problem
Check out the website for more details: http://www.bioinfotools.com

The information contained in this mail message is  confidential and
may be legally privileged.  Readers of this message who are not the
intended recipient are hereby notified that any use, dissemination,
distribution or reproduction of this message is prohibited.  If you
have received this message in error please notify the sender immed-
iately and destroy

Re: LyX 1.6.1 on OS X unable to handle diacritic marks in .bib files

[Konrad Hofbauer]

Make sure you have 'Unicode to TeX Conversion' enabled in BibDesk's 
'Files' Preferences !!!


HTH,
Konrad

Re: LyX 1.6.1 on OS X unable to handle diacritic marks in .bib files

[Konrad Hofbauer]

Grant Jacobs wrote:
One thing that bothers me about this being a BibTeX issue, is that 
BibDesk writes these characters to RTF just fine. (It would tempt me to 
think that there might a work-around that gets this directly from 
BibDesk, avoiding BibTex.)


0) You have not understood the concept behind BibDesk - it's only a 
sophisticated frontend to the database (bib) file that bibtex uses. Try 
to read up on this.


1) You can enter the TeX-replacements for the special characters in the 
bib-file (or in Bibdesk) instead of the unicode characters.

See 
Since you say it is mostly for European author names, this should be enough.

2) Make sure you use a recent version of BibDesk. Also, TeXLive-2008 and 
LyX 1.6.1 won't hurt.


3) BibDesk takes care of these characters automatically:
If I enter "åland sørenson" as author, the bib-file that Bibdesk 1.3.19 
creates looks like this:


%% Created for khofbaue at 2009-01-26 15:14:23 +0100
%% Saved with string encoding Unicode (UTF-8)
@phdthesis{sorenson:uq,
Author = {s{\o}renson, {\aa}land},
Title = {testing}}

So even though the file is saved in Unicode, Bibdesk replaces the 
characters by their corresponding latex macros (where possible, I 
assume), and everything "just works" in LyX.


HTH,
Konrad

P.S. Forget the path via RTF!

LyX 1.6.1 on OS X unable to handle diacritic marks in .bib files

[Grant Jacobs]

Thanks for explaining this. Its a damn nuisance I have admit, as I've 
spend a long time trying to resolve this when I have little time to 
spare really.


I thought I was starting a new thread: I set a new subject line. Is 
there some other thing I am supposed to do to make that happen?? I'm 
not working via a web interface, and most list software I've used 
just works off the subject lines...


If people are updating that part of the docs, it'd be good to see it 
*explain* how to make bibliographies work: the "explanation" 
currently there is leaving too much out. I'm too busy to help (grant 
application due, etc.)


One thing that bothers me about this being a BibTeX issue, is that 
BibDesk writes these characters to RTF just fine. (It would tempt me 
to think that there might a work-around that gets this directly from 
BibDesk, avoiding BibTex.)


I've tried saving the biblio as a RTF file from BibDesk, then 
importing that into a LyX document, but I got "An error occurred on 
while running rtf2latex 'file'". (Feels like nothing wants to work 
over here!)


Does rtf2latex suffer the same problem as BibTex? The idea was to set 
up a template in BibDesk to generate the right format, write that. In 
LyX, let LyX do the citations (which it seems to do OK, not that I 
can test it that well yet), import the RTF biblio (via BiBDesk) into 
the right place, and just delete the LyX-generated biblio from the 
final exported format using a PDF editor or whatnot.


I would try the equivalent using HTML and importing that, but the 
formatting/styling in HTML might be even more limited.




Grant Jacobs
--
---
Grant Jacobs Ph.D. BioinfoTools
ph. +64 3 478 0095  (office, after 10am)   PO Box 6129,
or  +64 27 601 5917 (mobile)   Dunedin,
gjac...@bioinfotools.com   NEW ZEALAND.
   Bioinformatics tools: deriving knowledge from biological data
Bioinformatics tools - software development - consulting - training
 15 years experience in bioinformatics ready to solve your problem
Check out the website for more details: http://www.bioinfotools.com

The information contained in this mail message is  confidential and
may be legally privileged.  Readers of this message who are not the
intended recipient are hereby notified that any use, dissemination,
distribution or reproduction of this message is prohibited.  If you
have received this message in error please notify the sender immed-
iately and destroy the original message.  This applies also to  any
attached documents.

Re: LyX 1.6.1 on OS X unable to handle diacritic marks in .bib files

[Jürgen Spitzmüller]

Jürgen Spitzmüller wrote:
> You need to use another encoding for the bib file, such as latin1, and
> insert the non-supported characters by means of the respective macros.

Also look here for some further information:
http://wiki.lyx.org/BibTeX/Tips#toc1

Jürgen

Re: LyX 1.6.1 on OS X unable to handle diacritic marks in .bib files

[Jürgen Spitzmüller]

Grant Jacobs wrote:
> Having *finally* managed to persuade LyX to generate bibliographies
> at all, I now find that LyX seemingly cannot handle characters with
> diacritic marks (i.e. UniCode) in .bib files when exporting them to
> PDF, etc. Unfortunately this really has to work for me as it will be
> the case with a very large number of European, etc., author's names
> and so on.

This is a BibTeX limitation. BibTeX is an old beast, the most recent release 
still predates the advent of unicode. Fact is that bibtex cannot deal with 
unicode (or any other multibyte encoding, for that matter) at all. There's 
noting that LyX can do here.

You need to use another encoding for the bib file, such as latin1, and insert 
the non-supported characters by means of the respective macros.

A decent bib file editor, such as JabRef or pybliographic, should actually 
handle this for you.

> If this is "how it is meant to be" or "the way it is", *please* add
> some information to this effect in the tutorial and user guide or
> somewhere prominent, so that users needing this can "quit early" and
> look for another another solution.

You are right, this needs to be documented in the UserGuide, sec. 6.5.2. 
CC:ing the documentation list.

Jürgen

PS.: please open a new thread for new topics.

LyX 1.6.1 on OS X unable to handle diacritic marks in .bib files

[Grant Jacobs]

Having *finally* managed to persuade LyX to generate bibliographies 
at all, I now find that LyX seemingly cannot handle characters with 
diacritic marks (i.e. UniCode) in .bib files when exporting them to 
PDF, etc. Unfortunately this really has to work for me as it will be 
the case with a very large number of European, etc., author's names 
and so on.


If this is "how it is meant to be" or "the way it is", *please* add 
some information to this effect in the tutorial and user guide or 
somewhere prominent, so that users needing this can "quit early" and 
look for another another solution. On the other hand, the 
"internationalisation" section of the "features" blurb mentions 
"unicode compliance" in passing...


Does anyone have any remedies for this? Preferably simple: I am all 
out of time to do more of the kind of fiddling I have had to do so 
far.


I've tried setting the language encoding to utf8, but it generates an 
empty document and complains that 'utf8.def' isn't present. Setting 
language encoding to 'Unicode (XeTeX) generates output (without 
"crashing" during the conversion), but the characters aren't correct. 
I can't generate plain text (ps2ascii) at all. Generating PostScript, 
generates output but also gets the characters wrong. And on it goes, 
randomly trying this, that and the other in vain attempts to make 
something work...! :-(


Is there some version or other of the main LaTeX installation that I 
am "supposed" to be using? (And if that's the case, shouldn't LyX be 
check that it's a sane choice, etc?)




Grant Jacobs
--
---
Grant Jacobs Ph.D. BioinfoTools
ph. +64 3 478 0095  (office, after 10am)   PO Box 6129,
or  +64 27 601 5917 (mobile)   Dunedin,
gjac...@bioinfotools.com   NEW ZEALAND.
   Bioinformatics tools: deriving knowledge from biological data
Bioinformatics tools - software development - consulting - training
 15 years experience in bioinformatics ready to solve your problem
Check out the website for more details: http://www.bioinfotools.com

The information contained in this mail message is  confidential and
may be legally privileged.  Readers of this message who are not the
intended recipient are hereby notified that any use, dissemination,
distribution or reproduction of this message is prohibited.  If you
have received this message in error please notify the sender immed-
iately and destroy the original message.  This applies also to  any
attached documents.