Re: [sword-devel] HTTPS Transport

[Michael Johnson]

Hi Troy,

Thanks for your work on this. Please add https://ebible.org/sword/ to 
the master repository list:


HTTPSPackagePreference=eBible.org|ebible.org|/sword/

I have added a symbolic link from https://eBible.org/sword/packages to 
https://eBible.org/sword/zip (which I thought was the standard place for 
the zip files). Wouldn't it make more sense to stick with the first 
standard?


On 12/16/23 12:11, Troy A. Griffitts wrote:

Hey guys,

For a while now, we've been working on a new mechanism for allowing a 
remote module installation repository to use only HTTPS traffic to 
supply their modules, if they wish.  A little history and how things 
work in the released SWORD engine and what we have in SVN trunk now... 
(skip to === NEW === if you don't care about the history and why)


From the beginning of SWORD we have had as a core value the simple 
enabling of Bible distribution.  The very first versions of our 
installer could use as an installation source any working installation 
of a SWORD library.  E.g., user 1 sets up Xiphos and manually unzips 
100 Bibles, commentaries, lexicons, dictionaries, etc. for use with 
Xiphos.  User 1 can then share (network drive, USB stick, FTP) their 
installation folder where they have unzipped all the data for their 
library, and user 2 can come along and install Bibletime or Xiphos or 
any other SWORD application and point their installer to this shared 
location and install from there any Bible, commentary, lexicon, 
dictionary, etc., from user 1's working installation.  Then user 2 can 
travel to their school in Zimbabwe, plug into their school's network 
and share their data folder from their working SWORD application and 
students on that network can install Bibles from them.


None of this has changed.  This is still a core value and still works 
with all the same mechanisms.


Over the years, we have added on top of this behavior optional 
optimizations for remote repositories.  For example, instead of 
looking for the mods.d/ folder and downloading individually all the 
.conf files found there to present to a user a list of which Bibles, 
commentaries, etc. are available, we first look for a mods.d.tar.gz 
file with all the .conf files bundled into a single download.  This 
saves a lot of time working with large remote repositories.  If we 
don't find this file, we still fallback to downloading the individual 
files.  We don't want a failure to happen when passing along Bibles if 
this optimization is not in place, but we do want to speed things up 
if the manager of the remote repository knows how to manage their 
repository optimally and is willing to do this extra work to keep this 
file in place and up to date.


Over the years, the FTP protocol, which SWORD has primarily used for 
remote module installation over the internet, has seen data providers 
block traffic due to its unencrypted nature.  Being Bible 
distributors, in most cases we don't care if anyone snoops on our data 
packets.  Generally, again in most cases, we WANT people to snoop.  We 
don't require user / password for distribution so the security issues 
involved in sending those in plain text don't apply to our 
applications, generally.  Now, of course there are scenarios which 
people may wish to distribute Bibles without public knowledge, and 
sometimes users may wish to protect their modules with username / 
password credentials, and for this we have historically also supported 
SFTP.


One driving factor for the latest improvement described below is that 
we have found some mobile data providers blocking FTP traffic on their 
network, requiring our users to get to a WiFi connection before they 
can install Bibles, etc.



=== NEW ===

In SVN trunk there is code to handle a new facility for remote module 
installation.  Like the optional optimization with the mods.d.tar.gz 
file, this new mechanism is optional.  All will work as before if 
nothing is changed.


Fully enabling the new mechanism consists of 4 steps:

1. assuring https access to the root folder of your module repository.

2. mods.d.tar.gz is required for this mechanism to be successful.

3. module.zip files must be available from a packages/ folder at the 
root of your module repository folder.  These .zip files have been 
historically required for JSword-based apps because JSword does not 
yet know how to install from an working installation of modules, as 
described at the beginning of this email.  So because many of our 
repository maintainers support JSword, this step might be as simple as 
creating a packages/ -> symbolic link to your JSword .zip module files 
folder, if you are already maintaining zip files.


4. adding an HTTPSPackagePreference entry into our master repository 
list telling us the server, and path on that server, to find your 
repository with https



The main CrossWire repository now has this mechanism enabled and can 
be used as a reference to test frontends and can be used as an example 

[sword-devel] HTTPS Transport

[Troy A. Griffitts]

Hey guys,

For a while now, we've been working on a new mechanism for allowing a 
remote module installation repository to use only HTTPS traffic to 
supply their modules, if they wish.  A little history and how things 
work in the released SWORD engine and what we have in SVN trunk now... 
(skip to === NEW === if you don't care about the history and why)


From the beginning of SWORD we have had as a core value the simple 
enabling of Bible distribution.  The very first versions of our 
installer could use as an installation source any working installation 
of a SWORD library.  E.g., user 1 sets up Xiphos and manually unzips 100 
Bibles, commentaries, lexicons, dictionaries, etc. for use with Xiphos.  
User 1 can then share (network drive, USB stick, FTP) their installation 
folder where they have unzipped all the data for their library, and user 
2 can come along and install Bibletime or Xiphos or any other SWORD 
application and point their installer to this shared location and 
install from there any Bible, commentary, lexicon, dictionary, etc., 
from user 1's working installation.  Then user 2 can travel to their 
school in Zimbabwe, plug into their school's network and share their 
data folder from their working SWORD application and students on that 
network can install Bibles from them.


None of this has changed.  This is still a core value and still works 
with all the same mechanisms.


Over the years, we have added on top of this behavior optional 
optimizations for remote repositories.  For example, instead of looking 
for the mods.d/ folder and downloading individually all the .conf files 
found there to present to a user a list of which Bibles, commentaries, 
etc. are available, we first look for a mods.d.tar.gz file with all the 
.conf files bundled into a single download.  This saves a lot of time 
working with large remote repositories.  If we don't find this file, we 
still fallback to downloading the individual files.  We don't want a 
failure to happen when passing along Bibles if this optimization is not 
in place, but we do want to speed things up if the manager of the remote 
repository knows how to manage their repository optimally and is willing 
to do this extra work to keep this file in place and up to date.


Over the years, the FTP protocol, which SWORD has primarily used for 
remote module installation over the internet, has seen data providers 
block traffic due to its unencrypted nature.  Being Bible distributors, 
in most cases we don't care if anyone snoops on our data packets.  
Generally, again in most cases, we WANT people to snoop.  We don't 
require user / password for distribution so the security issues involved 
in sending those in plain text don't apply to our applications, 
generally.  Now, of course there are scenarios which people may wish to 
distribute Bibles without public knowledge, and sometimes users may wish 
to protect their modules with username / password credentials, and for 
this we have historically also supported SFTP.


One driving factor for the latest improvement described below is that we 
have found some mobile data providers blocking FTP traffic on their 
network, requiring our users to get to a WiFi connection before they can 
install Bibles, etc.



=== NEW ===

In SVN trunk there is code to handle a new facility for remote module 
installation.  Like the optional optimization with the mods.d.tar.gz 
file, this new mechanism is optional.  All will work as before if 
nothing is changed.


Fully enabling the new mechanism consists of 4 steps:

1. assuring https access to the root folder of your module repository.

2. mods.d.tar.gz is required for this mechanism to be successful.

3. module.zip files must be available from a packages/ folder at the 
root of your module repository folder.  These .zip files have been 
historically required for JSword-based apps because JSword does not yet 
know how to install from an working installation of modules, as 
described at the beginning of this email.  So because many of our 
repository maintainers support JSword, this step might be as simple as 
creating a packages/ -> symbolic link to your JSword .zip module files 
folder, if you are already maintaining zip files.


4. adding an HTTPSPackagePreference entry into our master repository 
list telling us the server, and path on that server, to find your 
repository with https



The main CrossWire repository now has this mechanism enabled and can be 
used as a reference to test frontends and can be used as an example for 
remote module installation repository maintainers.


For CrossWire main, #1 is available here, and at the root of this 
location you can also see #2 mods.d.tar.gz and #3 packages/ :


https://crosswire.org/ftpmirror/pub/sword/raw/

Step 4 can be seen in our master repo list, the first entry under 
[Repos] here:


https://crosswire.org/ftpmirror/pub/sword/masterRepoList.conf


Any SWORD app compiled against SVN trunk should now only 

Re: [sword-devel] What markup format is SWORD's internal markup, and/or where is it documented?

[Troy A. Griffitts]

Yes, just be clear in my last message when I pointed to here in 
reference to the *XHTML filters:


https://crosswire.org/svn/sword/trunk/src/modules/filters/

I specifically referencing:

gbfxhtml.cpp
osisxhtml.cpp
teixhtml.cpp
thmlxhtml.cpp

These are the output filters to XHTML from our currently supported 
module markups: GBF, OSIS, TEI, and ThML.


I suggested XHTML because these are the most complete filters. They are 
the most used output filters.  My hope for markdown is that you won't 
need to understand everything they do, but you could simply find each 
XHTML tag and replace that with appropriate markdown.


In SWORD, as you've all discussed a bit, Storage Format is the way we 
break up a module into entries (verse, dictionary entries, etc.) and 
store it on disk. Markup is very different and is the rich tags within 
the module data which represent semantic data for words, paragraphs, 
quotes, etc.


Both of these are specified within the .conf file for the module.  We 
support various storage formats and various markups. The engine is 
suppose to handle all this for the user interface programmer.  They 
should simply be able to say, e.g., "Hey I want XHML output." and then 
instantiate a library factory (SWMgr) and ask for a module (Bible, 
dictionary, commentary, whatever), then position it, and then ask for 
the "RenderText" which should then give them XHTML output, regardless of 
the module's storage format or the markup within that module.


Hope this helps,

Troy


On 12/16/23 02:19, Peter von Kaehne wrote:

I have written a set of markdown filters which I never committed as it was 
incomplete and I never finished it.

I can look and see if I can find them.

One aspect is that our “plain” output actually contains a bit of markdown and 
is not _really_ plain. It would be good to fix that in the same go.

Also I think I found while leaning on the xhtml filters for inspiration was 
good, there were better, simpler starting points. The latex filters I think I 
built mainly on top of the rtf output.

I am not a programmer, i just copy and play with until things do as I want.

Peter

Sent from my phone. Please forgive misspellings and weird “corrections”


On 16 Dec 2023, at 06:43, Timothy Allen  wrote:

The SWORD project's command-line tool, diatheke, mentions the following in its 
command-line help:


Valid output_format values are: CGI, GBF, HTML, HTMLHREF, LaTeX, OSIS, RTF,
   ThML, WEBIF, XHTML, plain, and internal (def)
The option LaTeX will produce a compilable document, but may well require
   tweaking to be usable.

"HTML" produces a reasonable HTML document, and if you ask for footnotes to be 
included you'll get the footnote *markers* without the footnote *content*.

"CGI" produces something similar, but only the closing BODY and HTML tags, so 
you can have a CGI script that prints out a pretty header and lets the SWORD output close 
it off.

"HTMLHREF" produces the same document, but if you ask for footnotes, they are 
turned into links whose URLs match CrossWire's online Bible study tool, 
https://crosswire.org/study/

"XHTML" is like HTMLHREF but includes some static CSS and XML-style 
self-closing tags.

For conversion to Markdown, I'd recommend getting the HTML version, as it's the 
simplest and the closest to the document module Markdown uses.


Timothy.


On 16/12/23 15:12, Aaron Rainbolt wrote:
I think I'm beginning to understand - there is no one internal markup
format, but rather there's the binary format, and then the SWORD
engine converts that (sorta) into something very similar to the
original markup language. Is that about right? I think the markup
format I'm seeing then is a subset of OSIS, and probably other modules
will have a different syntax. Which would explain why the best way to
do this is via filters since I'll need to handle three or so different
markup formats depending on what each module uses.

In that instance, I guess my question becomes, what markup formats
does SWORD support emitting a subset of? OSIS is clearly one, and it
sounds like ThML is another. Is there a list of these so I can write
the needed Markdown filter for each one?

I may attempt to get the Markdown filter collection (if I manage to
write it) contributed back to SWORD if that's something the devs are
interested in. I'm not used to normal C++, but I can work my way
through Qt C++ with little difficulty, so I should hopefully be able
to adapt to Qt-less C++ without too many issues.

Thanks for your help!


On Fri, Dec 15, 2023 at 9:45 PM Troy A. Griffitts  wrote:
Hi Aaron,

The SWORD engine tries to preserve the best it can the markup from an imported 
text for few supported markup formats. The markup used by a SWORD module is 
specified in its .conf file. Most new SWORD modules from our Modules Team are 
in OSIS markup. If you want to add markdown as a new output format for SWORD, 
that would be great. To do this, you would want to follow the pattern you see 
for one of out existing 

Re: [sword-devel] What markup format is SWORD's internal markup, and/or where is it documented?

[Peter von Kaehne]

I have written a set of markdown filters which I never committed as it was 
incomplete and I never finished it. 

I can look and see if I can find them. 

One aspect is that our “plain” output actually contains a bit of markdown and 
is not _really_ plain. It would be good to fix that in the same go. 

Also I think I found while leaning on the xhtml filters for inspiration was 
good, there were better, simpler starting points. The latex filters I think I 
built mainly on top of the rtf output. 

I am not a programmer, i just copy and play with until things do as I want. 

Peter

Sent from my phone. Please forgive misspellings and weird “corrections”

> On 16 Dec 2023, at 06:43, Timothy Allen  wrote:
> 
> The SWORD project's command-line tool, diatheke, mentions the following in 
> its command-line help:
> 
> > Valid output_format values are: CGI, GBF, HTML, HTMLHREF, LaTeX, OSIS, RTF,
> >   ThML, WEBIF, XHTML, plain, and internal (def)
> > The option LaTeX will produce a compilable document, but may well require
> >   tweaking to be usable.
> 
> "HTML" produces a reasonable HTML document, and if you ask for footnotes to 
> be included you'll get the footnote *markers* without the footnote *content*.
> 
> "CGI" produces something similar, but only the closing BODY and HTML tags, so 
> you can have a CGI script that prints out a pretty header and lets the SWORD 
> output close it off.
> 
> "HTMLHREF" produces the same document, but if you ask for footnotes, they are 
> turned into links whose URLs match CrossWire's online Bible study tool, 
> https://crosswire.org/study/
> 
> "XHTML" is like HTMLHREF but includes some static CSS and XML-style 
> self-closing tags.
> 
> For conversion to Markdown, I'd recommend getting the HTML version, as it's 
> the simplest and the closest to the document module Markdown uses.
> 
> 
> Timothy.
> 
>> On 16/12/23 15:12, Aaron Rainbolt wrote:
>> I think I'm beginning to understand - there is no one internal markup
>> format, but rather there's the binary format, and then the SWORD
>> engine converts that (sorta) into something very similar to the
>> original markup language. Is that about right? I think the markup
>> format I'm seeing then is a subset of OSIS, and probably other modules
>> will have a different syntax. Which would explain why the best way to
>> do this is via filters since I'll need to handle three or so different
>> markup formats depending on what each module uses.
>> 
>> In that instance, I guess my question becomes, what markup formats
>> does SWORD support emitting a subset of? OSIS is clearly one, and it
>> sounds like ThML is another. Is there a list of these so I can write
>> the needed Markdown filter for each one?
>> 
>> I may attempt to get the Markdown filter collection (if I manage to
>> write it) contributed back to SWORD if that's something the devs are
>> interested in. I'm not used to normal C++, but I can work my way
>> through Qt C++ with little difficulty, so I should hopefully be able
>> to adapt to Qt-less C++ without too many issues.
>> 
>> Thanks for your help!
>> 
>>> On Fri, Dec 15, 2023 at 9:45 PM Troy A. Griffitts  
>>> wrote:
>>> Hi Aaron,
>>> 
>>> The SWORD engine tries to preserve the best it can the markup from an 
>>> imported text for few supported markup formats. The markup used by a SWORD 
>>> module is specified in its .conf file. Most new SWORD modules from our 
>>> Modules Team are in OSIS markup. If you want to add markdown as a new 
>>> output format for SWORD, that would be great. To do this, you would want to 
>>> follow the pattern you see for one of out existing output formats. I would 
>>> suggest XHTML. E.g., copy the 3 or so *XHTML filters you see here and 
>>> modify the output from XHTML to markdown.
>>> 
>>> https://crosswire.org/svn/sword/trunk/src/modules/filters/
>>> 
>>> Hope this gets you started. Let me know if you have questions,
>>> 
>>> Troy
>>> 
>>> 
>>> On December 15, 2023 20:26:22 MST, Aaron Rainbolt  
>>> wrote:
 I didn't mean the actual compiled files - I meant the kind of markup you 
 end up with if you use swModule->getRawEntry(). It's the same kind of 
 markup you see if you use mod2imp on a module. So far the modules I've 
 used this on seem to have some sort of pattern to them (there's  tags 
 with "lemma" and "morph" attributes for Strong's numbers and morphology, 
  tags for... something, looks like it's part of how red-letter Bibles 
 work because of the "who" attribute, and things like that). I assume it's 
 this markup that is parsed by the existing filters, and that I would need 
 to parse were I to write my own filter.
 
 My text renderer does indeed support HTML, but it's ability to output 
 Markdown is sorely lacking (I *can* tell it to give me whatever's in my 
 text editor widget in Markdown format, but it loses information that 
 Markdown is perfectly capable of containing). I need to be able to convert 
 between 

Re: [sword-devel] transChange element

[Tobias Klein]

On 12/11/23 5:47 PM, pinoaffe wrote:

First of all, the phrase "Words added for modern readers, where in the
original the meaning was implied", doesn't account for the case where
words were changed rather than added

In general, I think rephrasing these tooltips as full sentences would be
preferrable, e.g. something along the lines of

- (added) "These words were added in translation" or
- (added) "These words do not directly correspond to words in the source text"
- (amplified) "These words are amplifications of information in the source text"
- (changed) "These words were changed in translation"
- (deleted) "Some words in the source text are not present in the translation" 
or
- (deleted) "Some words in the source text do not directly correspond to any words 
in the translation"
- (implied) "These words were added or changed to translate information implied by 
the source text" or
- (implied) "These words were added or changed to translate information implied by 
(but not explicitly stated in) the source text"
- (moved) "These words were moved in translation" or
- (moved) "These words were in another location in the source text"
- (tense-change) "The gramatical tense of these words was changed in order to 
facilitate translation"


I have just changed 
 
the English text of the tooltips based on your suggestions.


These texts are the base for translations that are pending for the next 
release (in January).


Thank you once more for your help on this!

Best regards,
Tobias

___
sword-devel mailing list: sword-devel@crosswire.org
http://crosswire.org/mailman/listinfo/sword-devel
Instructions to unsubscribe/change your settings at above page

Re: [sword-devel] transChange element

[Tobias Klein]

Thanks for the feedback on this, David!

Best regards,
Tobias

On 12/12/23 9:00 AM, David Haslam wrote:

A bit of lateral thinking in this topic…

There’s no requirement that only whole words should be wrapped within 
a transChange element!
In some translations it’s not unusual for parts of words to be so 
marked. (eg. Spanish).


In some languages (eg Thai) one should also be aware that there are no 
spaces between words.
Determining word boundaries is not a straightforward task even for a 
computer.


And although SWORD renders in italics the text that’s marked with 
transChange, there are also writing systems (non-Roman scripts) for 
which there is no possibility to do this.


Best regards,

David

On Tue, Dec 12, 2023 at 07:45, Tobias Klein  wrote:

On 12/11/23 5:47 PM, pinoaffe wrote:
>> Would you suggest to change any of these for making the tooltip easier
>> to read?
> First of all, the phrase "Words added for modern readers, where in the
> original the meaning was implied", doesn't account for the case where
> words were changed rather than added
>
> In general, I think rephrasing these tooltips as full sentences 
would be

> preferrable, e.g. something along the lines of
>
> - (added) "These words were added in translation" or
> - (added) "These words do not directly correspond to words in the 
source text"
> - (amplified) "These words are amplifications of information in the 
source text"

> - (changed) "These words were changed in translation"
> - (deleted) "Some words in the source text are not present in the 
translation" or
> - (deleted) "Some words in the source text do not directly 
correspond to any words in the translation"
> - (implied) "These words were added or changed to translate 
information implied by the source text" or
> - (implied) "These words were added or changed to translate 
information implied by (but not explicitly stated in) the source text"

> - (moved) "These words were moved in translation" or
> - (moved) "These words were in another location in the source text"
> - (tense-change) "The gramatical tense of these words was changed 
in order to facilitate translation"

>
>> This will then become the basis for translating the tooltip to 
other languages as well.
> I think that having full sentences helps convey the tooltip meaning 
more

> clearly and that such sentences are more easy to translate to other
> languages

Thank you, that's helpful feedback! I'll consider your suggestions and
update the English locale accordingly.

How about the grammatical number of the words in the transChange
element: In your proposals, the tooltips/explanations are all plural
("These words"). When I look at examples in the Bible text, though I
often see only one word that is italic. Should we have singular/plural
variants of the tooltips? It is possible to detect dynamically how many
words are in the transChange element and determine the right tooltip
accordingly.

Best regards,
Tobias

___
sword-devel mailing list: sword-devel@crosswire.org
http://crosswire.org/mailman/listinfo/sword-devel
Instructions to unsubscribe/change your settings at above page


___
sword-devel mailing list: sword-devel@crosswire.org
http://crosswire.org/mailman/listinfo/sword-devel
Instructions to unsubscribe/change your settings at above page
___
sword-devel mailing list: sword-devel@crosswire.org
http://crosswire.org/mailman/listinfo/sword-devel
Instructions to unsubscribe/change your settings at above page