Re: [sword-devel] osis2mod bug - PLEASE HELP
On Fri, Mar 2, 2012 at 7:02 AM, Brian J Dumont brian.j.dum...@gmail.com wrote: Hi all, I've struggled with large commentaries before. It always seemed that once commentary sections get too big, then random sections of text start disappearing. I'm currently almost done repackaging a part of the EarlyFathers module to be the homilies of John Chrysostom in commentary form. Thing is, his homilies are long and I'm hitting the same problems again. Previously, I've been able to split sections between verses to awkwardly get things to work. This time I can't; some of these large homilies are on a single verse, and have another homily on the next. So this time I've had no choice but to give up or to buckle down and get something that the developers can really bite into. I've created an example OSIS module. One section: Matt.1.1. No special characters, no footnotes, no headers, almost no markup at all really. I'm not compiling it into a compressed format. My command/output is: [bjdasc@ascpc5] osis2mod mod debug.osis.xml You are running osis2mod: $Rev: 2671 $ SUCCESS: osis2mod: has finished its work and will now rest I can't always predict *exactly* at what size it will fail (it seems to depend on things like number of paragraph markers, etc). But it always fails when the section gets to be roughly 64,000 - 65,000 characters. This is a pretty suspicious number, being just below 2^16. I assume that I can't exactly predict the size because there's some overhead, some amount for paragraph markers, header, etc. The attached example file fails (I've also included the conf file that I'm using to make life easier for any developer that wishes to look at this). The output is drastically shorter than the input. Please delete one character from the text in Matt.1.1. It all works nicely. This is not a front end problem. I can reproduce the problem in either Xiphos or diatheke. My diatheke command has been: diatheke -b Chrysostom -k Mt 1:1 But you can even see when it has failed by the size of the nt file created by osis2mod. This osis file, as attached, creates an nt file that is 70 bytes. Delete one character and it goes up to 65607 bytes. Note that 70 bytes + 2^16 bytes = 65606 bytes. PLEASE PLEASE PLEASE PLEASE ... I beg you ... I'm stuck without a fix to this ... it seems like it's an array size limit or a compiler size limit for some data type or something like that. If you need some legwork, then let me know; I'll do it. Thanks in advance, Brian -- --- The question is not Have you accepted Jesus into your heart, but Has Jesus accepted you into His heart?; not have you given your whole life for God, but has God given His whole life for you - Pr Bryan Wolfmueller, 2010 ___ sword-devel mailing list: sword-devel@crosswire.org http://www.crosswire.org/mailman/listinfo/sword-devel Instructions to unsubscribe/change your settings at above page Brian, Try adding the switch '-s 4' to your import. Also change your driver in the conf file to RawCom4. This will increase the per-entry size limit from 2^16 bytes (the default) to 2^32 bytes giving you an effective limit of 2GB for each verse or other entry. --Greg ___ sword-devel mailing list: sword-devel@crosswire.org http://www.crosswire.org/mailman/listinfo/sword-devel Instructions to unsubscribe/change your settings at above page
Re: [sword-devel] osis2mod bug - PLEASE HELP
From memory -s 4 perhaps? Cent from my fone so theer mite be tipos. ;) On Mar 2, 2012, at 8:02 AM, Brian J Dumont brian.j.dum...@gmail.com wrote: Hi all, I've struggled with large commentaries before. It always seemed that once commentary sections get too big, then random sections of text start disappearing. I'm currently almost done repackaging a part of the EarlyFathers module to be the homilies of John Chrysostom in commentary form. Thing is, his homilies are long and I'm hitting the same problems again. Previously, I've been able to split sections between verses to awkwardly get things to work. This time I can't; some of these large homilies are on a single verse, and have another homily on the next. So this time I've had no choice but to give up or to buckle down and get something that the developers can really bite into. I've created an example OSIS module. One section: Matt.1.1. No special characters, no footnotes, no headers, almost no markup at all really. I'm not compiling it into a compressed format. My command/output is: [bjdasc@ascpc5] osis2mod mod debug.osis.xml You are running osis2mod: $Rev: 2671 $ SUCCESS: osis2mod: has finished its work and will now rest I can't always predict *exactly* at what size it will fail (it seems to depend on things like number of paragraph markers, etc). But it always fails when the section gets to be roughly 64,000 - 65,000 characters. This is a pretty suspicious number, being just below 2^16. I assume that I can't exactly predict the size because there's some overhead, some amount for paragraph markers, header, etc. The attached example file fails (I've also included the conf file that I'm using to make life easier for any developer that wishes to look at this). The output is drastically shorter than the input. Please delete one character from the text in Matt.1.1. It all works nicely. This is not a front end problem. I can reproduce the problem in either Xiphos or diatheke. My diatheke command has been: diatheke -b Chrysostom -k Mt 1:1 But you can even see when it has failed by the size of the nt file created by osis2mod. This osis file, as attached, creates an nt file that is 70 bytes. Delete one character and it goes up to 65607 bytes. Note that 70 bytes + 2^16 bytes = 65606 bytes. PLEASE PLEASE PLEASE PLEASE ... I beg you ... I'm stuck without a fix to this ... it seems like it's an array size limit or a compiler size limit for some data type or something like that. If you need some legwork, then let me know; I'll do it. Thanks in advance, Brian -- --- The question is not Have you accepted Jesus into your heart, but Has Jesus accepted you into His heart?; not have you given your whole life for God, but has God given His whole life for you - Pr Bryan Wolfmueller, 2010 debug.osis.xml chrysostom.conf ___ sword-devel mailing list: sword-devel@crosswire.org http://www.crosswire.org/mailman/listinfo/sword-devel Instructions to unsubscribe/change your settings at above page ___ sword-devel mailing list: sword-devel@crosswire.org http://www.crosswire.org/mailman/listinfo/sword-devel Instructions to unsubscribe/change your settings at above page
Re: [sword-devel] osis2mod bug - PLEASE HELP
I think osis2mod should detect the problem, output an error and a clear suggestion to use -s 4. Cent from my fone so theer mite be tipos. ;) On Mar 2, 2012, at 8:06 AM, Greg Hellings greg.helli...@gmail.com wrote: On Fri, Mar 2, 2012 at 7:02 AM, Brian J Dumont brian.j.dum...@gmail.com wrote: Hi all, I've struggled with large commentaries before. It always seemed that once commentary sections get too big, then random sections of text start disappearing. I'm currently almost done repackaging a part of the EarlyFathers module to be the homilies of John Chrysostom in commentary form. Thing is, his homilies are long and I'm hitting the same problems again. Previously, I've been able to split sections between verses to awkwardly get things to work. This time I can't; some of these large homilies are on a single verse, and have another homily on the next. So this time I've had no choice but to give up or to buckle down and get something that the developers can really bite into. I've created an example OSIS module. One section: Matt.1.1. No special characters, no footnotes, no headers, almost no markup at all really. I'm not compiling it into a compressed format. My command/output is: [bjdasc@ascpc5] osis2mod mod debug.osis.xml You are running osis2mod: $Rev: 2671 $ SUCCESS: osis2mod: has finished its work and will now rest I can't always predict *exactly* at what size it will fail (it seems to depend on things like number of paragraph markers, etc). But it always fails when the section gets to be roughly 64,000 - 65,000 characters. This is a pretty suspicious number, being just below 2^16. I assume that I can't exactly predict the size because there's some overhead, some amount for paragraph markers, header, etc. The attached example file fails (I've also included the conf file that I'm using to make life easier for any developer that wishes to look at this). The output is drastically shorter than the input. Please delete one character from the text in Matt.1.1. It all works nicely. This is not a front end problem. I can reproduce the problem in either Xiphos or diatheke. My diatheke command has been: diatheke -b Chrysostom -k Mt 1:1 But you can even see when it has failed by the size of the nt file created by osis2mod. This osis file, as attached, creates an nt file that is 70 bytes. Delete one character and it goes up to 65607 bytes. Note that 70 bytes + 2^16 bytes = 65606 bytes. PLEASE PLEASE PLEASE PLEASE ... I beg you ... I'm stuck without a fix to this ... it seems like it's an array size limit or a compiler size limit for some data type or something like that. If you need some legwork, then let me know; I'll do it. Thanks in advance, Brian -- --- The question is not Have you accepted Jesus into your heart, but Has Jesus accepted you into His heart?; not have you given your whole life for God, but has God given His whole life for you - Pr Bryan Wolfmueller, 2010 ___ sword-devel mailing list: sword-devel@crosswire.org http://www.crosswire.org/mailman/listinfo/sword-devel Instructions to unsubscribe/change your settings at above page Brian, Try adding the switch '-s 4' to your import. Also change your driver in the conf file to RawCom4. This will increase the per-entry size limit from 2^16 bytes (the default) to 2^32 bytes giving you an effective limit of 2GB for each verse or other entry. --Greg ___ sword-devel mailing list: sword-devel@crosswire.org http://www.crosswire.org/mailman/listinfo/sword-devel Instructions to unsubscribe/change your settings at above page ___ sword-devel mailing list: sword-devel@crosswire.org http://www.crosswire.org/mailman/listinfo/sword-devel Instructions to unsubscribe/change your settings at above page
Re: [sword-devel] osis2mod bug - PLEASE HELP
On 03/02/2012 08:06 AM, Greg Hellings wrote: On Fri, Mar 2, 2012 at 7:02 AM, Brian J Dumontbrian.j.dum...@gmail.com wrote: Hi all, I've struggled with large commentaries before. It always seemed that once commentary sections get too big, then random sections of text start disappearing. I'm currently almost done repackaging a part of the EarlyFathers module to be the homilies of John Chrysostom in commentary form. Thing is, his homilies are long and I'm hitting the same problems again. Previously, I've been able to split sections between verses to awkwardly get things to work. This time I can't; some of these large homilies are on a single verse, and have another homily on the next. So this time I've had no choice but to give up or to buckle down and get something that the developers can really bite into. I've created an example OSIS module. One section: Matt.1.1. No special characters, no footnotes, no headers, almost no markup at all really. I'm not compiling it into a compressed format. My command/output is: [bjdasc@ascpc5] osis2mod mod debug.osis.xml You are running osis2mod: $Rev: 2671 $ SUCCESS: osis2mod: has finished its work and will now rest I can't always predict *exactly* at what size it will fail (it seems to depend on things like number of paragraph markers, etc). But it always fails when the section gets to be roughly 64,000 - 65,000 characters. This is a pretty suspicious number, being just below 2^16. I assume that I can't exactly predict the size because there's some overhead, some amount for paragraph markers, header, etc. The attached example file fails (I've also included the conf file that I'm using to make life easier for any developer that wishes to look at this). The output is drastically shorter than the input. Please delete one character from the text in Matt.1.1. It all works nicely. This is not a front end problem. I can reproduce the problem in either Xiphos or diatheke. My diatheke command has been: diatheke -b Chrysostom -k Mt 1:1 But you can even see when it has failed by the size of the nt file created by osis2mod. This osis file, as attached, creates an nt file that is 70 bytes. Delete one character and it goes up to 65607 bytes. Note that 70 bytes + 2^16 bytes = 65606 bytes. PLEASE PLEASE PLEASE PLEASE ... I beg you ... I'm stuck without a fix to this ... it seems like it's an array size limit or a compiler size limit for some data type or something like that. If you need some legwork, then let me know; I'll do it. Thanks in advance, Brian -- --- The question is not Have you accepted Jesus into your heart, but Has Jesus accepted you into His heart?; not have you given your whole life for God, but has God given His whole life for you - Pr Bryan Wolfmueller, 2010 ___ sword-devel mailing list: sword-devel@crosswire.org http://www.crosswire.org/mailman/listinfo/sword-devel Instructions to unsubscribe/change your settings at above page Brian, Try adding the switch '-s 4' to your import. Also change your driver in the conf file to RawCom4. This will increase the per-entry size limit from 2^16 bytes (the default) to 2^32 bytes giving you an effective limit of 2GB for each verse or other entry. --Greg Thanks very, very much! That's so simple! I had tried each, but not together. Does this, however, mean that compressing the module is not an option? If I read correctly, the -s 4 option is only for uncompressed modules, and there seems to be no analog to RawCom4 that handles very large entries. Brian -- --- The question is not Have you accepted Jesus into your heart, but Has Jesus accepted you into His heart?; not have you given your whole life for God, but has God given His whole life for you - Pr Bryan Wolfmueller, 2010 ___ sword-devel mailing list: sword-devel@crosswire.org http://www.crosswire.org/mailman/listinfo/sword-devel Instructions to unsubscribe/change your settings at above page
Re: [sword-devel] osis2mod bug - PLEASE HELP
That would be super-keen DM. It would be nice if it also noted the need for ModDrv RawCom4 to go along with -s 4 Thanks, Brian On 03/02/2012 08:15 AM, DM Smith wrote: I think osis2mod should detect the problem, output an error and a clear suggestion to use -s 4. Cent from my fone so theer mite be tipos. ;) On Mar 2, 2012, at 8:06 AM, Greg Hellingsgreg.helli...@gmail.com wrote: On Fri, Mar 2, 2012 at 7:02 AM, Brian J Dumontbrian.j.dum...@gmail.com wrote: Hi all, I've struggled with large commentaries before. It always seemed that once commentary sections get too big, then random sections of text start disappearing. I'm currently almost done repackaging a part of the EarlyFathers module to be the homilies of John Chrysostom in commentary form. Thing is, his homilies are long and I'm hitting the same problems again. Previously, I've been able to split sections between verses to awkwardly get things to work. This time I can't; some of these large homilies are on a single verse, and have another homily on the next. So this time I've had no choice but to give up or to buckle down and get something that the developers can really bite into. I've created an example OSIS module. One section: Matt.1.1. No special characters, no footnotes, no headers, almost no markup at all really. I'm not compiling it into a compressed format. My command/output is: [bjdasc@ascpc5] osis2mod mod debug.osis.xml You are running osis2mod: $Rev: 2671 $ SUCCESS: osis2mod: has finished its work and will now rest I can't always predict *exactly* at what size it will fail (it seems to depend on things like number of paragraph markers, etc). But it always fails when the section gets to be roughly 64,000 - 65,000 characters. This is a pretty suspicious number, being just below 2^16. I assume that I can't exactly predict the size because there's some overhead, some amount for paragraph markers, header, etc. The attached example file fails (I've also included the conf file that I'm using to make life easier for any developer that wishes to look at this). The output is drastically shorter than the input. Please delete one character from the text in Matt.1.1. It all works nicely. This is not a front end problem. I can reproduce the problem in either Xiphos or diatheke. My diatheke command has been: diatheke -b Chrysostom -k Mt 1:1 But you can even see when it has failed by the size of the nt file created by osis2mod. This osis file, as attached, creates an nt file that is 70 bytes. Delete one character and it goes up to 65607 bytes. Note that 70 bytes + 2^16 bytes = 65606 bytes. PLEASE PLEASE PLEASE PLEASE ... I beg you ... I'm stuck without a fix to this ... it seems like it's an array size limit or a compiler size limit for some data type or something like that. If you need some legwork, then let me know; I'll do it. Thanks in advance, Brian -- --- The question is not Have you accepted Jesus into your heart, but Has Jesus accepted you into His heart?; not have you given your whole life for God, but has God given His whole life for you - Pr Bryan Wolfmueller, 2010 ___ sword-devel mailing list: sword-devel@crosswire.org http://www.crosswire.org/mailman/listinfo/sword-devel Instructions to unsubscribe/change your settings at above page Brian, Try adding the switch '-s 4' to your import. Also change your driver in the conf file to RawCom4. This will increase the per-entry size limit from 2^16 bytes (the default) to 2^32 bytes giving you an effective limit of 2GB for each verse or other entry. --Greg ___ sword-devel mailing list: sword-devel@crosswire.org http://www.crosswire.org/mailman/listinfo/sword-devel Instructions to unsubscribe/change your settings at above page -- --- The question is not Have you accepted Jesus into your heart, but Has Jesus accepted you into His heart?; not have you given your whole life for God, but has God given His whole life for you - Pr Bryan Wolfmueller, 2010 ___ sword-devel mailing list: sword-devel@crosswire.org http://www.crosswire.org/mailman/listinfo/sword-devel Instructions to unsubscribe/change your settings at above page
Re: [sword-devel] osis2mod bug - PLEASE HELP
On 2.3.2012 14:02, Brian J Dumont wrote: I'm not compiling it into a compressed format. My command/output is: [bjdasc@ascpc5] osis2mod mod debug.osis.xml You are running osis2mod: $Rev: 2671 $ SUCCESS: osis2mod: has finished its work and will now rest Interesting. Either it is in the version or in the fact that I have 64bit system and plenty of memory, but I don't see any problem with compiling your file here (Rev: 2686). The resulting mod directory is available on http://www.box.com/s/9jd8nixidljt9ckdogbi Best, Matěj ___ sword-devel mailing list: sword-devel@crosswire.org http://www.crosswire.org/mailman/listinfo/sword-devel Instructions to unsubscribe/change your settings at above page
Re: [sword-devel] osis2mod bug - PLEASE HELP
Hi Matej, The difference is that you compiled it as a compressed module. This makes the limit to be 64kb per section after it is zipped. If you triple the contents of the div then you'd have the same trouble. I avoided using a zipped module so that the issue wasn't confused, but it also happens with zipped modules. Thanks for looking at it, Brian On 03/02/2012 08:38 AM, Matej Cepl wrote: On 2.3.2012 14:02, Brian J Dumont wrote: I'm not compiling it into a compressed format. My command/output is: [bjdasc@ascpc5] osis2mod mod debug.osis.xml You are running osis2mod: $Rev: 2671 $ SUCCESS: osis2mod: has finished its work and will now rest Interesting. Either it is in the version or in the fact that I have 64bit system and plenty of memory, but I don't see any problem with compiling your file here (Rev: 2686). The resulting mod directory is available on http://www.box.com/s/9jd8nixidljt9ckdogbi Best, Matěj ___ sword-devel mailing list: sword-devel@crosswire.org http://www.crosswire.org/mailman/listinfo/sword-devel Instructions to unsubscribe/change your settings at above page -- --- The question is not Have you accepted Jesus into your heart, but Has Jesus accepted you into His heart?; not have you given your whole life for God, but has God given His whole life for you - Pr Bryan Wolfmueller, 2010 ___ sword-devel mailing list: sword-devel@crosswire.org http://www.crosswire.org/mailman/listinfo/sword-devel Instructions to unsubscribe/change your settings at above page
Re: [sword-devel] osis2mod bug - PLEASE HELP
2012/3/2 Brian J Dumont brian.j.dum...@gmail.com: Hi Matej, The difference is that you compiled it as a compressed module. This makes the limit to be 64kb per section after it is zipped. If you triple the contents of the div then you'd have the same trouble. I avoided using a zipped module so that the issue wasn't confused, but it also happens with zipped modules. I was under the impression that zipped modules do not suffer from the 64K section limit but perhaps I am mistaken. --Greg Thanks for looking at it, Brian On 03/02/2012 08:38 AM, Matej Cepl wrote: On 2.3.2012 14:02, Brian J Dumont wrote: I'm not compiling it into a compressed format. My command/output is: [bjdasc@ascpc5] osis2mod mod debug.osis.xml You are running osis2mod: $Rev: 2671 $ SUCCESS: osis2mod: has finished its work and will now rest Interesting. Either it is in the version or in the fact that I have 64bit system and plenty of memory, but I don't see any problem with compiling your file here (Rev: 2686). The resulting mod directory is available on http://www.box.com/s/9jd8nixidljt9ckdogbi Best, Matěj ___ sword-devel mailing list: sword-devel@crosswire.org http://www.crosswire.org/mailman/listinfo/sword-devel Instructions to unsubscribe/change your settings at above page -- --- The question is not Have you accepted Jesus into your heart, but Has Jesus accepted you into His heart?; not have you given your whole life for God, but has God given His whole life for you - Pr Bryan Wolfmueller, 2010 ___ sword-devel mailing list: sword-devel@crosswire.org http://www.crosswire.org/mailman/listinfo/sword-devel Instructions to unsubscribe/change your settings at above page ___ sword-devel mailing list: sword-devel@crosswire.org http://www.crosswire.org/mailman/listinfo/sword-devel Instructions to unsubscribe/change your settings at above page
Re: [sword-devel] osis2mod bug - PLEASE HELP
On 03/02/2012 09:13 AM, Greg Hellings wrote: 2012/3/2 Brian J Dumontbrian.j.dum...@gmail.com: Hi Matej, The difference is that you compiled it as a compressed module. This makes the limit to be 64kb per section after it is zipped. If you triple the contents of thediv then you'd have the same trouble. I avoided using a zipped module so that the issue wasn't confused, but it also happens with zipped modules. I was under the impression that zipped modules do not suffer from the 64K section limit but perhaps I am mistaken. --Greg I can't guarantee that the actual internal problem is identical, but if not then a very similar problem exists using zipped modules. I first noticed the problem when using compressed modules exclusively. Brian ___ sword-devel mailing list: sword-devel@crosswire.org http://www.crosswire.org/mailman/listinfo/sword-devel Instructions to unsubscribe/change your settings at above page
Re: [sword-devel] osis2mod bug - PLEASE HELP
On 03/02/2012 06:58 AM, Brian J Dumont wrote: On 03/02/2012 09:13 AM, Greg Hellings wrote: 2012/3/2 Brian J Dumontbrian.j.dum...@gmail.com: Hi Matej, The difference is that you compiled it as a compressed module. This makes the limit to be 64kb per section after it is zipped. If you triple the contents of thediv then you'd have the same trouble. I avoided using a zipped module so that the issue wasn't confused, but it also happens with zipped modules. I was under the impression that zipped modules do not suffer from the 64K section limit but perhaps I am mistaken. --Greg I can't guarantee that the actual internal problem is identical, but if not then a very similar problem exists using zipped modules. I first noticed the problem when using compressed modules exclusively. Brian I can confirm, to my surprise, that compressed modules appear unable to accommodate entries above 64kb. I tested osis2mod's ability to generate compressed modules as well as mod2zmod's ability to compress RawText4 modules, and neither work. I also tried out LZSS compression variable compression granularities, so it's probably something upstream in zText, zVerse, or such. --Chris ___ sword-devel mailing list: sword-devel@crosswire.org http://www.crosswire.org/mailman/listinfo/sword-devel Instructions to unsubscribe/change your settings at above page
