Hi Gerd and Nick
@nick, when you say it works on 4143 but not on 4179, did you test code
-page=1255 on 4143, or just unicode?
It looks like there are 2 problems.
The --unicode assertion error is a 'byte' to 'int' sign-extension
problem, introduced in r4167 or r4168 and I've attached a patch that
fixes this and also removes some other confusing/obscuring type
representation changes in this area of the code.
The patch also has 4 commented out (because not appropriate for -
-unicode) asserts that seem to point to the Hebrew/1255 problem being
in imgfmt/app/srt/Sort.java:finish():
$ java -ea -jar /mkgmap/mkgmap.jar --code-page=1255 --gmapsupp test
-map:all-elements
Time started: Mon Apr 08 09:30:06 BST 2019
Exception in thread "main" java.lang.AssertionError: 16
at
uk.me.parabola.imgfmt.app.srt.Sort$Page.setSecondary(Sort.java:642)
at uk.me.parabola.imgfmt.app.srt.Sort.finish(Sort.java:161)
at
uk.me.parabola.mkgmap.srt.SrtTextReader.read(SrtTextReader.java:182)
at
uk.me.parabola.mkgmap.srt.SrtTextReader.<init>(SrtTextReader.java:117)
at
uk.me.parabola.mkgmap.srt.SrtTextReader.<init>(SrtTextReader.java:106)
at
uk.me.parabola.mkgmap.srt.SrtTextReader.sortForCodepage(SrtTextReader.j
ava:141)
at uk.me.parabola.mkgmap.main.Main.getSort(Main.java:764)
at
uk.me.parabola.mkgmap.main.Main.processFilename(Main.java:283)
at
uk.me.parabola.mkgmap.CommandArgsReader$Filename.processArg(CommandArgs
Reader.java:272)
at
uk.me.parabola.mkgmap.CommandArgsReader.readArgs(CommandArgsReader.java
:125)
at uk.me.parabola.mkgmap.main.Main.mainStart(Main.java:143)
at uk.me.parabola.mkgmap.main.Main.main(Main.java:114)
this is corresponds to:
} else {
p.setSecondary(i,
p.getSecondary(i) + maxSecondary);
}
where this increase in the secondary value is governed by some bits in the
pages.flags and the flags testing was changed in r3968, along with the
introduction of the original assert that was failing (SRTFile.java:117) and
quite a few other changes in srt.
@gerd, I'm afraid I don't understand the srt logic so can you have a
look at it
Regards
Ticker
On Sat, 2019-04-06 at 08:16 +0000, Gerd Petermann wrote:
> Hi,
> maybe a regression from r4167?
> http://www.mkgmap.org.uk/websvn/revision.php?repname=mkgmap&rev=4167
>
> Gerd
>
> ________________________________________
> Von: mkgmap-dev <[email protected]> im Auftrag
> von osm@pinns <[email protected]>
> Gesendet: Samstag, 6. April 2019 10:01
> An: [email protected]
> Betreff: Re: [mkgmap-dev] Building map with Hebrew characters
>
> Have done some checking using older versions
>
> It works on 4143 but not on 4179
>
> I'm afraid I don't have anything in between
>
> r
>
> Nick
>
> On 05/04/2019 21:25, Carlos Dávila wrote:
> > I'm trying to build a map of Israel with Hebrew characters, but
> > using
> > --code-page=1255 a get the following assertion:
> >
> > at
> > uk.me.parabola.imgfmt.app.srt.SRTFile.writeWeights(SRTFile.java:117
> > )
> > at
> > uk.me.parabola.imgfmt.app.srt.SRTFile.writeSrt4Chars(SRTFile.java:9
> > 9)
> > at
> > uk.me.parabola.imgfmt.app.srt.SRTFile.write(SRTFile.java:63)
> > at
> > uk.me.parabola.mkgmap.combiners.MdrBuilder.init(MdrBuilder.java:117
> > )
> > at
> > uk.me.parabola.mkgmap.main.Main.endOptions(Main.java:605)
> > at
> > uk.me.parabola.mkgmap.CommandArgsReader.readArgs(CommandArgsReader.
> > java:128)
> > at uk.me.parabola.mkgmap.main.Main.mainStart(Main.java:143)
> > at uk.me.parabola.mkgmap.main.Main.main(Main.java:114)
> >
> > With --unicode I also get an assertion:
> > Exception in thread "main" java.lang.AssertionError: -125
> > at
> > uk.me.parabola.imgfmt.app.BufferedImgFileWriter.put1u(BufferedImgFi
> > leWriter.java:154)
> > at
> > uk.me.parabola.imgfmt.app.SectionWriter.put1u(SectionWriter.java:78
> > )
> > at
> > uk.me.parabola.imgfmt.app.srt.SRTFile.writeWeights(SRTFile.java:113
> > )
> > at
> > uk.me.parabola.imgfmt.app.srt.SRTFile.writeSrt8(SRTFile.java:166)
> > at
> > uk.me.parabola.imgfmt.app.srt.SRTFile.write(SRTFile.java:68)
> > at
> > uk.me.parabola.mkgmap.combiners.MdrBuilder.init(MdrBuilder.java:117
> > )
> > at
> > uk.me.parabola.mkgmap.main.Main.endOptions(Main.java:605)
> > at
> > uk.me.parabola.mkgmap.CommandArgsReader.readArgs(CommandArgsReader.
> > java:128)
> > at uk.me.parabola.mkgmap.main.Main.mainStart(Main.java:143)
> > at uk.me.parabola.mkgmap.main.Main.main(Main.java:114)
> >
> > Probably I'm missing something obvious, but I always used
> > --code-page=1251 and have no idea what it may be
> >
> > _______________________________________________
> > mkgmap-dev mailing list
> > [email protected]
> > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
> _______________________________________________
> mkgmap-dev mailing list
> [email protected]
> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
> _______________________________________________
> mkgmap-dev mailing list
> [email protected]
> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-devIndex: src/uk/me/parabola/imgfmt/app/srt/CodePosition.java
===================================================================
--- src/uk/me/parabola/imgfmt/app/srt/CodePosition.java (revision 4285)
+++ src/uk/me/parabola/imgfmt/app/srt/CodePosition.java (working copy)
@@ -20,19 +20,19 @@
* @author Steve Ratcliffe
*/
public class CodePosition {
- private char primary;
- private byte secondary;
- private byte tertiary;
+ private int primary;
+ private int secondary;
+ private int tertiary;
- public char getPrimary() {
+ public int getPrimary() {
return primary;
}
- public byte getSecondary() {
+ public int getSecondary() {
return secondary;
}
- public byte getTertiary() {
+ public int getTertiary() {
return tertiary;
}
@@ -47,23 +47,25 @@
case Collator.PRIMARY:
return primary;
case Collator.SECONDARY:
- return secondary & 0xff;
+ return secondary;
case Collator.TERTIARY:
- return tertiary & 0xff;
+ return tertiary;
default:
return 0;
}
}
- public void setPrimary(char primary) {
+ public void setPrimary(int primary) {
this.primary = primary;
}
- public void setSecondary(byte secondary) {
+ public void setSecondary(int secondary) {
+// assert secondary >= 0 && secondary <= 0xf: secondary;
this.secondary = secondary;
}
- public void setTertiary(byte tertiary) {
+ public void setTertiary(int tertiary) {
+// assert tertiary >= 0 && tertiary <= 0xf: tertiary;
this.tertiary = tertiary;
}
}
Index: src/uk/me/parabola/imgfmt/app/srt/SRTFile.java
===================================================================
--- src/uk/me/parabola/imgfmt/app/srt/SRTFile.java (revision 4285)
+++ src/uk/me/parabola/imgfmt/app/srt/SRTFile.java (working copy)
@@ -106,16 +106,16 @@
int secondary = sort.getSecondary(i);
int tertiary = sort.getTertiary(i);
if (isMulti) {
- assert primary <= 0xffff;
- assert secondary <= 0xff;
- assert tertiary <= 0xff;
+ assert primary <= 0xffff : primary;
+ assert secondary <= 0xff : secondary;
+ assert tertiary <= 0xff : tertiary;
writer.put2u(primary);
writer.put1u(secondary);
writer.put1u(tertiary);
} else {
- assert primary <= 0xff;
- assert secondary <= 0xf;
- assert tertiary <= 0xf;
+ assert primary <= 0xff : primary;
+ assert secondary <= 0xf : secondary;
+ assert tertiary <= 0xf : tertiary;
writer.put1u(primary);
writer.put1u((tertiary << 4) | (secondary & 0xf));
}
Index: src/uk/me/parabola/imgfmt/app/srt/Sort.java
===================================================================
--- src/uk/me/parabola/imgfmt/app/srt/Sort.java (revision 4285)
+++ src/uk/me/parabola/imgfmt/app/srt/Sort.java (working copy)
@@ -135,7 +135,7 @@
for (int i = 0; i < 256; i++) {
if (((p.flags[i] >>> 4) & 0xf) == 0) {
if (p.getPrimary(i) != 0) {
- byte second = p.getSecondary(i);
+ int second = p.getSecondary(i);
maxSecondary = Math.max(maxSecondary, second);
if (second != 0) {
maxTertiary = Math.max(maxTertiary, p.getTertiary(i));
@@ -626,7 +626,7 @@
private final byte[] tertiary = new byte[256];
private final byte[] flags = new byte[256];
- char getPrimary(int ch) {
+ int getPrimary(int ch) {
return primary[ch & 0xff];
}
@@ -634,19 +634,21 @@
primary[ch & 0xff] = (char) val;
}
- byte getSecondary(int ch) {
- return secondary[ch & 0xff];
+ int getSecondary(int ch) {
+ return secondary[ch & 0xff] & 0xff;
}
void setSecondary(int ch, int val) {
+// assert val >= 0 && val <= 0xf: val;
secondary[ch & 0xff] = (byte) val;
}
- byte getTertiary(int ch) {
- return tertiary[ch & 0xff];
+ int getTertiary(int ch) {
+ return tertiary[ch & 0xff] & 0xff;
}
void setTertiary(int ch, int val) {
+// assert val >= 0 && val <= 0xf: val;
tertiary[ch & 0xff] = (byte) val;
}
@@ -661,9 +663,9 @@
case Collator.PRIMARY:
return getPrimary(ch) & 0xffff;
case Collator.SECONDARY:
- return getSecondary(ch) & 0xff;
+ return getSecondary(ch);
case Collator.TERTIARY:
- return getTertiary(ch) & 0xff;
+ return getTertiary(ch);
default:
assert false : "bad collation type passed";
return 0;
Index: src/uk/me/parabola/imgfmt/app/srt/SrtFileReader.java
===================================================================
--- src/uk/me/parabola/imgfmt/app/srt/SrtFileReader.java (revision 4285)
+++ src/uk/me/parabola/imgfmt/app/srt/SrtFileReader.java (working copy)
@@ -173,20 +173,21 @@
int rec;
if (posLength == 2) {
rec = reader.get2u();
- cp.setPrimary((char) (rec & 0xff));
- cp.setSecondary((byte) ((rec >> 8) & 0xf));
- cp.setTertiary((byte) ((rec >> 12) & 0xf));
+ cp.setPrimary(rec & 0xff);
+ cp.setSecondary((rec >> 8) & 0xf);
+ cp.setTertiary((rec >> 12) & 0xf);
} else if (posLength == 3) {
+ // what is the extra byte for ???
rec = reader.get3u();
- cp.setPrimary((char) (rec & 0xff));
- cp.setSecondary((byte) ((rec >> 8) & 0xf));
- cp.setTertiary((byte) ((rec >> 12) & 0xf));
+ cp.setPrimary(rec & 0xff);
+ cp.setSecondary((rec >> 8) & 0xf);
+ cp.setTertiary((rec >> 12) & 0xf);
} else if (posLength == 4) {
rec = reader.get4();
- cp.setPrimary((char) (rec & 0xffff));
- cp.setSecondary((byte) ((rec >> 16) & 0xff));
- cp.setTertiary((byte) ((rec >> 24) & 0xff));
+ cp.setPrimary(rec & 0xffff);
+ cp.setSecondary((rec >> 16) & 0xff);
+ cp.setTertiary((rec >> 24) & 0xff);
} else {
throw new RuntimeException("unexpected value posLength " + posLength);
}
Index: src/uk/me/parabola/mkgmap/srt/SrtTextReader.java
===================================================================
--- src/uk/me/parabola/mkgmap/srt/SrtTextReader.java (revision 4285)
+++ src/uk/me/parabola/mkgmap/srt/SrtTextReader.java (working copy)
@@ -329,7 +329,7 @@
CodePosition cp = new CodePosition();
int b = r.getBval();
int primary = sort.getPrimary(b);
- cp.setPrimary((char) primary);
+ cp.setPrimary(primary);
// We do not want the character to sort fully equal to the expanded characters (or any other
// character so adjust the ordering at other strengths. May need further tweaks.
@@ -346,15 +346,15 @@
} else {
secondary = 1;
}
- cp.setSecondary((byte) (secondary));
- cp.setTertiary((byte) (tertiary));
+ cp.setSecondary(secondary);
+ cp.setTertiary(tertiary);
} else {
num++;
- secondary = sort.getSecondary(b) & 0xff;
- cp.setSecondary((byte) (secondary + 7));
+ secondary = sort.getSecondary(b);
+ cp.setSecondary(secondary + 7);
- tertiary = sort.getTertiary(b) & 0xff;
- cp.setTertiary((byte) (tertiary + 2));
+ tertiary = sort.getTertiary(b);
+ cp.setTertiary(tertiary + 2);
}
expansions.add(cp);
}
_______________________________________________
mkgmap-dev mailing list
[email protected]
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
