lawofcycles opened a new issue, #1300:
URL: https://github.com/apache/arrow-java/issues/1300
### Describe the bug, including details regarding any error messages,
version, and platform.
`Format.asString` builds the parameterised format strings with
`String.format("+w:%d", ...)` and no explicit `Locale`. `java.util.Formatter`
localises the digits of `%d` with `Locale.getDefault(Locale.Category.FORMAT)`,
so on a JVM whose default locale uses digits outside ASCII (`ar-EG`, `fa-IR`,
`bn-BD`, `mr-IN`, 98 of the 1069 locales available on JDK 21) a
`FixedSizeList(8)` is exported as `+w:٨` instead of `+w:8`. The same applies to
`FixedSizeBinary` and `Decimal`.
Affected calls:
*
https://github.com/apache/arrow-java/blob/v19.0.0/c/src/main/java/org/apache/arrow/c/Format.java#L66
*
https://github.com/apache/arrow-java/blob/v19.0.0/c/src/main/java/org/apache/arrow/c/Format.java#L68-L69
*
https://github.com/apache/arrow-java/blob/v19.0.0/c/src/main/java/org/apache/arrow/c/Format.java#L91
*
https://github.com/apache/arrow-java/blob/v19.0.0/c/src/main/java/org/apache/arrow/c/Format.java#L96
Other implementations parse the parameter as an ASCII integer and reject the
string. arrow-rs fails at
https://github.com/apache/arrow-rs/blob/58.0.0/arrow-schema/src/ffi.rs#L494-L503
with `C Data interface error: The FixedSizeList type requires an integer
parameter representing number of elements per list`. The Java importer reads
the parameter with `Integer.parseInt`
(https://github.com/apache/arrow-java/blob/v19.0.0/c/src/main/java/org/apache/arrow/c/Format.java#L352-L354),
which accepts Unicode digits, so a Java to Java round trip does not expose the
problem.
Reproduction (arrow-java 19.0.0 and 18.1.0, Temurin 21.0.6, run with
`--add-opens=java.base/java.nio=ALL-UNNAMED`):
```java
import java.nio.charset.StandardCharsets;
import java.util.HexFormat;
import java.util.List;
import java.util.Locale;
import org.apache.arrow.c.ArrowSchema;
import org.apache.arrow.c.Data;
import org.apache.arrow.c.NativeUtil;
import org.apache.arrow.memory.BufferAllocator;
import org.apache.arrow.memory.RootAllocator;
import org.apache.arrow.vector.types.FloatingPointPrecision;
import org.apache.arrow.vector.types.pojo.ArrowType;
import org.apache.arrow.vector.types.pojo.Field;
import org.apache.arrow.vector.types.pojo.FieldType;
public class FormatLocale {
public static void main(String[] args) {
Locale.setDefault(Locale.forLanguageTag("ar-EG"));
Field item = new Field("item",
FieldType.nullable(new
ArrowType.FloatingPoint(FloatingPointPrecision.SINGLE)), null);
Field vec = new Field("vec", FieldType.nullable(new
ArrowType.FixedSizeList(8)), List.of(item));
try (BufferAllocator allocator = new RootAllocator()) {
ArrowSchema schema = ArrowSchema.allocateNew(allocator);
Data.exportField(allocator, vec, null, schema);
String format = NativeUtil.toJavaString(schema.snapshot().format);
System.out.println(format + " "
+ HexFormat.ofDelimiter("
").formatHex(format.getBytes(StandardCharsets.UTF_8)));
System.out.println("Java import: " + Data.importField(allocator,
schema, null).getType());
}
}
}
```
```
+w:٨ 2b 77 3a d9 a8
Java import: FixedSizeList(8)
```
With `Locale.US` the first line is `+w:8 2b 77 3a 38`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
