[
https://issues.apache.org/jira/browse/AVRO-2904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dan Lipofsky updated AVRO-2904:
-------------------------------
Description:
In Avro 1.10.0 the setter for logical-type timestamp-millis will truncate the
timestamp to millis in the simplest condition but not when it is in a union
with null.
It looks like something similar was addressed in AVRO-2360 but that was
supposedly fixed in 1.9.0 and this is still broken in 1.10.0.
Note: I don't see this in Java8, only Java11 - it would appear that in Java8
the {{Instant}} is already truncated to millis so it is never a problem.
Here are my schemas
{noformat}
{
"type" : "record",
"name" : "TimestampTest1",
"fields" : [ {
"name" : "time",
"type" : [ "null", { "type" : "long", "logicalType" : "timestamp-millis" } ]
} ]
}
{
"type" : "record",
"name" : "TimestampTest2",
"fields" : [ {
"name" : "time",
"type" : { "type" : "long", "logicalType" : "timestamp-millis" }
} ]
}
{noformat}
Here is my test, in which {{TimestampTest2}} passes but {{TimestampTest1}}
fails.
{noformat}
public class Avro2904Test {
// If you just used Instant.now() test would randomly pass 1 times out of a
1000.
// If you use Instant.now().truncatedTo(ChronoUnit.MILLIS) it would always
pass.
// By making sure micros is non-zero we ensure it always fails and
demonstrates AVRO-2904.
private Instant now =
Instant.now().truncatedTo(ChronoUnit.MILLIS).plus(999, ChronoUnit.MICROS);
@Test
public void test1() throws IOException {
final TimestampTest1 t =
TimestampTest1.newBuilder().setTime(now).build();
byte[] bytes = serialize(t);
TimestampTest1 other = deserialize(t.getSchema(), bytes);
// fails because of AVRO-2904
assertEquals(t, other);
}
@Test
public void test2() throws IOException {
final TimestampTest2 t =
TimestampTest2.newBuilder().setTime(Instant.now()).build();
byte[] bytes = serialize(t);
TimestampTest2 other = deserialize(t.getSchema(), bytes);
// if logicalType timestamp-millis in not in union with null it works
fine
assertEquals(t, other);
}
private byte[] serialize(SpecificRecord record) throws IOException {
try (final ByteArrayOutputStream out = new ByteArrayOutputStream()) {
SpecificDatumWriter<SpecificRecord> writer = new
SpecificDatumWriter<>(
record.getSchema());
Encoder encoder = EncoderFactory.get().binaryEncoder(out, null);
writer.write(record, encoder);
encoder.flush();
final byte[] bytes = out.toByteArray();
return bytes;
}
}
private <T extends SpecificRecord> T deserialize(Schema schema, byte[]
bytes)
throws IOException {
Decoder decoder = DecoderFactory.get().binaryDecoder(bytes, null);
SpecificDatumReader<T> reader = new SpecificDatumReader<>(schema);
return reader.read(null, decoder);
}
}
{noformat}
{{TimestampTest1}} fails with
{noformat}
java.lang.AssertionError:
Expected :{"time": 2020-07-30T23:14:08.294999Z}
Actual :{"time": 2020-07-30T23:14:08.294Z}
{noformat}
If I change my tests to use {{Instant.now().truncatedTo(ChronoUnit.MILLIS)}}
they'll both pass, but it seems like that should not be necessary, particularly
given the code below.
Looking at the generated Java DTO, I see a few differences, but the most
important would seem to be that the working DTO has
{noformat}
public void setTime(java.time.Instant value) {
this.time = value.truncatedTo(java.time.temporal.ChronoUnit.MILLIS);
}
{noformat}
while the broken one has
{noformat}
public void setTime(java.time.Instant value) {
this.time = value;
}
{noformat}
The {{Builder}} is similar (only the non-union version has {{truncatedTo}} in
the setter).
The working one also has this code, which is missing from the broken one:
{noformat}
private static final org.apache.avro.Conversion<?>[] conversions =
new org.apache.avro.Conversion<?>[] {
new org.apache.avro.data.TimeConversions.TimestampMillisConversion(),
null
};
@Override
public org.apache.avro.Conversion<?> getConversion(int field) {
return conversions[field];
}
{noformat}
was:
In Avro 1.10.0 the setter for logical-type timestamp-millis will truncate the
timestamp to millis in the simplest condition but not when it is in a union
with null.
It looks like something similar was addressed in AVRO-2360 but that was
supposedly fixed in 1.9.0 and this is still broken in 1.10.0.
Here is my IDL
{noformat}
record TimestampTest1 {
union { null, timestamp_ms } ts;
}
record TimestampTest2 {
timestamp_ms ts;
}
{noformat}
Here is my test, in which {{TimestampTest2}} passes but {{TimestampTest1}}
fails.
{noformat}
public class SerDeTest {
@Test
public void TimestampTest1() throws IOException {
final TimestampTest1 ts =
TimestampTest1.newBuilder().setTs(Instant.now()).build();
byte[] bytes = serialize(ts);
TimestampTest1 other = deserialize(ts.getSchema(), bytes);
Assert.assertEquals(ts, other);
}
@Test
public void TimestampTest2() throws IOException {
final TimestampTest2 ts =
TimestampTest2.newBuilder().setTs(Instant.now()).build();
byte[] bytes = serialize(ts);
TimestampTest2 other = deserialize(ts.getSchema(), bytes);
Assert.assertEquals(ts, other);
}
private <T extends SpecificRecord> T deserialize(Schema schema, byte[]
bytes)
throws IOException {
Decoder decoder = DecoderFactory.get().binaryDecoder(bytes, null);
SpecificDatumReader<T> reader = new SpecificDatumReader<>(schema);
return reader.read(null, decoder);
}
private byte[] serialize(SpecificRecord record) throws IOException {
try (final ByteArrayOutputStream out = new ByteArrayOutputStream()) {
SpecificDatumWriter<SpecificRecord> writer = new
SpecificDatumWriter<>(
record.getSchema());
Encoder encoder = EncoderFactory.get().binaryEncoder(out, null);
writer.write(record, encoder);
encoder.flush();
return out.toByteArray();
}
}
}
{noformat}
{{TimestampTest1}} fails with
{noformat}
java.lang.AssertionError:
Expected :{"ts": 2020-07-23T21:35:12.348379Z}
Actual :{"ts": 2020-07-23T21:35:12.348Z}
{noformat}
If I change my tests to use {{Instant.now().truncatedTo(ChronoUnit.MILLIS)}}
they'll both pass, but it seems like that should not be necessary, particularly
given the code below.
Looking at the generated Java DTO, I see a few differences, but the most
important would seem to be that the working DTO has
{noformat}
public void setTs(java.time.Instant value) {
this.ts = value.truncatedTo(java.time.temporal.ChronoUnit.MILLIS);
}
{noformat}
while the broken one has
{noformat}
public void setTs(java.time.Instant value) {
this.ts = value;
}
{noformat}
The working one also has this code, which is missing from the broken one:
{noformat}
private static final org.apache.avro.Conversion<?>[] conversions =
new org.apache.avro.Conversion<?>[] {
new org.apache.avro.data.TimeConversions.TimestampMillisConversion(),
null
};
@Override
public org.apache.avro.Conversion<?> getConversion(int field) {
return conversions[field];
}
{noformat}
> timestamp-millis doesn't truncate when in a union with null
> -----------------------------------------------------------
>
> Key: AVRO-2904
> URL: https://issues.apache.org/jira/browse/AVRO-2904
> Project: Apache Avro
> Issue Type: Bug
> Components: java, logical types
> Affects Versions: 1.10.0
> Reporter: Dan Lipofsky
> Priority: Major
>
> In Avro 1.10.0 the setter for logical-type timestamp-millis will truncate the
> timestamp to millis in the simplest condition but not when it is in a union
> with null.
> It looks like something similar was addressed in AVRO-2360 but that was
> supposedly fixed in 1.9.0 and this is still broken in 1.10.0.
> Note: I don't see this in Java8, only Java11 - it would appear that in Java8
> the {{Instant}} is already truncated to millis so it is never a problem.
> Here are my schemas
> {noformat}
> {
> "type" : "record",
> "name" : "TimestampTest1",
> "fields" : [ {
> "name" : "time",
> "type" : [ "null", { "type" : "long", "logicalType" : "timestamp-millis"
> } ]
> } ]
> }
> {
> "type" : "record",
> "name" : "TimestampTest2",
> "fields" : [ {
> "name" : "time",
> "type" : { "type" : "long", "logicalType" : "timestamp-millis" }
> } ]
> }
> {noformat}
> Here is my test, in which {{TimestampTest2}} passes but {{TimestampTest1}}
> fails.
> {noformat}
> public class Avro2904Test {
> // If you just used Instant.now() test would randomly pass 1 times out of
> a 1000.
> // If you use Instant.now().truncatedTo(ChronoUnit.MILLIS) it would
> always pass.
> // By making sure micros is non-zero we ensure it always fails and
> demonstrates AVRO-2904.
> private Instant now =
> Instant.now().truncatedTo(ChronoUnit.MILLIS).plus(999, ChronoUnit.MICROS);
> @Test
> public void test1() throws IOException {
> final TimestampTest1 t =
> TimestampTest1.newBuilder().setTime(now).build();
> byte[] bytes = serialize(t);
> TimestampTest1 other = deserialize(t.getSchema(), bytes);
> // fails because of AVRO-2904
> assertEquals(t, other);
> }
> @Test
> public void test2() throws IOException {
> final TimestampTest2 t =
> TimestampTest2.newBuilder().setTime(Instant.now()).build();
> byte[] bytes = serialize(t);
> TimestampTest2 other = deserialize(t.getSchema(), bytes);
> // if logicalType timestamp-millis in not in union with null it works
> fine
> assertEquals(t, other);
> }
> private byte[] serialize(SpecificRecord record) throws IOException {
> try (final ByteArrayOutputStream out = new ByteArrayOutputStream()) {
> SpecificDatumWriter<SpecificRecord> writer = new
> SpecificDatumWriter<>(
> record.getSchema());
> Encoder encoder = EncoderFactory.get().binaryEncoder(out, null);
> writer.write(record, encoder);
> encoder.flush();
> final byte[] bytes = out.toByteArray();
> return bytes;
> }
> }
> private <T extends SpecificRecord> T deserialize(Schema schema, byte[]
> bytes)
> throws IOException {
> Decoder decoder = DecoderFactory.get().binaryDecoder(bytes, null);
> SpecificDatumReader<T> reader = new SpecificDatumReader<>(schema);
> return reader.read(null, decoder);
> }
> }
> {noformat}
> {{TimestampTest1}} fails with
> {noformat}
> java.lang.AssertionError:
> Expected :{"time": 2020-07-30T23:14:08.294999Z}
> Actual :{"time": 2020-07-30T23:14:08.294Z}
> {noformat}
> If I change my tests to use {{Instant.now().truncatedTo(ChronoUnit.MILLIS)}}
> they'll both pass, but it seems like that should not be necessary,
> particularly given the code below.
> Looking at the generated Java DTO, I see a few differences, but the most
> important would seem to be that the working DTO has
> {noformat}
> public void setTime(java.time.Instant value) {
> this.time = value.truncatedTo(java.time.temporal.ChronoUnit.MILLIS);
> }
> {noformat}
> while the broken one has
> {noformat}
> public void setTime(java.time.Instant value) {
> this.time = value;
> }
> {noformat}
> The {{Builder}} is similar (only the non-union version has {{truncatedTo}} in
> the setter).
> The working one also has this code, which is missing from the broken one:
> {noformat}
> private static final org.apache.avro.Conversion<?>[] conversions =
> new org.apache.avro.Conversion<?>[] {
> new org.apache.avro.data.TimeConversions.TimestampMillisConversion(),
> null
> };
> @Override
> public org.apache.avro.Conversion<?> getConversion(int field) {
> return conversions[field];
> }
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
