wgtmac commented on code in PR #1902:
URL: https://github.com/apache/orc/pull/1902#discussion_r1592651023
##########
java/bench/core/src/java/org/apache/orc/bench/core/convert/parquet/ParquetReader.java:
##########
@@ -54,8 +56,19 @@ public boolean nextBatch(VectorizedRowBatch batch) throws
IOException {
break;
}
int row = batch.size++;
- for(int c=0; c < converters.length; ++c) {
- converters[c].convert(batch.cols[c], row, value.get(c));
+ for (int c = 0; c < converters.length; ++c) {
+ Object o = value.get(c);
+ if (convertMicroseconds[c] == null) {
+ String fieldSchema =
value.getSchema().getFields().get(c).schema().toString();
+ convertMicroseconds[c] = fieldSchema.equals(
Review Comment:
I'm not familiar with Apache Avro. Is there a better way to get the
timestamp precision instead of comparing string representation?
##########
java/bench/core/src/java/org/apache/orc/bench/core/convert/parquet/ParquetReader.java:
##########
@@ -54,8 +56,19 @@ public boolean nextBatch(VectorizedRowBatch batch) throws
IOException {
break;
}
int row = batch.size++;
- for(int c=0; c < converters.length; ++c) {
- converters[c].convert(batch.cols[c], row, value.get(c));
+ for (int c = 0; c < converters.length; ++c) {
+ Object o = value.get(c);
+ if (convertMicroseconds[c] == null) {
+ String fieldSchema =
value.getSchema().getFields().get(c).schema().toString();
+ convertMicroseconds[c] = fieldSchema.equals(
+
"[\"null\",{\"type\":\"long\",\"logicalType\":\"timestamp-micros\"}]");
+ }
+ if (convertMicroseconds[c]) {
+ // The data format of taxi is parquet, and timestamp uses
microseconds,
+ // which is inconsistent with the milliseconds of Java's
java.sql.Timestamp.
+ o = (Long) o / 1000;
Review Comment:
Doesn't this lose the digits of microsecond?
##########
java/bench/core/src/java/org/apache/orc/bench/core/convert/parquet/ParquetReader.java:
##########
@@ -54,8 +56,19 @@ public boolean nextBatch(VectorizedRowBatch batch) throws
IOException {
break;
}
int row = batch.size++;
- for(int c=0; c < converters.length; ++c) {
- converters[c].convert(batch.cols[c], row, value.get(c));
+ for (int c = 0; c < converters.length; ++c) {
+ Object o = value.get(c);
+ if (convertMicroseconds[c] == null) {
+ String fieldSchema =
value.getSchema().getFields().get(c).schema().toString();
+ convertMicroseconds[c] = fieldSchema.equals(
+
"[\"null\",{\"type\":\"long\",\"logicalType\":\"timestamp-micros\"}]");
+ }
+ if (convertMicroseconds[c]) {
+ // The data format of taxi is parquet, and timestamp uses
microseconds,
Review Comment:
IMHO the comment should be agnostic to any dataset. We might simply say that
Avro record has two different timestamp precisions (micro, milli)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
