SEPURI-SAI-KRISHNA opened a new issue, #13042:
URL: https://github.com/apache/seatunnel/issues/13042
### Search before asking
- [X] I had searched in the issues and found no similar issues.
### What happened
When the Lance sink creates a dataset, it declares every `TIMESTAMP` column
as `Timestamp(MICROSECOND, "Asia/Shanghai")` (`SchemaUtils.java:125-130`), and
the sink documentation promises "timestamp (microsecond, Asia/Shanghai
timezone)" (`docs/en/connectors/sink/Lance.md:149`, zh `:141`).
The value that goes into that column comes from
`BaseTypeWriter.convertToEpochMicro`:
```java
if (value instanceof LocalDateTime) {
return ((LocalDateTime)
value).atZone(ZoneId.systemDefault()).toInstant().toEpochMilli()
* 1000;
} else if (value instanceof java.sql.Timestamp) {
return ((java.sql.Timestamp) value).getTime() * 1000;
```
`toEpochMilli() * 1000` and `getTime() * 1000` drop everything below the
millisecond, so the column has microsecond resolution but only ever holds whole
milliseconds. Every `TIMESTAMP` reaches this method: `TypeWriterFactory` maps
each `ArrowType.Timestamp` to `TimestampTypeWriter`, which calls it from
`writeToVector` and `writeToListWriter`.
Measured on `dev` with the real `LanceSinkWriter` writing through the native
Lance library into a new dataset, then reading the column back. The value shown
is the fraction of a second, in microseconds:
| id | written | stored |
| --- | --- | --- |
| 1 | `2025-02-15T10:21:23.123456` | `123000` |
| 2 | `1960-06-01T12:00:00.000250` | `0` |
| 3 | `1969-12-31T23:59:59.999500` | `999000` |
The conversion loses the same digits with the JVM in UTC, Asia/Shanghai and
America/New_York. A `java.sql.Timestamp` loses its sub-millisecond nanos the
same way.
`LanceSinkTest` does not catch this: its timestamp value is commented out
under `// TODO: support date/time/timestamp`.
### What you expected to happen
`2025-02-15T10:21:23.123456` is stored with `123456` microseconds, as the
declared unit and the documentation say.
### SeaTunnel Version
dev, verified at commit `f867a5410`.
### SeaTunnel Config
```conf
env {
parallelism = 1
job.mode = "BATCH"
}
source {
FakeSource {
schema = {
fields {
id = int
c_timestamp = timestamp
}
}
rows = [
{
kind = INSERT
fields = [1, "2025-02-15T10:21:23.123456"]
}
]
}
}
sink {
Lance {
dataset_path = "/tmp/lance_timestamp_table"
namespace_type = "dir"
namespace_id = "root"
table = "lance_timestamp_table"
}
}
```
### Running Command
```shell
./bin/seatunnel.sh --config config/fake_to_lance.conf -e local
```
### Error Exception
```log
None. The job succeeds, and the stored value has 000 in place of the last
three microsecond digits.
```
### Zeta or Flink or Spark Version
Not engine specific. The conversion happens in the sink writer.
### Java or Scala Version
Not version specific. Measured on JDK 17.
### Are you willing to submit PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's Code of Conduct
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
