cloud-fan commented on a change in pull request #28137: [SPARK-31361][SQL]
Rebase datetime in parquet/avro according to file written Spark version
URL: https://github.com/apache/spark/pull/28137#discussion_r404592294
##########
File path: sql/core/benchmarks/DateTimeRebaseBenchmark-jdk11-results.txt
##########
@@ -26,17 +15,6 @@ before 1582, vec off, rebase on 13836
13872
before 1582, vec on, rebase off 3664 3672
7 27.3 36.6 3.5X
before 1582, vec on, rebase on 6049 6078
30 16.5 60.5 2.1X
-OpenJDK 64-Bit Server VM 11.0.6+10-post-Ubuntu-1ubuntu118.04.1 on Linux
4.15.0-1063-aws
-Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
-Save timestamps to parquet: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
Review comment:
can't `toJavaTimestamp` benefit from your optimization?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
