HeartSaVioR commented on pull request #1487:
URL: https://github.com/apache/iceberg/pull/1487#issuecomment-698148650
One of new tests are failing on Spark 2.4 which didn't fail in IDEA. It
looks to be only failing with Gradle.
`gradle :iceberg-spark2:test`
```
org.apache.iceberg.spark.source.TestPartitionPruning24 >
testPartitionPruningBucketingInteger[0] FAILED
java.lang.AssertionError: Some of data files in partition range should
be read.
Read files in query:
[/tmp/junit2992201386166743058/logs-742071/metadata/snap-6042077398117050700-1-c705c4d6-eee7-4155-8959-c9c87c8ee790.avro,
/tmp/junit2992201386166743058/logs-742071/metadata/c705c4d6-eee7-4155-8959-c9c87c8ee790-m0.avro,
/tmp/junit2992201386166743058/logs-742071/metadata/version-hint.text,
/tmp/junit2992201386166743058/logs-742071/metadata/v2.metadata.json] /
data files in partition range:
[/tmp/junit2992201386166743058/logs-742071/data/date=2020-02-03/level=INFO/id_bucket=0/message_trunc=info+/timestamp_hour=2020-02-03-00/00005-104-9228c157-fa0d-4a0c-a7f0-0aa69acee184-00001.parquet,
/tmp/junit2992201386166743058/logs-742071/data/date=2020-02-02/level=INFO/id_bucket=0/message_trunc=info+/timestamp_hour=2020-02-02-01/00002-101-0fd6786b-fae7-400b-a696-bcbed73dfcde-00001.parquet,
/tmp/junit2992201386166743058/logs-742071/data/date=2020-02-04/level=DEBUG/id_bucket=0/message_trunc=debug/timestamp_hour=2020-02-04-01/00007-106-500c178d-079a-4afa-b0d3-4a396c4fb630-00001.parquet]
org.apache.iceberg.spark.source.TestPartitionPruning24 >
testPartitionPruningBucketingInteger[1] FAILED
java.lang.AssertionError: Some of data files in partition range should
be read.
Read files in query:
[/tmp/junit6453382097752467142/logs-387757/metadata/v2.metadata.json,
/tmp/junit6453382097752467142/logs-387757/metadata/snap-3808976267395158188-1-411c0913-6e93-43a5-a375-a7e6f7b0a5a8.avro,
/tmp/junit6453382097752467142/logs-387757/metadata/version-hint.text,
/tmp/junit6453382097752467142/logs-387757/metadata/411c0913-6e93-43a5-a375-a7e6f7b0a5a8-m0.avro]
/
data files in partition range:
[/tmp/junit6453382097752467142/logs-387757/data/date=2020-02-04/level=DEBUG/id_bucket=0/message_trunc=debug/timestamp_hour=2020-02-04-01/00007-217-b5ecf7bf-5207-4470-adf9-58b1571ff7b5-00001.parquet,
/tmp/junit6453382097752467142/logs-387757/data/date=2020-02-03/level=INFO/id_bucket=0/message_trunc=info+/timestamp_hour=2020-02-03-00/00005-215-8accdc56-06d0-4fec-9a18-ad10d350ccc8-00001.parquet,
/tmp/junit6453382097752467142/logs-387757/data/date=2020-02-02/level=INFO/id_bucket=0/message_trunc=info+/timestamp_hour=2020-02-02-01/00002-212-44a0d2fb-48d3-439a-8818-e800e98b5c7c-00001.parquet]
org.apache.iceberg.spark.source.TestPartitionPruning24 >
testPartitionPruningBucketingInteger[3] FAILED
java.lang.AssertionError: Some of data files in partition range should
be read.
Read files in query:
[/tmp/junit5114414994557432305/logs-606638/metadata/version-hint.text,
/tmp/junit5114414994557432305/logs-606638/metadata/snap-7702056315490708928-1-aa5dc06e-b173-4df3-9af0-e6f907ea3c37.avro,
/tmp/junit5114414994557432305/logs-606638/metadata/v2.metadata.json,
/tmp/junit5114414994557432305/logs-606638/metadata/aa5dc06e-b173-4df3-9af0-e6f907ea3c37-m0.avro]
/
data files in partition range:
[/tmp/junit5114414994557432305/logs-606638/data/date=2020-02-02/level=INFO/id_bucket=0/message_trunc=info+/timestamp_hour=2020-02-02-01/00002-434-39ad40a9-3b76-47ec-820b-5a305edabd79-00001.orc,
/tmp/junit5114414994557432305/logs-606638/data/date=2020-02-03/level=INFO/id_bucket=0/message_trunc=info+/timestamp_hour=2020-02-03-00/00005-437-e7545e12-e5f8-40cd-98e7-177969365c5a-00001.orc,
/tmp/junit5114414994557432305/logs-606638/data/date=2020-02-04/level=DEBUG/id_bucket=0/message_trunc=debug/timestamp_hour=2020-02-04-01/00007-439-d424d5d2-336f-484c-a499-6478f11f1602-00001.orc]
org.apache.iceberg.spark.source.TestPartitionPruning24 >
testPartitionPruningBucketingInteger[4] FAILED
java.lang.AssertionError: Some of data files in partition range should
be read.
Read files in query:
[/tmp/junit8082590320046964751/logs-144508/metadata/v2.metadata.json,
/tmp/junit8082590320046964751/logs-144508/metadata/361a3e56-58b7-4c4c-827b-cc83326d918a-m0.avro,
/tmp/junit8082590320046964751/logs-144508/metadata/snap-1068076646929597670-1-361a3e56-58b7-4c4c-827b-cc83326d918a.avro,
/tmp/junit8082590320046964751/logs-144508/metadata/version-hint.text] /
data files in partition range:
[/tmp/junit8082590320046964751/logs-144508/data/date=2020-02-03/level=INFO/id_bucket=0/message_trunc=info+/timestamp_hour=2020-02-03-00/00005-548-4b531da9-09c9-4d16-a851-ed6b5cd35379-00001.orc,
/tmp/junit8082590320046964751/logs-144508/data/date=2020-02-04/level=DEBUG/id_bucket=0/message_trunc=debug/timestamp_hour=2020-02-04-01/00007-550-6e838e99-33b0-43b7-b2af-41585e12b4ea-00001.orc,
/tmp/junit8082590320046964751/logs-144508/data/date=2020-02-02/level=INFO/id_bucket=0/message_trunc=info+/timestamp_hour=2020-02-02-01/00002-545-8c2a7444-646f-45ea-b082-f4db470db1c3-00001.orc]
```
That is not related to the timestamp issue - tracker doesn't see any "data"
files being read, and query result looks to be (surprisingly) empty. (I
modified the test a bit to check whether the query result is empty or not, and
it failed on the check.) It's not reproducible in IDEA, very weird.
Digging on this.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
