rdblue commented on a change in pull request #1747:
URL: https://github.com/apache/iceberg/pull/1747#discussion_r521667394
##########
File path:
api/src/test/java/org/apache/iceberg/expressions/TestInclusiveMetricsEvaluator.java
##########
@@ -156,6 +192,54 @@ public void testNoNulls() {
Assert.assertFalse("Should skip: non-null column contains no null values",
shouldRead);
}
+ @Test
+ public void testIsNaN() {
+ boolean shouldRead = new InclusiveMetricsEvaluator(SCHEMA,
isNaN("all_nans")).eval(FILE);
+ Assert.assertTrue("Should read: at least one nan value in all nan column",
shouldRead);
+
+ shouldRead = new InclusiveMetricsEvaluator(SCHEMA,
isNaN("some_nans")).eval(FILE);
+ Assert.assertTrue("Should read: at least one nan value in some nan
column", shouldRead);
+
+ shouldRead = new InclusiveMetricsEvaluator(SCHEMA,
isNaN("no_nans")).eval(FILE);
+ Assert.assertFalse("Should skip: no-nans column contains no nan values",
shouldRead);
+
+ shouldRead = new InclusiveMetricsEvaluator(SCHEMA,
isNaN("all_nulls_double")).eval(FILE);
+ Assert.assertFalse("Should skip: all-null column doesn't contain nan
value", shouldRead);
+
+ shouldRead = new InclusiveMetricsEvaluator(SCHEMA,
isNaN("no_nans")).eval(FILE_2);
+ Assert.assertTrue("Should read: no guarantee on if contains nan value
without nan stats", shouldRead);
+
+ shouldRead = new InclusiveMetricsEvaluator(SCHEMA,
isNaN("all_nans_v1_stats")).eval(FILE);
+ Assert.assertTrue("Should read: at least one nan value in all nan column",
shouldRead);
+
+ shouldRead = new InclusiveMetricsEvaluator(SCHEMA,
isNaN("nan_and_null_only")).eval(FILE);
+ Assert.assertTrue("Should read: at least one nan value in nan and nulls
only column", shouldRead);
+ }
+
+ @Test
+ public void testNotNaN() {
+ boolean shouldRead = new InclusiveMetricsEvaluator(SCHEMA,
notNaN("all_nans")).eval(FILE);
+ Assert.assertFalse("Should skip: column with all nans will not contain
non-nan", shouldRead);
+
+ shouldRead = new InclusiveMetricsEvaluator(SCHEMA,
notNaN("some_nans")).eval(FILE);
+ Assert.assertTrue("Should read: at least one non-nan value in some nan
column", shouldRead);
+
+ shouldRead = new InclusiveMetricsEvaluator(SCHEMA,
notNaN("no_nans")).eval(FILE);
+ Assert.assertTrue("Should read: at least one non-nan value in no nan
column", shouldRead);
+
+ shouldRead = new InclusiveMetricsEvaluator(SCHEMA,
notNaN("all_nulls_double")).eval(FILE);
+ Assert.assertTrue("Should read: at least one non-nan value in all null
column", shouldRead);
+
+ shouldRead = new InclusiveMetricsEvaluator(SCHEMA,
notNaN("no_nans")).eval(FILE_2);
Review comment:
Using FILE_2 is a little strange to read because the column is still
called "no_nans". Consider adding another column instead called `no_nan_stats`
just like the `no_stats` column that is used for `isNull`/`notNull`.
##########
File path:
api/src/test/java/org/apache/iceberg/expressions/TestInclusiveMetricsEvaluator.java
##########
@@ -156,6 +192,54 @@ public void testNoNulls() {
Assert.assertFalse("Should skip: non-null column contains no null values",
shouldRead);
}
+ @Test
+ public void testIsNaN() {
+ boolean shouldRead = new InclusiveMetricsEvaluator(SCHEMA,
isNaN("all_nans")).eval(FILE);
+ Assert.assertTrue("Should read: at least one nan value in all nan column",
shouldRead);
+
+ shouldRead = new InclusiveMetricsEvaluator(SCHEMA,
isNaN("some_nans")).eval(FILE);
+ Assert.assertTrue("Should read: at least one nan value in some nan
column", shouldRead);
+
+ shouldRead = new InclusiveMetricsEvaluator(SCHEMA,
isNaN("no_nans")).eval(FILE);
+ Assert.assertFalse("Should skip: no-nans column contains no nan values",
shouldRead);
+
+ shouldRead = new InclusiveMetricsEvaluator(SCHEMA,
isNaN("all_nulls_double")).eval(FILE);
+ Assert.assertFalse("Should skip: all-null column doesn't contain nan
value", shouldRead);
+
+ shouldRead = new InclusiveMetricsEvaluator(SCHEMA,
isNaN("no_nans")).eval(FILE_2);
+ Assert.assertTrue("Should read: no guarantee on if contains nan value
without nan stats", shouldRead);
+
+ shouldRead = new InclusiveMetricsEvaluator(SCHEMA,
isNaN("all_nans_v1_stats")).eval(FILE);
+ Assert.assertTrue("Should read: at least one nan value in all nan column",
shouldRead);
+
+ shouldRead = new InclusiveMetricsEvaluator(SCHEMA,
isNaN("nan_and_null_only")).eval(FILE);
+ Assert.assertTrue("Should read: at least one nan value in nan and nulls
only column", shouldRead);
+ }
+
+ @Test
+ public void testNotNaN() {
+ boolean shouldRead = new InclusiveMetricsEvaluator(SCHEMA,
notNaN("all_nans")).eval(FILE);
+ Assert.assertFalse("Should skip: column with all nans will not contain
non-nan", shouldRead);
+
+ shouldRead = new InclusiveMetricsEvaluator(SCHEMA,
notNaN("some_nans")).eval(FILE);
+ Assert.assertTrue("Should read: at least one non-nan value in some nan
column", shouldRead);
+
+ shouldRead = new InclusiveMetricsEvaluator(SCHEMA,
notNaN("no_nans")).eval(FILE);
+ Assert.assertTrue("Should read: at least one non-nan value in no nan
column", shouldRead);
+
+ shouldRead = new InclusiveMetricsEvaluator(SCHEMA,
notNaN("all_nulls_double")).eval(FILE);
+ Assert.assertTrue("Should read: at least one non-nan value in all null
column", shouldRead);
+
+ shouldRead = new InclusiveMetricsEvaluator(SCHEMA,
notNaN("no_nans")).eval(FILE_2);
Review comment:
Minor: Using FILE_2 is a little strange to read because the column is
still called "no_nans". Consider adding another column instead called
`no_nan_stats` just like the `no_stats` column that is used for
`isNull`/`notNull`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
