RussellSpitzer commented on code in PR #7636:
URL: https://github.com/apache/iceberg/pull/7636#discussion_r1198001598
##########
spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/source/SparkScanBuilder.java:
##########
@@ -221,15 +222,26 @@ public boolean pushAggregation(Aggregation aggregation) {
return false;
}
- TableScan scan = table.newScan().includeColumnStats();
- Snapshot snapshot = readSnapshot();
- if (snapshot == null) {
- LOG.info("Skipping aggregate pushdown: table snapshot is null");
- return false;
+ org.apache.iceberg.Scan scan;
Review Comment:
minor style thought?
Could we do something like
```java
Scan scan = null;
if (readConf.startSnapshotId() == null) {
TableScan tableScan = table.newScan()
Snapshot snapshot = readSnapshot();
if (snapshot == null) {
LOG.info("Skipping aggregate pushdown: table snapshot is null");
return false;
}
tableScan = tableScan.useSnapshot(snapshot.snapshotId());
scan = tableScan
} else {
IncrementalAppendScan incrementalScan =
table.newIncrementalAppendScan();
incrementalScan =
incrementalScan.fromSnapshotExclusive(readConf.startSnapshotId());
Long endSnapshotId = readConf.endSnapshotId();
if (endSnapshotId != null) {
incrementalScan = incrementalScan.toSnapshot(endSnapshotId);
}
scan = incrementalScan;
}
scan = scan.filter(filterExpression()).includeColumnStats();
```
I could also see extracting that branching part into another function. My
main suggestion is basically to structure it more like
```
if (tableScan) {
TableScanStuff
}
else () {
IncrementalStuff
}
Common Stuff
```
or
```java
def Scan getScan() {
...
}
getScan()
.filter()
.includeColumnStats()
```
So we can remove all the casting and keep all the common code in one place
just incase we need to add things in the future
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
