dmgcodevil opened a new issue #1713:
URL: https://github.com/apache/iceberg/issues/1713
I have two partitions created using transformation functions: `day` and
`bucket`.
Schema:
```
(
security_id varchar,
frontdoor_timestamp timestamp(3) with time zone
)
WITH (
format = 'PARQUET',
partitioning = ARRAY['day(frontdoor_timestamp)','bucket(security_id,
10)']
)
```
i.e. `struct: struct<1000: frontdoor_timestamp_day: optional date, 1001:
security_id_bucket: optional int>`
I'd like to query by day:
```
val tasks =
scan.filter(Expressions.equal(Expressions.day[String]("frontdoor_timestamp"),
"2020-10-20"))
.select("security_id")
.planTasks().toList
````
Getting the following error:
```
Exception in thread "main"
org.apache.iceberg.exceptions.ValidationException: Cannot find field 'day' in
struct: struct<1000: frontdoor_timestamp_day: optional date, 1001:
security_id_bucket: optional int>
at
org.apache.iceberg.exceptions.ValidationException.check(ValidationException.java:42)
at
org.apache.iceberg.expressions.NamedReference.bind(NamedReference.java:46)
at
org.apache.iceberg.expressions.NamedReference.bind(NamedReference.java:27)
at
org.apache.iceberg.expressions.UnboundPredicate.bind(UnboundPredicate.java:105)
at
org.apache.iceberg.expressions.Binder$BindVisitor.predicate(Binder.java:145)
at
org.apache.iceberg.expressions.Binder$BindVisitor.predicate(Binder.java:104)
at
org.apache.iceberg.expressions.ExpressionVisitors.visit(ExpressionVisitors.java:283)
at org.apache.iceberg.expressions.Binder.bind(Binder.java:62)
at
org.apache.iceberg.expressions.ManifestEvaluator.<init>(ManifestEvaluator.java:65)
at
org.apache.iceberg.expressions.ManifestEvaluator.forPartitionFilter(ManifestEvaluator.java:60)
at
org.apache.iceberg.ManifestGroup.lambda$entries$7(ManifestGroup.java:189)
at
com.github.benmanes.caffeine.cache.UnboundedLocalCache$UnboundedLocalLoadingCache.lambda$new$0(UnboundedLocalCache.java:928)
at
com.github.benmanes.caffeine.cache.UnboundedLocalCache.lambda$computeIfAbsent$2(UnboundedLocalCache.java:237)
at
java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1660)
at
com.github.benmanes.caffeine.cache.UnboundedLocalCache.computeIfAbsent(UnboundedLocalCache.java:233)
at
com.github.benmanes.caffeine.cache.LocalCache.computeIfAbsent(LocalCache.java:111)
at
com.github.benmanes.caffeine.cache.LocalLoadingCache.get(LocalLoadingCache.java:66)
at
org.apache.iceberg.ManifestGroup.lambda$entries$8(ManifestGroup.java:197)
at
org.apache.iceberg.relocated.com.google.common.collect.Iterators$5.computeNext(Iterators.java:637)
at
org.apache.iceberg.relocated.com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:141)
at
org.apache.iceberg.relocated.com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:136)
at
org.apache.iceberg.relocated.com.google.common.collect.Iterators$5.computeNext(Iterators.java:635)
at
org.apache.iceberg.relocated.com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:141)
at
org.apache.iceberg.relocated.com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:136)
at
org.apache.iceberg.relocated.com.google.common.collect.Iterators$5.computeNext(Iterators.java:635)
at
org.apache.iceberg.relocated.com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:141)
at
org.apache.iceberg.relocated.com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:136)
at
org.apache.iceberg.relocated.com.google.common.collect.TransformedIterator.hasNext(TransformedIterator.java:42)
at
org.apache.iceberg.relocated.com.google.common.collect.TransformedIterator.hasNext(TransformedIterator.java:42)
at
org.apache.iceberg.util.ParallelIterable$ParallelIterator.submitNextTask(ParallelIterable.java:113)
at
org.apache.iceberg.util.ParallelIterable$ParallelIterator.checkTasks(ParallelIterable.java:101)
at
org.apache.iceberg.util.ParallelIterable$ParallelIterator.hasNext(ParallelIterable.java:138)
at
org.apache.iceberg.relocated.com.google.common.collect.TransformedIterator.hasNext(TransformedIterator.java:42)
at
org.apache.iceberg.relocated.com.google.common.collect.TransformedIterator.hasNext(TransformedIterator.java:42)
at
org.apache.iceberg.relocated.com.google.common.collect.Iterators$ConcatenatedIterator.getTopMetaIterator(Iterators.java:1309)
at
org.apache.iceberg.relocated.com.google.common.collect.Iterators$ConcatenatedIterator.hasNext(Iterators.java:1325)
at
org.apache.iceberg.io.CloseableIterator$1.hasNext(CloseableIterator.java:48)
at
org.apache.iceberg.util.BinPacking$PackingIterator.hasNext(BinPacking.java:106)
at
org.apache.iceberg.io.CloseableIterator$1.hasNext(CloseableIterator.java:48)
at
org.apache.iceberg.io.CloseableIterable$3$1.hasNext(CloseableIterable.java:104)
at
scala.collection.convert.Wrappers$JIteratorWrapper.hasNext(Wrappers.scala:43)
at scala.collection.Iterator.foreach(Iterator.scala:943)
at scala.collection.Iterator.foreach$(Iterator.scala:943)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1431)
at scala.collection.IterableLike.foreach(IterableLike.scala:74)
at scala.collection.IterableLike.foreach$(IterableLike.scala:73)
at scala.collection.AbstractIterable.foreach(Iterable.scala:56)
at scala.collection.generic.Growable.$plus$plus$eq(Growable.scala:62)
at scala.collection.generic.Growable.$plus$plus$eq$(Growable.scala:53)
at
scala.collection.mutable.ListBuffer.$plus$plus$eq(ListBuffer.scala:184)
at
scala.collection.mutable.ListBuffer.$plus$plus$eq(ListBuffer.scala:47)
at scala.collection.TraversableLike.to(TraversableLike.scala:725)
at scala.collection.TraversableLike.to$(TraversableLike.scala:722)
at scala.collection.AbstractTraversable.to(Traversable.scala:108)
at scala.collection.TraversableOnce.toList(TraversableOnce.scala:332)
at scala.collection.TraversableOnce.toList$(TraversableOnce.scala:332)
at scala.collection.AbstractTraversable.toList(Traversable.scala:108)
at IcebergQueryApp$.main(IcebergQueryApp.scala:50)
at IcebergQueryApp.main(IcebergQueryApp.scala)
Process finished with exit code 1
```
What am I doing wrong?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
