findinpath commented on issue #5064:
URL: https://github.com/apache/iceberg/issues/5064#issuecomment-1160636601
This problem seems to be affecting Iceberg Spark implementation as well:
```
spark-sql> create table logs (level string, event_time timestamp , message
string) partitioned by (date(event_time), level);
Response code
Time taken: 2.377 seconds
spark-sql> drop table logs;
Response code
Time taken: 0.132 seconds
spark-sql> create table logs (level string, event_time timestamp , message
string) using iceberg partitioned by (date(event_time), level);
Response code
Time taken: 0.043 seconds
spark-sql> insert into logs values ('ERROR', timestamp '2021-04-01
12:23:53.383345', '1 message');
Response code
Time taken: 2.677 seconds
spark-sql> select * from logs.partitions;
22/06/20 18:14:13 WARN ObjectStore: Failed to get database global_temp,
returning NoSuchObjectException
partition record_count file_count
{"event_time_day":2021-04-01,"level":"ERROR"} 1 1
Time taken: 0.413 seconds, Fetched 1 row(s)
spark-sql> alter table logs drop partition field event_time_day;
ANTLR Runtime version 4.8 used for parser compilation does not match the
current runtime version 4.7.1ANTLR Runtime version 4.8 used for parser
compilation does not match the current runtime version 4.7.1Response code
Time taken: 0.109 seconds
spark-sql> alter table logs add partition field hours(event_time);
Response code
Time taken: 0.044 seconds
spark-sql> insert into logs values ('ERROR', timestamp '2021-04-01
15:55:23', '2 message');
Response code
Time taken: 0.225 seconds
spark-sql> insert into logs values ('WARN', timestamp '2021-04-01 16:55:23',
'3 message');
Response code
Time taken: 0.209 seconds
spark-sql> select * from logs.partitions;
```
```
22/06/20 18:17:09 ERROR Executor: Exception in task 0.0 in stage 4.0 (TID 4)
java.lang.IllegalStateException: Unknown type for long field. Type name:
java.lang.Integer
at
org.apache.iceberg.spark.source.StructInternalRow.getLong(StructInternalRow.java:146)
at
org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown
Source)
at
org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
at
org.apache.spark.sql.execution.WholeStageCodegenExec$$anon$1.hasNext(WholeStageCodegenExec.scala:759)
at
org.apache.spark.sql.execution.SparkPlan.$anonfun$getByteArrayRdd$1(SparkPlan.scala:349)
at
org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2(RDD.scala:898)
at
org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2$adapted(RDD.scala:898)
at
org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:373)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:337)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
at org.apache.spark.scheduler.Task.run(Task.scala:131)
at
org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:506)
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1462)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:509)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
22/06/20 18:17:09 WARN TaskSetManager: Lost task 0.0 in stage 4.0 (TID 4)
(10.0.0.1 executor driver): java.lang.IllegalStateException: Unknown type for
long field. Type name: java.lang.Integer
at
org.apache.iceberg.spark.source.StructInternalRow.getLong(StructInternalRow.java:146)
at
org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown
Source)
at
org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
at
org.apache.spark.sql.execution.WholeStageCodegenExec$$anon$1.hasNext(WholeStageCodegenExec.scala:759)
at
org.apache.spark.sql.execution.SparkPlan.$anonfun$getByteArrayRdd$1(SparkPlan.scala:349)
at
org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2(RDD.scala:898)
at
org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2$adapted(RDD.scala:898)
at
org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:373)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:337)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
at org.apache.spark.scheduler.Task.run(Task.scala:131)
at
org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:506)
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1462)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:509)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
