KnightChess commented on pull request #3459:
URL: https://github.com/apache/iceberg/pull/3459#issuecomment-971084426
@RussellSpitzer yes, I konw, `listPartitionIdentifiers` can return all
partitions of the "maximal" spec.
1. But in `ShowPartitionsExe`, spark use `partitionSchema` method to get
dataType in following code.
So that's mean I need return "maximal" spec when use `partitionSchema`
although the table was partitioned before but unpartitioned now.
```scala
val partitionIdentifiers = table.listPartitionIdentifiers(names.toArray,
ident)
// Converting partition identifiers as `InternalRow` of partition values,
// for instance InternalRow(value0, value1, ..., valueN), to `InternalRow`s
// with a string in the format: "col0=value0/col1=value1/.../colN=valueN".
val schema = table.partitionSchema()
val len = schema.length
val partitions = new Array[String](len)
val timeZoneId = conf.sessionLocalTimeZone
val output = partitionIdentifiers.map { row =>
var i = 0
while (i < len) {
val dataType = schema(i).dataType
val partValueUTF8String =
Cast(Literal(row.get(i, dataType), dataType), StringType,
Some(timeZoneId)).eval()
val partValueStr = if (partValueUTF8String == null) "null" else
partValueUTF8String.toString
partitions(i) = escapePathName(schema(i).name) + "=" +
escapePathName(partValueStr)
i += 1
}
partitions.mkString("/")
}
output.sorted.map(p => InternalRow(UTF8String.fromString(p)))
```
2. But if `partitionSchema` method return "maximal" spec. Some scenes may
lead to errors. like: `CheckAnalysis.checkAnalysis` in following code, although
the table is unpartitioned, but will pass the check if the table was
partitioned before.
```scala
case command: V2PartitionCommand =>
command.table match {
case r @ ResolvedTable(_, _, table, _) => table match {
case t: SupportsPartitionManagement =>
if (t.partitionSchema.isEmpty) {
failAnalysis(s"Table ${r.name} is not partitioned.")
}
case _ =>
failAnalysis(s"Table ${r.name} does not support partition management.")
}
case _ =>
}
```
3. And My fault, I didn't notice the following description before.
> `partitionSchema` must be consistent with `Table.partitioning`
The `Table.partitioning` will only return the physical partitioning of the
table (voiedTransform will be remove).
```scala
/**
* Get the partition schema of table,
* this must be consistent with ${@link Table#partitioning()}.
* @return the partition schema of table
*/
StructType partitionSchema();
```
I have a little confusion. So that's mean it can't achieve the same effect
by use `show partitions` and `select * form xxxx.partitions` when follow the
rules in points 2 and 3?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
