Maxim Gekk created SPARK-33505:
----------------------------------
Summary: Fix insert into `InMemoryPartitionTable`
Key: SPARK-33505
URL: https://issues.apache.org/jira/browse/SPARK-33505
Project: Spark
Issue Type: Sub-task
Components: SQL
Affects Versions: 3.1.0
Reporter: Maxim Gekk
Currently, INSERT INTO a partitioned table in V2 in-memory catalog doesn't
create partitions. The example below demonstrates the issue:
{code:scala}
test("insert into partitioned table") {
val t = "testpart.ns1.ns2.tbl"
withTable(t) {
spark.sql(
s"""
|CREATE TABLE $t (id bigint, name string, data string)
|USING foo
|PARTITIONED BY (id, name)""".stripMargin)
spark.sql(s"INSERT INTO $t PARTITION(id = 1, name = 'Max') SELECT 'abc'")
val partTable = catalog("testpart").asTableCatalog
.loadTable(Identifier.of(Array("ns1", "ns2"),
"tbl")).asInstanceOf[InMemoryPartitionTable]
assert(partTable.partitionExists(InternalRow.fromSeq(Seq(1,
UTF8String.fromString("Max")))))
}
}
{code}
The partitionExists() function return false for the partitions that must be
created.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
