zeruibao commented on code in PR #51621:
URL: https://github.com/apache/spark/pull/51621#discussion_r2274892282
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/python/EvaluatePython.scala:
##########
@@ -225,6 +226,42 @@ object EvaluatePython {
private val module = "pyspark.sql.types"
+ /**
+ * Pickler for ArraySeq
+ */
+ private class ArraySeqPickler extends IObjectPickler {
+
+ private val cls = classOf[ArraySeq[_]]
Review Comment:
When we put a value
```
row_value: Row(id=[0], tags=[['dummy1', 'dummy2'], ['dummy3']],
metadata=[{'key': 'env', 'value': 'prod'}, {'key': 'region', 'value':
'us-west'}])
schema: StructType([StructField('id', ArrayType(IntegerType(), True), True),
StructField('tags', ArrayType(ArrayType(StringType(), True), True), True),
StructField('metadata', ArrayType(StructType([StructField('key', StringType(),
True), StructField('value', StringType(), True)]), True), True)])
```
into the rocksdb and then fetch it back mapping it as
`GenericRowWithSchema`. The row looks like
```
pyRow: [ArraySeq(0),ArraySeq(ArraySeq(dummy1, dummy2),
ArraySeq(dummy3)),ArraySeq([env,prod], [region,us-west])]
pyRow schema:
StructType(StructField(id,ArrayType(IntegerType,true),true),StructField(tags,ArrayType(ArrayType(StringType,true),true),true),StructField(metadata,ArrayType(StructType(StructField(key,StringType,true),StructField(value,StringType,true)),true),true))
```
I printed it here
```
def toPyRow(row: Row): Array[Byte] = {
assert(row.isInstanceOf[GenericRowWithSchema])
println("pyRow: " + row)
println("pyRow schema: " + row.schema)
withInternalRowPickler(_.dumps(EvaluatePython.toJava(
CatalystTypeConverters.convertToCatalyst(row), row.schema)))
}
```
Array of elements is wrapped by `ArraySeq` type. It's where `ArraySeq` comes
from.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
