AngersZhuuuu commented on a change in pull request #25721:
[WIP][SPARK-29018][SQL] Implement Spark Thrift Server with it's own code base
on PROTOCOL_VERSION_V9
URL: https://github.com/apache/spark/pull/25721#discussion_r338350969
##########
File path:
sql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/cli/operation/SparkExecuteStatementOperation.scala
##########
@@ -134,31 +140,36 @@ private[hive] class SparkExecuteStatementOperation(
resultRowSet
} else {
// maxRowsL here typically maps to java.sql.Statement.getFetchSize,
which is an int
- val maxRows = maxRowsL.toInt
+ val maxRows = maxRowsL
var curRow = 0
while (curRow < maxRows && iter.hasNext) {
val sparkRow = iter.next()
- val row = ArrayBuffer[Any]()
- var curCol = 0
- while (curCol < sparkRow.length) {
- if (sparkRow.isNullAt(curCol)) {
- row += null
- } else {
- addNonNullColumnValue(sparkRow, row, curCol)
- }
- curCol += 1
- }
- resultRowSet.addRow(row.toArray.asInstanceOf[Array[Object]])
+ resultRowSet.addRow(sparkRow)
curRow += 1
}
resultRowSet
}
}
- def getResultSetSchema: TableSchema = resultSchema
+ def getResultSetSchema: StructType = {
Review comment:
> TableSchema in Hive format still needs to be returned as TTableSchema in
TGetResultSetMetadataResp. If you return a Spark StructType here, how do you
get TableSchema out of it for the result?
> Edit: Ok, I found that you implemented a SchemaMapper for that at the
thrift layer. 👍
Convert with same rule in SparkExecuteStatementOperation.
By the way , we may need to implement a spark's jdbc client to remove import
of `hive-beeline` `hive-jdbc` `hive-service`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
