zhenlineo commented on code in PR #40796:
URL: https://github.com/apache/spark/pull/40796#discussion_r1190193381
##########
connector/connect/client/jvm/src/main/scala/org/apache/spark/sql/KeyValueGroupedDataset.scala:
##########
@@ -322,41 +471,45 @@ abstract class KeyValueGroupedDataset[K, V] private[sql]
() extends Serializable
* [[KeyValueGroupedDataset]] behaves on the server.
*/
private class KeyValueGroupedDatasetImpl[K, V, IK, IV](
- private val ds: Dataset[IV],
private val sparkSession: SparkSession,
private val plan: proto.Plan,
private val ikEncoder: AgnosticEncoder[IK],
private val kEncoder: AgnosticEncoder[K],
- private val groupingFunc: IV => IK,
- private val valueMapFunc: IV => V)
+ private val ivEncoder: AgnosticEncoder[IV],
+ private val vEncoder: AgnosticEncoder[V],
+ private val groupingExprs: java.util.List[proto.Expression],
+ private val valueMapFunc: IV => V,
+ private val keysFunc: () => Dataset[IK])
Review Comment:
I do not see a better way to handle the keyFunc. If I pass it in, then I
need to carry the grouping_udf and grouping_expressions in the raw format. Now
I can just have a simple groupingExpr: proto.Expression. Happy to fix in
another PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
