mickjermsurawong-stripe edited a comment on pull request #27153: URL: https://github.com/apache/spark/pull/27153#issuecomment-873105068
Hey @eejbyfeldt sorry i realized that the internal fork we addressed this issue is in a different class of our custom encoder, so i didn't get to integrate it with the standard encoder here. The underlying issue we addressed is exactly the allocation you mentioned: From doc on value class: https://docs.scala-lang.org/overviews/core/value-classes.html Given: `class Wrapper(val underlying: Int) extends AnyVal`, - "The type at compile time is Wrapper, but at runtime, the representation is an Int" This implies that when our struct has a field of AnyVal case class, our generated code should support the underlying type during runtime execution. - The Wrapper "must be instantiated... when a value class is used as a type argument". This implies that scala.Tuple[Wrapper, ...], Seq[Wrapper], Map[String, Wrapper], Option[Wrapper] will still contain Wrapper as-is in during runtime instead of Int. Hope this could help. It's a long weekend here, so I will make sure I'll get to work on this. But please feel free to pursue independently as well. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
