[jira] [Commented] (SPARK-26991) Investigate difference of `returnNullable` between ScalaReflection.deserializerFor and JavaTypeInference.deserializerFor
[ https://issues.apache.org/jira/browse/SPARK-26991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16787585#comment-16787585 ] Jungtaek Lim commented on SPARK-26991: -- [~srowen] Yeah agreed. I myself am also lost tracking on this, and now follow up the issue. [~kiszk] I found your commit (though it was fairly long ago) only addressed ScalaReflection, not JavaTypeInference. [https://github.com/apache/spark/commit/7a63f5e82758345ff1f3322950f2bbea350c48b9] Could you remember there's some reason / intention on not applying to JavaTypeInference, or it is just missing spot? > Investigate difference of `returnNullable` between > ScalaReflection.deserializerFor and JavaTypeInference.deserializerFor > > > Key: SPARK-26991 > URL: https://issues.apache.org/jira/browse/SPARK-26991 > Project: Spark > Issue Type: Task > Components: SQL >Affects Versions: 3.0.0 >Reporter: Jungtaek Lim >Priority: Major > > This issue tracks the effort on investigation on difference between > ScalaReflection.deserializerFor and JavaTypeInference.deserializerFor, > especially the reason why Java side uses `returnNullable = true` whereas > `returnNullable = false`. > The origin discussion is linked here: > https://github.com/apache/spark/pull/23854#discussion_r260117702 -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-26991) Investigate difference of `returnNullable` between ScalaReflection.deserializerFor and JavaTypeInference.deserializerFor
[ https://issues.apache.org/jira/browse/SPARK-26991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16782796#comment-16782796 ] Sean Owen commented on SPARK-26991: --- This is a philosophical question about what JIRA is for. JIRA was built for short-lived issues that have a resolution, because it's built around a workflow from open to closed. It means JIRAs ought to be actionable and specific, so that we know what closed means and can imagine getting it there. Of course, there is a role for open-ended ideas and long-term goals. JIRA isn't great for that, but it's the only tool we have. JIRA has some tools for this type of thing, but we aren't able to control the workflow settings enough to use them. In theory that means, get another tool, not discourage open-ended ideas. In practice, we have an overwhelming number of JIRAs that don't fit what it's good for: underspecified issues, sketches of major changes, todos, etc -- things that either aren't actionable, are 'actionable' but just not a good idea, placeholders that will never be followed up on. This is why I don't think "let's investigate" JIRAs make much sense in theory or practice. Investigate, sure, then track the action that falls out in a JIRA. I don't object to this one given your clarification. My experience is that these don't get followed up on, they get left open, or they feel like a substitute for doing the investigation -- i.e. let me write this down for someone else to think about, which virtually never ever happens > Investigate difference of `returnNullable` between > ScalaReflection.deserializerFor and JavaTypeInference.deserializerFor > > > Key: SPARK-26991 > URL: https://issues.apache.org/jira/browse/SPARK-26991 > Project: Spark > Issue Type: Task > Components: SQL >Affects Versions: 3.0.0 >Reporter: Jungtaek Lim >Priority: Major > > This issue tracks the effort on investigation on difference between > ScalaReflection.deserializerFor and JavaTypeInference.deserializerFor, > especially the reason why Java side uses `returnNullable = true` whereas > `returnNullable = false`. > The origin discussion is linked here: > https://github.com/apache/spark/pull/23854#discussion_r260117702 -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-26991) Investigate difference of `returnNullable` between ScalaReflection.deserializerFor and JavaTypeInference.deserializerFor
[ https://issues.apache.org/jira/browse/SPARK-26991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16782128#comment-16782128 ] Jungtaek Lim commented on SPARK-26991: -- The outcome would be * "invalid" if there's reasonable reason for Java side to use `returnNullable = true` * new patch if there's no reason for Java side to use `returnNullable = true` separately and can follow Scala side Btw, is it discouraged to file an issue for TODO task? I agree this wouldn't be an issue after investigation: this is just not to miss for TODO - given this issue is opened to public, other than me can investigate this. > Investigate difference of `returnNullable` between > ScalaReflection.deserializerFor and JavaTypeInference.deserializerFor > > > Key: SPARK-26991 > URL: https://issues.apache.org/jira/browse/SPARK-26991 > Project: Spark > Issue Type: Task > Components: SQL >Affects Versions: 3.0.0 >Reporter: Jungtaek Lim >Priority: Major > > This issue tracks the effort on investigation on difference between > ScalaReflection.deserializerFor and JavaTypeInference.deserializerFor, > especially the reason why Java side uses `returnNullable = true` whereas > `returnNullable = false`. > The origin discussion is linked here: > https://github.com/apache/spark/pull/23854#discussion_r260117702 -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-26991) Investigate difference of `returnNullable` between ScalaReflection.deserializerFor and JavaTypeInference.deserializerFor
[ https://issues.apache.org/jira/browse/SPARK-26991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16782062#comment-16782062 ] Sean Owen commented on SPARK-26991: --- I'm not sure what the outcome would be here; let's open JIRAs for specific issues only. > Investigate difference of `returnNullable` between > ScalaReflection.deserializerFor and JavaTypeInference.deserializerFor > > > Key: SPARK-26991 > URL: https://issues.apache.org/jira/browse/SPARK-26991 > Project: Spark > Issue Type: Task > Components: SQL >Affects Versions: 3.0.0 >Reporter: Jungtaek Lim >Priority: Major > > This issue tracks the effort on investigation on difference between > ScalaReflection.deserializerFor and JavaTypeInference.deserializerFor, > especially the reason why Java side uses `returnNullable = true` whereas > `returnNullable = false`. > The origin discussion is linked here: > https://github.com/apache/spark/pull/23854#discussion_r260117702 -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-26991) Investigate difference of `returnNullable` between ScalaReflection.deserializerFor and JavaTypeInference.deserializerFor
[ https://issues.apache.org/jira/browse/SPARK-26991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16781487#comment-16781487 ] Jungtaek Lim commented on SPARK-26991: -- This issue could be extended to investigate the difference between ScalaReflection and JavaTypeInference, as we have found another case as well. [https://github.com/apache/spark/pull/23908#discussion_r261511454] > Investigate difference of `returnNullable` between > ScalaReflection.deserializerFor and JavaTypeInference.deserializerFor > > > Key: SPARK-26991 > URL: https://issues.apache.org/jira/browse/SPARK-26991 > Project: Spark > Issue Type: Task > Components: SQL >Affects Versions: 3.0.0 >Reporter: Jungtaek Lim >Priority: Major > > This issue tracks the effort on investigation on difference between > ScalaReflection.deserializerFor and JavaTypeInference.deserializerFor, > especially the reason why Java side uses `returnNullable = true` whereas > `returnNullable = false`. > The origin discussion is linked here: > https://github.com/apache/spark/pull/23854#discussion_r260117702 -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
