[GitHub] spark issue #16648: [SPARK-18016][SQL][CATALYST] Code Generation: Constant P...

2017-02-23 Thread bdrillard
Github user bdrillard commented on the issue: https://github.com/apache/spark/pull/16648 Thanks for that other test case. The one you provide I would say falls in the same class of error, however, this patch is still capable of addressing some others that still exist. While class

[GitHub] spark pull request #15453: [SPARK-17770] [CATALYST] making ObjectType public

2016-10-12 Thread bdrillard
GitHub user bdrillard opened a pull request: https://github.com/apache/spark/pull/15453 [SPARK-17770] [CATALYST] making ObjectType public ## What changes were proposed in this pull request? In order to facilitate the writing of additional Encoders, I proposed opening up

[GitHub] spark pull request #15453: [SPARK-17770] [CATALYST] making ObjectType public

2016-10-13 Thread bdrillard
Github user bdrillard commented on a diff in the pull request: https://github.com/apache/spark/pull/15453#discussion_r83343796 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/types/ObjectType.scala --- @@ -19,7 +19,7 @@ package org.apache.spark.sql.types

[GitHub] spark pull request #16648: [SPARK-18016][SQL][CATALYST] Code Generation: Con...

2017-03-10 Thread bdrillard
Github user bdrillard commented on a diff in the pull request: https://github.com/apache/spark/pull/16648#discussion_r105496928 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/DataFrameComplexTypeSuite.scala --- @@ -64,6 +65,24 @@ class DataFrameComplexTypeSuite extends

[GitHub] spark issue #16648: [SPARK-18016][SQL][CATALYST] Code Generation: Constant P...

2017-03-10 Thread bdrillard
Github user bdrillard commented on the issue: https://github.com/apache/spark/pull/16648 I've made some changes to this PR to address @mkiedys comments, and I'm using his test case, as it sets a higher bar for both class splitting and management of mutable state. Mutable

[GitHub] spark pull request #16648: [SPARK-18016][SQL][CATALYST] Code Generation: Con...

2017-03-10 Thread bdrillard
Github user bdrillard commented on a diff in the pull request: https://github.com/apache/spark/pull/16648#discussion_r105502401 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/MonotonicallyIncreasingID.scala --- @@ -67,14 +67,15 @@ case class

[GitHub] spark pull request #16648: [SPARK-18016][SQL][CATALYST] Code Generation: Con...

2017-01-19 Thread bdrillard
GitHub user bdrillard opened a pull request: https://github.com/apache/spark/pull/16648 [SPARK-18016][SQL][CATALYST] Code Generation: Constant Pool Limit [class_splitting] increasing stack size for Catalyst tests ## What changes were proposed in this pull request

[GitHub] spark pull request #20010: [SPARK-22826][SQL] findWiderTypeForTwo Fails over...

2017-12-18 Thread bdrillard
GitHub user bdrillard opened a pull request: https://github.com/apache/spark/pull/20010 [SPARK-22826][SQL] findWiderTypeForTwo Fails over StructField of Array ## What changes were proposed in this pull request? [SPARK-22826](https://issues.apache.org/jira/browse/SPARK

[GitHub] spark pull request #20010: [SPARK-22826][SQL] findWiderTypeForTwo Fails over...

2017-12-19 Thread bdrillard
Github user bdrillard commented on a diff in the pull request: https://github.com/apache/spark/pull/20010#discussion_r157778903 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercion.scala --- @@ -99,6 +99,17 @@ object TypeCoercion

[GitHub] spark pull request #20010: [SPARK-22826][SQL] findWiderTypeForTwo Fails over...

2017-12-19 Thread bdrillard
Github user bdrillard commented on a diff in the pull request: https://github.com/apache/spark/pull/20010#discussion_r157787536 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercion.scala --- @@ -99,6 +99,17 @@ object TypeCoercion

[GitHub] spark pull request #20010: [SPARK-22826][SQL] findWiderTypeForTwo Fails over...

2017-12-19 Thread bdrillard
Github user bdrillard commented on a diff in the pull request: https://github.com/apache/spark/pull/20010#discussion_r157792706 --- Diff: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercionSuite.scala --- @@ -389,6 +389,25 @@ class TypeCoercionSuite

[GitHub] spark pull request #20010: [SPARK-22826][SQL] findWiderTypeForTwo Fails over...

2017-12-19 Thread bdrillard
Github user bdrillard commented on a diff in the pull request: https://github.com/apache/spark/pull/20010#discussion_r157792361 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercion.scala --- @@ -158,11 +169,6 @@ object TypeCoercion

[GitHub] spark pull request #20010: [SPARK-22826][SQL] findWiderTypeForTwo Fails over...

2017-12-19 Thread bdrillard
Github user bdrillard commented on a diff in the pull request: https://github.com/apache/spark/pull/20010#discussion_r157792439 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercion.scala --- @@ -182,12 +188,6 @@ object TypeCoercion

[GitHub] spark pull request #20010: [SPARK-22826][SQL] findWiderTypeForTwo Fails over...

2017-12-19 Thread bdrillard
Github user bdrillard commented on a diff in the pull request: https://github.com/apache/spark/pull/20010#discussion_r157901873 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercion.scala --- @@ -158,11 +169,6 @@ object TypeCoercion

[GitHub] spark pull request #19518: [SPARK-18016][SQL][CATALYST] Code Generation: Con...

2017-12-19 Thread bdrillard
Github user bdrillard closed the pull request at: https://github.com/apache/spark/pull/19518 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19518: [SPARK-18016][SQL][CATALYST] Code Generation: Constant P...

2017-12-19 Thread bdrillard
Github user bdrillard commented on the issue: https://github.com/apache/spark/pull/19518 This PR was addressed by #19811, closing this one. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For

[GitHub] spark pull request #20010: [SPARK-22826][SQL] findWiderTypeForTwo Fails over...

2017-12-20 Thread bdrillard
Github user bdrillard commented on a diff in the pull request: https://github.com/apache/spark/pull/20010#discussion_r158081192 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercion.scala --- @@ -158,11 +169,6 @@ object TypeCoercion

[GitHub] spark pull request #20010: [SPARK-22826][SQL] findWiderTypeForTwo Fails over...

2017-12-26 Thread bdrillard
Github user bdrillard commented on a diff in the pull request: https://github.com/apache/spark/pull/20010#discussion_r158728293 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercion.scala --- @@ -158,11 +169,6 @@ object TypeCoercion

[GitHub] spark pull request #20010: [SPARK-22826][SQL] findWiderTypeForTwo Fails over...

2017-12-26 Thread bdrillard
Github user bdrillard commented on a diff in the pull request: https://github.com/apache/spark/pull/20010#discussion_r158724861 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercion.scala --- @@ -158,11 +213,8 @@ object TypeCoercion

[GitHub] spark pull request #20085: [SPARK-22739][Catalyst][WIP] Additional Expressio...

2017-12-26 Thread bdrillard
GitHub user bdrillard opened a pull request: https://github.com/apache/spark/pull/20085 [SPARK-22739][Catalyst][WIP] Additional Expression Support for Objects ## What changes were proposed in this pull request? This PR is a work-in-progress adding additional `Expression

[GitHub] spark issue #20085: [SPARK-22739][Catalyst][WIP] Additional Expression Suppo...

2017-12-26 Thread bdrillard
Github user bdrillard commented on the issue: https://github.com/apache/spark/pull/20085 cc: @marmbrus --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark pull request #20085: [SPARK-22739][Catalyst][WIP] Additional Expressio...

2017-12-26 Thread bdrillard
Github user bdrillard commented on a diff in the pull request: https://github.com/apache/spark/pull/20085#discussion_r158761155 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/objects/objects.scala --- @@ -106,27 +106,27 @@ trait InvokeLike extends

[GitHub] spark pull request #20085: [SPARK-22739][Catalyst][WIP] Additional Expressio...

2017-12-26 Thread bdrillard
Github user bdrillard commented on a diff in the pull request: https://github.com/apache/spark/pull/20085#discussion_r158761511 --- Diff: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CodeGenerationSuite.scala --- @@ -390,8 +391,8 @@ class

[GitHub] spark issue #20010: [SPARK-22826][SQL] findWiderTypeForTwo Fails over Struct...

2017-12-27 Thread bdrillard
Github user bdrillard commented on the issue: https://github.com/apache/spark/pull/20010 @gczsjdy, @mgaido91 If you all are comfortable with it, I think this PR is in a state where we could trigger a build

[GitHub] spark issue #20085: [SPARK-22739][Catalyst][WIP] Additional Expression Suppo...

2017-12-27 Thread bdrillard
Github user bdrillard commented on the issue: https://github.com/apache/spark/pull/20085 @viirya I've found the same intent of a `ValueIfType` function can be attained by adding a simpler `InstanceOf` [expressions](https://github.com/apache/spark/pull/20085/files

[GitHub] spark pull request #20010: [SPARK-22826][SQL] findWiderTypeForTwo Fails over...

2017-12-27 Thread bdrillard
Github user bdrillard commented on a diff in the pull request: https://github.com/apache/spark/pull/20010#discussion_r158851832 --- Diff: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercionSuite.scala --- @@ -469,12 +488,21 @@ class TypeCoercionSuite

[GitHub] spark pull request #20010: [SPARK-22826][SQL] findWiderTypeForTwo Fails over...

2017-12-27 Thread bdrillard
Github user bdrillard commented on a diff in the pull request: https://github.com/apache/spark/pull/20010#discussion_r158851887 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercion.scala --- @@ -158,11 +213,8 @@ object TypeCoercion

[GitHub] spark pull request #20010: [SPARK-22826][SQL] findWiderTypeForTwo Fails over...

2017-12-27 Thread bdrillard
Github user bdrillard commented on a diff in the pull request: https://github.com/apache/spark/pull/20010#discussion_r158851910 --- Diff: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercionSuite.scala --- @@ -389,6 +389,25 @@ class TypeCoercionSuite

[GitHub] spark pull request #20010: [SPARK-22826][SQL] findWiderTypeForTwo Fails over...

2017-12-27 Thread bdrillard
Github user bdrillard commented on a diff in the pull request: https://github.com/apache/spark/pull/20010#discussion_r158852095 --- Diff: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercionSuite.scala --- @@ -389,6 +389,25 @@ class TypeCoercionSuite

[GitHub] spark pull request #20010: [SPARK-22826][SQL] findWiderTypeForTwo Fails over...

2017-12-27 Thread bdrillard
Github user bdrillard commented on a diff in the pull request: https://github.com/apache/spark/pull/20010#discussion_r158852200 --- Diff: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercionSuite.scala --- @@ -469,12 +488,21 @@ class TypeCoercionSuite

[GitHub] spark pull request #20010: [SPARK-22826][SQL] findWiderTypeForTwo Fails over...

2017-12-27 Thread bdrillard
Github user bdrillard commented on a diff in the pull request: https://github.com/apache/spark/pull/20010#discussion_r158852830 --- Diff: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercionSuite.scala --- @@ -469,12 +488,21 @@ class TypeCoercionSuite

[GitHub] spark issue #20010: [SPARK-22826][SQL] findWiderTypeForTwo Fails over Struct...

2017-12-27 Thread bdrillard
Github user bdrillard commented on the issue: https://github.com/apache/spark/pull/20010 @mgaido91 Agreed with that concern. I think the last round of tests I've just added covers the permutation of cases well, where we have arrays and maps of structs, and structs of arrays and

[GitHub] spark pull request #20010: [SPARK-22826][SQL] findWiderTypeForTwo Fails over...

2018-01-02 Thread bdrillard
Github user bdrillard commented on a diff in the pull request: https://github.com/apache/spark/pull/20010#discussion_r159244795 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercion.scala --- @@ -148,6 +160,61 @@ object TypeCoercion

[GitHub] spark pull request #20085: [SPARK-22739][Catalyst][WIP] Additional Expressio...

2018-01-03 Thread bdrillard
Github user bdrillard commented on a diff in the pull request: https://github.com/apache/spark/pull/20085#discussion_r159520020 --- Diff: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CodeGenerationSuite.scala --- @@ -436,4 +437,16 @@ class

[GitHub] spark pull request #20085: [SPARK-22739][Catalyst][WIP] Additional Expressio...

2018-01-03 Thread bdrillard
Github user bdrillard commented on a diff in the pull request: https://github.com/apache/spark/pull/20085#discussion_r159519915 --- Diff: sql/catalyst/src/test/java/org/apache/spark/sql/catalyst/expressions/GenericBean.java --- @@ -0,0 +1,56 @@ +package

[GitHub] spark pull request #20085: [SPARK-22739][Catalyst][WIP] Additional Expressio...

2018-01-03 Thread bdrillard
Github user bdrillard commented on a diff in the pull request: https://github.com/apache/spark/pull/20085#discussion_r159519672 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/objects/objects.scala --- @@ -1237,47 +1342,91 @@ case class

[GitHub] spark issue #20085: [SPARK-22739][Catalyst][WIP] Additional Expression Suppo...

2018-01-03 Thread bdrillard
Github user bdrillard commented on the issue: https://github.com/apache/spark/pull/20085 I've added some comments describing an issue I've had with generalizing `InitializeJavaBean`, which I thought I'd added to this PR earlier but seem to have not

[GitHub] spark pull request #20010: [SPARK-22826][SQL] findWiderTypeForTwo Fails over...

2018-01-04 Thread bdrillard
Github user bdrillard commented on a diff in the pull request: https://github.com/apache/spark/pull/20010#discussion_r159675261 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercion.scala --- @@ -99,11 +100,22 @@ object TypeCoercion

[GitHub] spark pull request #20010: [SPARK-22826][SQL] findWiderTypeForTwo Fails over...

2018-01-04 Thread bdrillard
Github user bdrillard commented on a diff in the pull request: https://github.com/apache/spark/pull/20010#discussion_r159676537 --- Diff: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercionSuite.scala --- @@ -462,27 +507,139 @@ class TypeCoercionSuite

[GitHub] spark pull request #20085: [SPARK-22739][Catalyst][WIP] Additional Expressio...

2018-01-15 Thread bdrillard
Github user bdrillard commented on a diff in the pull request: https://github.com/apache/spark/pull/20085#discussion_r161607649 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/objects/objects.scala --- @@ -1237,47 +1342,91 @@ case class

[GitHub] spark issue #20010: [SPARK-22826][SQL] findWiderTypeForTwo Fails over Struct...

2018-01-15 Thread bdrillard
Github user bdrillard commented on the issue: https://github.com/apache/spark/pull/20010 Is there a consensus on the preferred behavior here? This issue would also be a blocker to encoders for Spark-Avro in Spark 2.3.0 that @marmbrus mentions in #20085

[GitHub] spark issue #16648: [SPARK-18016][SQL][CATALYST] Code Generation: Constant P...

2018-05-16 Thread bdrillard
Github user bdrillard commented on the issue: https://github.com/apache/spark/pull/16648 This PR was addressed in #18075. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e

[GitHub] spark pull request #16648: [SPARK-18016][SQL][CATALYST] Code Generation: Con...

2018-05-16 Thread bdrillard
Github user bdrillard closed the pull request at: https://github.com/apache/spark/pull/16648 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #21348: [SPARK-22739][Catalyst Additional Expression Supp...

2018-05-16 Thread bdrillard
GitHub user bdrillard opened a pull request: https://github.com/apache/spark/pull/21348 [SPARK-22739][Catalyst Additional Expression Support for Objects ## What changes were proposed in this pull request? This PR is a working followup to the expression work begun in #20085

[GitHub] spark pull request #21348: [SPARK-22739][Catalyst Additional Expression Supp...

2018-05-16 Thread bdrillard
Github user bdrillard commented on a diff in the pull request: https://github.com/apache/spark/pull/21348#discussion_r188810286 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/objects/objects.scala --- @@ -408,16 +439,19 @@ object NewInstance

[GitHub] spark issue #20085: [SPARK-22739][Catalyst][WIP] Additional Expression Suppo...

2018-05-16 Thread bdrillard
Github user bdrillard commented on the issue: https://github.com/apache/spark/pull/20085 Closing this PR in favor of #21348. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands

[GitHub] spark pull request #20085: [SPARK-22739][Catalyst][WIP] Additional Expressio...

2018-05-16 Thread bdrillard
Github user bdrillard closed the pull request at: https://github.com/apache/spark/pull/20085 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18579: [SPARK-18016][SQL][followup] merge declareAddedFunctions...

2017-07-09 Thread bdrillard
Github user bdrillard commented on the issue: https://github.com/apache/spark/pull/18579 Thanks everyone! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so

[GitHub] spark issue #19518: [SPARK-18016][SQL][CATALYST] Code Generation: Constant P...

2017-11-22 Thread bdrillard
Github user bdrillard commented on the issue: https://github.com/apache/spark/pull/19518 Thanks for giving this the attention to shepard it on through. I haven't had the time to do the additional coding work necessary to properly benchmark it in the last few weeks. @kiszk, if

[GitHub] spark pull request #19811: [SPARK-18016][SQL] Code Generation: Constant Pool...

2017-11-28 Thread bdrillard
Github user bdrillard commented on a diff in the pull request: https://github.com/apache/spark/pull/19811#discussion_r153534679 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeGenerator.scala --- @@ -177,11 +190,67 @@ class

[GitHub] spark issue #19811: [SPARK-18016][SQL] Code Generation: Constant Pool Limit ...

2017-11-30 Thread bdrillard
Github user bdrillard commented on the issue: https://github.com/apache/spark/pull/19811 As some context, I had initially found array initializations necessary because the number of `init` methods created to do line-by-line var initializations for large test cases was still

[GitHub] spark pull request #19518: [SPARK-18016][SQL][CATALYST] Code Generation: Con...

2017-10-17 Thread bdrillard
GitHub user bdrillard opened a pull request: https://github.com/apache/spark/pull/19518 [SPARK-18016][SQL][CATALYST] Code Generation: Constant Pool Limit - State Compaction ## What changes were proposed in this pull request? This PR is the part two followup to #18075

[GitHub] spark pull request #19518: [SPARK-18016][SQL][CATALYST] Code Generation: Con...

2017-10-17 Thread bdrillard
Github user bdrillard commented on a diff in the pull request: https://github.com/apache/spark/pull/19518#discussion_r145213781 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeGenerator.scala --- @@ -801,7 +908,10 @@ class

[GitHub] spark issue #16648: [SPARK-18016][SQL][CATALYST] Code Generation: Constant P...

2017-10-17 Thread bdrillard
Github user bdrillard commented on the issue: https://github.com/apache/spark/pull/16648 @kiszk please see #19518 for part 2 of this original PR, and thanks! --- - To unsubscribe, e-mail: reviews-unsubscr

[GitHub] spark issue #18075: [SPARK-18016][SQL][CATALYST] Code Generation: Constant P...

2017-10-17 Thread bdrillard
Github user bdrillard commented on the issue: https://github.com/apache/spark/pull/18075 The second part that follows this merged PR is up as #19518. --- - To unsubscribe, e-mail: reviews-unsubscr

[GitHub] spark issue #19518: [SPARK-18016][SQL][CATALYST] Code Generation: Constant P...

2017-10-19 Thread bdrillard
Github user bdrillard commented on the issue: https://github.com/apache/spark/pull/19518 @kiszk You are correct that the current implementation compacts all mutable state (where the state does not have to be _explicitly_ inlined). To your last question, I'd attempted

[GitHub] spark issue #19518: [SPARK-18016][SQL][CATALYST] Code Generation: Constant P...

2017-10-19 Thread bdrillard
Github user bdrillard commented on the issue: https://github.com/apache/spark/pull/19518 @kiszk Ah, thanks for the link back to that discussion. I'll make modifications to the trials for better data. --- ---

[GitHub] spark issue #16648: [SPARK-18016][SQL][CATALYST] Code Generation: Constant P...

2017-10-09 Thread bdrillard
Github user bdrillard commented on the issue: https://github.com/apache/spark/pull/16648 I'm blocking out time to prepare the part 2 PR for this issue starting today over this week, regarding compaction of excess primitive state. cc: @

[GitHub] spark pull request #19480: [SPARK-22226][SQL] splitExpression can create too...

2017-10-13 Thread bdrillard
Github user bdrillard commented on a diff in the pull request: https://github.com/apache/spark/pull/19480#discussion_r144564741 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeGenerator.scala --- @@ -277,13 +292,25 @@ class

[GitHub] spark issue #19480: [SPARK-22226][SQL] splitExpression can create too many m...

2017-10-16 Thread bdrillard
Github user bdrillard commented on the issue: https://github.com/apache/spark/pull/19480 @mgaido91 It is possible to increase the heap allocated during testing if that seems like a desirable thing to do (I did so for #18075, but the current default is already 4GB), see [1] and [2

[GitHub] spark issue #18377: [SPARK-18016][SQL][CATALYST][BRANCH-2.2] Code Generation...

2017-07-24 Thread bdrillard
Github user bdrillard commented on the issue: https://github.com/apache/spark/pull/18377 @cloud-fan @rxin @sameeragarwal What can we say is the status of these backports? Including this PR, #18354, and #18579, it was decided to revert the changes until the 2.2.0 release was finished

[GitHub] spark issue #16648: [SPARK-18016][SQL][CATALYST] Code Generation: Constant P...

2017-07-28 Thread bdrillard
Github user bdrillard commented on the issue: https://github.com/apache/spark/pull/16648 Thanks @kiszk, I'll work on preparing a PR for the second half of this issue. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as wel

[GitHub] spark issue #18075: [SPARK-18016][SQL][CATALYST] Code Generation: Constant P...

2017-05-31 Thread bdrillard
Github user bdrillard commented on the issue: https://github.com/apache/spark/pull/18075 The earlier failure occurred when the [`stopEarly()`](https://github.com/bdrillard/spark/blob/7fe5e4a84d4d8e71e2e63e6794e4ba13ac2e003f/sql/core/src/main/scala/org/apache/spark/sql/execution

[GitHub] spark issue #16648: [SPARK-18016][SQL][CATALYST] Code Generation: Constant P...

2017-06-01 Thread bdrillard
Github user bdrillard commented on the issue: https://github.com/apache/spark/pull/16648 @cloud-fan Good question, and I think we can resolve it by using different values for `N` in the [testcase](https://github.com/apache/spark/pull/18075/files#diff

[GitHub] spark pull request #18075: [SPARK-18016][SQL][CATALYST] Code Generation: Con...

2017-06-14 Thread bdrillard
Github user bdrillard commented on a diff in the pull request: https://github.com/apache/spark/pull/18075#discussion_r121968338 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeGenerator.scala --- @@ -233,10 +222,124 @@ class

[GitHub] spark pull request #18075: [SPARK-18016][SQL][CATALYST] Code Generation: Con...

2017-06-14 Thread bdrillard
Github user bdrillard commented on a diff in the pull request: https://github.com/apache/spark/pull/18075#discussion_r121975114 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeGenerator.scala --- @@ -233,10 +222,124 @@ class

[GitHub] spark pull request #18075: [SPARK-18016][SQL][CATALYST] Code Generation: Con...

2017-06-14 Thread bdrillard
Github user bdrillard commented on a diff in the pull request: https://github.com/apache/spark/pull/18075#discussion_r121989572 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/ColumnarBatchScan.scala --- @@ -93,7 +93,7 @@ private[sql] trait ColumnarBatchScan

[GitHub] spark pull request #18075: [SPARK-18016][SQL][CATALYST] Code Generation: Con...

2017-06-14 Thread bdrillard
Github user bdrillard commented on a diff in the pull request: https://github.com/apache/spark/pull/18075#discussion_r122016536 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeGenerator.scala --- @@ -233,10 +222,124 @@ class

[GitHub] spark pull request #18075: [SPARK-18016][SQL][CATALYST] Code Generation: Con...

2017-06-14 Thread bdrillard
Github user bdrillard commented on a diff in the pull request: https://github.com/apache/spark/pull/18075#discussion_r122016475 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeGenerator.scala --- @@ -233,10 +222,124 @@ class

[GitHub] spark pull request #18075: [SPARK-18016][SQL][CATALYST] Code Generation: Con...

2017-06-14 Thread bdrillard
Github user bdrillard commented on a diff in the pull request: https://github.com/apache/spark/pull/18075#discussion_r122016650 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeGenerator.scala --- @@ -629,7 +730,9 @@ class CodegenContext

[GitHub] spark pull request #18075: [SPARK-18016][SQL][CATALYST] Code Generation: Con...

2017-06-14 Thread bdrillard
Github user bdrillard commented on a diff in the pull request: https://github.com/apache/spark/pull/18075#discussion_r122016799 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeGenerator.scala --- @@ -233,10 +222,124 @@ class

[GitHub] spark pull request #18075: [SPARK-18016][SQL][CATALYST] Code Generation: Con...

2017-06-14 Thread bdrillard
Github user bdrillard commented on a diff in the pull request: https://github.com/apache/spark/pull/18075#discussion_r122016978 --- Diff: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/codegen/GeneratedProjectionSuite.scala --- @@ -83,6 +83,58 @@ class

[GitHub] spark pull request #18075: [SPARK-18016][SQL][CATALYST] Code Generation: Con...

2017-06-14 Thread bdrillard
Github user bdrillard commented on a diff in the pull request: https://github.com/apache/spark/pull/18075#discussion_r122017129 --- Diff: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/codegen/GeneratedProjectionSuite.scala --- @@ -83,6 +83,58 @@ class

[GitHub] spark pull request #18354: Class splitting 2.1

2017-06-19 Thread bdrillard
GitHub user bdrillard opened a pull request: https://github.com/apache/spark/pull/18354 Class splitting 2.1 ## What changes were proposed in this pull request? This is a backport patch for Spark 2.1.x of the class splitting feature over excess generated code as was merged

[GitHub] spark issue #18354: [SPARK-18016][SQL][Catalyst] Backport of Class Splitting

2017-06-20 Thread bdrillard
Github user bdrillard commented on the issue: https://github.com/apache/spark/pull/18354 @maor121 Yeah, I should be able to backpatch to 2.0.x as well in a separate PR. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If

[GitHub] spark issue #18354: [SPARK-18016][SQL][Catalyst] Backport of Class Splitting

2017-06-20 Thread bdrillard
Github user bdrillard commented on the issue: https://github.com/apache/spark/pull/18354 @dongjoon-hyun I had not yet. I hadn't realized master was already on a 2.3.0-SNAPSHOT. I can also write a backport for branch-2.2. --- If your project is set up for it, you can reply to

[GitHub] spark issue #18354: [SPARK-18016][SQL][Catalyst] Backport of Class Splitting

2017-06-20 Thread bdrillard
Github user bdrillard commented on the issue: https://github.com/apache/spark/pull/18354 @dongjoon-hyun Great, thanks for the heads up. I'll prepare a branch-2.2 backport PR and circle back to this PR with the link as confirmation. --- If your project is set up for it, you can

[GitHub] spark pull request #18377: [SPARK-18016][SQL][CATALYST][BRANCH-2.2] Code Gen...

2017-06-21 Thread bdrillard
GitHub user bdrillard opened a pull request: https://github.com/apache/spark/pull/18377 [SPARK-18016][SQL][CATALYST][BRANCH-2.2] Code Generation: Constant Pool Limit - Class Splitting ## What changes were proposed in this pull request? This is a backport patch for Spark

[GitHub] spark issue #18354: [SPARK-18016][SQL][CATALYST][BRANCH-2.1] Code Generation...

2017-06-21 Thread bdrillard
Github user bdrillard commented on the issue: https://github.com/apache/spark/pull/18354 @dongjoon-hyun I've prepared the PR for the branch-2.2 backport, #18377. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If

[GitHub] spark issue #18377: [SPARK-18016][SQL][CATALYST][BRANCH-2.2] Code Generation...

2017-06-22 Thread bdrillard
Github user bdrillard commented on the issue: https://github.com/apache/spark/pull/18377 @sameeragarwal @cloud-fan If I understand well what you mean by affect/regress, this patch would induce no change at all on the code generated for any class below the 1600k limit. For such

[GitHub] spark issue #18377: [SPARK-18016][SQL][CATALYST][BRANCH-2.2] Code Generation...

2017-06-23 Thread bdrillard
Github user bdrillard commented on the issue: https://github.com/apache/spark/pull/18377 @cloud-fan Sure! That makes perfect sense to me. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this

[GitHub] spark issue #16648: [SPARK-18016][SQL][CATALYST] Code Generation: Constant P...

2017-05-11 Thread bdrillard
Github user bdrillard commented on the issue: https://github.com/apache/spark/pull/16648 @HyukjinKwon @robert3005 I'll have some time soon to update this PR for the latest master. Thanks for the interest. It is a non-trivial change and would require a comprehensive code r

[GitHub] spark issue #16648: [SPARK-18016][SQL][CATALYST] Code Generation: Constant P...

2017-05-19 Thread bdrillard
Github user bdrillard commented on the issue: https://github.com/apache/spark/pull/16648 @kiszk, I've updated the pull-request description to include example code generation for mutable state compaction as well (which comes from inspecting the [generated code](https://githu

[GitHub] spark pull request #16648: [SPARK-18016][SQL][CATALYST] Code Generation: Con...

2017-05-22 Thread bdrillard
Github user bdrillard commented on a diff in the pull request: https://github.com/apache/spark/pull/16648#discussion_r117842789 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeGenerator.scala --- @@ -145,11 +145,85 @@ class

[GitHub] spark issue #16648: [SPARK-18016][SQL][CATALYST] Code Generation: Constant P...

2017-05-22 Thread bdrillard
Github user bdrillard commented on the issue: https://github.com/apache/spark/pull/16648 @kiszk We could do that, definitely. Changes in Feature 1 (splitting excess code among classes) are limited to the `CodeGeneration` class, and the few `Generate...` classes included with

[GitHub] spark pull request #16648: [SPARK-18016][SQL][CATALYST] Code Generation: Con...

2017-05-23 Thread bdrillard
Github user bdrillard commented on a diff in the pull request: https://github.com/apache/spark/pull/16648#discussion_r118050144 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeGenerator.scala --- @@ -145,11 +145,85 @@ class

[GitHub] spark issue #16648: [SPARK-18016][SQL][CATALYST] Code Generation: Constant P...

2017-05-23 Thread bdrillard
Github user bdrillard commented on the issue: https://github.com/apache/spark/pull/16648 @kiszk Sure, I'm glad to help make this change easier to review. I'll first make a PR that focuses on code splitting into nested classes. There should be a test case with a number of co

[GitHub] spark pull request #18075: [SPARK-18016][SQL][CATALYST] Code Generation: Con...

2017-05-23 Thread bdrillard
GitHub user bdrillard opened a pull request: https://github.com/apache/spark/pull/18075 [SPARK-18016][SQL][CATALYST] Code Generation: Constant Pool Limit - Class Splitting ## What changes were proposed in this pull request? This pull-request exclusively includes the class

[GitHub] spark issue #16648: [SPARK-18016][SQL][CATALYST] Code Generation: Constant P...

2017-05-23 Thread bdrillard
Github user bdrillard commented on the issue: https://github.com/apache/spark/pull/16648 I've created the first part of a pair of PRs to help make this review easier. Please see #18075 for a PR of the first feature (class splitting of excess code into nested sub-classes). If th

[GitHub] spark pull request #18075: [SPARK-18016][SQL][CATALYST] Code Generation: Con...

2017-05-24 Thread bdrillard
Github user bdrillard commented on a diff in the pull request: https://github.com/apache/spark/pull/18075#discussion_r118263506 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeGenerator.scala --- @@ -792,7 +887,18 @@ class

[GitHub] spark pull request #18075: [SPARK-18016][SQL][CATALYST] Code Generation: Con...

2017-05-24 Thread bdrillard
Github user bdrillard commented on a diff in the pull request: https://github.com/apache/spark/pull/18075#discussion_r118274094 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeGenerator.scala --- @@ -233,10 +223,129 @@ class

[GitHub] spark pull request #18075: [SPARK-18016][SQL][CATALYST] Code Generation: Con...

2017-05-24 Thread bdrillard
Github user bdrillard commented on a diff in the pull request: https://github.com/apache/spark/pull/18075#discussion_r118274179 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeGenerator.scala --- @@ -233,10 +223,129 @@ class

[GitHub] spark pull request #18075: [SPARK-18016][SQL][CATALYST] Code Generation: Con...

2017-05-24 Thread bdrillard
Github user bdrillard commented on a diff in the pull request: https://github.com/apache/spark/pull/18075#discussion_r118274597 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeGenerator.scala --- @@ -792,7 +887,18 @@ class

[GitHub] spark pull request #18075: [SPARK-18016][SQL][CATALYST] Code Generation: Con...

2017-05-24 Thread bdrillard
Github user bdrillard commented on a diff in the pull request: https://github.com/apache/spark/pull/18075#discussion_r118274669 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeGenerator.scala --- @@ -629,7 +736,9 @@ class CodegenContext

[GitHub] spark pull request #18075: [SPARK-18016][SQL][CATALYST] Code Generation: Con...

2017-05-24 Thread bdrillard
Github user bdrillard commented on a diff in the pull request: https://github.com/apache/spark/pull/18075#discussion_r118302075 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/basicPhysicalOperators.scala --- @@ -299,6 +297,9 @@ case class SampleExec

[GitHub] spark pull request #18075: [SPARK-18016][SQL][CATALYST] Code Generation: Con...

2017-05-24 Thread bdrillard
Github user bdrillard commented on a diff in the pull request: https://github.com/apache/spark/pull/18075#discussion_r118302508 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeGenerator.scala --- @@ -233,10 +223,124 @@ class

[GitHub] spark pull request #18075: [SPARK-18016][SQL][CATALYST] Code Generation: Con...

2017-05-24 Thread bdrillard
Github user bdrillard commented on a diff in the pull request: https://github.com/apache/spark/pull/18075#discussion_r118323003 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeGenerator.scala --- @@ -233,10 +223,124 @@ class

[GitHub] spark pull request #18075: [SPARK-18016][SQL][CATALYST] Code Generation: Con...

2017-05-24 Thread bdrillard
Github user bdrillard commented on a diff in the pull request: https://github.com/apache/spark/pull/18075#discussion_r118336805 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeGenerator.scala --- @@ -233,10 +223,124 @@ class

[GitHub] spark pull request #18075: [SPARK-18016][SQL][CATALYST] Code Generation: Con...

2017-05-24 Thread bdrillard
Github user bdrillard commented on a diff in the pull request: https://github.com/apache/spark/pull/18075#discussion_r118345866 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeGenerator.scala --- @@ -233,10 +223,124 @@ class

[GitHub] spark pull request #18075: [SPARK-18016][SQL][CATALYST] Code Generation: Con...

2017-05-25 Thread bdrillard
Github user bdrillard commented on a diff in the pull request: https://github.com/apache/spark/pull/18075#discussion_r118497931 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeGenerator.scala --- @@ -233,10 +223,124 @@ class

  1   2   >