[GitHub] spark pull request #21045: [SPARK-23931][SQL] Adds zip function to sparksql

2018-05-12 Thread DylanGuedes
Github user DylanGuedes commented on a diff in the pull request: https://github.com/apache/spark/pull/21045#discussion_r187775631 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala --- @@ -90,6 +90,112 @@ case class MapKeys

[GitHub] spark issue #21045: [SPARK-23931][SQL] Adds zip function to sparksql

2018-05-12 Thread DylanGuedes
Github user DylanGuedes commented on the issue: https://github.com/apache/spark/pull/21045 @mgaido91 Thank you for the suggestions and for being so patient. I updated the code with `zip` name, more tests in CollectionExpression (I'll add more after adding support to any number

[GitHub] spark issue #21045: [SPARK-23931][SQL] Adds zip function to sparksql

2018-05-11 Thread DylanGuedes
Github user DylanGuedes commented on the issue: https://github.com/apache/spark/pull/21045 Sorry for taking so long to update this - had a bad time looking for the best way to wrap the zip result. At the end I decided to use InternalRow/Struct, but I'm not sure if it is the best

[GitHub] spark pull request #21045: [WIP][SPARK-23931][SQL] Adds zip function to spar...

2018-05-15 Thread DylanGuedes
Github user DylanGuedes commented on a diff in the pull request: https://github.com/apache/spark/pull/21045#discussion_r188283550 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala --- @@ -90,6 +90,110 @@ case class MapKeys

[GitHub] spark issue #21045: [SPARK-23931][SQL] Adds zip function to sparksql

2018-05-15 Thread DylanGuedes
Github user DylanGuedes commented on the issue: https://github.com/apache/spark/pull/21045 Thank you so much for the suggestions! I tried to use IntelliJ a few times but at the end I always return to sbt/terminal/vim after some frustration (mainly due to not being able to configure

[GitHub] spark pull request #21045: [WIP][SPARK-23931][SQL] Adds zip function to spar...

2018-05-15 Thread DylanGuedes
Github user DylanGuedes commented on a diff in the pull request: https://github.com/apache/spark/pull/21045#discussion_r188283661 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala --- @@ -90,6 +90,110 @@ case class MapKeys

[GitHub] spark pull request #21045: [SPARK-23931][SQL] Adds zip function to sparksql

2018-05-17 Thread DylanGuedes
Github user DylanGuedes commented on a diff in the pull request: https://github.com/apache/spark/pull/21045#discussion_r188947896 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala --- @@ -90,6 +90,132 @@ case class MapKeys

[GitHub] spark pull request #21045: [SPARK-23931][SQL] Adds zip function to sparksql

2018-05-17 Thread DylanGuedes
Github user DylanGuedes commented on a diff in the pull request: https://github.com/apache/spark/pull/21045#discussion_r189023172 --- Diff: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CollectionExpressionsSuite.scala --- @@ -199,6 +200,20 @@ class

[GitHub] spark pull request #21045: [WIP][SPARK-23931][SQL] Adds zip function to spar...

2018-05-16 Thread DylanGuedes
Github user DylanGuedes commented on a diff in the pull request: https://github.com/apache/spark/pull/21045#discussion_r188693270 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala --- @@ -90,6 +90,117 @@ case class MapKeys

[GitHub] spark issue #21045: [WIP][SPARK-23931][SQL] Adds zip function to sparksql

2018-05-16 Thread DylanGuedes
Github user DylanGuedes commented on the issue: https://github.com/apache/spark/pull/21045 @mn-mikke I thought that `CodeGenerator.getValue` was directly used to retrieve values from 1d arrays (such as an arraydata) - but I don't get how to use it for 2d arrays (such as `Seq

[GitHub] spark pull request #21045: [WIP][SPARK-23931][SQL] Adds zip function to spar...

2018-05-16 Thread DylanGuedes
Github user DylanGuedes commented on a diff in the pull request: https://github.com/apache/spark/pull/21045#discussion_r188693715 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala --- @@ -90,6 +90,117 @@ case class MapKeys

[GitHub] spark pull request #21045: [WIP][SPARK-23931][SQL] Adds zip function to spar...

2018-05-16 Thread DylanGuedes
Github user DylanGuedes commented on a diff in the pull request: https://github.com/apache/spark/pull/21045#discussion_r188694041 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala --- @@ -90,6 +90,117 @@ case class MapKeys

[GitHub] spark pull request #21045: [WIP][SPARK-23931][SQL] Adds zip function to spar...

2018-05-16 Thread DylanGuedes
Github user DylanGuedes commented on a diff in the pull request: https://github.com/apache/spark/pull/21045#discussion_r188694557 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala --- @@ -90,6 +90,117 @@ case class MapKeys

[GitHub] spark pull request #21045: [WIP][SPARK-23931][SQL] Adds zip function to spar...

2018-05-15 Thread DylanGuedes
Github user DylanGuedes commented on a diff in the pull request: https://github.com/apache/spark/pull/21045#discussion_r188419498 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala --- @@ -90,6 +90,88 @@ case class MapKeys

[GitHub] spark pull request #21045: [WIP][SPARK-23931][SQL] Adds zip function to spar...

2018-05-15 Thread DylanGuedes
Github user DylanGuedes commented on a diff in the pull request: https://github.com/apache/spark/pull/21045#discussion_r188420006 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala --- @@ -90,6 +90,88 @@ case class MapKeys

[GitHub] spark issue #21045: [WIP][SPARK-23931][SQL] Adds zip function to sparksql

2018-05-15 Thread DylanGuedes
Github user DylanGuedes commented on the issue: https://github.com/apache/spark/pull/21045 @mn-mikke thank you! Any idea on how to access elements of individual arrays? In the old version I written a 'getValue' that uses `CodeGenerator.getValue`, but since now it is a 2d data

[GitHub] spark pull request #21045: [SPARK-23931][SQL] Adds zip function to sparksql

2018-05-21 Thread DylanGuedes
Github user DylanGuedes commented on a diff in the pull request: https://github.com/apache/spark/pull/21045#discussion_r189567647 --- Diff: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CollectionExpressionsSuite.scala --- @@ -265,6 +266,69 @@ class

[GitHub] spark pull request #21045: [SPARK-23931][SQL] Adds zip function to sparksql

2018-05-22 Thread DylanGuedes
Github user DylanGuedes commented on a diff in the pull request: https://github.com/apache/spark/pull/21045#discussion_r189925378 --- Diff: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CollectionExpressionsSuite.scala --- @@ -288,6 +289,88 @@ class

[GitHub] spark pull request #21045: [SPARK-23931][SQL] Adds zip function to sparksql

2018-05-22 Thread DylanGuedes
Github user DylanGuedes commented on a diff in the pull request: https://github.com/apache/spark/pull/21045#discussion_r189900854 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala --- @@ -127,6 +127,148 @@ case class

[GitHub] spark pull request #21045: [SPARK-23931][SQL] Adds zip function to sparksql

2018-05-22 Thread DylanGuedes
Github user DylanGuedes commented on a diff in the pull request: https://github.com/apache/spark/pull/21045#discussion_r189887317 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala --- @@ -127,6 +127,148 @@ case class

[GitHub] spark pull request #21045: [SPARK-23931][SQL] Adds zip function to sparksql

2018-05-22 Thread DylanGuedes
Github user DylanGuedes commented on a diff in the pull request: https://github.com/apache/spark/pull/21045#discussion_r190094810 --- Diff: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CollectionExpressionsSuite.scala --- @@ -288,6 +289,88 @@ class

[GitHub] spark pull request #21045: [SPARK-23931][SQL] Adds zip function to sparksql

2018-05-22 Thread DylanGuedes
Github user DylanGuedes commented on a diff in the pull request: https://github.com/apache/spark/pull/21045#discussion_r190094430 --- Diff: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CollectionExpressionsSuite.scala --- @@ -288,6 +289,88 @@ class

[GitHub] spark pull request #21045: [SPARK-23931][SQL] Adds zip function to sparksql

2018-05-22 Thread DylanGuedes
Github user DylanGuedes commented on a diff in the pull request: https://github.com/apache/spark/pull/21045#discussion_r190095025 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala --- @@ -127,6 +127,148 @@ case class

[GitHub] spark pull request #21045: [SPARK-23931][SQL] Adds zip function to sparksql

2018-05-23 Thread DylanGuedes
Github user DylanGuedes commented on a diff in the pull request: https://github.com/apache/spark/pull/21045#discussion_r190270795 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala --- @@ -127,6 +127,176 @@ case class

[GitHub] spark pull request #21045: [SPARK-23931][SQL] Adds zip function to sparksql

2018-05-22 Thread DylanGuedes
Github user DylanGuedes commented on a diff in the pull request: https://github.com/apache/spark/pull/21045#discussion_r190034892 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala --- @@ -127,6 +127,148 @@ case class

[GitHub] spark pull request #21045: [SPARK-23931][SQL] Adds zip function to sparksql

2018-05-22 Thread DylanGuedes
Github user DylanGuedes commented on a diff in the pull request: https://github.com/apache/spark/pull/21045#discussion_r190036385 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala --- @@ -127,6 +127,148 @@ case class

[GitHub] spark issue #21045: [SPARK-23931][SQL] Adds zip function to sparksql

2018-05-25 Thread DylanGuedes
Github user DylanGuedes commented on the issue: https://github.com/apache/spark/pull/21045 During the last days I looked for other ways to use `getValue` properly without such repetition, but didn't get success at all :( suggestions are (very) welcome

[GitHub] spark pull request #21045: [SPARK-23931][SQL] Adds zip function to sparksql

2018-05-18 Thread DylanGuedes
Github user DylanGuedes commented on a diff in the pull request: https://github.com/apache/spark/pull/21045#discussion_r189316316 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala --- @@ -126,6 +126,134 @@ case class

[GitHub] spark pull request #21045: [SPARK-23931][SQL] Adds zip function to sparksql

2018-05-18 Thread DylanGuedes
Github user DylanGuedes commented on a diff in the pull request: https://github.com/apache/spark/pull/21045#discussion_r189316555 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala --- @@ -126,6 +126,134 @@ case class

[GitHub] spark issue #21045: [SPARK-23931][SQL] Adds zip function to sparksql

2018-05-16 Thread DylanGuedes
Github user DylanGuedes commented on the issue: https://github.com/apache/spark/pull/21045 Great! Also I finally discovered the unit-tests log file that logs the generated java code :D Since now it works I'll remove the WIP tag and focus on the other suggestions (such as new

[GitHub] spark pull request #21045: [SPARK-23931][SQL] Adds zip function to sparksql

2018-05-23 Thread DylanGuedes
Github user DylanGuedes commented on a diff in the pull request: https://github.com/apache/spark/pull/21045#discussion_r190301638 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala --- @@ -127,6 +127,176 @@ case class

[GitHub] spark pull request #21045: [SPARK-23931][SQL] Adds zip function to sparksql

2018-05-22 Thread DylanGuedes
Github user DylanGuedes commented on a diff in the pull request: https://github.com/apache/spark/pull/21045#discussion_r190014678 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala --- @@ -127,6 +127,148 @@ case class

[GitHub] spark pull request #21045: [SPARK-23931][SQL] Adds zip function to sparksql

2018-05-22 Thread DylanGuedes
Github user DylanGuedes commented on a diff in the pull request: https://github.com/apache/spark/pull/21045#discussion_r189996412 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala --- @@ -127,6 +127,148 @@ case class

[GitHub] spark pull request #21045: [SPARK-23931][SQL] Adds zip function to sparksql

2018-05-18 Thread DylanGuedes
Github user DylanGuedes commented on a diff in the pull request: https://github.com/apache/spark/pull/21045#discussion_r189421719 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala --- @@ -126,6 +126,149 @@ case class

[GitHub] spark pull request #21045: [SPARK-23931][SQL] Adds zip function to sparksql

2018-05-18 Thread DylanGuedes
Github user DylanGuedes commented on a diff in the pull request: https://github.com/apache/spark/pull/21045#discussion_r189413681 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala --- @@ -126,6 +126,149 @@ case class

[GitHub] spark pull request #21045: [SPARK-23931][SQL] Adds zip function to sparksql

2018-06-07 Thread DylanGuedes
Github user DylanGuedes commented on a diff in the pull request: https://github.com/apache/spark/pull/21045#discussion_r193759057 --- Diff: python/pyspark/sql/functions.py --- @@ -2394,6 +2394,23 @@ def array_repeat(col, count): return Column(sc._jvm.functions.array_repeat

[GitHub] spark pull request #21045: [SPARK-23931][SQL] Adds arrays_zip function to sp...

2018-06-11 Thread DylanGuedes
Github user DylanGuedes commented on a diff in the pull request: https://github.com/apache/spark/pull/21045#discussion_r194530372 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala --- @@ -128,6 +128,170 @@ case class

[GitHub] spark issue #21045: [SPARK-23931][SQL] Adds zip function to sparksql

2018-06-11 Thread DylanGuedes
Github user DylanGuedes commented on the issue: https://github.com/apache/spark/pull/21045 Is the CI broken? oO --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail

[GitHub] spark pull request #21045: [SPARK-23931][SQL] Adds zip function to sparksql

2018-05-29 Thread DylanGuedes
Github user DylanGuedes commented on a diff in the pull request: https://github.com/apache/spark/pull/21045#discussion_r191412802 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala --- @@ -127,6 +127,165 @@ case class

[GitHub] spark pull request #21045: [SPARK-23931][SQL] Adds zip function to sparksql

2018-06-04 Thread DylanGuedes
Github user DylanGuedes commented on a diff in the pull request: https://github.com/apache/spark/pull/21045#discussion_r192729662 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala --- @@ -127,6 +127,165 @@ case class

[GitHub] spark issue #21045: [SPARK-23931][SQL] Adds zip function to sparksql

2018-06-04 Thread DylanGuedes
Github user DylanGuedes commented on the issue: https://github.com/apache/spark/pull/21045 @ueshin I took really long to update this because I'm getting a cache error and didn't manage to figure why. Just in case I pushed my last changes with your recommendation, and although looks

[GitHub] spark issue #21045: [SPARK-23931][SQL] Adds zip function to sparksql

2018-06-04 Thread DylanGuedes
Github user DylanGuedes commented on the issue: https://github.com/apache/spark/pull/21045 @ueshin Ok, looks like setting ev.isNull was causing the ev.value weird behaviour - thank you! Updated with a working version

[GitHub] spark pull request #21045: [SPARK-23931][SQL] Adds zip function to sparksql

2018-06-04 Thread DylanGuedes
Github user DylanGuedes commented on a diff in the pull request: https://github.com/apache/spark/pull/21045#discussion_r192862329 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala --- @@ -128,6 +128,175 @@ case class

[GitHub] spark pull request #21045: [SPARK-23931][SQL] Adds zip function to sparksql

2018-06-05 Thread DylanGuedes
Github user DylanGuedes commented on a diff in the pull request: https://github.com/apache/spark/pull/21045#discussion_r193053405 --- Diff: python/pyspark/sql/functions.py --- @@ -350,7 +350,7 @@ def corr(col1, col2): >>> a = range(20) >>>

[GitHub] spark pull request #21045: [SPARK-23931][SQL] Adds zip function to sparksql

2018-05-28 Thread DylanGuedes
Github user DylanGuedes commented on a diff in the pull request: https://github.com/apache/spark/pull/21045#discussion_r191202434 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala --- @@ -127,6 +127,165 @@ case class

[GitHub] spark pull request #21045: [SPARK-23931][SQL] Adds zip function to sparksql

2018-05-28 Thread DylanGuedes
Github user DylanGuedes commented on a diff in the pull request: https://github.com/apache/spark/pull/21045#discussion_r191208028 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala --- @@ -127,6 +127,176 @@ case class

[GitHub] spark issue #21045: [SPARK-23931][SQL][WIP] Adds zip function to sparksql

2018-06-04 Thread DylanGuedes
Github user DylanGuedes commented on the issue: https://github.com/apache/spark/pull/21045 Ok looks like the ternary if was the problem. Fixed it! --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org

[GitHub] spark pull request #21045: [SPARK-23931][SQL] Adds arrays_zip function to sp...

2018-07-16 Thread DylanGuedes
Github user DylanGuedes commented on a diff in the pull request: https://github.com/apache/spark/pull/21045#discussion_r202683445 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala --- @@ -128,6 +128,172 @@ case class

[GitHub] spark pull request #21045: [SPARK-23931][SQL] Adds arrays_zip function to sp...

2018-07-16 Thread DylanGuedes
Github user DylanGuedes commented on a diff in the pull request: https://github.com/apache/spark/pull/21045#discussion_r202688999 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala --- @@ -128,6 +128,172 @@ case class

[GitHub] spark issue #20788: [SPARK-23647][PYTHON][SQL] Adds more types for hint in p...

2018-03-11 Thread DylanGuedes
Github user DylanGuedes commented on the issue: https://github.com/apache/spark/pull/20788 Hi, I added two subtasks (one for Python and one for R) [here](https://issues.apache.org/jira/browse/SPARK-21030) Should I close this PR and open a new one or it is ok to just rename

[GitHub] spark pull request #20788: [SPARK-23647][PYTHON][SQL] Adds more types for hi...

2018-03-13 Thread DylanGuedes
Github user DylanGuedes commented on a diff in the pull request: https://github.com/apache/spark/pull/20788#discussion_r174208501 --- Diff: python/pyspark/sql/dataframe.py --- @@ -437,10 +437,11 @@ def hint(self, name, *parameters): if not isinstance(name, str

[GitHub] spark pull request #20775: Changes input variable to not conflict with built...

2018-03-08 Thread DylanGuedes
GitHub user DylanGuedes opened a pull request: https://github.com/apache/spark/pull/20775 Changes input variable to not conflict with built-in function Signed-off-by: DylanGuedes <djmggue...@gmail.com> ## What changes were proposed in this pull request? C

[GitHub] spark pull request #20788: [WIP][SPARK-21030][PYTHON][SQL] Adds more types f...

2018-03-10 Thread DylanGuedes
Github user DylanGuedes commented on a diff in the pull request: https://github.com/apache/spark/pull/20788#discussion_r173623998 --- Diff: python/pyspark/sql/dataframe.py --- @@ -437,10 +437,11 @@ def hint(self, name, *parameters): if not isinstance(name, str

[GitHub] spark issue #20775: [PYTHON] Changes input variable to not conflict with bui...

2018-03-10 Thread DylanGuedes
Github user DylanGuedes commented on the issue: https://github.com/apache/spark/pull/20775 Sure. Done. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark pull request #20788: [WIP][SPARK-21030][PYTHON][SQL] Adds more types f...

2018-03-10 Thread DylanGuedes
Github user DylanGuedes commented on a diff in the pull request: https://github.com/apache/spark/pull/20788#discussion_r173618711 --- Diff: python/pyspark/sql/dataframe.py --- @@ -437,10 +437,11 @@ def hint(self, name, *parameters): if not isinstance(name, str

[GitHub] spark issue #20775: Changes input variable to not conflict with built-in fun...

2018-03-09 Thread DylanGuedes
Github user DylanGuedes commented on the issue: https://github.com/apache/spark/pull/20775 Sure, and thanks for the review! I used the "dataset" name because I checked a few examples that also use it, but I'll rename to path then. I already checked all other examples

[GitHub] spark issue #20775: Changes input variable to not conflict with built-in fun...

2018-03-09 Thread DylanGuedes
Github user DylanGuedes commented on the issue: https://github.com/apache/spark/pull/20775 @HyukjinKwon I checked every other built in function and this was the only one being used. What you think

[GitHub] spark pull request #20775: Changes input variable to not conflict with built...

2018-03-09 Thread DylanGuedes
Github user DylanGuedes commented on a diff in the pull request: https://github.com/apache/spark/pull/20775#discussion_r173446874 --- Diff: examples/src/main/python/ml/dataframe_example.py --- @@ -35,18 +35,18 @@ print("Usage: dataframe_example.py ", file=

[GitHub] spark pull request #20788: [WIP][SPARK-21030][PYTHON][SQL] Adds more types f...

2018-03-09 Thread DylanGuedes
GitHub user DylanGuedes opened a pull request: https://github.com/apache/spark/pull/20788 [WIP][SPARK-21030][PYTHON][SQL] Adds more types for hint in pyspark Signed-off-by: DylanGuedes <djmggue...@gmail.com> ## What changes were proposed in this pull r

[GitHub] spark pull request #20788: [SPARK-23647][PYTHON][SQL] Adds more types for hi...

2018-03-14 Thread DylanGuedes
Github user DylanGuedes commented on a diff in the pull request: https://github.com/apache/spark/pull/20788#discussion_r174580902 --- Diff: python/pyspark/sql/dataframe.py --- @@ -437,10 +437,11 @@ def hint(self, name, *parameters): if not isinstance(name, str

[GitHub] spark issue #20788: [SPARK-23647][PYTHON][SQL] Adds more types for hint in p...

2018-04-10 Thread DylanGuedes
Github user DylanGuedes commented on the issue: https://github.com/apache/spark/pull/20788 Hi, any new feedback about this? thank you! --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org

[GitHub] spark pull request #21045: [WIP][SPARK-23931][SQLAdds zip function to sparks...

2018-04-11 Thread DylanGuedes
GitHub user DylanGuedes opened a pull request: https://github.com/apache/spark/pull/21045 [WIP][SPARK-23931][SQLAdds zip function to sparksql Signed-off-by: DylanGuedes <djmggue...@gmail.com> ## What changes were proposed in this pull request? (Pleas

[GitHub] spark pull request #21045: [WIP][SPARK-23931][SQL] Adds zip function to spar...

2018-04-12 Thread DylanGuedes
Github user DylanGuedes commented on a diff in the pull request: https://github.com/apache/spark/pull/21045#discussion_r181196604 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala --- @@ -87,6 +87,62 @@ case class MapKeys

[GitHub] spark pull request #21045: [WIP][SPARK-23931][SQL] Adds zip function to spar...

2018-04-12 Thread DylanGuedes
Github user DylanGuedes commented on a diff in the pull request: https://github.com/apache/spark/pull/21045#discussion_r181189546 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala --- @@ -87,6 +87,62 @@ case class MapKeys

[GitHub] spark pull request #21045: [WIP][SPARK-23931][SQL] Adds zip function to spar...

2018-04-12 Thread DylanGuedes
Github user DylanGuedes commented on a diff in the pull request: https://github.com/apache/spark/pull/21045#discussion_r181189694 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala --- @@ -87,6 +87,62 @@ case class MapKeys

[GitHub] spark issue #21045: [WIP][SPARK-23931][SQL] Adds zip function to sparksql

2018-04-13 Thread DylanGuedes
Github user DylanGuedes commented on the issue: https://github.com/apache/spark/pull/21045 Ok so It works fine in spark-shell but in pyspark I got this error: ```shell File "/home/dguedes/Workspace/spark/python/pyspark/sql/functions.py", line 2155, in pyspark.sql.fun

[GitHub] spark pull request #21045: [WIP][SPARK-23931][SQL] Adds zip function to spar...

2018-04-13 Thread DylanGuedes
Github user DylanGuedes commented on a diff in the pull request: https://github.com/apache/spark/pull/21045#discussion_r181371296 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala --- @@ -87,6 +87,62 @@ case class MapKeys

[GitHub] spark issue #21045: [WIP][SPARK-23931][SQL] Adds zip function to sparksql

2018-04-16 Thread DylanGuedes
Github user DylanGuedes commented on the issue: https://github.com/apache/spark/pull/21045 @mgaido91 thank you, the suggestions were VERY enlightening! You are correct, I tried to return the expected output in `doGenCode`, according with others implementations I thougth

[GitHub] spark pull request #20788: [SPARK-23647][PYTHON][SQL] Adds more types for hi...

2018-03-16 Thread DylanGuedes
Github user DylanGuedes commented on a diff in the pull request: https://github.com/apache/spark/pull/20788#discussion_r175181523 --- Diff: python/pyspark/sql/dataframe.py --- @@ -437,10 +437,12 @@ def hint(self, name, *parameters): if not isinstance(name, str

[GitHub] spark issue #20788: [SPARK-23647][PYTHON][SQL] Adds more types for hint in p...

2018-11-10 Thread DylanGuedes
Github user DylanGuedes commented on the issue: https://github.com/apache/spark/pull/20788 @HyukjinKwon Sure! I'll be working on this, then. Do you have any specific scenarios in mind, so that I could start with them

[GitHub] spark issue #20788: [SPARK-23647][PYTHON][SQL] Adds more types for hint in p...

2018-11-13 Thread DylanGuedes
Github user DylanGuedes commented on the issue: https://github.com/apache/spark/pull/20788 @HyukjinKwon I added a test but I checked only the logical plan - is it ok? --- - To unsubscribe, e-mail: reviews-unsubscr

[GitHub] spark issue #20788: [SPARK-23647][PYTHON][SQL] Adds more types for hint in p...

2018-11-14 Thread DylanGuedes
Github user DylanGuedes commented on the issue: https://github.com/apache/spark/pull/20788 > **[Test build #98824 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/98824/testReport)** for PR 20788 at commit [`de3257f`](https://github.com/apache/sp