[GitHub] spark pull request: [SPARK-14883][DOCS] Fix wrong R examples and m...

2016-04-24 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the pull request: https://github.com/apache/spark/pull/12649#issuecomment-213907830 Hi, @davies , @shivaram, @felixcheung . Could you review this PR when you have some time? --- If your project is set up for it, you can reply to this email

[GitHub] spark pull request: [SPARK-14664][SQL] Fix DecimalAggregates optim...

2016-04-24 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the pull request: https://github.com/apache/spark/pull/12421#issuecomment-213907928 Rebased. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SPARK-15020][SQL] GROUP-BY should support Ali...

2016-04-29 Thread dongjoon-hyun
GitHub user dongjoon-hyun opened a pull request: https://github.com/apache/spark/pull/12794 [SPARK-15020][SQL] GROUP-BY should support Aliases ## What changes were proposed in this pull request? `GROUP-BY` clauses raise **AnalysisException** for aliases while `ORDER

[GitHub] spark pull request: [SPARK-15020][SQL] GROUP-BY should support Ali...

2016-04-29 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the pull request: https://github.com/apache/spark/pull/12794#issuecomment-215910465 Oh, Spark prefers `ordinal`. Thank you again, @gatorsmile . I'll close this PR and JIRA together now. --- If your project is set up for it, you can reply

[GitHub] spark pull request: [SPARK-15020][SQL] GROUP-BY should support Ali...

2016-04-29 Thread dongjoon-hyun
Github user dongjoon-hyun closed the pull request at: https://github.com/apache/spark/pull/12794 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so

[GitHub] spark pull request: [SPARK-15020][SQL] GROUP-BY should support Ali...

2016-04-29 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the pull request: https://github.com/apache/spark/pull/12794#issuecomment-215910061 Opps. Thank you for notifying me. I'll. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark pull request: [SPARK-14939][SQL] Add FoldablePropagation opt...

2016-04-30 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the pull request: https://github.com/apache/spark/pull/12719#issuecomment-215940750 Hi, @cloud-fan . Could you review this PR again when you have some time? --- If your project is set up for it, you can reply to this email and have your

[GitHub] spark pull request: [SPARK-15031][EXAMPLE] Fix SQL Python example.

2016-04-30 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the pull request: https://github.com/apache/spark/pull/12809#issuecomment-215943715 I thought this is a bug fix, but I think I need to update to use `SparkSession`, too. I will add a commit very soon. --- If your project is set up

[GitHub] spark pull request: [SPARK-15031][EXAMPLE] Fix SQL Python example.

2016-04-30 Thread dongjoon-hyun
GitHub user dongjoon-hyun opened a pull request: https://github.com/apache/spark/pull/12809 [SPARK-15031][EXAMPLE] Fix SQL Python example. ## What changes were proposed in this pull request? Currently, Python SQL example, `sql.py`, fails due to the following two lines

[GitHub] spark pull request: [SPARK-15031][EXAMPLE] Fix SQL Python example.

2016-04-30 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the pull request: https://github.com/apache/spark/pull/12809#issuecomment-215945687 As you know, for the example, I need to verify the result manually. I'll proceed testsuite one first because it's automatically verified while this one

[GitHub] spark pull request: [MINOR][EXAMPLE] Use SparkSession instead of S...

2016-04-30 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the pull request: https://github.com/apache/spark/pull/12808#issuecomment-215943551 Oh, sure! I will take a look at SQL testsuites and make a single PR for that. Thank you, @rxin . --- If your project is set up for it, you can reply

[GitHub] spark pull request: [SPARK-15031][EXAMPLE] Fix SQL Python example.

2016-04-30 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the pull request: https://github.com/apache/spark/pull/12809#issuecomment-215943997 I mean `SparkSession` updating here since I didn't run all examples. --- If your project is set up for it, you can reply to this email and have your reply appear

[GitHub] spark pull request: [MINOR][EXAMPLE] Use SparkSession instead of S...

2016-04-30 Thread dongjoon-hyun
GitHub user dongjoon-hyun opened a pull request: https://github.com/apache/spark/pull/12808 [MINOR][EXAMPLE] Use SparkSession instead of SQLContext in RDDRelation.scala ## What changes were proposed in this pull request? Now, `SQLContext` is used for backward-compatibility

[GitHub] spark pull request: [SPARK-15031][EXAMPLE] Fix SQL Python example.

2016-04-30 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the pull request: https://github.com/apache/spark/pull/12809#issuecomment-215943886 Oh, sure. May I proceed this PR for all examples and all testsuites together? --- If your project is set up for it, you can reply to this email and have your

[GitHub] spark pull request: [SPARK-15031][EXAMPLE] Fix SQL Python example.

2016-04-30 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the pull request: https://github.com/apache/spark/pull/12809#issuecomment-215943910 If you don't mind, I prefer to do as a single one. I can update the JIRA and PR description. --- If your project is set up for it, you can reply

[GitHub] spark pull request: [SPARK-15031][EXAMPLE] Fix SQL Python example.

2016-04-30 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the pull request: https://github.com/apache/spark/pull/12809#issuecomment-215945517 I see. No problem. Then, I'll use this one for all the example changes (including some fix like here). --- If your project is set up for it, you can reply

[GitHub] spark pull request: [SPARK-15031][EXAMPLE] Use SparkSession in Sca...

2016-04-30 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the pull request: https://github.com/apache/spark/pull/12809#issuecomment-215996680 Now, I addressed all comments so far. Thank you for fast reviews, @rxin . --- If your project is set up for it, you can reply to this email and have your

[GitHub] spark pull request: [SPARK-15031][EXAMPLE] Use SparkSession in Sca...

2016-04-30 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the pull request: https://github.com/apache/spark/pull/12809#issuecomment-215992910 Hi, @rxin . For this issue, I'll add new constructor for `SparkSession` and proceed to Java examples. ``` def this(sparkContext: JavaSparkContext

[GitHub] spark pull request: [SPARK-15031][EXAMPLE] Use SparkSession in Sca...

2016-04-30 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the pull request: https://github.com/apache/spark/pull/12809#issuecomment-215995243 I updated Java examples with SparkSession(JavaSparkContext). For `SparkSession(SparkConf)`, I'll handle soon. --- If your project is set up for it, you can

[GitHub] spark pull request: [SPARK-15031][EXAMPLE] Use SparkSession in Sca...

2016-04-30 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the pull request: https://github.com/apache/spark/pull/12809#issuecomment-216012424 Rebased. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SPARK-14939][SQL] Improve EliminateSorts opti...

2016-04-28 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the pull request: https://github.com/apache/spark/pull/12719#issuecomment-215616875 Thank you for review, @cloud-fan ! Do you mean removing aliases by replacing the base expression(?) by using `transformUp`? Maybe, except the top most

[GitHub] spark pull request: [SPARK-14939][SQL] Improve EliminateSorts opti...

2016-04-28 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the pull request: https://github.com/apache/spark/pull/12719#issuecomment-215620369 If that's just about how to handle `Sort(_, Project(_,_))` expressions in `EliminateSorts`, I can easily modify this PR according to your advice. After moving

[GitHub] spark pull request: [SPARK-14939][SQL] Improve EliminateSorts opti...

2016-04-28 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the pull request: https://github.com/apache/spark/pull/12719#issuecomment-215620961 Right. Thank you so much for enriching ideas! I'll update this PR with `FoldablePropagation`. --- If your project is set up for it, you can reply

[GitHub] spark pull request: [SPARK-14939][SQL] Improve EliminateSorts opti...

2016-04-28 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the pull request: https://github.com/apache/spark/pull/12719#issuecomment-215619413 Actually, `Sort` is dead end, we can not propagate up anymore. So, in that case, removing looks more efficient. Do you mean more generalized

[GitHub] spark pull request: [SPARK-14939][SQL] Improve EliminateSorts opti...

2016-04-28 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the pull request: https://github.com/apache/spark/pull/12719#issuecomment-215620522 Oh, I got. Thanks. I will try to generalize. * Sort(_, Project(_)) * Project(_, Project(...)) And so on. --- If your project is set up for it, you

[GitHub] spark pull request: [SPARK-14907][MLLIB] Use repartition in GLMReg...

2016-04-26 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the pull request: https://github.com/apache/spark/pull/12676#issuecomment-214822460 Hi, @mengxr . Could you review this too? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well

[GitHub] spark pull request: [SPARK-14830][SQL] Add RemoveRepetitionFromGro...

2016-04-26 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the pull request: https://github.com/apache/spark/pull/12590#issuecomment-214831287 Hi, @marmbrus . Could you review this PR when you have some time? --- If your project is set up for it, you can reply to this email and have your reply

[GitHub] spark pull request: [SPARK-14907][MLLIB] Use repartition in GLMReg...

2016-04-25 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the pull request: https://github.com/apache/spark/pull/12676#issuecomment-214605818 Hi, @jkbradley . Could you review this PR when you have some time? --- If your project is set up for it, you can reply to this email and have your reply

[GitHub] spark pull request: [MINOR][BUILD] Enable RAT checking on `LZ4Bloc...

2016-04-25 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the pull request: https://github.com/apache/spark/pull/12677#issuecomment-214606445 Hi, @davies and @srowen . This PR just removes `LZ4BlockInputStream.java` from `dev/.rat-exclude` and passed the RAT test. Could you merge this PR

[GitHub] spark pull request: [SPARK-14907][MLLIB] Use repartition in GLMReg...

2016-04-26 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the pull request: https://github.com/apache/spark/pull/12676#issuecomment-214842843 In fact, I didn't try to change that if it's just a style problem. --- If your project is set up for it, you can reply to this email and have your reply appear

[GitHub] spark pull request: [SPARK-14907][MLLIB] Use repartition in GLMReg...

2016-04-26 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the pull request: https://github.com/apache/spark/pull/12676#issuecomment-214842611 If you think so, it's okay, @jkbradley . But, if you don't mind, could you remove those TODO by yourself. Do you have any reason to maintain

[GitHub] spark pull request: [HOTFIX][SQL] sparkSession can't be private.

2016-04-25 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the pull request: https://github.com/apache/spark/pull/12684#issuecomment-214612978 It's a clean build after `git clean -fdx`. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well

[GitHub] spark pull request: [HOTFIX][SQL] sparkSession can't be private.

2016-04-25 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the pull request: https://github.com/apache/spark/pull/12684#issuecomment-214612753 Ur, @rxin . Could you check that again? ``` [error] /Users/dongjoon/spark/sql/hivecontext-compatibility/src/main/scala/org/apache/spark/sql/hive

[GitHub] spark pull request: [HOTFIX][SQL] sparkSession can't be private.

2016-04-25 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the pull request: https://github.com/apache/spark/pull/12684#issuecomment-214615564 Thank you! :) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SPARK-14664][SQL] Implement DecimalAggregates...

2016-04-25 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the pull request: https://github.com/apache/spark/pull/12421#issuecomment-214616662 Rebased. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [HOTFIX][SQL] sparkSession can't be private.

2016-04-25 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the pull request: https://github.com/apache/spark/pull/12684#issuecomment-214608530 cc @rxin . --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [HOTFIX][SQL] sparkSession can't be private.

2016-04-25 Thread dongjoon-hyun
GitHub user dongjoon-hyun opened a pull request: https://github.com/apache/spark/pull/12684 [HOTFIX][SQL] sparkSession can't be private. ## What changes were proposed in this pull request? This fixes the following errors. ``` -@transient private val sparkSession

[GitHub] spark pull request: [HOTFIX][SQL] sparkSession can't be private.

2016-04-25 Thread dongjoon-hyun
Github user dongjoon-hyun closed the pull request at: https://github.com/apache/spark/pull/12684 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so

[GitHub] spark pull request: [HOTFIX][SQL] sparkSession can't be private.

2016-04-25 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the pull request: https://github.com/apache/spark/pull/12684#issuecomment-214609078 Sure! Thank for quick fix. I found that when I rebase my PR. --- If your project is set up for it, you can reply to this email and have your reply appear

[GitHub] spark pull request: [SPARK-14796][SQL] Add spark.sql.optimizer.inS...

2016-04-21 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the pull request: https://github.com/apache/spark/pull/12562#issuecomment-213177718 Hi, @rxin and @marmbrus . How do you think about the updated PR? It's just first update. If there is something to do more, please let me know. Thank

[GitHub] spark pull request: [SPARK-14830][SQL] Add RemoveRepetitionFromGro...

2016-04-21 Thread dongjoon-hyun
GitHub user dongjoon-hyun opened a pull request: https://github.com/apache/spark/pull/12590 [SPARK-14830][SQL] Add RemoveRepetitionFromGroupExpressions optimizer. ## What changes were proposed in this pull request? This PR aims to optimize GroupExpressions by removing

[GitHub] spark pull request: [SPARK-14883][DOCS] Fix wrong R examples and m...

2016-04-24 Thread dongjoon-hyun
GitHub user dongjoon-hyun opened a pull request: https://github.com/apache/spark/pull/12649 [SPARK-14883][DOCS] Fix wrong R examples and make them up-to-date ## What changes were proposed in this pull request? This issue aims to fix some errors in R examples and make them

[GitHub] spark pull request: [SPARK-14868][BUILD] Enable NewLineAtEofChecke...

2016-04-24 Thread dongjoon-hyun
Github user dongjoon-hyun commented on a diff in the pull request: https://github.com/apache/spark/pull/12632#discussion_r60839935 --- Diff: core/src/main/java/org/apache/spark/util/collection/unsafe/sort/PrefixComparators.java --- @@ -82,37 +81,37 @@ public static long

[GitHub] spark pull request: [SPARK-14868][BUILD] Enable NewLineAtEofChecke...

2016-04-24 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the pull request: https://github.com/apache/spark/pull/12632#issuecomment-213906023 Hi, @rxin . Thank you for review. FYI, here is the result of `dev/lint-java` of current master branch. ```bash spark:master$ dev/lint-java Using `mvn

[GitHub] spark pull request: [SPARK-14868][BUILD] Enable NewLineAtEofChecke...

2016-04-24 Thread dongjoon-hyun
Github user dongjoon-hyun commented on a diff in the pull request: https://github.com/apache/spark/pull/12632#discussion_r60839977 --- Diff: mllib/src/test/java/org/apache/spark/ml/classification/JavaRandomForestClassifierSuite.java --- @@ -81,15 +81,15 @@ public void runDT

[GitHub] spark pull request: [SPARK-14868][BUILD] Enable NewLineAtEofChecke...

2016-04-24 Thread dongjoon-hyun
Github user dongjoon-hyun commented on a diff in the pull request: https://github.com/apache/spark/pull/12632#discussion_r60840026 --- Diff: mllib/src/test/java/org/apache/spark/ml/classification/JavaRandomForestClassifierSuite.java --- @@ -81,15 +81,15 @@ public void runDT

[GitHub] spark pull request: [SPARK-14868][BUILD] Enable NewLineAtEofChecke...

2016-04-24 Thread dongjoon-hyun
Github user dongjoon-hyun commented on a diff in the pull request: https://github.com/apache/spark/pull/12632#discussion_r60840040 --- Diff: mllib/src/test/java/org/apache/spark/ml/classification/JavaRandomForestClassifierSuite.java --- @@ -81,15 +81,15 @@ public void runDT

[GitHub] spark pull request: [SPARK-14868][BUILD] Enable NewLineAtEofChecke...

2016-04-24 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the pull request: https://github.com/apache/spark/pull/12632#issuecomment-213906191 Sure! It's just one line change. May I turn it one right now? --- If your project is set up for it, you can reply to this email and have your reply appear

[GitHub] spark pull request: [SPARK-14868][BUILD] Enable NewLineAtEofChecke...

2016-04-24 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the pull request: https://github.com/apache/spark/pull/12632#issuecomment-213907254 Interesting. @rxin . Jenkins is trying to use Maven 3.1.1 due to the mismatch between `--force` option and `lint-java`. ``` Using `mvn` from path: /home

[GitHub] spark pull request: [SPARK-14868][BUILD] Enable NewLineAtEofChecke...

2016-04-24 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the pull request: https://github.com/apache/spark/pull/12632#issuecomment-213907583 I reverted the last commit about Jenkins Java Linter. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub

[GitHub] spark pull request: [SPARK-14867][BUILD] Remove `--force` option i...

2016-04-23 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the pull request: https://github.com/apache/spark/pull/12631#issuecomment-213880918 Rebased. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SPARK-13135][SQL] Don't print expressions rec...

2016-05-19 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the pull request: https://github.com/apache/spark/pull/13192#issuecomment-220257517 Hi, @rxin . This is the first attempt according to your request. I removed some obsolete code in #11019 in order to pass the tests. Please let me know

[GitHub] spark pull request: [SPARK-15058][MLLIB][TEST] Enable Java Decisio...

2016-05-19 Thread dongjoon-hyun
Github user dongjoon-hyun closed the pull request at: https://github.com/apache/spark/pull/12840 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so

[GitHub] spark pull request: [SPARK-15282][SQL] PushDownPredicate should no...

2016-05-20 Thread dongjoon-hyun
Github user dongjoon-hyun commented on a diff in the pull request: https://github.com/apache/spark/pull/13087#discussion_r64002341 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala --- @@ -1025,7 +1025,8 @@ object PushDownPredicate

[GitHub] spark pull request: [SPARK-13135][SQL] Don't print expressions rec...

2016-05-20 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the pull request: https://github.com/apache/spark/pull/13192#issuecomment-220532622 As @rxin told, what was really needed is removing `overlapping` comments. So, I rethink about that and revert the change on `Expression.gen` which removes

[GitHub] spark pull request: [SPARK-15282][SQL] PushDownPredicate should no...

2016-05-20 Thread dongjoon-hyun
Github user dongjoon-hyun commented on a diff in the pull request: https://github.com/apache/spark/pull/13087#discussion_r63997574 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala --- @@ -1025,7 +1025,8 @@ object PushDownPredicate

[GitHub] spark pull request: [SPARK-15057][GRAPHX] Remove stale TODO commen...

2016-05-20 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the pull request: https://github.com/apache/spark/pull/12839#issuecomment-220530065 Thank you, @rxin . :) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does

[GitHub] spark pull request: [SPARK-15282][SQL] Make ScalaUDF nondeterminis...

2016-05-20 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the pull request: https://github.com/apache/spark/pull/13087#issuecomment-220659068 Thank you for review, @marmbrus and @markhamstra ! Actually, it's huge change. Although I'm not aware of the real background, the reported case can

[GitHub] spark pull request: [SPARK-13135][SQL] Don't print expressions rec...

2016-05-20 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the pull request: https://github.com/apache/spark/pull/13192#issuecomment-220667461 Hi, @davies . It's ready for review, again! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well

[GitHub] spark pull request: [SPARK-15282][SQL] PushDownPredicate should no...

2016-05-20 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the pull request: https://github.com/apache/spark/pull/13087#issuecomment-220662788 @marmbrus @markhamstra @thunterdb . Now, this code and description of this PR is rollbacked my second commit 7 days ago. For `common subexpression

[GitHub] spark pull request: [SPARK-13135][SQL] Don't print expressions rec...

2016-05-20 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the pull request: https://github.com/apache/spark/pull/13192#issuecomment-220682262 Ya. There were a huge change. I've saw the PR before, but I didn't consider that in this PR. My bad. Let me think how to solve the original goal with new

[GitHub] spark pull request: [SPARK-15282][SQL] Make ScalaUDF nondeterminis...

2016-05-20 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the pull request: https://github.com/apache/spark/pull/13087#issuecomment-220547374 According to @cloud-fan 's advice, the goal of this PR is now making `ScalaUDF` as a non-deterministic expression. Although this is a correct fix, one noticeable

[GitHub] spark pull request: [SPARK-15282][SQL] Make ScalaUDF nondeterminis...

2016-05-20 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the pull request: https://github.com/apache/spark/pull/13087#issuecomment-220563755 Hmm. @cloud-fan . There is bad news. **ALS.scala** uses UDF on aggregation. So, there are 7 failures on **ALSSuite.scala**. ```scala override def

[GitHub] spark pull request: [SPARK-13135][SQL] Don't print expressions rec...

2016-05-20 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the pull request: https://github.com/apache/spark/pull/13192#issuecomment-220547752 Retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SPARK-15282][SQL] Make ScalaUDF nondeterminis...

2016-05-20 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the pull request: https://github.com/apache/spark/pull/13087#issuecomment-220572951 I updated `ALS` and `ALSSuite` just in order to pass the Jenkins for further discussion. --- If your project is set up for it, you can reply to this email

[GitHub] spark pull request: [SPARK-15462][SQL][TEST] `unresolved === false...

2016-05-21 Thread dongjoon-hyun
GitHub user dongjoon-hyun opened a pull request: https://github.com/apache/spark/pull/13241 [SPARK-15462][SQL][TEST] `unresolved === false` is enough in testcases. ## What changes were proposed in this pull request? In only `catalyst` module, there exists 7 evaluation test

[GitHub] spark pull request: [SPARK-15462][SQL][TEST] `unresolved === false...

2016-05-21 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the pull request: https://github.com/apache/spark/pull/13241#issuecomment-220763620 cc @cloud-fan . --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SPARK-15282][SQL][DOCS] Add notes of the dete...

2016-05-23 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the pull request: https://github.com/apache/spark/pull/13087#issuecomment-221049934 Hi, @marmbrus . I replaced 'should' with 'must', and added the detail description for `functions.py`, `SQLContext.scala`, `SparkSession.scala

[GitHub] spark pull request: [SPARK-15282][SQL][DOCS] Add notes of the dete...

2016-05-23 Thread dongjoon-hyun
Github user dongjoon-hyun commented on a diff in the pull request: https://github.com/apache/spark/pull/13087#discussion_r64259325 --- Diff: python/pyspark/sql/functions.py --- @@ -1756,6 +1756,7 @@ def __call__(self, *cols): @since(1.3) def udf(f, returnType=StringType

[GitHub] spark pull request: [SPARK-15282][SQL][DOCS] Add notes of the dete...

2016-05-23 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the pull request: https://github.com/apache/spark/pull/13087#issuecomment-221093278 Hi, @marmbrus . Instead of creating new JIRA, I think we had better change the title of this PR into `[MINOR][SQL][DOC] ...`. Initially, I tried to handle

[GitHub] spark pull request: [MINOR][SQL][DOCS] Add notes of the determinis...

2016-05-23 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the pull request: https://github.com/apache/spark/pull/13087#issuecomment-221098550 Oops. I didn't change the title yet. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark pull request: [SPARK-15512][CORE] repartition(0) should rais...

2016-05-24 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the pull request: https://github.com/apache/spark/pull/13282#issuecomment-221425046 Thank you for review, @rxin . I'll check them whether they need this. --- If your project is set up for it, you can reply to this email and have your reply

[GitHub] spark pull request: [SPARK-15512][CORE] repartition(0) should rais...

2016-05-24 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the pull request: https://github.com/apache/spark/pull/13282#issuecomment-221437321 retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SPARK-15512][CORE] repartition(0) should rais...

2016-05-24 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the pull request: https://github.com/apache/spark/pull/13282#issuecomment-221462008 Thank you, @rxin ! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SPARK-15512][CORE] repartition(0) should rais...

2016-05-24 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the pull request: https://github.com/apache/spark/pull/13282#issuecomment-221431064 Hi, @rxin . I added that to catalyst/DataSetSuite/DataFrameSuite, too. So far, I cannot find a proper place in Analyzer. If you mind, could you give me

[GitHub] spark pull request: [SPARK-15512][CORE] repartition(0) should rais...

2016-05-24 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the pull request: https://github.com/apache/spark/pull/13282#issuecomment-221435072 It seems not necessary since this PR prevents at the level of `Repartition` ctor. ``` case class Repartition(numPartitions: Int, shuffle: Boolean, child

[GitHub] spark pull request: [MINOR][SQL][DOCS] Add notes of the determinis...

2016-05-23 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the pull request: https://github.com/apache/spark/pull/13087#issuecomment-221099176 I'm not sure what happen. I'll remove this PR information from @linbojin 's JIRA issue anyway. --- If your project is set up for it, you can reply to this email

[GitHub] spark pull request: [MINOR][SQL][DOCS] Add notes of the determinis...

2016-05-23 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the pull request: https://github.com/apache/spark/pull/13087#issuecomment-22101 Thank you so much! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [MINOR][SQL][DOCS] Add notes of the determinis...

2016-05-23 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the pull request: https://github.com/apache/spark/pull/13087#issuecomment-221101369 Thank you all for reviewing and helping this PR! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well

[GitHub] spark pull request: [SPARK-14939][SQL] Add FoldablePropagation opt...

2016-05-18 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the pull request: https://github.com/apache/spark/pull/12719#issuecomment-220134254 In addition, sorry for that I wrote a wrong example before `checkEvaluation(FormatNumber(Literal(4.asInstanceOf[Byte]), Literal(3)), "4.000")`. -

[GitHub] spark pull request: [SPARK-14939][SQL] Add FoldablePropagation opt...

2016-05-18 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the pull request: https://github.com/apache/spark/pull/12719#issuecomment-220128276 Hi, @cloud-fan . I found the root cause. There is an exceptional case for **Literal.create(null, NullType)**. ```scala scala> imp

[GitHub] spark pull request: [SPARK-15373][WEB UI] Spark UI should show con...

2016-05-18 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the pull request: https://github.com/apache/spark/pull/13158#issuecomment-220173646 Hi, @zsxwing . Finally, it passes the Jenkins test. :) --- If your project is set up for it, you can reply to this email and have your reply appear

[GitHub] spark pull request: [SPARK-15373][WEB UI] Spark UI should show con...

2016-05-18 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the pull request: https://github.com/apache/spark/pull/13158#issuecomment-220175949 Oh, thank you, @srowen ! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does

[GitHub] spark pull request: [SPARK-15058][MLLIB][TEST] Enable Java Decisio...

2016-05-18 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the pull request: https://github.com/apache/spark/pull/12840#issuecomment-220161025 Hi, @jkbradley , @mengxr , @MLnick , @rxin , @srowen . Such a long silence definitely means I did this in a wrong way, didn't it? There were already

[GitHub] spark pull request: [SPARK-14939][SQL] Add FoldablePropagation opt...

2016-05-18 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the pull request: https://github.com/apache/spark/pull/12719#issuecomment-220212842 For the second suggestion, `the optimizer is not tested but skipped`, you mean skipping `FoldablePropagation` optimizer? --- If your project is set up

[GitHub] spark pull request: [SPARK-14939][SQL] Add FoldablePropagation opt...

2016-05-18 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the pull request: https://github.com/apache/spark/pull/12719#issuecomment-220216995 Thank you for understanding. I'll try to handle those test issues in another PR. --- If your project is set up for it, you can reply to this email and have your

[GitHub] spark pull request: [SPARK-14939][SQL] Add FoldablePropagation opt...

2016-05-18 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the pull request: https://github.com/apache/spark/pull/12719#issuecomment-220207591 No, there are more test suite failures. That is just one example. --- If your project is set up for it, you can reply to this email and have your reply appear

[GitHub] spark pull request: [SPARK-14939][SQL] Add FoldablePropagation opt...

2016-05-18 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the pull request: https://github.com/apache/spark/pull/12719#issuecomment-220217482 Oh, amazing. According to the last Jenkins results. The seven test failures in `catalyst` are all of them. ``` [info] *** 7 TESTS FAILED *** [error

[GitHub] spark pull request: [SPARK-14939][SQL] Add FoldablePropagation opt...

2016-05-18 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the pull request: https://github.com/apache/spark/pull/12719#issuecomment-220207782 Actually, I made two separate Jenkins run to show you the comparison. Those fails on MiMa errors. --- If your project is set up for it, you can reply

[GitHub] spark pull request: [SPARK-14939][SQL] Add FoldablePropagation opt...

2016-05-18 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the pull request: https://github.com/apache/spark/pull/12719#issuecomment-220207983 May I rollback the last commit? Let's see the Jenkins result. I think It's worth of doing that. --- If your project is set up for it, you can reply

[GitHub] spark pull request: [SPARK-14939][SQL] Add FoldablePropagation opt...

2016-05-18 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the pull request: https://github.com/apache/spark/pull/12719#issuecomment-220210913 Should I touch those too? I agree with you that this situation will not occur in **real** testcases. Maybe, only `catalyst`-related problems? --- If your

[GitHub] spark pull request: [SPARK-14939][SQL] Add FoldablePropagation opt...

2016-05-18 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the pull request: https://github.com/apache/spark/pull/12719#issuecomment-220208494 Here is another test run (without resolved checking and the test case `checkEvaluation(FormatNumber(Literal.create(null, NullType), Literal(3)), null

[GitHub] spark pull request: [SPARK-14939][SQL] Add FoldablePropagation opt...

2016-05-18 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the pull request: https://github.com/apache/spark/pull/12719#issuecomment-220210664 @cloud-fan . Here is the result of `catalyst` first. - **catalyst**: 7 failures - DateExpressionsSuite: 3 failures - CastSuite: 4 failures

[GitHub] spark pull request: [SPARK-14939][SQL] Add FoldablePropagation opt...

2016-05-18 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the pull request: https://github.com/apache/spark/pull/12719#issuecomment-220211380 Okay, I will proceed with modifying `checkEvaluation`. Thank you for fast decision, @cloud-fan . --- If your project is set up for it, you can reply

[GitHub] spark pull request: [SPARK-14939][SQL] Add FoldablePropagation opt...

2016-05-18 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the pull request: https://github.com/apache/spark/pull/12719#issuecomment-220212554 Oops. Analyzing by modifying `checkEvaluation` is not helpful in this case. For example, `CastSuite`, ``` { val ret = cast(array_notNull

[GitHub] spark pull request: [SPARK-14939][SQL] Add FoldablePropagation opt...

2016-05-18 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the pull request: https://github.com/apache/spark/pull/12719#issuecomment-220213423 Hmm. @cloud-fan . What about just using `resolved` checking simply? IMHO, it provides just robustness. And, in fact, I'm reluctant to change testsuite when

[GitHub] spark pull request: [SPARK-14939][SQL] Add FoldablePropagation opt...

2016-05-18 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the pull request: https://github.com/apache/spark/pull/12719#issuecomment-220217246 I removed the last test commit. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project

[GitHub] spark pull request: [SPARK-13135][SQL] Don't print expressions rec...

2016-05-19 Thread dongjoon-hyun
Github user dongjoon-hyun commented on a diff in the pull request: https://github.com/apache/spark/pull/13192#discussion_r63916714 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/GenerateMutableProjection.scala --- @@ -124,6 +124,7

[GitHub] spark pull request: [SPARK-13135][SQL] Don't print expressions rec...

2016-05-19 Thread dongjoon-hyun
Github user dongjoon-hyun commented on a diff in the pull request: https://github.com/apache/spark/pull/13192#discussion_r63918627 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/GenerateMutableProjection.scala --- @@ -124,6 +124,7

[GitHub] spark pull request: [SPARK-13135][SQL] Don't print expressions rec...

2016-05-19 Thread dongjoon-hyun
Github user dongjoon-hyun commented on a diff in the pull request: https://github.com/apache/spark/pull/13192#discussion_r63915288 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeFormatter.scala --- @@ -24,13 +24,13 @@ package

<    5   6   7   8   9   10   11   12   13   14   >