[GitHub] spark pull request: [SPARK-10577] [PySpark] DataFrame hint for bro...
Github user Jianfeng-chs commented on the pull request: https://github.com/apache/spark/pull/8801#issuecomment-142208649 Thanks @rxin. My JIRA username is jzhang. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10577] [PySpark] DataFrame hint for bro...
Github user Jianfeng-chs commented on the pull request: https://github.com/apache/spark/pull/8801#issuecomment-141660403 @rxin Sorry for inconvenience. Can you trigger a new test for this. I have merged changes into one single commit. Thanks --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10577] [PySpark] DataFrame hint for bro...
Github user Jianfeng-chs commented on a diff in the pull request: https://github.com/apache/spark/pull/8801#discussion_r39914305 --- Diff: python/pyspark/sql/functions.py --- @@ -189,6 +190,14 @@ def approxCountDistinct(col, rsd=None): return Column(jc) +@since(1.6) +def broadcast(df): +"""Marks a DataFrame as small enough for use in broadcast joins.""" + +sc = SparkContext._active_spark_context +return DataFrame(sc._jvm.functions.broadcast(df._jdf), sc._jsc) --- End diff -- @davies changed it. Thank you! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10577] [PySpark] DataFrame hint for bro...
Github user Jianfeng-chs closed the pull request at: https://github.com/apache/spark/pull/8777 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10577] [PySpark] DataFrame hint for bro...
Github user Jianfeng-chs commented on the pull request: https://github.com/apache/spark/pull/8777#issuecomment-141106831 Since messed commits, closing it --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10577] [PySpark] DataFrame hint for bro...
GitHub user Jianfeng-chs opened a pull request: https://github.com/apache/spark/pull/8801 [SPARK-10577] [PySpark] DataFrame hint for broadcast join https://issues.apache.org/jira/browse/SPARK-10577 You can merge this pull request into a Git repository by running: $ git pull https://github.com/Jianfeng-chs/spark master Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/8801.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #8801 commit 794b283adc6bf1adff1d6af10d19c2af10931109 Author: Jian Feng <jzhang@gmail.com> Date: 2015-09-17T21:33:06Z [SPARK-10577] [PySpark] DataFrame hint for broadcast join https://issues.apache.org/jira/browse/SPARK-10577 commit c8267ee31464cd383522d27992fb88978a973ca9 Author: Jian Feng <jzhang@gmail.com> Date: 2015-09-17T21:47:12Z [SPARK-10577] [PySpark] DataFrame hint for broadcast join https://issues.apache.org/jira/browse/SPARK-10577 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10577] [PySpark] DataFrame hint for bro...
Github user Jianfeng-chs commented on a diff in the pull request: https://github.com/apache/spark/pull/8777#discussion_r39635628 --- Diff: python/pyspark/sql/functions.py --- @@ -189,6 +190,14 @@ def approxCountDistinct(col, rsd=None): return Column(jc) +@since(1.6) +def broadcast(df): +"""Marks a DataFrame as small enough for use in broadcast joins.""" --- End diff -- Reynold Xin, I have added one test for this. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10577] [PySpark] DataFrame hint for bro...
Github user Jianfeng-chs closed the pull request at: https://github.com/apache/spark/pull/8770 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10577] [PySpark] DataFrame hint for bro...
Github user Jianfeng-chs commented on the pull request: https://github.com/apache/spark/pull/8770#issuecomment-140579799 um, let me close this one to submit a new one. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10577] [PySpark] DataFrame hint for bro...
GitHub user Jianfeng-chs opened a pull request: https://github.com/apache/spark/pull/8777 [SPARK-10577] [PySpark] DataFrame hint for broadcast join https://issues.apache.org/jira/browse/SPARK-10577 You can merge this pull request into a Git repository by running: $ git pull https://github.com/Jianfeng-chs/spark master Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/8777.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #8777 commit ded210be4bb7c01f6df1ede6269cfb2b1db325d2 Author: Jian Feng <jzhang@gmail.com> Date: 2015-09-16T02:18:55Z [SPARK-10577] [PySpark] DataFrame hint for broadcast join https://issues.apache.org/jira/browse/SPARK-10577 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10577] [PySpark] DataFrame hint for bro...
Github user Jianfeng-chs commented on the pull request: https://github.com/apache/spark/pull/8770#issuecomment-140579825 Thanks. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10577] [PySpark] DataFrame hint for bro...
Github user Jianfeng-chs commented on the pull request: https://github.com/apache/spark/pull/8770#issuecomment-140436530 https://issues.apache.org/jira/browse/SPARK-10577 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10577] [PySpark] DataFrame hint for bro...
GitHub user Jianfeng-chs opened a pull request: https://github.com/apache/spark/pull/8770 [SPARK-10577] [PySpark] DataFrame hint for broadcast join https://issues.apache.org/jira/browse/SPARK-10577 You can merge this pull request into a Git repository by running: $ git pull https://github.com/Jianfeng-chs/spark master Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/8770.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #8770 commit 0d0f4d5a1a8136bc8e2c1d72733979d7d3a659de Author: Jianfeng-chs <jzhang@gmail.com> Date: 2015-05-14T13:45:56Z Merge pull request #1 from apache/master Syn apache spark commit 43e23314a2ca4d248f1bb8796bd6f093cbcf62b0 Author: Jianfeng <jzhang@gmail.com> Date: 2015-09-10T05:44:59Z Merge remote-tracking branch 'apache/master' commit 6b221b9c53856611c58eb548bed32986e608b80c Author: Jianfeng <jzhang@gmail.com> Date: 2015-09-10T15:58:48Z Merge remote-tracking branch 'apache/master' commit cbd485a87d0f7607a3e118aeb9176ad870e14712 Author: Jianfeng <jzhang@gmail.com> Date: 2015-09-11T01:01:00Z Merge remote-tracking branch 'apache/master' commit 4807a6e8a46b31158f72e3a0d8b3a40f8651dde7 Author: Jian Feng <jzhang@gmail.com> Date: 2015-09-15T07:55:42Z Merge remote-tracking branch 'apache/master' commit 5d643aa919e0aa35024a673c7c36f6e98e1429a4 Author: Jian Feng <jzhang@gmail.com> Date: 2015-09-15T08:09:41Z For [PySpark] DataFrame hint for broadcast join commit bbd68c42aaa554cba50c9711768d98e28fbd78a7 Author: Jian Feng <jzhang@gmail.com> Date: 2015-09-15T08:22:43Z Revert "For [PySpark] DataFrame hint for broadcast join" This reverts commit 5d643aa919e0aa35024a673c7c36f6e98e1429a4. commit 5168fa6f897de52303efc94bbe3c2728e7c0597d Author: Jian Feng <jzhang@gmail.com> Date: 2015-09-15T08:25:48Z [SPARK-10577] [PySpark] DataFrame hint for broadcast join --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: Merge pull request #1 from apache/master
GitHub user Jianfeng-chs opened a pull request: https://github.com/apache/spark/pull/8576 Merge pull request #1 from apache/master Syn apache spark You can merge this pull request into a Git repository by running: $ git pull https://github.com/Jianfeng-chs/spark master Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/8576.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #8576 commit 0d0f4d5a1a8136bc8e2c1d72733979d7d3a659de Author: Jianfeng-chs <jzhang@gmail.com> Date: 2015-05-14T13:45:56Z Merge pull request #1 from apache/master Syn apache spark --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
