date:20151127

[GitHub] spark pull request: [SPARK-5682][Core] Add encrypted shuffle in sp...

2015-11-27 Thread AmplabJenkins

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/8880#issuecomment-160167903
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5682][Core] Add encrypted shuffle in sp...

2015-11-27 Thread SparkQA

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/8880#issuecomment-160167851
  
**[Test build #46822 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/46822/consoleFull)**
 for PR 8880 at commit 
[`6d001ca`](https://github.com/apache/spark/commit/6d001ca57e5d92faba88b2439437d304b9a51988).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11102] [SQL] Uninformative exception wh...

2015-11-27 Thread AmplabJenkins

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9490#issuecomment-160172809
  
Build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12012] [SQL] Show more comprehensive Ph...

2015-11-27 Thread SparkQA

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/10004#issuecomment-160175554
  
**[Test build #46825 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/46825/consoleFull)**
 for PR 10004 at commit 
[`8cbf0ff`](https://github.com/apache/spark/commit/8cbf0ff166f57f0592149381bf3a8daf7ed311c4).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12023][BUILD] Fix warnings while packag...

2015-11-27 Thread SparkQA

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/10014#issuecomment-160175326
  
**[Test build #2124 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/2124/consoleFull)**
 for PR 10014 at commit 
[`6e9a37b`](https://github.com/apache/spark/commit/6e9a37b86c10f223590decfc7b3903e988daad28).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12010][SQL] Spark JDBC requires support...

2015-11-27 Thread srowen

Github user srowen commented on the pull request:

https://github.com/apache/spark/pull/10003#issuecomment-160189345
  
I'm not sure what the issue is with SBT (do you get download errors or 
something?) but you can also see the results of the style checker in the build 
output: 
https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/2119/consoleFull
Search for "[error]"

I tend to think it's better practice to specify column names explicitly. Is 
there a downside? I don't recall a database that _wouldn't_ let you specify 
them, right?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12028] [SQL] get_json_object returns an...

2015-11-27 Thread gatorsmile

GitHub user gatorsmile opened a pull request:

https://github.com/apache/spark/pull/10018

[SPARK-12028] [SQL] get_json_object returns an incorrect result when the 
value is null literals

When calling `get_json_object` for the following two cases, both results 
are `"null"`:

```scala
val tuple: Seq[(String, String)] = ("5", """{"f1": null}""") :: Nil
val df: DataFrame = tuple.toDF("key", "jstring")
val res = df.select(functions.get_json_object($"jstring", 
"$.f1")).collect()
```
```scala
val tuple2: Seq[(String, String)] = ("5", """{"f1": "null"}""") :: Nil
val df2: DataFrame = tuple2.toDF("key", "jstring")
val res3 = df2.select(functions.get_json_object($"jstring", 
"$.f1")).collect()
```

Fixed the problem and also added a test case. 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gatorsmile/spark get_json_object

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/10018.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #10018


commit 06d9eae73e4b40a0451d1d21f7174aacbf05f780
Author: gatorsmile 
Date:   2015-11-27T18:59:31Z

fixed a bug of get_json_object

commit 54edc84f21918c5cb69a0abfc51f680190f27a1f
Author: gatorsmile 
Date:   2015-11-27T19:04:13Z

Merge remote-tracking branch 'upstream/master' into get_json_object




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12010][SQL] Spark JDBC requires support...

2015-11-27 Thread CK50

Github user CK50 commented on the pull request:

https://github.com/apache/spark/pull/10003#issuecomment-160193463

Do I take this as "SBT works fine for me on master"?

Yes, build/sbt fails for me when doing downloads:

Getting org.scala-sbt sbt 0.13.7 ...

:: problems summary ::
 WARNINGS
module not found: org.scala-sbt#sbt;0.13.7
    local: tried
 /home/ckurz/.ivy2/local/org.scala-sbt/sbt/0.13.7/ivys/ivy.xml
 -- artifact org.scala-sbt#sbt;0.13.7!sbt.jar:
 /home/ckurz/.ivy2/local/org.scala-sbt/sbt/0.13.7/jars/sbt.jar
    jcenter: tried
https://jcenter.bintray.com/org/scala-sbt/sbt/0.13.7/sbt-0.13.7.pom
 -- artifact org.scala-sbt#sbt;0.13.7!sbt.jar:
https://jcenter.bintray.com/org/scala-sbt/sbt/0.13.7/sbt-0.13.7.jar
    typesafe-ivy-releases: tried

https://repo.typesafe.com/typesafe/ivy-releases/org.scala-sbt/sbt/0.13.7/ivys/ivy.xml
    Maven Central: tried
https://repo1.maven.org/maven2/org/scala-sbt/sbt/0.13.7/sbt-0.13.7.pom
 -- artifact org.scala-sbt#sbt;0.13.7!sbt.jar:
https://repo1.maven.org/maven2/org/scala-sbt/sbt/0.13.7/sbt-0.13.7.jar
::
::  UNRESOLVED DEPENDENCIES ::
::
:: org.scala-sbt#sbt;0.13.7: not found
::

 ERRORS
   Server access Error: Connection timed out 
url=https://jcenter.bintray.com/org/scala-sbt/sbt/0.13.7/sbt-0.13.7.pom
   Server access Error: Connection timed out 
url=https://jcenter.bintray.com/org/scala-sbt/sbt/0.13.7/sbt-0.13.7.jar
   Server access Error: Connection timed out 

url=https://repo.typesafe.com/typesafe/ivy-releases/org.scala-sbt/sbt/0.13.7/ivys/ivy.xml
   Server access Error: Connection timed out 
url=https://repo1.maven.org/maven2/org/scala-sbt/sbt/0.13.7/sbt-0.13.7.pom
   Server access Error: Connection timed out 
url=https://repo1.maven.org/maven2/org/scala-sbt/sbt/0.13.7/sbt-0.13.7.jar

I am behind a proxy and have tried configuring linux env vars and 
JAVA_OPTS/SBT_OPTS, but this did not help.

The thing, which strikes me is that when I try accessing these urls: 
https://jcenter.bintray.com/org/scala-sbt/sbt/0.13.7/sbt-0.13.7.pom
from within a browser, I also get 404. When looking into 
https://jcenter.bintray.com/org/scala-sbt there is no sub-folder called 
sbt. - No idea what is going wrong here :-(

--
Back to the PR:
I will upload the suggested style changes.

In general I agree that column names should be given by default. I just 
have two concerns re making this the new default behaviour:

1) does this work on all supported DBs? - Does Spark have automated 
tests with appropriate envs for this?
2) is it okay to assume that DataFrame column names always match target 
col names? (Spark currently does not rely on this. Instead it relies on 
column positions, which is usually not good practice.)
3) is such a change okay with Spark's rules for backwards compatibility?

Just some concerns, I am fine with making column names the new default 
behaviour. But as I said, I do not have the background to judge this.

Please let me know how to proceed.

On 27.11.2015 19:49, Sean Owen wrote:
>
> I'm not sure what the issue is with SBT (do you get download errors or 
> something?) but you can also see the results of the style checker in 
> the build output: 
> 
https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/2119/consoleFull
> Search for "[error]"
>
> I tend to think it's better practice to specify column names 
> explicitly. Is there a downside? I don't recall a database that 
> /wouldn't/ let you specify them, right?
>
> â
> Reply to this email directly or view it on GitHub 
> .
>

---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12025] [SparkR] Rename some window rank...

2015-11-27 Thread shivaram

Github user shivaram commented on the pull request:

https://github.com/apache/spark/pull/10016#issuecomment-160195289
  
LGTM. Thanks @yanboliang -- BTW were these APIs were present in SparkR 1.5 
? If so we'll need to update the release docs about this change.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12021][Streaming][Tests]Fix the potenti...

2015-11-27 Thread zsxwing

Github user zsxwing commented on the pull request:

https://github.com/apache/spark/pull/10011#issuecomment-160195297
  
I saw this issue in several PR builds. I'm going to merge this one because 
it only touches the test codes and it bores other people.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5682][Core] Add encrypted shuffle in sp...

2015-11-27 Thread AmplabJenkins

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/8880#issuecomment-160167905
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/46822/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5337][Mesos][Standalone] respect spark....

2015-11-27 Thread CodingCat

Github user CodingCat commented on the pull request:

https://github.com/apache/spark/pull/8610#issuecomment-160196347
  
Jenkins, retest this please


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11206] (Followup) Fix SQLListenerMemory...

2015-11-27 Thread SparkQA

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/9991#issuecomment-160196391
  
**[Test build #46828 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/46828/consoleFull)**
 for PR 9991 at commit 
[`b694e27`](https://github.com/apache/spark/commit/b694e27979a2acad3e9653e082c08e8b3f7e41b7).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12021][Streaming][Tests]Fix the potenti...

2015-11-27 Thread zsxwing

Github user zsxwing commented on the pull request:

https://github.com/apache/spark/pull/10011#issuecomment-160196396
  
@tedyu Happy Thanksgiving, too :)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12012] [SQL] Show more comprehensive Ph...

2015-11-27 Thread SparkQA

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/10004#issuecomment-160171989
  
**[Test build #46825 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/46825/consoleFull)**
 for PR 10004 at commit 
[`8cbf0ff`](https://github.com/apache/spark/commit/8cbf0ff166f57f0592149381bf3a8daf7ed311c4).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11206] (Followup) Fix SQLListenerMemory...

2015-11-27 Thread zsxwing

Github user zsxwing commented on the pull request:

https://github.com/apache/spark/pull/9991#issuecomment-160195567
  
> Test build #46807 has finished for PR 9991 at commit b694e27.
> 
> This patch fails from timeout after a configured wait of `250m`.
> This patch merges cleanly.
> This patch adds no public classes.

This one should be fixed in #10011


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12021][Streaming][Tests]Fix the potenti...

2015-11-27 Thread asfgit

Github user asfgit closed the pull request at:

https://github.com/apache/spark/pull/10011


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11206] (Followup) Fix SQLListenerMemory...

2015-11-27 Thread zsxwing

Github user zsxwing commented on the pull request:

https://github.com/apache/spark/pull/9991#issuecomment-160195573
  
retest this please


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11788][SQL] surround timestamp/date val...

2015-11-27 Thread huaxingao

Github user huaxingao commented on the pull request:

https://github.com/apache/spark/pull/9872#issuecomment-160182654
  
Fixed title and scala style problem.  Thanks!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12012] [SQL] Show more comprehensive Ph...

2015-11-27 Thread liancheng

Github user liancheng commented on the pull request:

https://github.com/apache/spark/pull/10004#issuecomment-160170582
  
@rxin Updated. Now it shows `ParquetRelation` and `OrcRelation` etc. 
instead of `PhysicalRDD`. Hive metastore table name is also attached. For the 
long input paths scenario, yes it makes the box super wide. So in the last 
commit, I truncate the text if it's longer than 100 characters. This is 
somewhat brutal, but I'm not sure about the best practice here. Maybe doing the 
truncation in JS side is a better idea?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11102] [SQL] Uninformative exception wh...

2015-11-27 Thread SparkQA

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/9490#issuecomment-160172679
  
**[Test build #46821 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/46821/consoleFull)**
 for PR 9490 at commit 
[`9e67a3d`](https://github.com/apache/spark/commit/9e67a3d3f36d4c8be9c016f8cbb4db885e341610).
 * This patch passes all tests.
 * This patch **does not merge cleanly**.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11102] [SQL] Uninformative exception wh...

2015-11-27 Thread AmplabJenkins

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9490#issuecomment-160172813
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/46821/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12010][SQL] Spark JDBC requires support...

2015-11-27 Thread CK50

Github user CK50 commented on the pull request:

https://github.com/apache/spark/pull/10003#issuecomment-160178110

@rxin
I wish I could run dev/lint-scala, but even after hours of struggling I 
cannot get build/sbt running. I have downloaded sbt-launcher.jar 
manually, but now I am stuck with sbt trying to download extra files 
from a variety of invalid urls such as
https://jcenter.bintray.com/org/scala-sbt/sbt/0.13.7/sbt-0.13.7.pom
https://jcenter.bintray.com/org/scala-sbt/sbt/0.13.7/sbt-0.13.7.jar
Any ideas appreciated. The Spark 1.6 build page did not help. - Any 
other page I should follow?

The old generated statement is

INSERT INTO mytable VALUES (?, ?, ..., ?)

whereas the new statement is

INSERT INTO mytable (col1, col2,  ..., colN) VALUES (?, ?, ..., ?)

So the old syntax relies on the column positions, whereas the new one 
relies on the column names.

Column names are taken from the DataFrame. So DataFrame column names 
must match target table column names.

At least for Oracle adding the column names is fine. The benefit of this 
change is that you can write out DataFrames with having less columns 
than the target table, which I think is not possible today. The downside 
is that DataFrame column names *must* match. For best backwards 
compatibility I only wanted to provide column names when it is really 
needed, like in case of Cassandra.

WDYT?

On 26.11.2015 20:11, Reynold Xin wrote:
>
> @CK50  this needs some style cleanup - you 
> can run dev/lint-scala to check styles locally.
>
> Can you please also attach the generated query before / after this 
> change? IIUC, maybe it's ok to have this for all dialects.
>
> â
> Reply to this email directly or view it on GitHub 
> .
>

---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12028] [SQL] get_json_object returns an...

2015-11-27 Thread SparkQA

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/10018#issuecomment-160192964
  
**[Test build #46827 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/46827/consoleFull)**
 for PR 10018 at commit 
[`54edc84`](https://github.com/apache/spark/commit/54edc84f21918c5cb69a0abfc51f680190f27a1f).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12012] [SQL] Show more comprehensive Ph...

2015-11-27 Thread AmplabJenkins

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/10004#issuecomment-160175612
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/46825/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12012] [SQL] Show more comprehensive Ph...

2015-11-27 Thread AmplabJenkins

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/10004#issuecomment-160175611
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12023][BUILD] Fix warnings while packag...

2015-11-27 Thread SparkQA

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/10014#issuecomment-160187760
  
**[Test build #2124 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/2124/consoleFull)**
 for PR 10014 at commit 
[`6e9a37b`](https://github.com/apache/spark/commit/6e9a37b86c10f223590decfc7b3903e988daad28).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12025] [SparkR] Rename some window rank...

2015-11-27 Thread asfgit

Github user asfgit closed the pull request at:

https://github.com/apache/spark/pull/10016


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-1537] [YARN] [WiP] Add history provider...

2015-11-27 Thread SparkQA

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/5423#issuecomment-160185630
  
**[Test build #46826 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/46826/consoleFull)**
 for PR 5423 at commit 
[`033a0a0`](https://github.com/apache/spark/commit/033a0a08c3f37612e2426cf502797bc106767ab7).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12024][SQL] More efficient multi-column...

2015-11-27 Thread SparkQA

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/10015#issuecomment-160117691
  
**[Test build #46814 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/46814/consoleFull)**
 for PR 10015 at commit 
[`3fe9cf3`](https://github.com/apache/spark/commit/3fe9cf3373a90c0a0d0bd6e75d44e13905e15458).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-6521][Core] Bypass unnecessary network ...

2015-11-27 Thread AmplabJenkins

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9478#issuecomment-160117763
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/46806/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-6521][Core] Bypass unnecessary network ...

2015-11-27 Thread SparkQA

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/9478#issuecomment-160117706
  
**[Test build #46806 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/46806/consoleFull)**
 for PR 9478 at commit 
[`303abcd`](https://github.com/apache/spark/commit/303abcd0f37d137c6a8ce4a0147466bb8feb9d9e).
 * This patch **fails from timeout after a configured wait of \`250m\`**.
 * This patch **does not merge cleanly**.
 * This patch adds the following public classes _(experimental)_:\n  * `  
case class GetLocalDirsPath(blockManagerId: BlockManagerId) extends 
ToBlockManagerMaster`\n


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11206] (Followup) Fix SQLListenerMemory...

2015-11-27 Thread AmplabJenkins

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9991#issuecomment-160118552
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12025] [SparkR] Rename some window rank...

2015-11-27 Thread SparkQA

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/10016#issuecomment-160118563
  
**[Test build #46815 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/46815/consoleFull)**
 for PR 10016 at commit 
[`c56a7bc`](https://github.com/apache/spark/commit/c56a7bcce530b604edd2f486af6514587dd59e8e).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12025] [SparkR] Rename some window rank...

2015-11-27 Thread AmplabJenkins

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/10016#issuecomment-160121465
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [Spark-10625] [SQL] Spark SQL JDBC read/write ...

2015-11-27 Thread srowen

Github user srowen commented on a diff in the pull request:

https://github.com/apache/spark/pull/8785#discussion_r46041751
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/DataFrameWriter.scala ---
@@ -280,7 +275,8 @@ final class DataFrameWriter private[sql](df: DataFrame) 
{
   conn.close()
 }
 
-JdbcUtils.saveTable(df, url, table, props)
+val props2 = JDBCRelation.getEffectiveProperties(connectionProperties, 
this.extraOptions)
--- End diff --

Why does this need to be created twice? isn't `props` the same?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [MINOR][BUILD] Changed the comment to reflect ...

2015-11-27 Thread srowen

Github user srowen commented on the pull request:

https://github.com/apache/spark/pull/10012#issuecomment-160129061
  
OK


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12023][BUILD] Fix warnings while packag...

2015-11-27 Thread srowen

Github user srowen commented on the pull request:

https://github.com/apache/spark/pull/10014#issuecomment-160129296
  
Seems OK, but the warning indicates it was interpreted as an absolute path 
before, but it wasn't. Hm.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [Doc] Explicitly state that top maintains the ...

2015-11-27 Thread mineo

Github user mineo commented on the pull request:

https://github.com/apache/spark/pull/10013#issuecomment-160133244
  
I've used the same wording that the docs of `takeOrdered` use.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11859][Mesos] SparkContext accepts inva...

2015-11-27 Thread SparkQA

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/9886#issuecomment-160132881
  
**[Test build #2123 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/2123/consoleFull)**
 for PR 9886 at commit 
[`1eeb933`](https://github.com/apache/spark/commit/1eeb933cff4451744a65e16c4a4694e2212cbcbc).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11102] [SQL] Uninformative exception wh...

2015-11-27 Thread SparkQA

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/9490#issuecomment-160117323
  
**[Test build #46810 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/46810/consoleFull)**
 for PR 9490 at commit 
[`d8e7f5c`](https://github.com/apache/spark/commit/d8e7f5c1183edbb039050ccf71f20b0c8fc20556).
 * This patch passes all tests.
 * This patch **does not merge cleanly**.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11102] [SQL] Uninformative exception wh...

2015-11-27 Thread AmplabJenkins

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9490#issuecomment-160117437
  
Build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11102] [SQL] Uninformative exception wh...

2015-11-27 Thread AmplabJenkins

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9490#issuecomment-160117440
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/46810/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12025] [SparkR] Rename some window rank...

2015-11-27 Thread AmplabJenkins

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/10016#issuecomment-160121466
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/46815/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12025] [SparkR] Rename some window rank...

2015-11-27 Thread SparkQA

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/10016#issuecomment-160121371
  
**[Test build #46815 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/46815/consoleFull)**
 for PR 10016 at commit 
[`c56a7bc`](https://github.com/apache/spark/commit/c56a7bcce530b604edd2f486af6514587dd59e8e).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12010][SQL] Spark JDBC requires support...

2015-11-27 Thread srowen

Github user srowen commented on the pull request:

https://github.com/apache/spark/pull/10003#issuecomment-160128013
  
This is an orthogonal issue, but it occurs to me that we have a bunch of 
dialects, and a bunch of behavior variations, but typically a variation isn't 
specific to a dialect, necessarily. How do we let a new variation apply to N 
dialects at once? this may become a problem soon and need some refactoring.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-8641][SQL] Native Spark Window function...

2015-11-27 Thread AmplabJenkins

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9819#issuecomment-160132031
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-8641][SQL] Native Spark Window function...

2015-11-27 Thread AmplabJenkins

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9819#issuecomment-160132033
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/46817/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-8641][SQL] Native Spark Window function...

2015-11-27 Thread SparkQA

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/9819#issuecomment-160132136
  
**[Test build #46820 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/46820/consoleFull)**
 for PR 9819 at commit 
[`31c6fb3`](https://github.com/apache/spark/commit/31c6fb3e7708b59e80f0ab24dd77c3241487e0e7).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-8641][SQL] Native Spark Window function...

2015-11-27 Thread AmplabJenkins

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9819#issuecomment-160133535
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/46819/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-8641][SQL] Native Spark Window function...

2015-11-27 Thread AmplabJenkins

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9819#issuecomment-160133533
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12024][SQL] More efficient multi-column...

2015-11-27 Thread hvanhovell

GitHub user hvanhovell opened a pull request:

https://github.com/apache/spark/pull/10015

[SPARK-12024][SQL] More efficient multi-column counting.

In https://github.com/apache/spark/pull/9409 we enabled multi-column 
counting. The approach taken in that PR introduces a bit of overhead by first 
creating a row only to check if all of the columns are non-null. 

This PR fixes that technical debt. Count now takes multiple columns as its 
input. In order to make this work I have also added support for multiple 
columns in the single distinct code path.

cc @yhuai

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/hvanhovell/spark SPARK-12024

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/10015.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #10015


commit 3fe9cf3373a90c0a0d0bd6e75d44e13905e15458
Author: Herman van Hovell 
Date:   2015-11-27T11:03:28Z

More efficient multi-column counting.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11206] (Followup) Fix SQLListenerMemory...

2015-11-27 Thread AmplabJenkins

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9991#issuecomment-160118554
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/46807/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11206] (Followup) Fix SQLListenerMemory...

2015-11-27 Thread SparkQA

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/9991#issuecomment-160118520
  
**[Test build #46807 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/46807/consoleFull)**
 for PR 9991 at commit 
[`b694e27`](https://github.com/apache/spark/commit/b694e27979a2acad3e9653e082c08e8b3f7e41b7).
 * This patch **fails from timeout after a configured wait of \`250m\`**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12010][SQL] Spark JDBC requires support...

2015-11-27 Thread CK50

Github user CK50 commented on a diff in the pull request:

https://github.com/apache/spark/pull/10003#discussion_r46038687
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/jdbc/ProgressCassandraDialect.scala
 ---
@@ -0,0 +1,51 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.jdbc
+
+import java.sql.Types
+
+import org.apache.spark.sql.types._
+
+
+private case object ProgressCassandraDialect extends JdbcDialect {
--- End diff --

Yes, there is a bunch of other Cassandra JDBC drivers out there. Progress 
one of them. But seen that the Cassandra SQL syntax mandates the list of column 
names in the INSERT statement, it is likely that other drivers will require the 
same syntax. So I will remove "Progress" from the class name.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11753][SQL] Make allowNonNumericNumbers...

2015-11-27 Thread SparkQA

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/9759#issuecomment-160120215
  
**[Test build #46812 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/46812/consoleFull)**
 for PR 9759 at commit 
[`6d90b24`](https://github.com/apache/spark/commit/6d90b2464501fa69c4475b3f756f4062c2863d2f).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11753][SQL] Make allowNonNumericNumbers...

2015-11-27 Thread AmplabJenkins

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9759#issuecomment-160120337
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11753][SQL] Make allowNonNumericNumbers...

2015-11-27 Thread AmplabJenkins

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9759#issuecomment-160120340
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/46812/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [Doc] Explicitly state that top maintains the ...

2015-11-27 Thread srowen

Github user srowen commented on the pull request:

https://github.com/apache/spark/pull/10013#issuecomment-160128842
  
I think that's fine, though might say it a little differently. It returns 
the top k elements according to some ordering, and the k elements are in order 
ascending/descending.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-8641][SQL] Native Spark Window function...

2015-11-27 Thread AmplabJenkins

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9819#issuecomment-160131454
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-8641][SQL] Native Spark Window function...

2015-11-27 Thread AmplabJenkins

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9819#issuecomment-160132666
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11826][MLlib] Reimplement add() and sub...

2015-11-27 Thread srowen

Github user srowen commented on the pull request:

https://github.com/apache/spark/pull/9916#issuecomment-160133764
  
I'd defer to @mengxr on this one for final approval but yes that's 
definitely what I had in mind. It's nice that you refactored add and subtract 
together


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12024][SQL] More efficient multi-column...

2015-11-27 Thread AmplabJenkins

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/10015#issuecomment-160136500
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12024][SQL] More efficient multi-column...

2015-11-27 Thread AmplabJenkins

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/10015#issuecomment-160136502
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/46814/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12024][SQL] More efficient multi-column...

2015-11-27 Thread SparkQA

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/10015#issuecomment-160136407
  
**[Test build #46814 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/46814/consoleFull)**
 for PR 10015 at commit 
[`3fe9cf3`](https://github.com/apache/spark/commit/3fe9cf3373a90c0a0d0bd6e75d44e13905e15458).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds the following public classes _(experimental)_:\n  * 
`case class Count(children: Seq[Expression]) extends DeclarativeAggregate `\n


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12002][Streaming][PySpark] Fix python d...

2015-11-27 Thread AmplabJenkins

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/10017#issuecomment-160150075
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/46824/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-8641][SQL] Native Spark Window function...

2015-11-27 Thread SparkQA

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/9819#issuecomment-160150804
  
**[Test build #46820 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/46820/consoleFull)**
 for PR 9819 at commit 
[`31c6fb3`](https://github.com/apache/spark/commit/31c6fb3e7708b59e80f0ab24dd77c3241487e0e7).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12002][Streaming][PySpark] Fix python d...

2015-11-27 Thread SparkQA

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/10017#issuecomment-160149986
  
**[Test build #46824 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/46824/consoleFull)**
 for PR 10017 at commit 
[`bfae0ab`](https://github.com/apache/spark/commit/bfae0ab21c75eefcee2bb276ec152cfdfb2a4228).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12002][Streaming][PySpark] Fix python d...

2015-11-27 Thread AmplabJenkins

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/10017#issuecomment-160150072
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-8641][SQL] Native Spark Window function...

2015-11-27 Thread AmplabJenkins

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9819#issuecomment-160150936
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/46820/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-8641][SQL] Native Spark Window function...

2015-11-27 Thread AmplabJenkins

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9819#issuecomment-160150935
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12025] [SparkR] Rename some window rank...

2015-11-27 Thread sun-rui

Github user sun-rui commented on the pull request:

https://github.com/apache/spark/pull/10016#issuecomment-160151261
  
@yanboliang, thanks for your info. I didn't look at the latest code:)

LGTM


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-7857][MLLIB] Prevent IDFModel from retu...

2015-11-27 Thread karlhigley

Github user karlhigley commented on a diff in the pull request:

https://github.com/apache/spark/pull/9843#discussion_r46051533
  
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/feature/IDF.scala ---
@@ -211,14 +213,17 @@ private object IDFModel {
 val n = v.size
 v match {
   case SparseVector(size, indices, values) =>
+val newElements = new ArrayBuffer[(Int, Double)]
 val nnz = indices.size
-val newValues = new Array[Double](nnz)
 var k = 0
 while (k < nnz) {
-  newValues(k) = values(k) * idf(indices(k))
+  val newValue = values(k) * idf(indices(k))
--- End diff --

This change doesn't affect the `idf` vector -- it's still always dense. The 
existing `transform` code was producing `SparseVector`s containing explicit 
zeros. This change removes the explicit zeros, but only when the input to 
`transform` (not the `idf`) is a `SparseVector`.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-7857][MLLIB] Prevent IDFModel from retu...

2015-11-27 Thread karlhigley

Github user karlhigley commented on a diff in the pull request:

https://github.com/apache/spark/pull/9843#discussion_r46051762
  
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/feature/IDF.scala ---
@@ -211,14 +213,17 @@ private object IDFModel {
 val n = v.size
 v match {
   case SparseVector(size, indices, values) =>
+val newElements = new ArrayBuffer[(Int, Double)]
 val nnz = indices.size
-val newValues = new Array[Double](nnz)
 var k = 0
 while (k < nnz) {
-  newValues(k) = values(k) * idf(indices(k))
+  val newValue = values(k) * idf(indices(k))
--- End diff --

If I were to follow the suggestion of updating line 58 to `new 
IDFModel(idf.compressed)`, then it would be possible for the `idf` vector to be 
sparse, but I'm not doing that here. That's part of the performance 
optimization I'm suggesting could happen in a separate PR.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12025] [SparkR] Rename some window rank...

2015-11-27 Thread yanboliang

GitHub user yanboliang opened a pull request:

https://github.com/apache/spark/pull/10016

[SPARK-12025] [SparkR] Rename some window rank function names for SparkR

Rename some window rank function names for SparkR.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/yanboliang/spark SPARK-12025

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/10016.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #10016


commit c56a7bcce530b604edd2f486af6514587dd59e8e
Author: Yanbo Liang 
Date:   2015-11-27T11:11:40Z

Rename window rank function names for SparkR




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-6521][Core] Bypass unnecessary network ...

2015-11-27 Thread AmplabJenkins

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9478#issuecomment-160117761
  
Build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11401] [MLLIB] PMML export for Logistic...

2015-11-27 Thread selvinsource

Github user selvinsource commented on the pull request:

https://github.com/apache/spark/pull/9397#issuecomment-160121192
  
@dbtsai thanks for the suggestion, rebasing from master seems to have fixed 
it.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [Spark-10625] [SQL] Spark SQL JDBC read/write ...

2015-11-27 Thread srowen

Github user srowen commented on the pull request:

https://github.com/apache/spark/pull/8785#issuecomment-160127622
  
I guess I'm missing why a deep copy solves a problem of an unserializable 
property value. It still exists in the copy right?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12025] [SparkR] Rename some window rank...

2015-11-27 Thread sun-rui

Github user sun-rui commented on the pull request:

https://github.com/apache/spark/pull/10016#issuecomment-160129665
  
code change looks good. New names have no conflict with R base package.

But I don't see they are deprecated now? Could you point to me the 
discussion or JIRA that they will be removed in Spark 2.0?
If they are deprecated, I am fine with the name change.
If they are not deprecated, could we add new names as aliases, while 
keeping the old names at the same time?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-8641][SQL] Native Spark Window function...

2015-11-27 Thread AmplabJenkins

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9819#issuecomment-160131455
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/46816/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-8641][SQL] Native Spark Window function...

2015-11-27 Thread AmplabJenkins

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9819#issuecomment-160132668
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/46818/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [Doc] Explicitly state that top maintains the ...

2015-11-27 Thread srowen

Github user srowen commented on the pull request:

https://github.com/apache/spark/pull/10013#issuecomment-160133967
  
I see your point, let's go with the same wording then, yes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12002][Streaming][PySpark] Fix python d...

2015-11-27 Thread AmplabJenkins

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/10017#issuecomment-160143837
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12002][Streaming][PySpark] Fix python d...

2015-11-27 Thread AmplabJenkins

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/10017#issuecomment-160143838
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/46823/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12002][Streaming][PySpark] Fix python d...

2015-11-27 Thread SparkQA

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/10017#issuecomment-160143835
  
**[Test build #46823 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/46823/consoleFull)**
 for PR 10017 at commit 
[`3ff4b09`](https://github.com/apache/spark/commit/3ff4b099e3deb37aec784ee67cf4a83e10c644b7).
 * This patch **fails Python style tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12002][Streaming][PySpark] Fix python d...

2015-11-27 Thread SparkQA

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/10017#issuecomment-160145767
  
**[Test build #46824 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/46824/consoleFull)**
 for PR 10017 at commit 
[`bfae0ab`](https://github.com/apache/spark/commit/bfae0ab21c75eefcee2bb276ec152cfdfb2a4228).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12025] [SparkR] Rename some window rank...

2015-11-27 Thread yanboliang

Github user yanboliang commented on the pull request:

https://github.com/apache/spark/pull/10016#issuecomment-160139585
  
@sun-rui Thanks for your comments. You can get the issue from the [source 
code](https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/functions.scala#L563)
 and this PR(#9930).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11102] [SQL] Uninformative exception wh...

2015-11-27 Thread SparkQA

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/9490#issuecomment-160140929
  
**[Test build #46821 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/46821/consoleFull)**
 for PR 9490 at commit 
[`9e67a3d`](https://github.com/apache/spark/commit/9e67a3d3f36d4c8be9c016f8cbb4db885e341610).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12002][Streaming][PySpark] Fix python d...

2015-11-27 Thread jerryshao

GitHub user jerryshao opened a pull request:

https://github.com/apache/spark/pull/10017

[SPARK-12002][Streaming][PySpark] Fix python direct stream checkpoint 
recovery issue

Currently when serializing `TransformFunction`, `rdd_wrap_func` object is 
not serialized into bytearray. So after deserializing, it will use default 
value, which will make Kafka direct stream `transform` wrongly recovered and 
throw exception.

Please help to review, thanks a lot.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/jerryshao/apache-spark SPARK-12002

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/10017.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #10017


commit 3ff4b099e3deb37aec784ee67cf4a83e10c644b7
Author: jerryshao 
Date:   2015-11-27T13:30:13Z

Fix python direct stream checkpoint recovery issue




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5682][Core] Add encrypted shuffle in sp...

2015-11-27 Thread SparkQA

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/8880#issuecomment-160141802
  
**[Test build #46822 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/46822/consoleFull)**
 for PR 8880 at commit 
[`6d001ca`](https://github.com/apache/spark/commit/6d001ca57e5d92faba88b2439437d304b9a51988).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12002][Streaming][PySpark] Fix python d...

2015-11-27 Thread SparkQA

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/10017#issuecomment-160143731
  
**[Test build #46823 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/46823/consoleFull)**
 for PR 10017 at commit 
[`3ff4b09`](https://github.com/apache/spark/commit/3ff4b099e3deb37aec784ee67cf4a83e10c644b7).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12029][SPARKR] Improve column functions...

2015-11-27 Thread felixcheung

GitHub user felixcheung opened a pull request:

https://github.com/apache/spark/pull/10019

[SPARK-12029][SPARKR] Improve column functions signature, param check, 
tests, fix doc and add examples

@shivaram @sun-rui 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/felixcheung/spark rfunctionsdoc

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/10019.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #10019


commit 7e61064af7ddb62b2056d6192933a4516d153bdb
Author: felixcheung 
Date:   2015-11-20T02:00:27Z

functions doc fix

commit 8d18e1f7f812742b07a99b976074b2d27297061a
Author: felixcheung 
Date:   2015-11-27T02:59:11Z

add param check, more examples, test

commit e2b1ba0d3947227c98e1b29c37aba75478aff54b
Author: felixcheung 
Date:   2015-11-27T06:02:13Z

fix typos




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: test case demonstrating SPARK-10625:

2015-11-27 Thread AmplabJenkins

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/10022#issuecomment-160209666
  
Can one of the admins verify this patch?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12021][Streaming][Tests]Fix the potenti...

2015-11-27 Thread CodingCat

Github user CodingCat commented on the pull request:

https://github.com/apache/spark/pull/10011#issuecomment-160217707
  
Thanks, @zsxwing, at least it works for my PR


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12020] [TESTS] [test-hadoop2.0] PR buil...

2015-11-27 Thread yhuai

Github user yhuai commented on the pull request:

https://github.com/apache/spark/pull/10010#issuecomment-160226172
  
oh, actually I probably to merge this to branch 1.6 (otherwise, pr for 1.6 
branch will not be able to test with hadoop 2.0 profile?).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12028] [SQL] get_json_object returns an...

2015-11-27 Thread SparkQA

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/10018#issuecomment-160242055
  
**[Test build #46832 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/46832/consoleFull)**
 for PR 10018 at commit 
[`54edc84`](https://github.com/apache/spark/commit/54edc84f21918c5cb69a0abfc51f680190f27a1f).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12028] [SQL] get_json_object returns an...

2015-11-27 Thread AmplabJenkins

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/10018#issuecomment-160242074
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/46832/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11206] (Followup) Fix SQLListenerMemory...

2015-11-27 Thread SparkQA

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/9991#issuecomment-160242817
  
**[Test build #46834 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/46834/consoleFull)**
 for PR 9991 at commit 
[`b694e27`](https://github.com/apache/spark/commit/b694e27979a2acad3e9653e082c08e8b3f7e41b7).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12012] [SQL] Show more comprehensive Ph...

2015-11-27 Thread SparkQA

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/10004#issuecomment-160250890
  
**[Test build #46835 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/46835/consoleFull)**
 for PR 10004 at commit 
[`6925eb3`](https://github.com/apache/spark/commit/6925eb310bdfcdf465b315dabf18f03a6578a633).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-9319][SPARKR] Add support for setting c...

2015-11-27 Thread felixcheung

Github user felixcheung commented on the pull request:

https://github.com/apache/spark/pull/9654#issuecomment-160202381
  
Could we please review this? I think it would be great to get matching 
coltype<- in Spark 1.6


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

1 2 >

1 - 100 of 183 matches

Mail list logo