date:20140904

[GitHub] spark pull request: [SPARK-3393] [SQL] add configuration template ...

2014-09-04 Thread liancheng

Github user liancheng commented on a diff in the pull request:

https://github.com/apache/spark/pull/2263#discussion_r17096828
  
--- Diff: conf/hive-log4j.properties.template ---
@@ -0,0 +1,84 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# License); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an AS IS BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Define some default values that can be overridden by system properties
+hive.root.logger=WARN,console
+hive.log.dir=/tmp/${user.name}
+hive.log.file=hive.log
+
+# Define the root logger to the system property hadoop.root.logger.
+log4j.rootLogger=${hive.root.logger}, EventCounter
+
+# Logging Threshold
+log4j.threshhold=WARN
+
+#
+# Daily Rolling File Appender
+#
+# Use the PidDailyerRollingFileAppend class instead if you want to use 
separate log files
+# for different CLI session.
+#
+# 
log4j.appender.DRFA=org.apache.hadoop.hive.ql.log.PidDailyRollingFileAppender
+
+log4j.appender.DRFA=org.apache.log4j.DailyRollingFileAppender
+
+log4j.appender.DRFA.File=${hive.log.dir}/${hive.log.file}
+
+# Rollver at midnight
+log4j.appender.DRFA.DatePattern=.-MM-dd
+
+# 30-day backup
--- End diff --

`# Uncomment the this line to enable 30-day log backup`


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-3393] [SQL] add configuration template ...

2014-09-04 Thread liancheng

Github user liancheng commented on a diff in the pull request:

https://github.com/apache/spark/pull/2263#discussion_r17096824
  
--- Diff: conf/hive-log4j.properties.template ---
@@ -0,0 +1,84 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# License); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an AS IS BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Define some default values that can be overridden by system properties
+hive.root.logger=WARN,console
+hive.log.dir=/tmp/${user.name}
+hive.log.file=hive.log
+
+# Define the root logger to the system property hadoop.root.logger.
+log4j.rootLogger=${hive.root.logger}, EventCounter
+
+# Logging Threshold
+log4j.threshhold=WARN
+
+#
+# Daily Rolling File Appender
+#
+# Use the PidDailyerRollingFileAppend class instead if you want to use 
separate log files
+# for different CLI session.
+#
+# 
log4j.appender.DRFA=org.apache.hadoop.hive.ql.log.PidDailyRollingFileAppender
--- End diff --

I guess this line should be removed?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-3393] [SQL] add configuration template ...

2014-09-04 Thread liancheng

Github user liancheng commented on a diff in the pull request:

https://github.com/apache/spark/pull/2263#discussion_r17096833
  
--- Diff: conf/hive-log4j.properties.template ---
@@ -0,0 +1,84 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# License); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an AS IS BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Define some default values that can be overridden by system properties
+hive.root.logger=WARN,console
+hive.log.dir=/tmp/${user.name}
+hive.log.file=hive.log
+
+# Define the root logger to the system property hadoop.root.logger.
+log4j.rootLogger=${hive.root.logger}, EventCounter
+
+# Logging Threshold
+log4j.threshhold=WARN
+
+#
+# Daily Rolling File Appender
+#
+# Use the PidDailyerRollingFileAppend class instead if you want to use 
separate log files
+# for different CLI session.
+#
+# 
log4j.appender.DRFA=org.apache.hadoop.hive.ql.log.PidDailyRollingFileAppender
+
+log4j.appender.DRFA=org.apache.log4j.DailyRollingFileAppender
+
+log4j.appender.DRFA.File=${hive.log.dir}/${hive.log.file}
+
+# Rollver at midnight
+log4j.appender.DRFA.DatePattern=.-MM-dd
+
+# 30-day backup
+#log4j.appender.DRFA.MaxBackupIndex=30
+log4j.appender.DRFA.layout=org.apache.log4j.PatternLayout
+
+# Pattern format: Date LogLevel LoggerName LogMessage
+#log4j.appender.DRFA.layout.ConversionPattern=%d{ISO8601} %p %c: %m%n
+# Debugging Pattern format
+log4j.appender.DRFA.layout.ConversionPattern=%d{ISO8601} %-5p %c{2} 
(%F:%M(%L)) - %m%n
+
+
+#
+# console
--- End diff --

Remove this line


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-3393] [SQL] add configuration template ...

2014-09-04 Thread liancheng

Github user liancheng commented on the pull request:

https://github.com/apache/spark/pull/2263#issuecomment-54410818
  
I know that `hive-log4j.properties.template` is copied from Hive, but this 
file really makes me itchy...

Besides that, I think these are generally good to have, thanks!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-3349] [SQL] Output partitioning of limi...

2014-09-04 Thread chenghao-intel

Github user chenghao-intel commented on a diff in the pull request:

https://github.com/apache/spark/pull/2262#discussion_r17096863
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/execution/basicOperators.scala ---
@@ -97,6 +97,7 @@ case class Limit(limit: Int, child: SparkPlan)
   // partition local limit - exchange into one partition - partition 
local limit again
 
   override def output = child.output
+  override def outputPartitioning = SinglePartition
--- End diff --

`SinglePartition` for `LIMIT` may cause performance issue for large number 
of records(in multiple partitions), do we really need to change this?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-3349] [SQL] Output partitioning of limi...

2014-09-04 Thread marmbrus

Github user marmbrus commented on a diff in the pull request:

https://github.com/apache/spark/pull/2262#discussion_r17096919
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/execution/basicOperators.scala ---
@@ -97,6 +97,7 @@ case class Limit(limit: Int, child: SparkPlan)
   // partition local limit - exchange into one partition - partition 
local limit again
 
   override def output = child.output
+  override def outputPartitioning = SinglePartition
--- End diff --

This is not changing the implementation, just correcting a bug that
prevents exchange operators from being inserted when we need them.
On Sep 3, 2014 11:00 PM, Cheng Hao notificati...@github.com wrote:

 In
 
sql/core/src/main/scala/org/apache/spark/sql/execution/basicOperators.scala:

  @@ -97,6 +97,7 @@ case class Limit(limit: Int, child: SparkPlan)
 // partition local limit - exchange into one partition - partition 
local limit again
 
 override def output = child.output
  +  override def outputPartitioning = SinglePartition

 SinglePartition for LIMIT may cause performance issue for large number of
 records(in multiple partitions), do we really need to change this?

 â
 Reply to this email directly or view it on GitHub
 https://github.com/apache/spark/pull/2262/files#r17096863.



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SQL][WIP] Refined Thrift server test suite

2014-09-04 Thread SparkQA

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/2214#issuecomment-54411030
  
**[Tests timed 
out](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19735/consoleFull)**
 after a configured wait of `120m`.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-3393] [SQL] add configuration template ...

2014-09-04 Thread liancheng

Github user liancheng commented on a diff in the pull request:

https://github.com/apache/spark/pull/2263#discussion_r17097013
  
--- Diff: conf/hive-log4j.properties.template ---
@@ -0,0 +1,84 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# License); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an AS IS BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Define some default values that can be overridden by system properties
+hive.root.logger=WARN,console
+hive.log.dir=/tmp/${user.name}
+hive.log.file=hive.log
+
+# Define the root logger to the system property hadoop.root.logger.
+log4j.rootLogger=${hive.root.logger}, EventCounter
+
+# Logging Threshold
+log4j.threshhold=WARN
+
+#
+# Daily Rolling File Appender
+#
+# Use the PidDailyerRollingFileAppend class instead if you want to use 
separate log files
+# for different CLI session.
+#
+# 
log4j.appender.DRFA=org.apache.hadoop.hive.ql.log.PidDailyRollingFileAppender
+
+log4j.appender.DRFA=org.apache.log4j.DailyRollingFileAppender
+
+log4j.appender.DRFA.File=${hive.log.dir}/${hive.log.file}
+
+# Rollver at midnight
+log4j.appender.DRFA.DatePattern=.-MM-dd
+
+# 30-day backup
+#log4j.appender.DRFA.MaxBackupIndex=30
+log4j.appender.DRFA.layout=org.apache.log4j.PatternLayout
+
+# Pattern format: Date LogLevel LoggerName LogMessage
+#log4j.appender.DRFA.layout.ConversionPattern=%d{ISO8601} %p %c: %m%n
+# Debugging Pattern format
+log4j.appender.DRFA.layout.ConversionPattern=%d{ISO8601} %-5p %c{2} 
(%F:%M(%L)) - %m%n
+
+
+#
+# console
+# Add console to rootlogger above if you want to use this 
+#
+
+log4j.appender.console=org.apache.log4j.ConsoleAppender
+log4j.appender.console.target=System.err
+log4j.appender.console.layout=org.apache.log4j.PatternLayout
+log4j.appender.console.layout.ConversionPattern=%d{yy/MM/dd HH:mm:ss} %p 
%c{2}: %m%n
+log4j.appender.console.encoding=UTF-8
+
+#custom logging levels
+#log4j.logger.xxx=DEBUG
+
+#
+# Event Counter Appender
+# Sends counts of logging messages at different severity levels to Hadoop 
Metrics.
+#
+log4j.appender.EventCounter=org.apache.hadoop.metrics.jvm.EventCounter
+
+
+log4j.category.DataNucleus=ERROR,DRFA
+log4j.category.Datastore=ERROR,DRFA
+log4j.category.Datastore.Schema=ERROR,DRFA
+log4j.category.JPOX.Datastore=ERROR,DRFA
+log4j.category.JPOX.Plugin=ERROR,DRFA
+log4j.category.JPOX.MetaData=ERROR,DRFA
+log4j.category.JPOX.Query=ERROR,DRFA
+log4j.category.JPOX.General=ERROR,DRFA
+log4j.category.JPOX.Enhancer=ERROR,DRFA
+log4j.category.scheduler.DAGScheduler=ERROR,DRFA
+
--- End diff --

Remove this line


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-3349] [SQL] Output partitioning of limi...

2014-09-04 Thread chenghao-intel

Github user chenghao-intel commented on a diff in the pull request:

https://github.com/apache/spark/pull/2262#discussion_r17097010
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/execution/basicOperators.scala ---
@@ -164,6 +165,7 @@ case class Limit(limit: Int, child: SparkPlan)
 case class TakeOrdered(limit: Int, sortOrder: Seq[SortOrder], child: 
SparkPlan) extends UnaryNode {
 
   override def output = child.output
+  override def outputPartitioning = SinglePartition
--- End diff --

This probably not what we want, since the `limit` provided, I think the 
original implementation is better for performance reason.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-3325] Add a parameter to the method pri...

2014-09-04 Thread watermen

Github user watermen commented on the pull request:

https://github.com/apache/spark/pull/2216#issuecomment-54411613
  
@tdas When i use the function of updateStateByKey, and the number of key 
are more than 10. I want to watch all values of keys to make sure the code is 
right.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-3363][SQL] Type Coercion should support...

2014-09-04 Thread adrian-wang

Github user adrian-wang commented on the pull request:

https://github.com/apache/spark/pull/2246#issuecomment-54412078
  
I have changed the code, now `compatibleType` and `findTightestCommonType` 
looks different.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-3363][SQL] Type Coercion should support...

2014-09-04 Thread SparkQA

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/2246#issuecomment-54412046
  
  [QA tests have 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19754/consoleFull)
 for   PR 2246 at commit 
[`ef6f986`](https://github.com/apache/spark/commit/ef6f986093b90d19bc98772a752b4809686c4347).
 * This patch merges cleanly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-3349] [SQL] Output partitioning of limi...

2014-09-04 Thread chenghao-intel

Github user chenghao-intel commented on a diff in the pull request:

https://github.com/apache/spark/pull/2262#discussion_r17097138
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/execution/basicOperators.scala ---
@@ -97,6 +97,7 @@ case class Limit(limit: Int, child: SparkPlan)
   // partition local limit - exchange into one partition - partition 
local limit again
 
   override def output = child.output
+  override def outputPartitioning = SinglePartition
--- End diff --

OK, understood, thanks for explanation.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-3377] [Metrics] codahale base Metrics d...

2014-09-04 Thread sarutak

Github user sarutak commented on the pull request:

https://github.com/apache/spark/pull/2250#issuecomment-54412947
  
retest this please.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SQL] Update SQL Programming Guide

2014-09-04 Thread davies

Github user davies commented on a diff in the pull request:

https://github.com/apache/spark/pull/2258#discussion_r17097268
  
--- Diff: docs/sql-programming-guide.md ---
@@ -263,15 +300,195 @@ for teenName in teenNames.collect():
 
 /div
 
-**Note that Spark SQL currently uses a very basic SQL parser.**
-Users that want a more complete dialect of SQL should look at the HiveQL 
support provided by
-`HiveContext`.
+### Programmatically Specifying the Schema
+
+div class=codetabs
+
+div data-lang=scala  markdown=1
+
+In cases that case classes cannot be defined ahead of time (for example,
+the structure of records is encoded in a string or a text dataset will be 
parsed
+and fields will be projected differently for different users),
+a `SchemaRDD` can be created programmatically with three steps.
+
+1. Create an RDD of `Row`s from the original RDD;
+2. Create the schema represented by a `StructType` matching the structure 
of
+`Row`s in the RDD created in the step 1.
+3. Apply the schema to the RDD of `Row`s via `applySchema` method provided
+by `SQLContext`.
+
+For example:
+{% highlight scala %}
+// sc is an existing SparkContext.
+val sqlContext = new org.apache.spark.sql.SQLContext(sc)
+
+// Create an RDD
+val people = sc.textFile(examples/src/main/resources/people.txt)
+
+// The schema is encoded in a string
+val schemaString = name age
+
+// Import Spark SQL data types and Row.
+import org.apache.spark.sql._
+
+// Generate the schema based on the string of schema
+val schema =
+  StructType(
+schemaString.split( ).map(fieldName = StructField(fieldName, 
StringType, true)))
+
+// Convert records of the RDD (people) to Rows.
+val rowRDD = people.map(_.split(,)).map(p = Row(p(0), p(1).trim))
+
+// Apply the schema to the RDD.
+val peopleSchemaRDD = sqlContext.applySchema(rowRDD, schema)
+
+// Register the SchemaRDD as a table.
+peopleSchemaRDD.registerTempTable(people)
+
+// SQL statements can be run by using the sql methods provided by 
sqlContext.
+val results = sqlContext.sql(SELECT name FROM people)
+
+// The results of SQL queries are SchemaRDDs and support all the normal 
RDD operations.
+// The columns of a row in the result can be accessed by ordinal.
+results.map(t = Name:  + t(0)).collect().foreach(println)
+{% endhighlight %}
+
+
+/div
+
+div data-lang=java  markdown=1
+
+In cases that JavaBean classes cannot be defined ahead of time (for 
example,
+the structure of records is encoded in a string or a text dataset will be 
parsed and
+fields will be projected differently for different users),
+a `SchemaRDD` can be created programmatically with three steps.
+
+1. Create an RDD of `Row`s from the original RDD;
+2. Create the schema represented by a `StructType` matching the structure 
of
+`Row`s in the RDD created in the step 1.
+3. Apply the schema to the RDD of `Row`s via `applySchema` method provided
+by `JavaSQLContext`.
+
+For example:
+{% highlight java %}
+// Import factory methods provided by DataType.
+import org.apache.spark.sql.api.java.DataType
+// Import StructType and StructField
+import org.apache.spark.sql.api.java.StructType
+import org.apache.spark.sql.api.java.StructField
+// Import Row.
+import org.apache.spark.sql.api.java.Row
+
+// sc is an existing JavaSparkContext.
+JavaSQLContext sqlContext = new 
org.apache.spark.sql.api.java.JavaSQLContext(sc);
+
+// Load a text file and convert each line to a JavaBean.
+JavaRDDString people = 
sc.textFile(examples/src/main/resources/people.txt);
+
+// The schema is encoded in a string
+String schemaString = name age;
+
+// Generate the schema based on the string of schema
+ListStructField fields = new ArrayListStructField();
+for (String fieldName: schemaString.split( )) {
+  fields.add(DataType.createStructField(fieldName, DataType.StringType, 
true));
+}
+StructType schema = DataType.createStructType(fields);
+
+// Convert records of the RDD (people) to Rows.
+JavaRDDRow rowRDD = people.map(
+  new FunctionString, Row() {
+public Row call(String record) throws Exception {
+  String[] fields = record.split(,);
+  return Row.create(fields[0], fields[1].trim());
+}
+  });
+
+// Apply the schema to the RDD.
+JavaSchemaRDD peopleSchemaRDD = sqlContext.applySchema(rowRDD, schema);
+
+// Register the SchemaRDD as a table.
+peopleSchemaRDD.registerTempTable(people);
+
+// SQL can be run over RDDs that have been registered as tables.
+JavaSchemaRDD results = sqlContext.sql(SELECT name FROM people);
+
+// The results of SQL queries are

[GitHub] spark pull request: [Build] Removed -Phive-thriftserver since this...

2014-09-04 Thread liancheng

GitHub user liancheng opened a pull request:

https://github.com/apache/spark/pull/2269

[Build] Removed -Phive-thriftserver since this profile has been removed



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/liancheng/spark clean-run-tests-profile

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/2269.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2269


commit 08617bd7eeadd639ff10ee1a2b6f5d37ee2123f2
Author: Cheng Lian lian.cs@gmail.com
Date:   2014-09-04T06:22:47Z

Removed -Phive-thriftserver since this profile has been removed




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-3181][MLLIB]: Add Robust Regression Alg...

2014-09-04 Thread SparkQA

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/2096#issuecomment-54413134
  
  [QA tests have 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19749/consoleFull)
 for   PR 2096 at commit 
[`ab0f539`](https://github.com/apache/spark/commit/ab0f53946b6b43fa8934f112c0c5b11e2224dfd8).
 * This patch **fails** unit tests.
 * This patch merges cleanly.
 * This patch adds the following public classes _(experimental)_:
  * `  case class Params(`
  * `class HuberRobustGradient extends Gradient `
  * `class HuberRobustRegressionModel (`



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-3377] [Metrics] codahale base Metrics d...

2014-09-04 Thread SparkQA

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/2250#issuecomment-54413324
  
  [QA tests have 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19755/consoleFull)
 for   PR 2250 at commit 
[`15f88a3`](https://github.com/apache/spark/commit/15f88a390fc78181e88e030f47903c0576b664bd).
 * This patch merges cleanly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-2713] Executors of same application in ...

2014-09-04 Thread li-zhihui

Github user li-zhihui commented on a diff in the pull request:

https://github.com/apache/spark/pull/1616#discussion_r17097313
  
--- Diff: core/src/main/scala/org/apache/spark/util/Utils.scala ---
@@ -317,13 +317,58 @@ private[spark] object Utils extends Logging {
   }
 
   /**
+   * Copy cached file to targetDir, if not exists, download it from url 
firstly.
+   * If useCache == false, download file to targetDir directly.
--- End diff --

I want to keep the util method flexible primarily, because we make the old 
codefetchFile/code private. But I am not sure if the tag is necessary too.
@JoshRosen How do you think about it?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SQL] Update SQL Programming Guide

2014-09-04 Thread davies

Github user davies commented on a diff in the pull request:

https://github.com/apache/spark/pull/2258#discussion_r17097430
  
--- Diff: python/pyspark/sql.py ---
@@ -287,7 +287,7 @@ class StructType(DataType):
 
 Spark SQL StructType
 
-The data type representing rows.
+The data type representing tuple or list values.
--- End diff --

I think that the object with StructType in SchemaRDD will convert into 
Row() in Python, also ArrayType will be used to represent list values, so why 
change this?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-3325] Add a parameter to the method pri...

2014-09-04 Thread tdas

Github user tdas commented on the pull request:

https://github.com/apache/spark/pull/2216#issuecomment-54414169
  
Ahh okay. As @ScrapCodes said, this can be done pretty easily with 
foreachRDD, but I can see how this makes it a tad bit convenient. I am okay to 
add this. However, please add this to the Java API as well.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: SPARK-3337 Paranoid quoting in shell to allow ...

2014-09-04 Thread ScrapCodes

Github user ScrapCodes commented on a diff in the pull request:

https://github.com/apache/spark/pull/2229#discussion_r17097578
  
--- Diff: bin/compute-classpath.sh ---
@@ -63,7 +63,7 @@ else
   assembly_folder=$ASSEMBLY_DIR
 fi
 
-num_jars=$(ls $assembly_folder | grep spark-assembly.*hadoop.*\.jar | 
wc -l)
+num_jars=$(ls $assembly_folder | grep spark-assembly.*hadoop.*\.jar | 
wc -l)
--- End diff --

You are right !


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-3399] Test for PySpark should ignore HA...

2014-09-04 Thread sarutak

GitHub user sarutak opened a pull request:

https://github.com/apache/spark/pull/2270

[SPARK-3399] Test for PySpark should ignore HADOOP_CONF_DIR and 
YARN_CONF_DIR



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/sarutak/spark SPARK-3399

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/2270.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2270


commit 7613be65f38d133fefd41f0a982cd97b25c12c0a
Author: Kousuke Saruta saru...@oss.nttdata.co.jp
Date:   2014-09-04T06:37:08Z

Modified pyspark script to ignore environment variables YARN_CONF_DIR and 
HADOOP_CONF_DIR while testing




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: SPARK-3337 Paranoid quoting in shell to allow ...

2014-09-04 Thread ScrapCodes

Github user ScrapCodes commented on a diff in the pull request:

https://github.com/apache/spark/pull/2229#discussion_r17097618
  
--- Diff: sbt/sbt-launch-lib.bash ---
@@ -190,5 +190,5 @@ runAlternateBoot() {
   local bootpropsfile=$1
   shift
   addJava -Dsbt.boot.properties=$bootpropsfile
-  run $@
+  run $@
--- End diff --

This should be harmless, AFAIK. But it does not help either. So I will get 
rid of this.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [HOTFIX] [SPARK-3400] Revert 9b225ac fix Grap...

2014-09-04 Thread ankurdave

GitHub user ankurdave opened a pull request:

https://github.com/apache/spark/pull/2271

[HOTFIX] [SPARK-3400] Revert 9b225ac fix GraphX EdgeRDD zipPartitions

9b225ac3072de522b40b46aba6df1f1c231f13ef has been causing GraphX tests
to fail nondeterministically, which is blocking development for others.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ankurdave/spark SPARK-3400

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/2271.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2271


commit 10c2a97107b2043e67dfaee4e8a85a69f3b2fb78
Author: Ankur Dave ankurd...@gmail.com
Date:   2014-09-04T06:30:53Z

[HOTFIX] [SPARK-3400] Revert 9b225ac fix GraphX EdgeRDD zipPartitions

9b225ac3072de522b40b46aba6df1f1c231f13ef has been causing GraphX tests
to fail nondeterministically, which is blocking development for others.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [HOTFIX] [SPARK-3400] Revert 9b225ac fix Grap...

2014-09-04 Thread ankurdave

Github user ankurdave commented on the pull request:

https://github.com/apache/spark/pull/2271#issuecomment-54416619
  
@rxin @liancheng I'm going to merge this immediately.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: SPARK-3337 Paranoid quoting in shell to allow ...

2014-09-04 Thread SparkQA

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/2229#issuecomment-54416704
  
  [QA tests have 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19760/consoleFull)
 for   PR 2229 at commit 
[`8002918`](https://github.com/apache/spark/commit/8002918dde107e7f79e87573cfb2d33e8c4956e1).
 * This patch merges cleanly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [HOTFIX] [SPARK-3400] Revert 9b225ac fix Grap...

2014-09-04 Thread rxin

Github user rxin commented on the pull request:

https://github.com/apache/spark/pull/2271#issuecomment-54416983
  
Thanks.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [HOTFIX] [SPARK-3400] Revert 9b225ac fix Grap...

2014-09-04 Thread asfgit

Github user asfgit closed the pull request at:

https://github.com/apache/spark/pull/2271


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-3378] [DOCS] Replace the word SparkSQL...

2014-09-04 Thread SparkQA

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/2251#issuecomment-54417319
  
  [QA tests have 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19746/consoleFull)
 for   PR 2251 at commit 
[`0bfe234`](https://github.com/apache/spark/commit/0bfe23410d58de3547d1285400716a117cc30d43).
 * This patch **passes** unit tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-3377] [Metrics] codahale base Metrics d...

2014-09-04 Thread SparkQA

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/2250#issuecomment-54420383
  
  [QA tests have 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19755/consoleFull)
 for   PR 2250 at commit 
[`15f88a3`](https://github.com/apache/spark/commit/15f88a390fc78181e88e030f47903c0576b664bd).
 * This patch **fails** unit tests.
 * This patch merges cleanly.
 * This patch adds the following public classes _(experimental)_:
  * `class BlockManagerMaster(`
  * `class AttributeMap[A](baseMap: Map[ExprId, (Attribute, A)])`



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [HOTFIX] [SPARK-3400] Revert 9b225ac fix Grap...

2014-09-04 Thread liancheng

Github user liancheng commented on the pull request:

https://github.com/apache/spark/pull/2271#issuecomment-54420735
  
Thanks!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-3377] [Metrics] codahale base Metrics d...

2014-09-04 Thread sarutak

Github user sarutak commented on the pull request:

https://github.com/apache/spark/pull/2250#issuecomment-54421851
  
retest this please.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-3377] [Metrics] codahale base Metrics d...

2014-09-04 Thread SparkQA

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/2250#issuecomment-54422284
  
  [QA tests have 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19764/consoleFull)
 for   PR 2250 at commit 
[`15f88a3`](https://github.com/apache/spark/commit/15f88a390fc78181e88e030f47903c0576b664bd).
 * This patch merges cleanly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-3353] parent stage should have lower st...

2014-09-04 Thread rxin

GitHub user rxin opened a pull request:

https://github.com/apache/spark/pull/2273

[SPARK-3353] parent stage should have lower stage id.

Previously parent stages had higher stage id, but parent stages are 
executed first. This pull request changes the behavior so parent stages would 
have lower stage id.

For example, command: 
```scala
sc.parallelize(1 to 10).map(x=(x,x)).reduceByKey(_+_).count
```
breaks down into 2 stages.

The old web UI:
![screen shot 2014-09-04 at 12 42 44 
am](https://cloud.githubusercontent.com/assets/323388/4146177/60fb4f42-3407-11e4-819f-853eb0e22b25.png)


Web UI with this patch:
![screen shot 2014-09-04 at 12 44 55 
am](https://cloud.githubusercontent.com/assets/323388/4146178/62e08e62-3407-11e4-867b-a36b10534464.png)



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/rxin/spark lower-stage-id

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/2273.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2273


commit 54ccea3b08b629dd21e6083d1fcd88c75e531067
Author: Reynold Xin r...@apache.org
Date:   2014-09-04T07:41:32Z

[SPARK-3353] parent stage should have lower stage id.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-3181][MLLIB]: Add Robust Regression Alg...

2014-09-04 Thread ankurdave

Github user ankurdave commented on the pull request:

https://github.com/apache/spark/pull/2096#issuecomment-54423889
  
Jenkins, retest this please.

This was a problematic commit in GraphX which I just 
[reverted](https://github.com/apache/spark/commit/00362dac976cd05b06638deb11d990d612429e0b),
 so a retest should fix the problem.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-3393] [SQL] add configuration template ...

2014-09-04 Thread chenghao-intel

Github user chenghao-intel commented on the pull request:

https://github.com/apache/spark/pull/2263#issuecomment-54424705
  
Thanks for reviewing this, you're right, the log4j configuration template 
mainly for Hive usage (particularly for MapReduce), I've updated the code and 
keep it simple for demo purpose.

I also checked the other template files under `conf`, seems the `licence 
header` is not necessary for `.properties` files. Let's do it the same way?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-3363][SQL] Type Coercion should support...

2014-09-04 Thread SparkQA

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/2246#issuecomment-54424875
  
  [QA tests have 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19754/consoleFull)
 for   PR 2246 at commit 
[`ef6f986`](https://github.com/apache/spark/commit/ef6f986093b90d19bc98772a752b4809686c4347).
 * This patch **fails** unit tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-3401][PySpark] Wrong usage of tee comma...

2014-09-04 Thread rxin

Github user rxin commented on the pull request:

https://github.com/apache/spark/pull/2272#issuecomment-54425075
  
Jenkins, test this please.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-3363][SQL] Type Coercion should support...

2014-09-04 Thread adrian-wang

Github user adrian-wang commented on the pull request:

https://github.com/apache/spark/pull/2246#issuecomment-54425114
  
I hate graphx :(


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-3401][PySpark] Wrong usage of tee comma...

2014-09-04 Thread rxin

Github user rxin commented on the pull request:

https://github.com/apache/spark/pull/2272#issuecomment-54425102
  
LGTM pending Jenkins.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-3363][SQL] Type Coercion should support...

2014-09-04 Thread rxin

Github user rxin commented on the pull request:

https://github.com/apache/spark/pull/2246#issuecomment-54425261
  
Don't hate it :)




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-3363][SQL] Type Coercion should support...

2014-09-04 Thread rxin

Github user rxin commented on the pull request:

https://github.com/apache/spark/pull/2246#issuecomment-54425279
  
Jenkins, retest this please.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-3353] parent stage should have lower st...

2014-09-04 Thread rxin

Github user rxin commented on the pull request:

https://github.com/apache/spark/pull/2273#issuecomment-54425345
  
Jenkins, ok to test.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: SPARK-2895: Add mapPartitionsWithContext relat...

2014-09-04 Thread ChengXiangLi

Github user ChengXiangLi commented on the pull request:

https://github.com/apache/spark/pull/2194#issuecomment-54425347
  
we hit the binary incompatibilities error here, i already annotated new 
added methods as DeveloperApi, do i miss something here?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-3401][PySpark] Wrong usage of tee comma...

2014-09-04 Thread SparkQA

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/2272#issuecomment-54425969
  
  [QA tests have 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19767/consoleFull)
 for   PR 2272 at commit 
[`2b35a59`](https://github.com/apache/spark/commit/2b35a592b38195e5d54f68f74efaf386faad7d1c).
 * This patch merges cleanly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-3353] parent stage should have lower st...

2014-09-04 Thread rxin

Github user rxin commented on the pull request:

https://github.com/apache/spark/pull/2273#issuecomment-54426303
  
Jenkins, add to whitelist.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-3353] parent stage should have lower st...

2014-09-04 Thread SparkQA

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/2273#issuecomment-54426801
  
  [QA tests have 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19770/consoleFull)
 for   PR 2273 at commit 
[`54ccea3`](https://github.com/apache/spark/commit/54ccea3b08b629dd21e6083d1fcd88c75e531067).
 * This patch merges cleanly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-3280] Made sort-based shuffle the defau...

2014-09-04 Thread rxin

Github user rxin commented on the pull request:

https://github.com/apache/spark/pull/2178#issuecomment-54429684
  
Ok I updated the PR to set spark.shuffle.spill.compress and 
spark.shuffle.compress consistent for now.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-3280] Made sort-based shuffle the defau...

2014-09-04 Thread SparkQA

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/2178#issuecomment-54430749
  
  [QA tests have 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19771/consoleFull)
 for   PR 2178 at commit 
[`713d341`](https://github.com/apache/spark/commit/713d341748c7a65f3a354b290b1edf1a5fe5fd0f).
 * This patch merges cleanly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: SPARK-3337 Paranoid quoting in shell to allow ...

2014-09-04 Thread SparkQA

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/2229#issuecomment-54431089
  
  [QA tests have 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19760/consoleFull)
 for   PR 2229 at commit 
[`8002918`](https://github.com/apache/spark/commit/8002918dde107e7f79e87573cfb2d33e8c4956e1).
 * This patch **fails** unit tests.
 * This patch merges cleanly.
 * This patch adds the following public classes _(experimental)_:
  * `case class SparkListenerBlockManagerAdded(time: Long, blockManagerId: 
BlockManagerId, maxMem: Long)`
  * `case class SparkListenerBlockManagerRemoved(time: Long, 
blockManagerId: BlockManagerId)`
  * `case class SparkListenerApplicationStart(appName: String, appId: 
Option[String], time: Long,`
  * `class BlockManagerMaster(`
  * `class AttributeMap[A](baseMap: Map[ExprId, (Attribute, A)])`



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [Minor]Remove extra semicolon in FlumeStreamSu...

2014-09-04 Thread SparkQA

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/2265#issuecomment-54431508
  
  [QA tests have 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19772/consoleFull)
 for   PR 2265 at commit 
[`6c99e6e`](https://github.com/apache/spark/commit/6c99e6e133ca2e0872aa5841ab6fb30009aa58bd).
 * This patch merges cleanly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-3325] Add a parameter to the method pri...

2014-09-04 Thread srowen

Github user srowen commented on the pull request:

https://github.com/apache/spark/pull/2216#issuecomment-54431716
  
@watermen @tdas PS do you think this can be added to the Java API too? it 
needs to be a new additional method not a change to the existing one. Worth the 
extra method, you guys think?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-3399][PySpark] Test for PySpark should ...

2014-09-04 Thread sarutak

Github user sarutak commented on the pull request:

https://github.com/apache/spark/pull/2270#issuecomment-54431833
  
Jenkins, test this please.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: SPARK-2978. Transformation with MR shuffle sem...

2014-09-04 Thread sryza

GitHub user sryza opened a pull request:

https://github.com/apache/spark/pull/2274

SPARK-2978. Transformation with MR shuffle semantics

I didn't add this to the transformations list in the docs because it's kind 
of obscure, but would be happy to do so if others think it would be helpful.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/sryza/spark sandy-spark-2978

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/2274.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2274


commit a75f277943ad9c8d0aa8f2a3c98967bce1a7deac
Author: Sandy Ryza sa...@cloudera.com
Date:   2014-08-21T16:02:00Z

SPARK-2978. Transformation with MR shuffle semantics




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-3377] [Metrics] codahale base Metrics d...

2014-09-04 Thread SparkQA

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/2250#issuecomment-54432073
  
  [QA tests have 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19764/consoleFull)
 for   PR 2250 at commit 
[`15f88a3`](https://github.com/apache/spark/commit/15f88a390fc78181e88e030f47903c0576b664bd).
 * This patch **passes** unit tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: SPARK-2978. Transformation with MR shuffle sem...

2014-09-04 Thread SparkQA

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/2274#issuecomment-54432312
  
  [QA tests have 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19773/consoleFull)
 for   PR 2274 at commit 
[`a75f277`](https://github.com/apache/spark/commit/a75f277943ad9c8d0aa8f2a3c98967bce1a7deac).
 * This patch merges cleanly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: SPARK-2978. Transformation with MR shuffle sem...

2014-09-04 Thread SparkQA

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/2274#issuecomment-54432465
  
  [QA tests have 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19773/consoleFull)
 for   PR 2274 at commit 
[`a75f277`](https://github.com/apache/spark/commit/a75f277943ad9c8d0aa8f2a3c98967bce1a7deac).
 * This patch **fails** unit tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-2710] [SQL] Build SchemaRDD from a Jdbc...

2014-09-04 Thread chutium

Github user chutium commented on a diff in the pull request:

https://github.com/apache/spark/pull/1612#discussion_r17101449
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/jdbc/JdbcResultSetRDD.scala ---
@@ -0,0 +1,72 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the License); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an AS IS BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.jdbc
+
+import java.sql.ResultSet
+
+import org.apache.spark.rdd.RDD
+import org.apache.spark.rdd.JdbcRDD
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.plans.logical._
+import org.apache.spark.sql.catalyst.types._
+import org.apache.spark.sql.execution.{ExistingRdd, SparkLogicalPlan}
+import org.apache.spark.Logging
+
+private[sql] object JdbcResultSetRDD extends Logging {
+
+  private[sql] def inferSchema(
+  jdbcResultSet: JdbcRDD[ResultSet]): StructType = {
+StructType(createSchema(jdbcResultSet.getSchema))
+  }
+
+  private def createSchema(metaSchema: Seq[(String, Int, Boolean)]): 
Seq[StructField] = {
+metaSchema.map(e = StructField(e._1, 
JdbcTypes.toPrimitiveDataType(e._2), e._3))
+  }
+
+  private[sql] def jdbcResultSetToRow(
+  jdbcResultSet: JdbcRDD[ResultSet],
+  schema: StructType) : RDD[Row] = {
+val row = new GenericMutableRow(schema.fields.length)
+jdbcResultSet.map(asRow(_, row, schema.fields))
+  }
+
+  private def asRow(rs: ResultSet, row: GenericMutableRow, schema: 
Seq[StructField]): Row = {
+schema.zipWithIndex.foreach {
--- End diff --

changed to while, learned to write from your ```fillObject``` in hive 
```TableReader``` :D 
https://github.com/apache/spark/blob/2b8d89e30ebfe2272229a1eddd7542d7437c9924/sql/hive/src/main/scala/org/apache/spark/sql/hive/TableReader.scala#L279-L288

thanks.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-3397] Bump pom.xml version number of ma...

2014-09-04 Thread srowen

Github user srowen commented on the pull request:

https://github.com/apache/spark/pull/2268#issuecomment-54432582
  
@witgo The release process does this, I believe. I don't think you need to 
open a PR for this especially before 1.1.0 is released


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: Augmented updateStateByKey API

2014-09-04 Thread srowen

Github user srowen commented on a diff in the pull request:

https://github.com/apache/spark/pull/2267#discussion_r17101501
  
--- Diff: spark2 ---
@@ -0,0 +1 @@
+Some changes
--- End diff --

What is this file? accidentally added?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-2710] [SQL] Build SchemaRDD from a Jdbc...

2014-09-04 Thread SparkQA

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/1612#issuecomment-54433054
  
  [QA tests have 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19775/consoleFull)
 for   PR 1612 at commit 
[`566d154`](https://github.com/apache/spark/commit/566d154b23d0d7163b48889cfcd523e706c9a480).
 * This patch merges cleanly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: Augmented updateStateByKey API

2014-09-04 Thread srowen

Github user srowen commented on a diff in the pull request:

https://github.com/apache/spark/pull/2267#discussion_r17101578
  
--- Diff: 
streaming/src/main/scala/org/apache/spark/streaming/dstream/PairDStreamFunctions.scala
 ---
@@ -396,6 +396,26 @@ class PairDStreamFunctions[K, V](self: DStream[(K,V)])
 
   /**
* Return a new state DStream where the state for each key is updated 
by applying
+   * the given function on the previous state of the key and the new 
values of the key.
+   * org.apache.spark.Partitioner is used to control the partitioning of 
each RDD.
+   * @param updateFunc State update function. If `this` function returns 
None, then
+   *   corresponding state key-value pair will be 
eliminated.
+   * @param partitioner Partitioner for controlling the partitioning of 
each RDD in the new
+   *DStream.
+   * @tparam S State type
+   */
+  def updateStateByKey[S: ClassTag](
+  updateFunc: (Time, K, Seq[V], Option[S]) = Option[S],
+  partitioner: Partitioner
+): DStream[(K, S)] = {
+val newUpdateFunc = (time: Time, iterator: Iterator[(K, Seq[V], 
Option[S])]) = {
--- End diff --

The existing method could call the new method rather than duplicate the 
logic. If the user supplies a function with Seq[V], Option[S] as args, that can 
be made into a function that also accepts and does nothing with Time.

What's the use case for this though?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: SPARK-2461. Add a toString method to Generaliz...

2014-09-04 Thread sryza

Github user sryza commented on a diff in the pull request:

https://github.com/apache/spark/pull/1388#discussion_r17101588
  
--- Diff: python/pyspark/mllib/regression.py ---
@@ -66,6 +66,9 @@ def weights(self):
 def intercept(self):
 return self._intercept
 
+def __str__(self):
--- End diff --

It seems like __str__ is widely used across pyspark.  Updated patch 
replaces all __str__'s with __repr__'s.  Any reason I'm missing not to do this?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-2710] [SQL] Build SchemaRDD from a Jdbc...

2014-09-04 Thread SparkQA

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/1612#issuecomment-54433211
  
  [QA tests have 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19775/consoleFull)
 for   PR 1612 at commit 
[`566d154`](https://github.com/apache/spark/commit/566d154b23d0d7163b48889cfcd523e706c9a480).
 * This patch **fails** unit tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: SPARK-2461. Add a toString method to Generaliz...

2014-09-04 Thread SparkQA

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/1388#issuecomment-54433745
  
  [QA tests have 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19776/consoleFull)
 for   PR 1388 at commit 
[`e6f29cd`](https://github.com/apache/spark/commit/e6f29cd0b70cdd2bc81ddf7ac01db456b16cc55f).
 * This patch merges cleanly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: SPARK-2978. Transformation with MR shuffle sem...

2014-09-04 Thread SparkQA

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/2274#issuecomment-54434254
  
  [QA tests have 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19777/consoleFull)
 for   PR 2274 at commit 
[`a1ef807`](https://github.com/apache/spark/commit/a1ef807c05b6a088a36d3305781d773457b61335).
 * This patch merges cleanly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-3124] Fix the jar version conflict in u...

2014-09-04 Thread chenghao-intel

Github user chenghao-intel commented on the pull request:

https://github.com/apache/spark/pull/2035#issuecomment-54434388
  
test this please


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-2663] [SQL] Support the Grouping Set

2014-09-04 Thread chenghao-intel

Github user chenghao-intel commented on the pull request:

https://github.com/apache/spark/pull/1567#issuecomment-54434357
  
test this please


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-3392] [SQL] Show value spark.sql.shuffl...

2014-09-04 Thread chenghao-intel

Github user chenghao-intel commented on the pull request:

https://github.com/apache/spark/pull/2261#issuecomment-54434450
  
Jenkins doesn't work?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-3392] [SQL] Show value spark.sql.shuffl...

2014-09-04 Thread chenghao-intel

Github user chenghao-intel commented on the pull request:

https://github.com/apache/spark/pull/2261#issuecomment-54434457
  
test this please


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-3353] parent stage should have lower st...

2014-09-04 Thread SparkQA

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/2273#issuecomment-54434600
  
  [QA tests have 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19770/consoleFull)
 for   PR 2273 at commit 
[`54ccea3`](https://github.com/apache/spark/commit/54ccea3b08b629dd21e6083d1fcd88c75e531067).
 * This patch **fails** unit tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-3392] [SQL] Show value spark.sql.shuffl...

2014-09-04 Thread SparkQA

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/2261#issuecomment-54434920
  
  [QA tests have 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19780/consoleFull)
 for   PR 2261 at commit 
[`653858a`](https://github.com/apache/spark/commit/653858ac2ec505f68bd4b4843d502dafad4a17b1).
 * This patch merges cleanly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-2663] [SQL] Support the Grouping Set

2014-09-04 Thread SparkQA

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/1567#issuecomment-54434939
  
  [QA tests have 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19781/consoleFull)
 for   PR 1567 at commit 
[`0325be5`](https://github.com/apache/spark/commit/0325be5e94501620eecbd16ba4b0d42dc7ac3a8e).
 * This patch merges cleanly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-2713] Executors of same application in ...

2014-09-04 Thread SparkQA

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/1616#issuecomment-54434954
  
  [QA tests have 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19778/consoleFull)
 for   PR 1616 at commit 
[`76a7b66`](https://github.com/apache/spark/commit/76a7b66eed73594ad425213b8a230b837f7c4da4).
 * This patch merges cleanly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-3124] Fix the jar version conflict in u...

2014-09-04 Thread SparkQA

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/2035#issuecomment-54434901
  
  [QA tests have 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19779/consoleFull)
 for   PR 2035 at commit 
[`f96ec21`](https://github.com/apache/spark/commit/f96ec211f582c72af8e8c5d4189ed661e6ad8286).
 * This patch merges cleanly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-3353] parent stage should have lower st...

2014-09-04 Thread SparkQA

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/2273#issuecomment-54435580
  
  [QA tests have 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19768/consoleFull)
 for   PR 2273 at commit 
[`54ccea3`](https://github.com/apache/spark/commit/54ccea3b08b629dd21e6083d1fcd88c75e531067).
 * This patch **fails** unit tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-3401][PySpark] Wrong usage of tee comma...

2014-09-04 Thread SparkQA

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/2272#issuecomment-54436390
  
  [QA tests have 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19767/consoleFull)
 for   PR 2272 at commit 
[`2b35a59`](https://github.com/apache/spark/commit/2b35a592b38195e5d54f68f74efaf386faad7d1c).
 * This patch **passes** unit tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-2710] [SQL] Build SchemaRDD from a Jdbc...

2014-09-04 Thread SparkQA

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/1612#issuecomment-54436556
  
  [QA tests have 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19782/consoleFull)
 for   PR 1612 at commit 
[`2013303`](https://github.com/apache/spark/commit/2013303c37eb8408ea31d285d2d3ac23ded7b3e9).
 * This patch merges cleanly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-3280] Made sort-based shuffle the defau...

2014-09-04 Thread SparkQA

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/2178#issuecomment-54438651
  
  [QA tests have 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19771/consoleFull)
 for   PR 2178 at commit 
[`713d341`](https://github.com/apache/spark/commit/713d341748c7a65f3a354b290b1edf1a5fe5fd0f).
 * This patch **fails** unit tests.
 * This patch merges cleanly.
 * This patch adds the following public classes _(experimental)_:
  * `class BlockManagerMaster(`
  * `class AttributeMap[A](baseMap: Map[ExprId, (Attribute, A)])`



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [Minor]Remove extra semicolon in FlumeStreamSu...

2014-09-04 Thread SparkQA

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/2265#issuecomment-54441308
  
  [QA tests have 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19772/consoleFull)
 for   PR 2265 at commit 
[`6c99e6e`](https://github.com/apache/spark/commit/6c99e6e133ca2e0872aa5841ab6fb30009aa58bd).
 * This patch **passes** unit tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-3397] Bump pom.xml version number of ma...

2014-09-04 Thread witgo

Github user witgo commented on the pull request:

https://github.com/apache/spark/pull/2268#issuecomment-54441310
  
@srowen I agree with you.  But 
[SparkContext.SPARK_VERSION](https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/SparkContext.scala#L1300)
 has been modified to `1.2.0-SNAPSHOT`


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-3399][PySpark] Test for PySpark should ...

2014-09-04 Thread SparkQA

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/2270#issuecomment-54441790
  
  [QA tests have 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19774/consoleFull)
 for   PR 2270 at commit 
[`7613be6`](https://github.com/apache/spark/commit/7613be65f38d133fefd41f0a982cd97b25c12c0a).
 * This patch **passes** unit tests.
 * This patch merges cleanly.
 * This patch adds the following public classes _(experimental)_:
  * `class BlockManagerMaster(`
  * `class AttributeMap[A](baseMap: Map[ExprId, (Attribute, A)])`



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: SPARK-2461. Add a toString method to Generaliz...

2014-09-04 Thread SparkQA

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/1388#issuecomment-54441755
  
  [QA tests have 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19776/consoleFull)
 for   PR 1388 at commit 
[`e6f29cd`](https://github.com/apache/spark/commit/e6f29cd0b70cdd2bc81ddf7ac01db456b16cc55f).
 * This patch **fails** unit tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: SPARK-2978. Transformation with MR shuffle sem...

2014-09-04 Thread SparkQA

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/2274#issuecomment-54443225
  
  [QA tests have 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19777/consoleFull)
 for   PR 2274 at commit 
[`a1ef807`](https://github.com/apache/spark/commit/a1ef807c05b6a088a36d3305781d773457b61335).
 * This patch **fails** unit tests.
 * This patch merges cleanly.
 * This patch adds the following public classes _(experimental)_:
  * `class BlockManagerMaster(`
  * `class AttributeMap[A](baseMap: Map[ExprId, (Attribute, A)])`



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-3124] Fix the jar version conflict in u...

2014-09-04 Thread witgo

Github user witgo commented on the pull request:

https://github.com/apache/spark/pull/2035#issuecomment-54443276
  
LGTM


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-3363][SQL] Type Coercion should support...

2014-09-04 Thread SparkQA

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/2246#issuecomment-54443352
  
  [QA tests have 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19769/consoleFull)
 for   PR 2246 at commit 
[`ef6f986`](https://github.com/apache/spark/commit/ef6f986093b90d19bc98772a752b4809686c4347).
 * This patch **passes** unit tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-2713] Executors of same application in ...

2014-09-04 Thread SparkQA

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/1616#issuecomment-5557
  
  [QA tests have 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19778/consoleFull)
 for   PR 1616 at commit 
[`76a7b66`](https://github.com/apache/spark/commit/76a7b66eed73594ad425213b8a230b837f7c4da4).
 * This patch **fails** unit tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-2663] [SQL] Support the Grouping Set

2014-09-04 Thread SparkQA

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/1567#issuecomment-54449841
  
  [QA tests have 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19781/consoleFull)
 for   PR 1567 at commit 
[`0325be5`](https://github.com/apache/spark/commit/0325be5e94501620eecbd16ba4b0d42dc7ac3a8e).
 * This patch **fails** unit tests.
 * This patch merges cleanly.
 * This patch adds the following public classes _(experimental)_:
  * `case class GroupingSet(bitmasks: Seq[Int], `
  * `case class Cube(groupByExprs: Seq[Expression],`
  * `case class Rollup(groupByExprs: Seq[Expression],`
  * `protected case class AttributeEquals(val a: Attribute) `
  * `case class VirtualColumn(name: String, dataType: DataType = 
StringType, nullable: Boolean = false)`
  * `case class GroupingSetExpansion(`
  * `case class GroupingSetExpansion(`



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-3392] [SQL] Show value spark.sql.shuffl...

2014-09-04 Thread SparkQA

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/2261#issuecomment-54451537
  
  [QA tests have 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19780/consoleFull)
 for   PR 2261 at commit 
[`653858a`](https://github.com/apache/spark/commit/653858ac2ec505f68bd4b4843d502dafad4a17b1).
 * This patch **passes** unit tests.
 * This patch merges cleanly.
 * This patch adds the following public classes _(experimental)_:
  * `case class SparkListenerBlockManagerAdded(time: Long, blockManagerId: 
BlockManagerId, maxMem: Long)`
  * `case class SparkListenerBlockManagerRemoved(time: Long, 
blockManagerId: BlockManagerId)`
  * `case class SparkListenerApplicationStart(appName: String, appId: 
Option[String], time: Long,`
  * `class BlockManagerMaster(`
  * `class AttributeMap[A](baseMap: Map[ExprId, (Attribute, A)])`
  * `  protected trait YarnAllocateResponse `



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-3124] Fix the jar version conflict in u...

2014-09-04 Thread SparkQA

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/2035#issuecomment-54452278
  
  [QA tests have 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19779/consoleFull)
 for   PR 2035 at commit 
[`f96ec21`](https://github.com/apache/spark/commit/f96ec211f582c72af8e8c5d4189ed661e6ad8286).
 * This patch **passes** unit tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-2710] [SQL] Build SchemaRDD from a Jdbc...

2014-09-04 Thread SparkQA

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/1612#issuecomment-54453005
  
  [QA tests have 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19782/consoleFull)
 for   PR 1612 at commit 
[`2013303`](https://github.com/apache/spark/commit/2013303c37eb8408ea31d285d2d3ac23ded7b3e9).
 * This patch **passes** unit tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: add ability to submit multiple jars for Driver

2014-09-04 Thread lianhuiwang

Github user lianhuiwang commented on the pull request:

https://github.com/apache/spark/pull/1113#issuecomment-54457835
  
@JoshRosen @andrewor14 I have update comment. has any question for it?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-3396][MLLIB] Use SquaredL2Updater in Lo...

2014-09-04 Thread BigCrunsh

Github user BigCrunsh commented on the pull request:

https://github.com/apache/spark/pull/2231#issuecomment-54459193
  
Jenkins, retest this please.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-3396][MLLIB] Use SquaredL2Updater in Lo...

2014-09-04 Thread BigCrunsh

Github user BigCrunsh commented on the pull request:

https://github.com/apache/spark/pull/2231#issuecomment-54459168
  
@mengxr: done ;)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-3396][MLLIB] Use SquaredL2Updater in Lo...

2014-09-04 Thread SparkQA

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/2231#issuecomment-54460133
  
  [QA tests have 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19783/consoleFull)
 for   PR 2231 at commit 
[`1ccc4d8`](https://github.com/apache/spark/commit/1ccc4d864463025defab2cee83ff2b7fd6bf9658).
 * This patch merges cleanly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: Fixing AWS instance type information based upo...

2014-09-04 Thread jerry86

Github user jerry86 commented on the pull request:

https://github.com/apache/spark/pull/1156#issuecomment-54463121
  
@pwendell I think this seems to be the best choice at this moment. Not sure 
if Amazon has made or will make any improvements to PVM that fits PySpark.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: SPARK-2895: Add mapPartitionsWithContext relat...

2014-09-04 Thread ScrapCodes

Github user ScrapCodes commented on the pull request:

https://github.com/apache/spark/pull/2194#issuecomment-54465038
  
I am looking at this. Mima check should have excluded those methods. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-3396][MLLIB] Use SquaredL2Updater in Lo...

2014-09-04 Thread SparkQA

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/2231#issuecomment-54473057
  
  [QA tests have 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19783/consoleFull)
 for   PR 2231 at commit 
[`1ccc4d8`](https://github.com/apache/spark/commit/1ccc4d864463025defab2cee83ff2b7fd6bf9658).
 * This patch **passes** unit tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-3367] Remove spark.shuffle.spill.compre...

2014-09-04 Thread tgravescs

Github user tgravescs commented on the pull request:

https://github.com/apache/spark/pull/2247#issuecomment-54474045
  
Perhaps we should have a separate discussion on the dev mailing list as 
what we consider public interfaces? That way everyone is in agreement or at 
least is following the same rules when things get committed. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

1 2 3 >

1 - 100 of 253 matches

Mail list logo