[GitHub] spark pull request: [SPARK-11827][SQL] Fix compilation against Jav...
Github user ted-yu closed the pull request at: https://github.com/apache/spark/pull/13233 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11827][SQL] Fix compilation against Jav...
Github user ted-yu commented on the pull request: https://github.com/apache/spark/pull/13233#issuecomment-220731442 Should reflection be used :-) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11827][SQL] Fix compilation against Jav...
GitHub user ted-yu opened a pull request: https://github.com/apache/spark/pull/13233 [SPARK-11827][SQL] Fix compilation against Java 7 ## What changes were proposed in this pull request? This PR fixes compilation error against Java 7: ``` sql/catalyst/src/main/scala/org/apache/spark/sql/types/Decimal.scala:137: value longValueExact is not a member of java.math.BigInteger [ERROR] this.longVal = bigintval.longValueExact() [ERROR]^ [ERROR] one error found ``` ## How was this patch tested? Existing tests. You can merge this pull request into a Git repository by running: $ git pull https://github.com/ted-yu/spark master Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/13233.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #13233 commit 422ccadb031a6b39465ce97b89963a8b37821615 Author: tedyu <yuzhih...@gmail.com> Date: 2016-05-20T21:13:00Z [SPARK-11827][SQL] Fix compilation against Java 7 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12181] Check Cached unaligned-access ca...
Github user ted-yu closed the pull request at: https://github.com/apache/spark/pull/10181 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13203] Add scalastyle rule banning use ...
Github user ted-yu closed the pull request at: https://github.com/apache/spark/pull/11082 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13203] Add scalastyle rule banning use ...
Github user ted-yu commented on the pull request: https://github.com/apache/spark/pull/11082#issuecomment-181036649 DirectKafkaStreamSuite.scala is not covered by #11067 Addition of scalastyle rule would not pass even after #11067 goes in. Should I open a seperate PR for DirectKafkaStreamSuite.scala ? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13203] Add scalastyle rule banning use ...
Github user ted-yu commented on the pull request: https://github.com/apache/spark/pull/11082#issuecomment-181051841 bq. If an existing PR means to catch all occurrences of the issue the rule would flag, then it needs to cover this case too In this case, looks like SPARK-13151 should cover all occurrences. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13203] Add scalastyle rule banning use ...
Github user ted-yu commented on the pull request: https://github.com/apache/spark/pull/11082#issuecomment-180616613 There is no error in DirectKafkaStreamSuite.scala Once [~holdenk]'s PR goes in, I will rebase. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13203] Add scalastyle rule banning use ...
Github user ted-yu commented on the pull request:
https://github.com/apache/spark/pull/11082#issuecomment-180498087
I am trying to add an Array to ConcurrentLinkedQueue (collectedData) in
Scala 2.11:
```
stream.foreachRDD { rdd => Collections.addAll(collectedData, rdd.collect())
}
```
I got the following compilation error:
```
[error] found : java.util.concurrent.ConcurrentLinkedQueue[(String,
String)]
[error] required: java.util.Collection[_ >: java.io.Serializable]
[error] Note: (String, String) <: Any, but Java-defined trait Collection is
invariant in type E.
[error] You may wish to investigate a wildcard type such as `_ <: Any`.
(SLS 3.2.10)
[error] stream.foreachRDD { rdd => Collections.addAll(allReceived,
rdd.collect()) }
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13203] Add scalastyle rule banning use ...
Github user ted-yu commented on the pull request: https://github.com/apache/spark/pull/11082#issuecomment-180546842 Jenkins, test this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13203] Add scalastyle rule banning use ...
Github user ted-yu commented on the pull request: https://github.com/apache/spark/pull/11082#issuecomment-180536530 Jenkins, test this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13203] Add scalastyle rule banning use ...
Github user ted-yu commented on the pull request: https://github.com/apache/spark/pull/11082#issuecomment-180318313 Jenkins, test this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13203] Add scalastyle rule banning use ...
Github user ted-yu commented on the pull request: https://github.com/apache/spark/pull/11082#issuecomment-180064615 ``` external/kafka/src/test/scala/org/apache/spark/streaming/kafka/DirectKafkaStreamSuite.scala: new ArrayBuffer[(String, String)] with mutable.SynchronizedBuffer[(String, Strin external/kafka/src/test/scala/org/apache/spark/streaming/kafka/DirectKafkaStreamSuite.scala: val collectedData = new mutable.ArrayBuffer[String]() with mutable.SynchronizedBuf external/kafka/src/test/scala/org/apache/spark/streaming/kafka/DirectKafkaStreamSuite.scala: val collectedData = new mutable.ArrayBuffer[String]() with mutable.SynchronizedBuf external/kafka/src/test/scala/org/apache/spark/streaming/kafka/DirectKafkaStreamSuite.scala: new ArrayBuffer[(String, String)] with mutable.SynchronizedBuffer[(String, Strin external/kafka/src/test/scala/org/apache/spark/streaming/kafka/DirectKafkaStreamSuite.scala: new mutable.ArrayBuffer[Array[String]]() with mutable.SynchronizedBuffer[Array[S external/kafka/src/test/scala/org/apache/spark/streaming/kafka/DirectKafkaStreamSuite.scala: val collectedData = new mutable.ArrayBuffer[String]() with mutable.SynchronizedBuffe streaming/src/main/scala/org/apache/spark/streaming/receiver/ReceiverSupervisorImpl.scala: with mutable.SynchronizedBuffer[BlockGenerator] streaming/src/test/scala/org/apache/spark/streaming/TestSuiteBase.scala:import scala.collection.mutable.SynchronizedBuffer streaming/src/test/scala/org/apache/spark/streaming/receiver/BlockGeneratorSuite.scala: val pushedData = new mutable.ArrayBuffer[Any] with mutable.SynchronizedBuffer[Any] streaming/src/test/scala/org/apache/spark/streaming/receiver/BlockGeneratorSuite.scala: val addedData = new mutable.ArrayBuffer[Any] with mutable.SynchronizedBuffer[Any] streaming/src/test/scala/org/apache/spark/streaming/receiver/BlockGeneratorSuite.scala: val addedMetadata = new mutable.ArrayBuffer[Any] with mutable.SynchronizedBuffer[Any] streaming/src/test/scala/org/apache/spark/streaming/util/RecurringTimerSuite.scala: val results = new mutable.ArrayBuffer[Long]() with mutable.SynchronizedBuffer[Long] streaming/src/test/scala/org/apache/spark/streaming/util/RecurringTimerSuite.scala: val results = new mutable.ArrayBuffer[Long]() with mutable.SynchronizedBuffer[Long] ``` Should the above be cleaned as addendum to #11059 or, using this PR ? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13203] Add scalastyle rule banning use ...
Github user ted-yu commented on the pull request: https://github.com/apache/spark/pull/11082#issuecomment-180095881 Jenkins, test this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13203] Add scalastyle rule banning use ...
Github user ted-yu commented on the pull request: https://github.com/apache/spark/pull/11082#issuecomment-180104115 Jenkins, test this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13203] Add scalastyle rule banning use ...
Github user ted-yu commented on the pull request: https://github.com/apache/spark/pull/11082#issuecomment-180084459 Jenkins, test this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13203] Add scalastyle rule banning use ...
Github user ted-yu commented on the pull request:
https://github.com/apache/spark/pull/11082#issuecomment-180063800
```
external/kafka/src/test/scala/org/apache/spark/streaming/kafka/KafkaStreamSuite.scala:
val result = new mutable.HashMap[String, Long]() with
mutable.SynchronizedMap[String, Lo
extras/kinesis-asl/src/test/scala/org/apache/spark/streaming/kinesis/KinesisStreamSuite.scala:
with mutable.SynchronizedMap[Time, (Array[SequenceNumberRanges],
Seq[Int])]
streaming/src/main/scala/org/apache/spark/streaming/dstream/FileInputDStream.scala:
new mutable.HashMap[Time, Array[String]] with mutable.SynchronizedMap[Time,
Array[String]]
streaming/src/main/scala/org/apache/spark/streaming/dstream/FileInputDStream.scala:
new mutable.HashMap[Time, Array[String]] with
mutable.SynchronizedMap[Time, Array[String]
streaming/src/test/scala/org/apache/spark/streaming/StreamingListenerSuite.scala:import
scala.collection.mutable.{ArrayBuffer, HashMap, SynchronizedBuffer,
SynchronizedMap}
streaming/src/test/scala/org/apache/spark/streaming/StreamingListenerSuite.scala:
val failureReasons = new HashMap[Int, String] with SynchronizedMap[Int,
String]
```
Since SynchronizedMap is used, should I open another JIRA ?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: Add scalastyle rule banning use of mutable.Syn...
GitHub user ted-yu opened a pull request: https://github.com/apache/spark/pull/11082 Add scalastyle rule banning use of mutable.SynchronizedBuffer See discussion at the tail of #11059 @andrewor14 Please review You can merge this pull request into a Git repository by running: $ git pull https://github.com/ted-yu/spark master Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/11082.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #11082 commit 4375f65982dea740dbba3bbe868ee0b7dd55eacf Author: tedyu <yuzhih...@gmail.com> Date: 2016-02-04T21:32:07Z Add scalastyle rule banning use of mutable.SynchronizedBuffer --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13203] Add scalastyle rule banning use ...
Github user ted-yu commented on the pull request: https://github.com/apache/spark/pull/11082#issuecomment-180074493 Logged SPARK-13204 for replacing mutable.SynchronizedMap --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13203] Add scalastyle rule banning use ...
Github user ted-yu commented on the pull request: https://github.com/apache/spark/pull/11082#issuecomment-180062533 Jenkins, test this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13203] Add scalastyle rule banning use ...
Github user ted-yu commented on the pull request: https://github.com/apache/spark/pull/11082#issuecomment-180072504 Jenkins, test this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13203] Add scalastyle rule banning use ...
Github user ted-yu commented on the pull request: https://github.com/apache/spark/pull/11082#issuecomment-180172718 Jenkins, test this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13203] Add scalastyle rule banning use ...
Github user ted-yu commented on the pull request: https://github.com/apache/spark/pull/11082#issuecomment-180162600 Some files in this PR are not covered by #11067 @holdenk I can base my PR on yours. Let me know what you think. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13203] Add scalastyle rule banning use ...
Github user ted-yu commented on the pull request: https://github.com/apache/spark/pull/11082#issuecomment-180163281 Jenkins, test this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13203] Add scalastyle rule banning use ...
Github user ted-yu commented on the pull request: https://github.com/apache/spark/pull/11082#issuecomment-180181416 Jenkins, test this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13203] Add scalastyle rule banning use ...
Github user ted-yu commented on the pull request: https://github.com/apache/spark/pull/11082#issuecomment-180208925 Jenkins, test this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [DOC] Adjust coverage for partitionBy()
GitHub user ted-yu opened a pull request: https://github.com/apache/spark/pull/10499 [DOC] Adjust coverage for partitionBy() This is the related thread: http://search-hadoop.com/m/q3RTtO3ReeJ1iF02=Re+partitioning+json+data+in+spark Michael suggested fixing the doc. Please review. You can merge this pull request into a Git repository by running: $ git pull https://github.com/ted-yu/spark master Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/10499.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #10499 commit 7884e87975e8655f0e3a20cc0455e0d7cd614fe4 Author: tedyu <yuzhih...@gmail.com> Date: 2015-12-28T20:47:26Z Adjust coverage for partitionBy() --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12527] Add private val after @transient...
Github user ted-yu closed the pull request at: https://github.com/apache/spark/pull/10482 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12527] Add private val after @transient...
Github user ted-yu commented on the pull request: https://github.com/apache/spark/pull/10482#issuecomment-167309902 I noticed the Jenkins sbt build jobs against Scala 2.11 were recreated yesterday. Do you know when the previous 'SPARK-branch-1.5-COMPILE-SBT-SCALA-2.11' started failing ? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12527] Add private val after @transient...
Github user ted-yu commented on the pull request: https://github.com/apache/spark/pull/10482#issuecomment-167291348 Jenkins, test this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12527] Add private val after @transient...
Github user ted-yu commented on the pull request: https://github.com/apache/spark/pull/10482#issuecomment-167287706 Jenkins, test this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12527] Add private val after @transient...
GitHub user ted-yu opened a pull request: https://github.com/apache/spark/pull/10482 [SPARK-12527] Add private val after @transient for kinesis-asl module In SBT build using Scala 2.11, the following warnings were reported which resulted in build failure: ``` [error] [warn] /dev/shm/spark-workspaces/8/extras/kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/KinesisInputDStream.scala:33: no valid targets for annotation on value _ssc - it is discarded unused. You may specify targets with meta-annotations, e.g. @(transient @param) [error] [warn] @transient _ssc: StreamingContext, [error] [warn] [error] [warn] /dev/shm/spark-workspaces/8/extras/kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/KinesisBackedBlockRDD.scala:73: no valid targets for annotation on value sc - it is discarded unused. You may specify targets with meta-annotations, e.g. @(transient @param) [error] [warn] @transient sc: SparkContext, [error] [warn] [error] [warn] /dev/shm/spark-workspaces/8/extras/kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/KinesisBackedBlockRDD.scala:76: no valid targets for annotation on value blockIds - it is discarded unused. You may specify targets with meta-annotations, e.g. @(transient @param) [error] [warn] @transient blockIds: Array[BlockId], [error] [warn] [error] [warn] /dev/shm/spark-workspaces/8/extras/kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/KinesisBackedBlockRDD.scala:78: no valid targets for annotation on value isBlockIdValid - it is discarded unused. You may specify targets with meta-annotations, e.g. @(transient @param) [error] [warn] @transient isBlockIdValid: Array[Boolean] = Array.empty, ``` You can merge this pull request into a Git repository by running: $ git pull https://github.com/ted-yu/spark master Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/10482.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #10482 commit 2e1e58e34ccb42c693822431f53b6d4e292fbf8d Author: tedyu <yuzhih...@gmail.com> Date: 2015-12-26T06:18:25Z [SPARK-12527] Add private val after @transient for kinesis-asl module --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12527] Add private val after @transient...
Github user ted-yu commented on the pull request: https://github.com/apache/spark/pull/10482#issuecomment-167291583 Jenkins, test this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12527] Add private val after @transient...
Github user ted-yu commented on the pull request: https://github.com/apache/spark/pull/10482#issuecomment-167284530 Jenkins, test this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12365][CORE] Use ShutdownHookManager wh...
Github user ted-yu commented on the pull request: https://github.com/apache/spark/pull/10325#issuecomment-165191304 ``` org.apache.spark.streaming.JavaAPISuite.testStreamingContextTransform Failing for the past 1 build (Since Failed#47817 ) Took 62 ms. Error Message org.apache.spark.SparkException: Only one SparkContext may be running in this JVM (see SPARK-2243). To ignore this error, set spark.driver.allowMultipleContexts = true. The currently running SparkContext was created at: org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2287) org.apache.spark.streaming.StreamingContext.(StreamingContext.scala:140) org.apache.spark.streaming.StreamingContext$$anonfun$getOrCreate$1.apply(StreamingContext.scala:869) org.apache.spark.streaming.StreamingContext$$anonfun$getOrCreate$1.apply(StreamingContext.scala:869) scala.Option.map(Option.scala:145) org.apache.spark.streaming.StreamingContext$.getOrCreate(StreamingContext.scala:869) org.apache.spark.streaming.MasterFailureTest$.runStreams(MasterFailureTest.scala:278) org.apache.spark.streaming.MasterFailureTest$.testOperation(MasterFailureTest.scala:165) ``` I don't think the above is caused by this PR. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12365][CORE] Use ShutdownHookManager wh...
Github user ted-yu commented on the pull request: https://github.com/apache/spark/pull/10325#issuecomment-165191392 Jenkins, test this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12365][CORE] Use ShutdownHookManager wh...
Github user ted-yu commented on the pull request: https://github.com/apache/spark/pull/10325#issuecomment-165266815 ``` > git fetch --tags --progress https://github.com/apache/spark.git +refs/pull/10325/*:refs/remotes/origin/pr/10325/* # timeout=15 ERROR: Timeout after 15 minutes ERROR: Error fetching remote repo 'origin' ``` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12365][CORE] Use ShutdownHookManager wh...
Github user ted-yu commented on the pull request: https://github.com/apache/spark/pull/10325#issuecomment-165266890 Jenkins, test this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [CORE] Use ShutdownHookManager where Runtime.g...
GitHub user ted-yu opened a pull request: https://github.com/apache/spark/pull/10325 [CORE] Use ShutdownHookManager where Runtime.getRuntime.addShutdownHook() is called SPARK-9886 fixed ExternalBlockStore.scala This PR fixes the remaining references to Runtime.getRuntime.addShutdownHook() You can merge this pull request into a Git repository by running: $ git pull https://github.com/ted-yu/spark master Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/10325.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #10325 commit 8b4b261aee2859c63ddd7a9e9672c64a9a8b01c3 Author: tedyu <yuzhih...@gmail.com> Date: 2015-12-16T11:30:00Z [CORE] Use ShutdownHookManager where Runtime.getRuntime.addShutdownHook() is called --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12365][CORE] Use ShutdownHookManager wh...
Github user ted-yu commented on the pull request: https://github.com/apache/spark/pull/10325#issuecomment-165143050 For build 47811: ``` Had test failures in pyspark.mllib.tests with python2.6; see logs. ``` I don't think the above was related to this PR. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12365][CORE] Use ShutdownHookManager wh...
Github user ted-yu commented on the pull request: https://github.com/apache/spark/pull/10325#issuecomment-165076521 Jenkins, test this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12365][CORE] Use ShutdownHookManager wh...
Github user ted-yu commented on the pull request: https://github.com/apache/spark/pull/10325#issuecomment-165079256 @srowen Thanks for the quick review --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12181] Check Cached unaligned-access ca...
Github user ted-yu commented on the pull request: https://github.com/apache/spark/pull/10181#issuecomment-163290885 @davies @rxin @zsxwing Let me what I need to do. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12181] Check Cached unaligned-access ca...
Github user ted-yu commented on the pull request: https://github.com/apache/spark/pull/10181#issuecomment-162982916 @rxin @zsxwing FYI --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12181] Check Cached unaligned-access ca...
Github user ted-yu commented on the pull request: https://github.com/apache/spark/pull/10181#issuecomment-163003200 See line 231: ``` case MemoryMode.OFF_HEAP => MemoryAllocator.UNSAFE ``` When unaligned() is false, we shouldn't use Unsafe (corresponding with OFF_HEAP mode). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12181] Check Cached unaligned-access ca...
Github user ted-yu commented on the pull request: https://github.com/apache/spark/pull/10181#issuecomment-163056766 Jenkins, test this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12181] Check Cached unaligned-access ca...
Github user ted-yu commented on the pull request: https://github.com/apache/spark/pull/10181#issuecomment-163046834 Jenkins, test this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12181] Check Cached unaligned-access ca...
Github user ted-yu commented on the pull request:
https://github.com/apache/spark/pull/10181#issuecomment-163061030
```
[error]
/home/jenkins/workspace/SparkPullRequestBuilder@2/unsafe/src/main/java/org/apache/spark/unsafe/Platform.java:21:
error: Bits is not public in java.nio; cannot be accessed from outside package
[error] import java.nio.Bits;
[error]^
[error]
/home/jenkins/workspace/SparkPullRequestBuilder@2/unsafe/src/main/java/org/apache/spark/unsafe/Platform.java:41:
error: cannot find symbol
[error] Field unalignedField =
Bits.class.getDeclaredField("unaligned");
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12181] Check Cached unaligned-access ca...
Github user ted-yu commented on the pull request: https://github.com/apache/spark/pull/10181#issuecomment-163085130 Jenkins, test this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11884] Drop multiple columns in the Dat...
Github user ted-yu commented on the pull request: https://github.com/apache/spark/pull/9862#issuecomment-162591780 @marmbrus What do you think ? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11884] Drop multiple columns in the Dat...
Github user ted-yu commented on the pull request: https://github.com/apache/spark/pull/9862#issuecomment-162699140 Thanks for the reviews, Michael and Wenchen --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12181] Check Cached unaligned-access ca...
GitHub user ted-yu opened a pull request: https://github.com/apache/spark/pull/10181 [SPARK-12181] Check Cached unaligned-access capability before using Unsafe For MemoryMode.OFF_HEAP, Unsafe.getInt etc. are used with no restriction. However, the Oracle implementation uses these methods only if the class variable unaligned (commented as "Cached unaligned-access capability") is true, which seems to be calculated whether the architecture is i386, x86, amd64, or x86_64. I think we should perform similar check for the use of Unsafe. You can merge this pull request into a Git repository by running: $ git pull https://github.com/ted-yu/spark master Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/10181.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #10181 commit 846877a999e8f48a1d44c4f32f1364d8e9114ff9 Author: tedyu <yuzhih...@gmail.com> Date: 2015-12-07T23:06:02Z [SPARK-12181] Check Cached unaligned-access capability before using Unsafe --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12181] Check Cached unaligned-access ca...
Github user ted-yu commented on the pull request: https://github.com/apache/spark/pull/10181#issuecomment-162709842 Jenkins, retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12181] Check Cached unaligned-access ca...
Github user ted-yu commented on the pull request: https://github.com/apache/spark/pull/10181#issuecomment-162709947 @davies @nongli @marmbrus Can you review this ? Thanks --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11884] Drop multiple columns in the Dat...
Github user ted-yu commented on the pull request: https://github.com/apache/spark/pull/9862#issuecomment-162037207 @cloud-fan @marmbrus Kindly let me know what else needs to be done --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11884] Drop multiple columns in the Dat...
Github user ted-yu commented on the pull request: https://github.com/apache/spark/pull/9862#issuecomment-161840743 Jenkins, test this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11884] Drop multiple columns in the Dat...
Github user ted-yu commented on the pull request: https://github.com/apache/spark/pull/9862#issuecomment-161847016 Jenkins, test this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11884] Drop multiple columns in the Dat...
Github user ted-yu commented on the pull request: https://github.com/apache/spark/pull/9862#issuecomment-161849699 Jenkins, test this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11884] Drop multiple columns in the Dat...
Github user ted-yu commented on the pull request: https://github.com/apache/spark/pull/9862#issuecomment-161858186 Jenkins, test this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11884] Drop multiple columns in the Dat...
Github user ted-yu commented on the pull request: https://github.com/apache/spark/pull/9862#issuecomment-158959620 @marmbrus Is there anything I need to do ? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: Drop multiple columns in the DataFrame API
GitHub user ted-yu opened a pull request: https://github.com/apache/spark/pull/9862 Drop multiple columns in the DataFrame API See the thread Ben started: http://search-hadoop.com/m/q3RTtveEuhjsr7g/ This PR adds drop() method to DataFrame which accepts multiple column names You can merge this pull request into a Git repository by running: $ git pull https://github.com/ted-yu/spark master Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/9862.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #9862 commit f2ca6d0e05a7c7c65213a05781b1d69cdc991bea Author: tedyu <yuzhih...@gmail.com> Date: 2015-11-20T18:26:01Z Drop multiple columns in the DataFrame API --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11884] Drop multiple columns in the Dat...
Github user ted-yu commented on the pull request: https://github.com/apache/spark/pull/9862#issuecomment-158484757 Jenkins, test this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11884] Drop multiple columns in the Dat...
Github user ted-yu commented on the pull request: https://github.com/apache/spark/pull/9862#issuecomment-158511346 I am open to rewriting column resolution logic in the new method but may need some pointer since I am not familiar with this area of the codebase --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11884] Drop multiple columns in the Dat...
Github user ted-yu commented on the pull request: https://github.com/apache/spark/pull/9862#issuecomment-158527785 Jenkins, test this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11884] Drop multiple columns in the Dat...
Github user ted-yu commented on the pull request: https://github.com/apache/spark/pull/9862#issuecomment-158565956 Jenkins, test this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11884] Drop multiple columns in the Dat...
Github user ted-yu commented on the pull request: https://github.com/apache/spark/pull/9862#issuecomment-158549960 Thanks for the prompt hint, Michael. New test coming shortly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11884] Drop multiple columns in the Dat...
Github user ted-yu commented on the pull request: https://github.com/apache/spark/pull/9862#issuecomment-158555946 Thanks, Ben --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11884] Drop multiple columns in the Dat...
Github user ted-yu commented on the pull request:
https://github.com/apache/spark/pull/9862#issuecomment-158551264
With
```
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/DataFrameSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/DataFrameSuite.scala
index dd6d065..fedb0df 100644
--- a/sql/core/src/test/scala/org/apache/spark/sql/DataFrameSuite.scala
+++ b/sql/core/src/test/scala/org/apache/spark/sql/DataFrameSuite.scala
@@ -378,6 +378,15 @@ class DataFrameSuite extends QueryTest with
SharedSQLContext {
assert(df.schema.map(_.name) === Seq("value"))
}
+ test("drop columns using drop") {
+val src = Seq((1,2,3)).toDF("a", "b", "c")
+val df = src.drop("a", "b")
+checkAnswer(
+ df,
+ src.collect().map(x => Row(x.getInt(1))).toSeq)
+assert(df.schema.map(_.name) === Seq("c"))
+ }
+
test("drop unknown column (no-op)") {
val df = testData.drop("random")
checkAnswer(
```
I got:
```
- drop columns using drop *** FAILED ***
Results do not match for query:
== Parsed Logical Plan ==
'Project [unresolvedalias('c)]
Project [b#134,c#135]
Project [_1#130 AS a#133,_2#131 AS b#134,_3#132 AS c#135]
LocalRelation [_1#130,_2#131,_3#132], [[1,2,3]]
== Analyzed Logical Plan ==
c: int
Project [c#135]
Project [b#134,c#135]
Project [_1#130 AS a#133,_2#131 AS b#134,_3#132 AS c#135]
LocalRelation [_1#130,_2#131,_3#132], [[1,2,3]]
== Optimized Logical Plan ==
LocalRelation [c#135], [[3]]
== Physical Plan ==
LocalTableScan [c#135], [[3]]
== Results ==
!== Correct Answer - 1 == == Spark Answer - 1 ==
![2][3] (QueryTest.scala:126)
```
Some hint ?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11884] Drop multiple columns in the Dat...
Github user ted-yu commented on the pull request: https://github.com/apache/spark/pull/9862#issuecomment-158544889 I looked at sql/core/src/test/scala/org/apache/spark/sql/DataFrameSuite.scala but testData has only one column Suggestion on where the test should be added is welcome --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11884] Drop multiple columns in the Dat...
Github user ted-yu commented on the pull request: https://github.com/apache/spark/pull/9862#issuecomment-158561800 Jenkins, retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11884] Drop multiple columns in the Dat...
Github user ted-yu commented on the pull request: https://github.com/apache/spark/pull/9862#issuecomment-158561780 ``` > git fetch --tags --progress https://github.com/apache/spark.git +refs/pull/9862/*:refs/remotes/origin/pr/9862/* # timeout=15 ERROR: Timeout after 15 minutes ERROR: Error fetching remote repo 'origin' hudson.plugins.git.GitException: Failed to fetch from https://github.com/apache/spark.git at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:763) ``` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11884] Drop multiple columns in the Dat...
Github user ted-yu commented on the pull request: https://github.com/apache/spark/pull/9862#issuecomment-158562846 Jenkins, test this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11572] Exit AsynchronousListenerBus thr...
Github user ted-yu commented on the pull request: https://github.com/apache/spark/pull/9546#issuecomment-156850070 I checked https://amplab.cs.berkeley.edu/jenkins/job/Spark-Master-Maven-with-YARN/HADOOP_PROFILE=hadoop-2.4,label=spark-test/4122/consoleFull back till: https://amplab.cs.berkeley.edu/jenkins/job/Spark-Master-Maven-with-YARN/HADOOP_PROFILE=hadoop-2.4,label=spark-test/4118/consoleFull https://amplab.cs.berkeley.edu/jenkins/job/Spark-Master-Maven-with-YARN/HADOOP_PROFILE=hadoop-2.3,label=spark-test/4122/consoleFull back until: https://amplab.cs.berkeley.edu/jenkins/job/Spark-Master-Maven-with-YARN/HADOOP_PROFILE=hadoop-2.3,label=spark-test/4119/consoleFull EventLoggingListenerSuite passed in every build above --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11572] Exit AsynchronousListenerBus thr...
Github user ted-yu commented on the pull request: https://github.com/apache/spark/pull/9546#issuecomment-156866484 Planning to send out a PR to fix the regression by keeping count of queued events first time seeing the stop flag. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11572] Exit AsynchronousListenerBus thr...
Github user ted-yu commented on the pull request:
https://github.com/apache/spark/pull/9546#issuecomment-156867526
Cloning git repo was extremely slow.
Here is proposed fix:
```
diff --git
a/core/src/main/scala/org/apache/spark/util/AsynchronousListenerBus.scala
b/core/src/main/scala/org/apache/spark/util/AsynchronousListenerBus.scala
index b3b54af..cc58bc5 100644
---
a/core/src/main/scala/org/apache/spark/util/AsynchronousListenerBus.scala
+++
b/core/src/main/scala/org/apache/spark/util/AsynchronousListenerBus.scala
@@ -56,19 +56,24 @@ private[spark] abstract class AsynchronousListenerBus[L
<: AnyRef, E](name: Stri
// A counter that represents the number of events produced and consumed
in the queue
private val eventLock = new Semaphore(0)
+ // limit on the number of events to process before exiting. -1 means no
limit
+ private val eventLimit = -1
private val listenerThread = new Thread(name) {
setDaemon(true)
override def run(): Unit = Utils.tryOrStopSparkContext(sparkContext) {
- while (true) {
+ while (eventLimit != 0) {
eventLock.acquire()
self.synchronized {
processingEvent = true
}
try {
if (stopped.get()) {
-// Get out of the while loop and shutdown the daemon thread
-return
+eventLimit = eventQueue.size
+if (eventLimit == 0) {
+ // Get out of the while loop and shutdown the daemon thread
+ return
+}
}
val event = eventQueue.poll
assert(event != null, "event queue was empty but the listener
bus was not stopped")
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11572] Exit AsynchronousListenerBus thr...
Github user ted-yu commented on the pull request: https://github.com/apache/spark/pull/9546#issuecomment-155533447 @andrewor14 See if all comments have been addressed --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11572] Exit AsynchronousListenerBus thr...
Github user ted-yu commented on the pull request: https://github.com/apache/spark/pull/9546#issuecomment-155276724 I see several errors in the following form (https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45460/consoleFull): ``` [error] SERVER ERROR: Service Temporarily Unavailable url=http://repository.mapr.com/maven/org/apache/hadoop/hadoop-yarn-server/2.2.0/hadoop-yarn-server-2.2.0.jar ``` Not related to the PR. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: Exit AsynchronousListenerBus thread when stop(...
GitHub user ted-yu opened a pull request: https://github.com/apache/spark/pull/9546 Exit AsynchronousListenerBus thread when stop() is called As vonnagy reported in the following thread: http://search-hadoop.com/m/q3RTtk982kvIow22 Attempts to join the thread in AsynchronousListenerBus resulted in lock up because AsynchronousListenerBus thread was still getting messages `SparkListenerExecutorMetricsUpdate` from the DAGScheduler You can merge this pull request into a Git repository by running: $ git pull https://github.com/ted-yu/spark master Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/9546.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #9546 commit c60b8607201ac7094f39499439fe5022d0e26de3 Author: tedyu <yuzhih...@gmail.com> Date: 2015-11-08T17:08:54Z Exit AsynchronousListenerBus thread when stop() is called --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-11371 Make "mean" an alias for "avg" ope...
GitHub user ted-yu opened a pull request: https://github.com/apache/spark/pull/9332 SPARK-11371 Make "mean" an alias for "avg" operator From Reynold in the thread 'Exception when using some aggregate operators' (http://search-hadoop.com/m/q3RTt0xFr22nXB4/): I don't think these are bugs. The SQL standard for average is "avg", not "mean". Similarly, a distinct count is supposed to be written as "count(distinct col)", not "countDistinct(col)". We can, however, make "mean" an alias for "avg" to improve compatibility between DataFrame and SQL. You can merge this pull request into a Git repository by running: $ git pull https://github.com/ted-yu/spark master Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/9332.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #9332 commit f1447f0cba860a84ed60929b4871936198fe4150 Author: tedyu <yuzhih...@gmail.com> Date: 2015-10-28T14:12:12Z SPARK-11371 Make "mean" an alias for "avg" operator --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: Close HBaseAdmin at the end of HBaseTest
GitHub user ted-yu opened a pull request: https://github.com/apache/spark/pull/6381 Close HBaseAdmin at the end of HBaseTest You can merge this pull request into a Git repository by running: $ git pull https://github.com/ted-yu/spark master Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/6381.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #6381 commit e2f0ea14fdc256caf5e41ba44a865926642849a8 Author: tedyu yuzhih...@gmail.com Date: 2015-05-23T23:48:20Z Close HBaseAdmin at the end of HBaseTest --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: Make SPARK prefix a variable
GitHub user ted-yu opened a pull request: https://github.com/apache/spark/pull/6153 Make SPARK prefix a variable You can merge this pull request into a Git repository by running: $ git pull https://github.com/ted-yu/spark master Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/6153.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #6153 commit ab982aa0da5ca8beefa6a662aad02b13ef7192a1 Author: tedyu yuzhih...@gmail.com Date: 2015-05-14T19:10:09Z Make SPARK prefix a variable --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-7237 Clean function in several RDD metho...
Github user ted-yu commented on the pull request: https://github.com/apache/spark/pull/5959#issuecomment-100261058 Build was aborted by shane: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/32231/consoleFull ClosureCleanerSuite and RDDSuite passed in above run. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-7237 Clean function in several RDD metho...
Github user ted-yu commented on the pull request: https://github.com/apache/spark/pull/5959#issuecomment-100317495 @rxin @andrewor14 Please let me know if there is any review comment which I haven't addressed --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-7237 Clean function in several RDD metho...
Github user ted-yu commented on the pull request: https://github.com/apache/spark/pull/5959#issuecomment-100371211 From https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/32195/consoleFull : [info] - filterWith *** FAILED *** (13 milliseconds) [info] org.apache.spark.SparkException: Job aborted due to stage failure: Task 1 in stage 154.0 failed 1 times, most recent failure: Lost task 1.0 in stage 154.0 (TID 865, localhost): org.apache.spark.SparkException: RDD transformations and actions can only be invoked by the driver, not inside of other transformations; for example, rdd1.map(x = rdd2.values.count() * x) is invalid because the values transformation and count action cannot be performed inside of the rdd1.map transformation. For more information, see SPARK-5063. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-7237 Clean function in several RDD metho...
Github user ted-yu commented on the pull request: https://github.com/apache/spark/pull/5959#issuecomment-100388849 Should have looked closer :-) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: Clean function in several RDD methods
Github user ted-yu commented on a diff in the pull request:
https://github.com/apache/spark/pull/5959#discussion_r29882493
--- Diff: core/src/main/scala/org/apache/spark/rdd/RDD.scala ---
@@ -741,9 +741,10 @@ abstract class RDD[T: ClassTag](
def mapWith[A, U: ClassTag]
(constructA: Int = A, preservesPartitioning: Boolean = false)
(f: (T, A) = U): RDD[U] = withScope {
+val cleanF = sc.clean(f)
mapPartitionsWithIndex((index, iter) = {
val a = constructA(index)
- iter.map(t = f(t, a))
+ iter.map(t = cleanF(t, a))
--- End diff --
Looks like Reynold is correct.
I can update the PR for collect() and undo the change for other methods
touched.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: Clean function in several RDD methods
GitHub user ted-yu opened a pull request: https://github.com/apache/spark/pull/5959 Clean function in several RDD methods You can merge this pull request into a Git repository by running: $ git pull https://github.com/ted-yu/spark master Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/5959.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #5959 commit 6c124a9ddeb19541c381c3eea0bc13db38ce7574 Author: tedyu yuzhih...@gmail.com Date: 2015-05-07T02:23:46Z Clean function in several RDD methods --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
