Github user ted-yu closed the pull request at:
https://github.com/apache/spark/pull/13233
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature
Github user ted-yu commented on the pull request:
https://github.com/apache/spark/pull/13233#issuecomment-220731442
Should reflection be used :-)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does
GitHub user ted-yu opened a pull request:
https://github.com/apache/spark/pull/13233
[SPARK-11827][SQL] Fix compilation against Java 7
## What changes were proposed in this pull request?
This PR fixes compilation error against Java 7:
```
sql/catalyst/src/main/scala/org
Github user ted-yu closed the pull request at:
https://github.com/apache/spark/pull/10181
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature
Github user ted-yu closed the pull request at:
https://github.com/apache/spark/pull/11082
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature
Github user ted-yu commented on the pull request:
https://github.com/apache/spark/pull/11082#issuecomment-181036649
DirectKafkaStreamSuite.scala is not covered by #11067
Addition of scalastyle rule would not pass even after #11067 goes in.
Should I open a seperate PR
Github user ted-yu commented on the pull request:
https://github.com/apache/spark/pull/11082#issuecomment-181051841
bq. If an existing PR means to catch all occurrences of the issue the rule
would flag, then it needs to cover this case too
In this case, looks like SPARK-13151
Github user ted-yu commented on the pull request:
https://github.com/apache/spark/pull/11082#issuecomment-180616613
There is no error in DirectKafkaStreamSuite.scala
Once [~holdenk]'s PR goes in, I will rebase.
---
If your project is set up for it, you can reply
Github user ted-yu commented on the pull request:
https://github.com/apache/spark/pull/11082#issuecomment-180498087
I am trying to add an Array to ConcurrentLinkedQueue (collectedData) in
Scala 2.11:
```
stream.foreachRDD { rdd => Collections.addAll(collectedData, rdd.coll
Github user ted-yu commented on the pull request:
https://github.com/apache/spark/pull/11082#issuecomment-180546842
Jenkins, test this please.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user ted-yu commented on the pull request:
https://github.com/apache/spark/pull/11082#issuecomment-180536530
Jenkins, test this please.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user ted-yu commented on the pull request:
https://github.com/apache/spark/pull/11082#issuecomment-180318313
Jenkins, test this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user ted-yu commented on the pull request:
https://github.com/apache/spark/pull/11082#issuecomment-180064615
```
external/kafka/src/test/scala/org/apache/spark/streaming/kafka/DirectKafkaStreamSuite.scala:
new ArrayBuffer[(String, String
Github user ted-yu commented on the pull request:
https://github.com/apache/spark/pull/11082#issuecomment-180095881
Jenkins, test this please.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user ted-yu commented on the pull request:
https://github.com/apache/spark/pull/11082#issuecomment-180104115
Jenkins, test this please.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user ted-yu commented on the pull request:
https://github.com/apache/spark/pull/11082#issuecomment-180084459
Jenkins, test this please.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user ted-yu commented on the pull request:
https://github.com/apache/spark/pull/11082#issuecomment-180063800
```
external/kafka/src/test/scala/org/apache/spark/streaming/kafka/KafkaStreamSuite.scala:
val result = new mutable.HashMap[String, Long
GitHub user ted-yu opened a pull request:
https://github.com/apache/spark/pull/11082
Add scalastyle rule banning use of mutable.SynchronizedBuffer
See discussion at the tail of #11059
@andrewor14
Please review
You can merge this pull request into a Git repository
Github user ted-yu commented on the pull request:
https://github.com/apache/spark/pull/11082#issuecomment-180074493
Logged SPARK-13204 for replacing mutable.SynchronizedMap
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well
Github user ted-yu commented on the pull request:
https://github.com/apache/spark/pull/11082#issuecomment-180062533
Jenkins, test this please.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user ted-yu commented on the pull request:
https://github.com/apache/spark/pull/11082#issuecomment-180072504
Jenkins, test this please.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user ted-yu commented on the pull request:
https://github.com/apache/spark/pull/11082#issuecomment-180172718
Jenkins, test this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user ted-yu commented on the pull request:
https://github.com/apache/spark/pull/11082#issuecomment-180162600
Some files in this PR are not covered by #11067
@holdenk
I can base my PR on yours.
Let me know what you think.
---
If your project is set up
Github user ted-yu commented on the pull request:
https://github.com/apache/spark/pull/11082#issuecomment-180163281
Jenkins, test this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user ted-yu commented on the pull request:
https://github.com/apache/spark/pull/11082#issuecomment-180181416
Jenkins, test this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user ted-yu commented on the pull request:
https://github.com/apache/spark/pull/11082#issuecomment-180208925
Jenkins, test this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
GitHub user ted-yu opened a pull request:
https://github.com/apache/spark/pull/10499
[DOC] Adjust coverage for partitionBy()
This is the related thread:
http://search-hadoop.com/m/q3RTtO3ReeJ1iF02=Re+partitioning+json+data+in+spark
Michael suggested fixing the doc
Github user ted-yu closed the pull request at:
https://github.com/apache/spark/pull/10482
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature
Github user ted-yu commented on the pull request:
https://github.com/apache/spark/pull/10482#issuecomment-167309902
I noticed the Jenkins sbt build jobs against Scala 2.11 were recreated
yesterday.
Do you know when the previous 'SPARK-branch-1.5-COMPILE-SBT-SCALA-2.11
Github user ted-yu commented on the pull request:
https://github.com/apache/spark/pull/10482#issuecomment-167291348
Jenkins, test this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user ted-yu commented on the pull request:
https://github.com/apache/spark/pull/10482#issuecomment-167287706
Jenkins, test this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
GitHub user ted-yu opened a pull request:
https://github.com/apache/spark/pull/10482
[SPARK-12527] Add private val after @transient for kinesis-asl module
In SBT build using Scala 2.11, the following warnings were reported which
resulted in build failure:
```
[error] [warn
Github user ted-yu commented on the pull request:
https://github.com/apache/spark/pull/10482#issuecomment-167291583
Jenkins, test this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user ted-yu commented on the pull request:
https://github.com/apache/spark/pull/10482#issuecomment-167284530
Jenkins, test this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user ted-yu commented on the pull request:
https://github.com/apache/spark/pull/10325#issuecomment-165191304
```
org.apache.spark.streaming.JavaAPISuite.testStreamingContextTransform
Failing for the past 1 build (Since Failed#47817 )
Took 62 ms.
Error
Github user ted-yu commented on the pull request:
https://github.com/apache/spark/pull/10325#issuecomment-165191392
Jenkins, test this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user ted-yu commented on the pull request:
https://github.com/apache/spark/pull/10325#issuecomment-165266815
```
> git fetch --tags --progress https://github.com/apache/spark.git
+refs/pull/10325/*:refs/remotes/origin/pr/10325/* # timeout=15
ERROR: Timeout after
Github user ted-yu commented on the pull request:
https://github.com/apache/spark/pull/10325#issuecomment-165266890
Jenkins, test this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
GitHub user ted-yu opened a pull request:
https://github.com/apache/spark/pull/10325
[CORE] Use ShutdownHookManager where Runtime.getRuntime.addShutdownHook()
is called
SPARK-9886 fixed ExternalBlockStore.scala
This PR fixes the remaining references
Github user ted-yu commented on the pull request:
https://github.com/apache/spark/pull/10325#issuecomment-165143050
For build 47811:
```
Had test failures in pyspark.mllib.tests with python2.6; see logs.
```
I don't think the above was related to this PR.
---
If your
Github user ted-yu commented on the pull request:
https://github.com/apache/spark/pull/10325#issuecomment-165076521
Jenkins, test this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user ted-yu commented on the pull request:
https://github.com/apache/spark/pull/10325#issuecomment-165079256
@srowen
Thanks for the quick review
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project
Github user ted-yu commented on the pull request:
https://github.com/apache/spark/pull/10181#issuecomment-163290885
@davies @rxin @zsxwing
Let me what I need to do.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well
Github user ted-yu commented on the pull request:
https://github.com/apache/spark/pull/10181#issuecomment-162982916
@rxin @zsxwing
FYI
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user ted-yu commented on the pull request:
https://github.com/apache/spark/pull/10181#issuecomment-163003200
See line 231:
```
case MemoryMode.OFF_HEAP => MemoryAllocator.UNSAFE
```
When unaligned() is false, we shouldn't use Unsafe (correspond
Github user ted-yu commented on the pull request:
https://github.com/apache/spark/pull/10181#issuecomment-163056766
Jenkins, test this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user ted-yu commented on the pull request:
https://github.com/apache/spark/pull/10181#issuecomment-163046834
Jenkins, test this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user ted-yu commented on the pull request:
https://github.com/apache/spark/pull/10181#issuecomment-163061030
```
[error]
/home/jenkins/workspace/SparkPullRequestBuilder@2/unsafe/src/main/java/org/apache/spark/unsafe/Platform.java:21:
error: Bits is not public in java.nio
Github user ted-yu commented on the pull request:
https://github.com/apache/spark/pull/10181#issuecomment-163085130
Jenkins, test this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user ted-yu commented on the pull request:
https://github.com/apache/spark/pull/9862#issuecomment-162591780
@marmbrus
What do you think ?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does
Github user ted-yu commented on the pull request:
https://github.com/apache/spark/pull/9862#issuecomment-162699140
Thanks for the reviews, Michael and Wenchen
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
GitHub user ted-yu opened a pull request:
https://github.com/apache/spark/pull/10181
[SPARK-12181] Check Cached unaligned-access capability before using Unsafe
For MemoryMode.OFF_HEAP, Unsafe.getInt etc. are used with no restriction.
However, the Oracle implementation uses
Github user ted-yu commented on the pull request:
https://github.com/apache/spark/pull/10181#issuecomment-162709842
Jenkins, retest this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does
Github user ted-yu commented on the pull request:
https://github.com/apache/spark/pull/10181#issuecomment-162709947
@davies @nongli @marmbrus
Can you review this ?
Thanks
---
If your project is set up for it, you can reply to this email and have your
reply appear
Github user ted-yu commented on the pull request:
https://github.com/apache/spark/pull/9862#issuecomment-162037207
@cloud-fan @marmbrus
Kindly let me know what else needs to be done
---
If your project is set up for it, you can reply to this email and have your
reply appear
Github user ted-yu commented on the pull request:
https://github.com/apache/spark/pull/9862#issuecomment-161840743
Jenkins, test this please.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user ted-yu commented on the pull request:
https://github.com/apache/spark/pull/9862#issuecomment-161847016
Jenkins, test this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user ted-yu commented on the pull request:
https://github.com/apache/spark/pull/9862#issuecomment-161849699
Jenkins, test this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user ted-yu commented on the pull request:
https://github.com/apache/spark/pull/9862#issuecomment-161858186
Jenkins, test this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user ted-yu commented on the pull request:
https://github.com/apache/spark/pull/9862#issuecomment-158959620
@marmbrus
Is there anything I need to do ?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
GitHub user ted-yu opened a pull request:
https://github.com/apache/spark/pull/9862
Drop multiple columns in the DataFrame API
See the thread Ben started:
http://search-hadoop.com/m/q3RTtveEuhjsr7g/
This PR adds drop() method to DataFrame which accepts multiple column
Github user ted-yu commented on the pull request:
https://github.com/apache/spark/pull/9862#issuecomment-158484757
Jenkins, test this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user ted-yu commented on the pull request:
https://github.com/apache/spark/pull/9862#issuecomment-158511346
I am open to rewriting column resolution logic in the new method but may
need some pointer since I am not familiar with this area of the codebase
---
If your project
Github user ted-yu commented on the pull request:
https://github.com/apache/spark/pull/9862#issuecomment-158527785
Jenkins, test this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user ted-yu commented on the pull request:
https://github.com/apache/spark/pull/9862#issuecomment-158565956
Jenkins, test this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user ted-yu commented on the pull request:
https://github.com/apache/spark/pull/9862#issuecomment-158549960
Thanks for the prompt hint, Michael.
New test coming shortly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub
Github user ted-yu commented on the pull request:
https://github.com/apache/spark/pull/9862#issuecomment-158555946
Thanks, Ben
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user ted-yu commented on the pull request:
https://github.com/apache/spark/pull/9862#issuecomment-158551264
With
```
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/DataFrameSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/DataFrameSuite.scala
Github user ted-yu commented on the pull request:
https://github.com/apache/spark/pull/9862#issuecomment-158544889
I looked at
sql/core/src/test/scala/org/apache/spark/sql/DataFrameSuite.scala but testData
has only one column
Suggestion on where the test should be added
Github user ted-yu commented on the pull request:
https://github.com/apache/spark/pull/9862#issuecomment-158561800
Jenkins, retest this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user ted-yu commented on the pull request:
https://github.com/apache/spark/pull/9862#issuecomment-158561780
```
> git fetch --tags --progress https://github.com/apache/spark.git
+refs/pull/9862/*:refs/remotes/origin/pr/9862/* # timeout=15
ERROR: Timeout after
Github user ted-yu commented on the pull request:
https://github.com/apache/spark/pull/9862#issuecomment-158562846
Jenkins, test this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user ted-yu commented on the pull request:
https://github.com/apache/spark/pull/9546#issuecomment-156850070
I checked
https://amplab.cs.berkeley.edu/jenkins/job/Spark-Master-Maven-with-YARN/HADOOP_PROFILE=hadoop-2.4,label=spark-test/4122/consoleFull
back till
Github user ted-yu commented on the pull request:
https://github.com/apache/spark/pull/9546#issuecomment-156866484
Planning to send out a PR to fix the regression by keeping count of queued
events first time seeing the stop flag.
---
If your project is set up for it, you can reply
Github user ted-yu commented on the pull request:
https://github.com/apache/spark/pull/9546#issuecomment-156867526
Cloning git repo was extremely slow.
Here is proposed fix:
```
diff --git
a/core/src/main/scala/org/apache/spark/util/AsynchronousListenerBus.scala
b/core
Github user ted-yu commented on the pull request:
https://github.com/apache/spark/pull/9546#issuecomment-155533447
@andrewor14
See if all comments have been addressed
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well
Github user ted-yu commented on the pull request:
https://github.com/apache/spark/pull/9546#issuecomment-155276724
I see several errors in the following form
(https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45460/consoleFull):
```
[error] SERVER ERROR
GitHub user ted-yu opened a pull request:
https://github.com/apache/spark/pull/9546
Exit AsynchronousListenerBus thread when stop() is called
As vonnagy reported in the following thread:
http://search-hadoop.com/m/q3RTtk982kvIow22
Attempts to join the thread
GitHub user ted-yu opened a pull request:
https://github.com/apache/spark/pull/9332
SPARK-11371 Make "mean" an alias for "avg" operator
From Reynold in the thread 'Exception when using some aggregate operators'
(http://search-hadoop.com/m/q3RTt0xFr22nXB4/):
GitHub user ted-yu opened a pull request:
https://github.com/apache/spark/pull/6381
Close HBaseAdmin at the end of HBaseTest
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/ted-yu/spark master
Alternatively you can review
GitHub user ted-yu opened a pull request:
https://github.com/apache/spark/pull/6153
Make SPARK prefix a variable
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/ted-yu/spark master
Alternatively you can review and apply
Github user ted-yu commented on the pull request:
https://github.com/apache/spark/pull/5959#issuecomment-100261058
Build was aborted by shane:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/32231/consoleFull
ClosureCleanerSuite and RDDSuite passed
Github user ted-yu commented on the pull request:
https://github.com/apache/spark/pull/5959#issuecomment-100317495
@rxin @andrewor14
Please let me know if there is any review comment which I haven't addressed
---
If your project is set up for it, you can reply to this email
Github user ted-yu commented on the pull request:
https://github.com/apache/spark/pull/5959#issuecomment-100371211
From
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/32195/consoleFull
:
[info] - filterWith *** FAILED *** (13 milliseconds)
[info
Github user ted-yu commented on the pull request:
https://github.com/apache/spark/pull/5959#issuecomment-100388849
Should have looked closer :-)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does
Github user ted-yu commented on a diff in the pull request:
https://github.com/apache/spark/pull/5959#discussion_r29882493
--- Diff: core/src/main/scala/org/apache/spark/rdd/RDD.scala ---
@@ -741,9 +741,10 @@ abstract class RDD[T: ClassTag](
def mapWith[A, U: ClassTag
GitHub user ted-yu opened a pull request:
https://github.com/apache/spark/pull/5959
Clean function in several RDD methods
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/ted-yu/spark master
Alternatively you can review
87 matches
Mail list logo
