Github user tejasapatil commented on a diff in the pull request:
https://github.com/apache/spark/pull/13042#discussion_r62794845
--- Diff:
common/network-common/src/main/java/org/apache/spark/network/util/JavaUtils.java
---
@@ -79,14 +80,27 @@ public static String bytesToString
Github user tejasapatil commented on the pull request:
https://github.com/apache/spark/pull/13042#issuecomment-218344802
ok to test
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
GitHub user tejasapatil opened a pull request:
https://github.com/apache/spark/pull/13042
[SPARK-15263][Core] Make shuffle service dir cleanup faster by using `rm
-rf`
## What changes were proposed in this pull request?
Jira: https://issues.apache.org/jira/browse/SPARK
Github user tejasapatil commented on the pull request:
https://github.com/apache/spark/pull/12194#issuecomment-218310372
ok to test
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user tejasapatil commented on the pull request:
https://github.com/apache/spark/pull/10938#issuecomment-216563850
Major chunks of this PR is already in trunk after
https://github.com/apache/spark/pull/11891/files so this PR is not necessary
anymore (unless I missed something
Github user tejasapatil commented on a diff in the pull request:
https://github.com/apache/spark/pull/12309#discussion_r60339435
--- Diff: core/src/main/scala/org/apache/spark/rdd/PipedRDD.scala ---
@@ -144,7 +142,8 @@ private[spark] class PipedRDD[T: ClassTag](
new Thread
Github user tejasapatil commented on the pull request:
https://github.com/apache/spark/pull/12194#issuecomment-209185057
ok to test
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user tejasapatil commented on the pull request:
https://github.com/apache/spark/pull/12194#issuecomment-207508838
ok to test
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user tejasapatil commented on the pull request:
https://github.com/apache/spark/pull/12194#issuecomment-206075956
ok to test
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
GitHub user tejasapatil opened a pull request:
https://github.com/apache/spark/pull/12194
[SPARK-14400] [SQL] ScriptTransformation does not fail the job for bad user
command
## What changes were proposed in this pull request?
- Refer to the Jira for the problem: jira
Github user tejasapatil commented on the pull request:
https://github.com/apache/spark/pull/11723#issuecomment-206071824
Can someone review this PR ?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does
Github user tejasapatil commented on the pull request:
https://github.com/apache/spark/pull/11891#issuecomment-204259326
ping @liancheng
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user tejasapatil commented on the pull request:
https://github.com/apache/spark/pull/11723#issuecomment-203252858
sure. @rxin can you please review the PR ?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user tejasapatil commented on the pull request:
https://github.com/apache/spark/pull/11891#issuecomment-201966523
@liancheng : I have made all requested changes as per review and also
rebased. Can you please take a look ?
---
If your project is set up for it, you can reply
Github user tejasapatil commented on the pull request:
https://github.com/apache/spark/pull/11891#issuecomment-201945359
ok to test
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user tejasapatil commented on a diff in the pull request:
https://github.com/apache/spark/pull/11891#discussion_r57393457
--- Diff:
sql/hive/src/test/scala/org/apache/spark/sql/hive/orc/OrcQuerySuite.scala ---
@@ -400,4 +400,33 @@ class OrcQuerySuite extends QueryTest
Github user tejasapatil commented on a diff in the pull request:
https://github.com/apache/spark/pull/11723#discussion_r57337365
--- Diff:
core/src/test/scala/org/apache/spark/scheduler/ExternalClusterManagerSuite.scala
---
@@ -0,0 +1,67 @@
+/*
+ * Licensed to the Apache
Github user tejasapatil commented on a diff in the pull request:
https://github.com/apache/spark/pull/11723#discussion_r57337311
--- Diff:
core/src/test/scala/org/apache/spark/scheduler/ExternalClusterManagerSuite.scala
---
@@ -0,0 +1,67 @@
+/*
+ * Licensed to the Apache
Github user tejasapatil commented on a diff in the pull request:
https://github.com/apache/spark/pull/11723#discussion_r57337345
--- Diff:
core/src/test/scala/org/apache/spark/scheduler/ExternalClusterManagerSuite.scala
---
@@ -0,0 +1,67 @@
+/*
+ * Licensed to the Apache
Github user tejasapatil commented on a diff in the pull request:
https://github.com/apache/spark/pull/11723#discussion_r57336395
--- Diff: dev/.rat-excludes ---
@@ -98,3 +98,4 @@ LZ4BlockInputStream.java
spark-deps-.*
.*csv
.*tsv
Github user tejasapatil commented on the pull request:
https://github.com/apache/spark/pull/11723#issuecomment-200704530
@rxin : Yes !! At Facebook we are using an internal scheduler to run Spark
executors. Maintaining an internal patch to have that "glue" and merging i
Github user tejasapatil commented on the pull request:
https://github.com/apache/spark/pull/11723#issuecomment-200687233
@rxin : I would really like to have this PR in trunk. As things stand, for
anyone using their own scheduler, one has to maintain a patch over open source
release
Github user tejasapatil commented on a diff in the pull request:
https://github.com/apache/spark/pull/11723#discussion_r57276779
--- Diff:
core/src/test/scala/org/apache/spark/scheduler/ExternalClusterManagerSuite.scala
---
@@ -0,0 +1,65 @@
+/*
+ * Licensed to the Apache
Github user tejasapatil commented on a diff in the pull request:
https://github.com/apache/spark/pull/11723#discussion_r57276711
--- Diff: dev/.rat-excludes ---
@@ -98,3 +98,4 @@ LZ4BlockInputStream.java
spark-deps-.*
.*csv
.*tsv
Github user tejasapatil commented on a diff in the pull request:
https://github.com/apache/spark/pull/11723#discussion_r57276691
--- Diff:
core/src/test/resources/META-INF/services/org.apache.spark.scheduler.ExternalClusterManager
---
@@ -0,0 +1
Github user tejasapatil commented on a diff in the pull request:
https://github.com/apache/spark/pull/11723#discussion_r57276663
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/ExternalClusterManager.scala ---
@@ -0,0 +1,62 @@
+/*
+ * Licensed to the Apache
Github user tejasapatil commented on a diff in the pull request:
https://github.com/apache/spark/pull/11723#discussion_r57276660
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/ExternalClusterManager.scala ---
@@ -0,0 +1,62 @@
+/*
+ * Licensed to the Apache
Github user tejasapatil commented on a diff in the pull request:
https://github.com/apache/spark/pull/11723#discussion_r57276618
--- Diff:
core/src/test/scala/org/apache/spark/scheduler/ExternalClusterManagerSuite.scala
---
@@ -0,0 +1,65 @@
+/*
+ * Licensed to the Apache
Github user tejasapatil commented on a diff in the pull request:
https://github.com/apache/spark/pull/11723#discussion_r57276344
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/ExternalClusterManager.scala ---
@@ -0,0 +1,62 @@
+/*
+ * Licensed to the Apache
Github user tejasapatil commented on a diff in the pull request:
https://github.com/apache/spark/pull/11723#discussion_r57276268
--- Diff: core/src/main/scala/org/apache/spark/executor/Executor.scala ---
@@ -149,7 +149,14 @@ private[spark] class Executor(
tr.kill
Github user tejasapatil commented on a diff in the pull request:
https://github.com/apache/spark/pull/11723#discussion_r57276201
--- Diff: core/src/main/scala/org/apache/spark/executor/Executor.scala ---
@@ -149,7 +149,14 @@ private[spark] class Executor(
tr.kill
Github user tejasapatil commented on a diff in the pull request:
https://github.com/apache/spark/pull/11723#discussion_r57276211
--- Diff: core/src/main/scala/org/apache/spark/executor/Executor.scala ---
@@ -149,7 +149,14 @@ private[spark] class Executor(
tr.kill
Github user tejasapatil commented on a diff in the pull request:
https://github.com/apache/spark/pull/11723#discussion_r57276135
--- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala ---
@@ -2443,8 +2443,34 @@ object SparkContext extends Logging
Github user tejasapatil commented on a diff in the pull request:
https://github.com/apache/spark/pull/11927#discussion_r57275794
--- Diff: core/src/main/scala/org/apache/spark/rdd/PipedRDD.scala ---
@@ -205,6 +206,8 @@ private[spark] class PipedRDD[T: ClassTag](
private
GitHub user tejasapatil opened a pull request:
https://github.com/apache/spark/pull/11927
[SPARK-14110] [CORE] PipedRDD to print the command ran on non zero exit
## What changes were proposed in this pull request?
In case of failure in subprocess launched in PipedRDD
Github user tejasapatil commented on the pull request:
https://github.com/apache/spark/pull/11891#issuecomment-200627141
ok to test
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user tejasapatil commented on the pull request:
https://github.com/apache/spark/pull/11891#issuecomment-200607485
@liancheng : I rebased and fixed the problem. Also, thanks for the pointer
for debugging the tests.
---
If your project is set up for it, you can reply
Github user tejasapatil commented on the pull request:
https://github.com/apache/spark/pull/11891#issuecomment-200562818
@liancheng + @marmbrus : Thanks for your comments. I have made the
suggested changes except the one related to test case which I am not sure how
to do
Github user tejasapatil commented on a diff in the pull request:
https://github.com/apache/spark/pull/11891#discussion_r57240228
--- Diff:
sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveMetastoreCatalog.scala ---
@@ -597,6 +619,107 @@ private[hive] class
Github user tejasapatil commented on a diff in the pull request:
https://github.com/apache/spark/pull/11891#discussion_r57240256
--- Diff:
sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveMetastoreCatalog.scala ---
@@ -597,6 +619,107 @@ private[hive] class
Github user tejasapatil commented on a diff in the pull request:
https://github.com/apache/spark/pull/11891#discussion_r57240207
--- Diff:
sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveMetastoreCatalog.scala ---
@@ -439,6 +440,56 @@ private[hive] class HiveMetastoreCatalog
Github user tejasapatil commented on a diff in the pull request:
https://github.com/apache/spark/pull/11891#discussion_r57240111
--- Diff:
sql/hive/src/test/scala/org/apache/spark/sql/hive/orc/OrcQuerySuite.scala ---
@@ -400,4 +400,33 @@ class OrcQuerySuite extends QueryTest
Github user tejasapatil commented on a diff in the pull request:
https://github.com/apache/spark/pull/11891#discussion_r57240048
--- Diff:
sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveMetastoreCatalog.scala ---
@@ -597,6 +619,107 @@ private[hive] class
Github user tejasapatil commented on a diff in the pull request:
https://github.com/apache/spark/pull/11891#discussion_r57239903
--- Diff:
sql/hive/src/test/scala/org/apache/spark/sql/hive/orc/OrcQuerySuite.scala ---
@@ -400,4 +400,33 @@ class OrcQuerySuite extends QueryTest
Github user tejasapatil commented on a diff in the pull request:
https://github.com/apache/spark/pull/11891#discussion_r57239593
--- Diff:
sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveMetastoreCatalog.scala ---
@@ -439,6 +440,56 @@ private[hive] class HiveMetastoreCatalog
Github user tejasapatil commented on the pull request:
https://github.com/apache/spark/pull/11891#issuecomment-200078443
@rxin : Can you point me to specific features / changes in Parquet which
are not in ORC ? I am happy to work on adding that to ORC.
---
If your project is set up
Github user tejasapatil commented on the pull request:
https://github.com/apache/spark/pull/11891#issuecomment-200077504
@marmbrus : I looked at the build failures trying to figure out the cause :
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/53806/
I
GitHub user tejasapatil opened a pull request:
https://github.com/apache/spark/pull/11891
Use ORC data source for SQL queries on ORC tables
## What changes were proposed in this pull request?
This patch enables use of OrcRelation for SQL queries which read data from
Hive
Github user tejasapatil commented on the pull request:
https://github.com/apache/spark/pull/11628#issuecomment-197008054
@srowen : The problem sequence you identified is correct. I agree that
doing check both at the start and end is overkill (not to mention the overhead
and ugliness
Github user tejasapatil commented on the pull request:
https://github.com/apache/spark/pull/11628#issuecomment-196901467
@srowen : That looks cleaner :)
- Calling `hasNext()` inside `next()` looks somewhat un-conventional to me.
But again, its based on an expectation that all
Github user tejasapatil commented on a diff in the pull request:
https://github.com/apache/spark/pull/11628#discussion_r56189655
--- Diff: core/src/test/scala/org/apache/spark/rdd/PipedRDDSuite.scala ---
@@ -50,6 +50,29 @@ class PipedRDDSuite extends SparkFunSuite
Github user tejasapatil commented on the pull request:
https://github.com/apache/spark/pull/11628#issuecomment-196540199
@srowen : As suggested, I am changing `next()` to do
`propagateChildThreadException`. The iterator itself uses a
`BufferedLineIterator` which would throw
Github user tejasapatil commented on a diff in the pull request:
https://github.com/apache/spark/pull/11628#discussion_r55752933
--- Diff: core/src/main/scala/org/apache/spark/rdd/PipedRDD.scala ---
@@ -133,23 +136,30 @@ private[spark] class PipedRDD[T: ClassTag](
// Start
Github user tejasapatil commented on a diff in the pull request:
https://github.com/apache/spark/pull/11628#discussion_r55752244
--- Diff: core/src/main/scala/org/apache/spark/rdd/PipedRDD.scala ---
@@ -157,8 +167,16 @@ private[spark] class PipedRDD[T: ClassTag](
val lines
Github user tejasapatil commented on the pull request:
https://github.com/apache/spark/pull/11628#issuecomment-194724125
The build failed because I was using some Process apis added in Java 8 but
Jenkins builds with Java 7. Fixed that.
@rxin : Can you please re-trigger
Github user tejasapatil commented on the pull request:
https://github.com/apache/spark/pull/11628#issuecomment-194694929
@rxin : Thanks for the review. I have updated the PR
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well
Github user tejasapatil commented on a diff in the pull request:
https://github.com/apache/spark/pull/11628#discussion_r55639115
--- Diff: core/src/main/scala/org/apache/spark/rdd/PipedRDD.scala ---
@@ -133,23 +134,30 @@ private[spark] class PipedRDD[T: ClassTag](
// Start
Github user tejasapatil commented on a diff in the pull request:
https://github.com/apache/spark/pull/11628#discussion_r55639013
--- Diff: core/src/main/scala/org/apache/spark/rdd/PipedRDD.scala ---
@@ -118,6 +118,7 @@ private[spark] class PipedRDD[T: ClassTag](
val
GitHub user tejasapatil opened a pull request:
https://github.com/apache/spark/pull/11628
[SPARK-13793] [CORE] PipedRDD doesn't propagate exceptions while reading
parent RDD
## What changes were proposed in this pull request?
PipedRDD creates a child thread to read output
Github user tejasapatil commented on a diff in the pull request:
https://github.com/apache/spark/pull/10964#discussion_r51380304
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala
---
@@ -2056,4 +2056,11 @@ class SQLQuerySuite extends QueryTest
Github user tejasapatil commented on a diff in the pull request:
https://github.com/apache/spark/pull/10964#discussion_r51380186
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala
---
@@ -2056,4 +2056,11 @@ class SQLQuerySuite extends QueryTest
Github user tejasapatil commented on the pull request:
https://github.com/apache/spark/pull/10731#issuecomment-177531893
I would like to have this patch in trunk. If @zhichao-li is not planning to
do more changes, can one of the admins review ?
---
If your project is set up
Github user tejasapatil commented on the pull request:
https://github.com/apache/spark/pull/10849#issuecomment-176122957
@marmbrus : Updated the title and description.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well
Github user tejasapatil commented on the pull request:
https://github.com/apache/spark/pull/10849#issuecomment-175755363
ping !!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user tejasapatil commented on the pull request:
https://github.com/apache/spark/pull/10849#issuecomment-174873141
Fixed scala style test
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does
Github user tejasapatil commented on the pull request:
https://github.com/apache/spark/pull/10849#issuecomment-174125561
ping !!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user tejasapatil commented on the pull request:
https://github.com/apache/spark/pull/10849#issuecomment-173436587
@marmbrus : Makes sense. I have updated the diff with your suggestion.
---
If your project is set up for it, you can reply to this email and have your
reply appear
GitHub user tejasapatil opened a pull request:
https://github.com/apache/spark/pull/10849
[SPARK-12926][SQL] SQLContext to disallow users passing non-sql configs
Testing:
```
./bin/spark-shell
scala> sqlContext.sql("SET spark.sql.orc.filterPushdo
701 - 768 of 768 matches
Mail list logo
