[GitHub] spark issue #20742: [SPARK-23572][docs] Bring "security.md" up to date.

2018-03-16 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20742 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark pull request #20844: [SPARK-23707][SQL] Fresh 'initRange' name to avoi...

2018-03-16 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/20844#discussion_r175230004 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/basicPhysicalOperators.scala --- @@ -396,9 +396,11 @@ case class RangeExec(range:

[GitHub] spark issue #20742: [SPARK-23572][docs] Bring "security.md" up to date.

2018-03-16 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20742 **[Test build #88324 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/88324/testReport)** for PR 20742 at commit

[GitHub] spark issue #20803: [SPARK-23653][SQL] Show sql statement in spark SQL UI

2018-03-16 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/20803 Sorry I didn't clarify it clearly enough. I was not suggesting to show sql text for all of these cases, but tried to raise a discussion about when we should show sql text. e.g. for `df.count()`

[GitHub] spark issue #20579: [SPARK-23372][SQL] Writing empty struct in parquet fails...

2018-03-16 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/20579 After some more thoughts, we should try our best to not introduce behavior change to existing data sources. How about we only add this check for file-based data sources(all of them are built-in)?

[GitHub] spark issue #20726: [SPARK-23574][SQL] Report SinglePartition in DataSourceV...

2018-03-16 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/20726 LGTM, pending jenkins --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #20726: [SPARK-23574][SQL] Report SinglePartition in DataSourceV...

2018-03-16 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20726 **[Test build #88329 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/88329/testReport)** for PR 20726 at commit

[GitHub] spark issue #20726: [SPARK-23574][SQL] Report SinglePartition in DataSourceV...

2018-03-16 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20726 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #20726: [SPARK-23574][SQL] Report SinglePartition in DataSourceV...

2018-03-16 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20726 **[Test build #88328 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/88328/testReport)** for PR 20726 at commit

[GitHub] spark issue #20726: [SPARK-23574][SQL] Report SinglePartition in DataSourceV...

2018-03-16 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20726 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/88328/ Test FAILed. ---

[GitHub] spark issue #20726: [SPARK-23574][SQL] Report SinglePartition in DataSourceV...

2018-03-16 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20726 **[Test build #88328 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/88328/testReport)** for PR 20726 at commit

[GitHub] spark issue #20745: [SPARK-23288][SS] Fix output metrics with parquet sink

2018-03-16 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20745 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #20745: [SPARK-23288][SS] Fix output metrics with parquet sink

2018-03-16 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20745 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/88323/ Test PASSed. ---

[GitHub] spark issue #20767: [SPARK-23623] [SS] Avoid concurrent use of cached consum...

2018-03-16 Thread tedyu
Github user tedyu commented on the issue: https://github.com/apache/spark/pull/20767 I did a quick search for 'apache commons pool metrics' which didn't show up directly related links. --- - To unsubscribe, e-mail:

[GitHub] spark issue #20745: [SPARK-23288][SS] Fix output metrics with parquet sink

2018-03-16 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20745 **[Test build #88323 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/88323/testReport)** for PR 20745 at commit

[GitHub] spark pull request #20726: [SPARK-23574][SQL] Report SinglePartition in Data...

2018-03-16 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/20726#discussion_r175222350 --- Diff: sql/core/src/main/java/org/apache/spark/sql/sources/v2/reader/SupportsReportPartitioning.java --- @@ -23,6 +23,11 @@ /** * A mix in

[GitHub] spark issue #20767: [SPARK-23623] [SS] Avoid concurrent use of cached consum...

2018-03-16 Thread tdas
Github user tdas commented on the issue: https://github.com/apache/spark/pull/20767 @tedyu @zsxwing My thoughts on this is that we should consider migrating to something like Apache Common Pool (assuming it does not require additional maven libraries), which might be less maintenance

[GitHub] spark issue #20752: [SPARK-23559][SS] Create StreamingDataWriterFactory for ...

2018-03-16 Thread jose-torres
Github user jose-torres commented on the issue: https://github.com/apache/spark/pull/20752 There've been no comments on the doc. Should we move forward with this PR? --- - To unsubscribe, e-mail:

[GitHub] spark pull request #20796: [SPARK-23649][SQL] Prevent crashes on schema infe...

2018-03-16 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/20796#discussion_r175215287 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/csv/CSVSuite.scala --- @@ -1279,4 +1280,57 @@ class CSVSuite extends

[GitHub] spark issue #20343: [SPARK-23167][SQL] Add TPCDS queries v2.7 in TPCDSQueryS...

2018-03-16 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20343 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/88321/ Test PASSed. ---

[GitHub] spark issue #20343: [SPARK-23167][SQL] Add TPCDS queries v2.7 in TPCDSQueryS...

2018-03-16 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20343 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #20343: [SPARK-23167][SQL] Add TPCDS queries v2.7 in TPCDSQueryS...

2018-03-16 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20343 **[Test build #88321 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/88321/testReport)** for PR 20343 at commit

[GitHub] spark issue #20579: [SPARK-23372][SQL] Writing empty struct in parquet fails...

2018-03-16 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20579 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #20579: [SPARK-23372][SQL] Writing empty struct in parquet fails...

2018-03-16 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20579 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/1572/

[GitHub] spark issue #20579: [SPARK-23372][SQL] Writing empty struct in parquet fails...

2018-03-16 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20579 **[Test build #88327 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/88327/testReport)** for PR 20579 at commit

[GitHub] spark issue #20809: [SPARK-23667][CORE] Better scala version check

2018-03-16 Thread vanzin
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/20809 Can you provide more information in the bug report? e.g. a sample application and a sample error. I don't think this is the correct change, but without your use case I'm not sure what the

[GitHub] spark issue #20327: [SPARK-12963][CORE] NM host for driver end points

2018-03-16 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20327 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #20327: [SPARK-12963][CORE] NM host for driver end points

2018-03-16 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20327 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/88318/ Test PASSed. ---

[GitHub] spark issue #20327: [SPARK-12963][CORE] NM host for driver end points

2018-03-16 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20327 **[Test build #88318 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/88318/testReport)** for PR 20327 at commit

[GitHub] spark issue #20796: [SPARK-23649][SQL] Prevent crashes on schema inferring o...

2018-03-16 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20796 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #20796: [SPARK-23649][SQL] Prevent crashes on schema inferring o...

2018-03-16 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20796 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/88320/ Test FAILed. ---

[GitHub] spark issue #20796: [SPARK-23649][SQL] Prevent crashes on schema inferring o...

2018-03-16 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20796 **[Test build #88320 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/88320/testReport)** for PR 20796 at commit

[GitHub] spark issue #20659: [DO-NOT-MERGE] Try to update Hive to 2.3.2

2018-03-16 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20659 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #20659: [DO-NOT-MERGE] Try to update Hive to 2.3.2

2018-03-16 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20659 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/88317/ Test PASSed. ---

[GitHub] spark issue #20659: [DO-NOT-MERGE] Try to update Hive to 2.3.2

2018-03-16 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20659 **[Test build #88317 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/88317/testReport)** for PR 20659 at commit

[GitHub] spark issue #20807: SPARK-23660: Fix exception in yarn cluster mode when app...

2018-03-16 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20807 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #20807: SPARK-23660: Fix exception in yarn cluster mode when app...

2018-03-16 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20807 **[Test build #88326 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/88326/testReport)** for PR 20807 at commit

[GitHub] spark issue #20807: SPARK-23660: Fix exception in yarn cluster mode when app...

2018-03-16 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20807 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/88326/ Test PASSed. ---

[GitHub] spark issue #20742: [SPARK-23572][docs] Bring "security.md" up to date.

2018-03-16 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/20742 lgtm --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #20807: SPARK-23660: Fix exception in yarn cluster mode when app...

2018-03-16 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20807 **[Test build #88326 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/88326/testReport)** for PR 20807 at commit

[GitHub] spark issue #20807: SPARK-23660: Fix exception in yarn cluster mode when app...

2018-03-16 Thread gaborgsomogyi
Github user gaborgsomogyi commented on the issue: https://github.com/apache/spark/pull/20807 @vanzin sorry, one useless comment left in the code. Just removed. --- - To unsubscribe, e-mail:

[GitHub] spark issue #20726: [SPARK-23574][SQL] Report SinglePartition in DataSourceV...

2018-03-16 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20726 **[Test build #88325 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/88325/testReport)** for PR 20726 at commit

[GitHub] spark issue #20726: [SPARK-23574][SQL] Report SinglePartition in DataSourceV...

2018-03-16 Thread jose-torres
Github user jose-torres commented on the issue: https://github.com/apache/spark/pull/20726 Addressed last comment. Sorry it took me so long. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For

[GitHub] spark issue #20742: [SPARK-23572][docs] Bring "security.md" up to date.

2018-03-16 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20742 **[Test build #88324 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/88324/testReport)** for PR 20742 at commit

[GitHub] spark issue #20742: [SPARK-23572][docs] Bring "security.md" up to date.

2018-03-16 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20742 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/1571/

[GitHub] spark issue #20742: [SPARK-23572][docs] Bring "security.md" up to date.

2018-03-16 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20742 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #20745: [SPARK-23288][SS] Fix output metrics with parquet sink

2018-03-16 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20745 **[Test build #88323 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/88323/testReport)** for PR 20745 at commit

[GitHub] spark pull request #20745: [SPARK-23288][SS] Fix output metrics with parquet...

2018-03-16 Thread gaborgsomogyi
Github user gaborgsomogyi commented on a diff in the pull request: https://github.com/apache/spark/pull/20745#discussion_r175186820 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/streaming/FileStreamSinkSuite.scala --- @@ -405,4 +406,52 @@ class FileStreamSinkSuite

[GitHub] spark issue #17774: [SPARK-18371][Streaming] Spark Streaming backpressure ge...

2018-03-16 Thread koeninger
Github user koeninger commented on the issue: https://github.com/apache/spark/pull/17774 merged to master Thanks @arzt ! --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark pull request #20745: [SPARK-23288][SS] Fix output metrics with parquet...

2018-03-16 Thread gaborgsomogyi
Github user gaborgsomogyi commented on a diff in the pull request: https://github.com/apache/spark/pull/20745#discussion_r175186800 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/streaming/FileStreamSinkSuite.scala --- @@ -405,4 +406,52 @@ class FileStreamSinkSuite

[GitHub] spark pull request #20777: [SPARK-23615][ML][PYSPARK]Add maxDF Parameter to ...

2018-03-16 Thread BryanCutler
Github user BryanCutler commented on a diff in the pull request: https://github.com/apache/spark/pull/20777#discussion_r175184951 --- Diff: mllib/src/main/scala/org/apache/spark/ml/feature/CountVectorizer.scala --- @@ -70,19 +70,21 @@ private[feature] trait CountVectorizerParams

[GitHub] spark pull request #20777: [SPARK-23615][ML][PYSPARK]Add maxDF Parameter to ...

2018-03-16 Thread BryanCutler
Github user BryanCutler commented on a diff in the pull request: https://github.com/apache/spark/pull/20777#discussion_r175184795 --- Diff: mllib/src/main/scala/org/apache/spark/ml/feature/CountVectorizer.scala --- @@ -70,19 +70,21 @@ private[feature] trait CountVectorizerParams

[GitHub] spark pull request #20777: [SPARK-23615][ML][PYSPARK]Add maxDF Parameter to ...

2018-03-16 Thread BryanCutler
Github user BryanCutler commented on a diff in the pull request: https://github.com/apache/spark/pull/20777#discussion_r175184503 --- Diff: mllib/src/main/scala/org/apache/spark/ml/feature/CountVectorizer.scala --- @@ -70,19 +70,21 @@ private[feature] trait CountVectorizerParams

[GitHub] spark pull request #20816: [SPARK-21479][SQL] Outer join filter pushdown in ...

2018-03-16 Thread maryannxue
Github user maryannxue commented on a diff in the pull request: https://github.com/apache/spark/pull/20816#discussion_r175184304 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala --- @@ -669,11 +672,42 @@ object

[GitHub] spark pull request #16770: [SPARK-15009][PYTHON][ML] Construct a CountVector...

2018-03-16 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/16770 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16770: [SPARK-15009][PYTHON][ML] Construct a CountVectorizerMod...

2018-03-16 Thread holdenk
Github user holdenk commented on the issue: https://github.com/apache/spark/pull/16770 Looks good to me, merged to master. Thanks! --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #20847: [SPARK-23644][CORE][UI][BACKPORT-2.3] Use absolute path ...

2018-03-16 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20847 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #20847: [SPARK-23644][CORE][UI][BACKPORT-2.3] Use absolute path ...

2018-03-16 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20847 **[Test build #88322 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/88322/testReport)** for PR 20847 at commit

[GitHub] spark issue #20847: [SPARK-23644][CORE][UI][BACKPORT-2.3] Use absolute path ...

2018-03-16 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20847 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/1570/

[GitHub] spark issue #20847: [SPARK-23644][CORE][UI][BACKPORT-2.3] Use absolute path ...

2018-03-16 Thread kiszk
Github user kiszk commented on the issue: https://github.com/apache/spark/pull/20847 retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark pull request #20788: [SPARK-23647][PYTHON][SQL] Adds more types for hi...

2018-03-16 Thread DylanGuedes
Github user DylanGuedes commented on a diff in the pull request: https://github.com/apache/spark/pull/20788#discussion_r175181523 --- Diff: python/pyspark/sql/dataframe.py --- @@ -437,10 +437,12 @@ def hint(self, name, *parameters): if not isinstance(name, str):

[GitHub] spark pull request #16770: [SPARK-15009][PYTHON][ML] Construct a CountVector...

2018-03-16 Thread holdenk
Github user holdenk commented on a diff in the pull request: https://github.com/apache/spark/pull/16770#discussion_r175177201 --- Diff: python/pyspark/ml/feature.py --- @@ -497,66 +544,61 @@ def setMinTF(self, value): """ return self._set(minTF=value)

[GitHub] spark pull request #16770: [SPARK-15009][PYTHON][ML] Construct a CountVector...

2018-03-16 Thread holdenk
Github user holdenk commented on a diff in the pull request: https://github.com/apache/spark/pull/16770#discussion_r175179623 --- Diff: python/pyspark/ml/feature.py --- @@ -565,6 +607,20 @@ def vocabulary(self): """ return self._call_java("vocabulary")

[GitHub] spark pull request #16770: [SPARK-15009][PYTHON][ML] Construct a CountVector...

2018-03-16 Thread holdenk
Github user holdenk commented on a diff in the pull request: https://github.com/apache/spark/pull/16770#discussion_r175180172 --- Diff: python/pyspark/ml/tests.py --- @@ -1980,8 +1997,8 @@ def test_java_params(self): pyspark.ml.regression] for

[GitHub] spark pull request #16770: [SPARK-15009][PYTHON][ML] Construct a CountVector...

2018-03-16 Thread holdenk
Github user holdenk commented on a diff in the pull request: https://github.com/apache/spark/pull/16770#discussion_r175179845 --- Diff: python/pyspark/ml/tests.py --- @@ -640,6 +640,34 @@ def test_count_vectorizer_with_binary(self): feature, expected = r

[GitHub] spark pull request #20745: [SPARK-23288][SS] Fix output metrics with parquet...

2018-03-16 Thread zsxwing
Github user zsxwing commented on a diff in the pull request: https://github.com/apache/spark/pull/20745#discussion_r175178609 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/streaming/FileStreamSinkSuite.scala --- @@ -405,4 +406,52 @@ class FileStreamSinkSuite extends

[GitHub] spark pull request #20745: [SPARK-23288][SS] Fix output metrics with parquet...

2018-03-16 Thread zsxwing
Github user zsxwing commented on a diff in the pull request: https://github.com/apache/spark/pull/20745#discussion_r175178230 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/streaming/FileStreamSinkSuite.scala --- @@ -405,4 +406,52 @@ class FileStreamSinkSuite extends

[GitHub] spark issue #20796: [SPARK-23649][SQL] Prevent crashes on schema inferring o...

2018-03-16 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20796 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #20796: [SPARK-23649][SQL] Prevent crashes on schema inferring o...

2018-03-16 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20796 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/88316/ Test FAILed. ---

[GitHub] spark issue #20767: [SPARK-23623] [SS] Avoid concurrent use of cached consum...

2018-03-16 Thread zsxwing
Github user zsxwing commented on the issue: https://github.com/apache/spark/pull/20767 @tedyu that's a good idea. Could you create a ticket? Thanks! --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org

[GitHub] spark issue #20705: [SPARK-23553][TESTS] Tests should not assume the default...

2018-03-16 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/20705 Thank you, @gatorsmile , @HyukjinKwon , @bersprockets . --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org

[GitHub] spark issue #20796: [SPARK-23649][SQL] Prevent crashes on schema inferring o...

2018-03-16 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20796 **[Test build #88316 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/88316/testReport)** for PR 20796 at commit

[GitHub] spark issue #20767: [SPARK-23623] [SS] Avoid concurrent use of cached consum...

2018-03-16 Thread tedyu
Github user tedyu commented on the issue: https://github.com/apache/spark/pull/20767 @tdas Do you think a follow on JIRA can be logged for adding metrics for the cache operations ? Thanks --- - To

[GitHub] spark pull request #20767: [SPARK-23623] [SS] Avoid concurrent use of cached...

2018-03-16 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/20767 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #20742: [SPARK-23572][docs] Bring "security.md" up to dat...

2018-03-16 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/20742#discussion_r175171592 --- Diff: docs/security.md --- @@ -3,47 +3,291 @@ layout: global displayTitle: Spark Security title: Security --- +* This will become a

[GitHub] spark pull request #20742: [SPARK-23572][docs] Bring "security.md" up to dat...

2018-03-16 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/20742#discussion_r175173523 --- Diff: docs/security.md --- @@ -182,54 +582,70 @@ configure those ports. -### HTTP Security Headers -Apache Spark can be

[GitHub] spark pull request #20742: [SPARK-23572][docs] Bring "security.md" up to dat...

2018-03-16 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/20742#discussion_r175170426 --- Diff: docs/security.md --- @@ -3,47 +3,291 @@ layout: global displayTitle: Spark Security title: Security --- +* This will become a

[GitHub] spark issue #20767: [SPARK-23623] [SS] Avoid concurrent use of cached consum...

2018-03-16 Thread zsxwing
Github user zsxwing commented on the issue: https://github.com/apache/spark/pull/20767 @tdas this is much simpler!!! LGTM. Merging to master. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For

[GitHub] spark issue #20847: [SPARK-23644][CORE][UI][BACKPORT-2.3] Use absolute path ...

2018-03-16 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20847 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #20847: [SPARK-23644][CORE][UI][BACKPORT-2.3] Use absolute path ...

2018-03-16 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20847 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/88315/ Test FAILed. ---

[GitHub] spark issue #20847: [SPARK-23644][CORE][UI][BACKPORT-2.3] Use absolute path ...

2018-03-16 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20847 **[Test build #88315 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/88315/testReport)** for PR 20847 at commit

[GitHub] spark issue #20343: [SPARK-23167][SQL] Add TPCDS queries v2.7 in TPCDSQueryS...

2018-03-16 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20343 **[Test build #88321 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/88321/testReport)** for PR 20343 at commit

[GitHub] spark pull request #20822: [SPARK-23680] Fix entrypoint.sh to properly suppo...

2018-03-16 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/20822 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20822: [SPARK-23680] Fix entrypoint.sh to properly support Arbi...

2018-03-16 Thread erikerlandson
Github user erikerlandson commented on the issue: https://github.com/apache/spark/pull/20822 Merged to master --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #19041: [SPARK-21097][CORE] Add option to recover cached data

2018-03-16 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/19041 Thanks @brad-kaiser -- want to re-iterate my comment from Feb 2nd, I think that is really the most important part to address before getting into the details of the current implementation: >

[GitHub] spark issue #19431: [SPARK-18580] [DStreams] [external/kafka-0-10][external/...

2018-03-16 Thread koeninger
Github user koeninger commented on the issue: https://github.com/apache/spark/pull/19431 @akonopko thanks for this, if you can resolve merge conflict I think we can get this in --- - To unsubscribe, e-mail:

[GitHub] spark pull request #19041: [SPARK-21097][CORE] Add option to recover cached ...

2018-03-16 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/19041#discussion_r175164254 --- Diff: core/src/main/scala/org/apache/spark/storage/BlockManagerMasterEndpoint.scala --- @@ -246,6 +251,38 @@ class BlockManagerMasterEndpoint(

[GitHub] spark pull request #20841: [SPARK-23706][PYTHON] spark.conf.get(value, defau...

2018-03-16 Thread ueshin
Github user ueshin commented on a diff in the pull request: https://github.com/apache/spark/pull/20841#discussion_r175163502 --- Diff: python/pyspark/sql/tests.py --- @@ -2504,6 +2504,10 @@ def test_conf(self): spark.conf.unset("bogo")

[GitHub] spark issue #20807: SPARK-23660: Fix exception in yarn cluster mode when app...

2018-03-16 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20807 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #20807: SPARK-23660: Fix exception in yarn cluster mode when app...

2018-03-16 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20807 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/88319/ Test PASSed. ---

[GitHub] spark issue #20807: SPARK-23660: Fix exception in yarn cluster mode when app...

2018-03-16 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20807 **[Test build #88319 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/88319/testReport)** for PR 20807 at commit

[GitHub] spark pull request #20750: [SPARK-23581][SQL] Add interpreted unsafe project...

2018-03-16 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/20750 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #17774: [SPARK-18371][Streaming] Spark Streaming backpres...

2018-03-16 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/17774 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20796: [SPARK-23649][SQL] Prevent crashes on schema inferring o...

2018-03-16 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20796 **[Test build #88320 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/88320/testReport)** for PR 20796 at commit

[GitHub] spark issue #20750: [SPARK-23581][SQL] Add interpreted unsafe projection

2018-03-16 Thread hvanhovell
Github user hvanhovell commented on the issue: https://github.com/apache/spark/pull/20750 @cloud-fan has some issue with his mac, so I will be merging :)... Thanks for the reviews! --- - To unsubscribe,

[GitHub] spark issue #20796: [SPARK-23649][SQL] Prevent crashes on schema inferring o...

2018-03-16 Thread MaxGekk
Github user MaxGekk commented on the issue: https://github.com/apache/spark/pull/20796 test this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #20750: [SPARK-23581][SQL] Add interpreted unsafe projection

2018-03-16 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/20750 thanks, merging to master! --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #20046: [SPARK-22362][SQL] Add unit test for Window Aggregate Fu...

2018-03-16 Thread attilapiros
Github user attilapiros commented on the issue: https://github.com/apache/spark/pull/20046 gentle ping @gatorsmile @hvanhovell --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark pull request #20788: [SPARK-23647][PYTHON][SQL] Adds more types for hi...

2018-03-16 Thread ueshin
Github user ueshin commented on a diff in the pull request: https://github.com/apache/spark/pull/20788#discussion_r175156065 --- Diff: python/pyspark/sql/dataframe.py --- @@ -437,10 +437,12 @@ def hint(self, name, *parameters): if not isinstance(name, str):

[GitHub] spark pull request #20795: [SPARK-23486]cache the function name from the cat...

2018-03-16 Thread kevinyu98
Github user kevinyu98 commented on a diff in the pull request: https://github.com/apache/spark/pull/20795#discussion_r175159061 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala --- @@ -1192,11 +1195,24 @@ class Analyzer( * @see

<    1   2   3   4   5   >