[GitHub] spark issue #22470: [SPARK-25454][SQL] should not generate negative scale as...

[mgaido91]

Github user mgaido91 commented on the issue:

https://github.com/apache/spark/pull/22470
  
@cloud-fan I think the main problem about this (and it is the reason why I 
haven't proposed it) is that the range of operations supported would be 
smaller, so we may forbid operations which now can happen. For instance, the 
following code has been always working on Spark: `lit(BigDecimal(1e36)) * 
lit(BigDecimal(1))`. Indeed now this would become a `decimal(6, -36)`. With 
your change, this is going to be a `decimal(42, 0)` which is out of the range 
of supported values (ie. an overflow would occur).

I am not sure if any user has something like this, but it is possible and I 
think we cannot exclude it. We may, though, restrict again the condition when 
it happens, ie. in case we are just parsing from a literal we can avoid 
returning a negative scale. But the other fix would be needed anyway in this 
case, as we could still have to deal with negative scales, so IMO this would be 
quite useless.

I'd agree though about forbidding negative scale in 3.0.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22465: [SPARK-25457][SQL] IntegralDivide returns data type of t...

[mgaido91]

Github user mgaido91 commented on the issue:

https://github.com/apache/spark/pull/22465
  
retest this please


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22173: [SPARK-24335] Spark external shuffle server improvement ...

[tgravescs]

Github user tgravescs commented on the issue:

https://github.com/apache/spark/pull/22173
  
also please fix jira number, should be SPARK-24355


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22413: [SPARK-25425][SQL] Extra options should override session...

[MaxGekk]

Github user MaxGekk commented on the issue:

https://github.com/apache/spark/pull/22413
  
@dongjoon-hyun It cannot be merged to 2.3 easily because 
`DataSourceV2Relation` doesn't have the `options` field, and the test for read 
is not compilable. I will try to fix it tomorrow.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22460: DO NOT MERGE

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/22460
  
**[Test build #96292 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96292/testReport)**
 for PR 22460 at commit 
[`53387f0`](https://github.com/apache/spark/commit/53387f05eecc5a9a2d13254ce9bea1cd4102c96c).


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22173: [SPARK-24355] Spark external shuffle server improvement ...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/22173
  
**[Test build #96293 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96293/testReport)**
 for PR 22173 at commit 
[`40cfbed`](https://github.com/apache/spark/commit/40cfbed70bd51e30ac451cb2204f34c7105fa15f).


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22173: [SPARK-24355] Spark external shuffle server improvement ...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/22173
  
**[Test build #96291 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96291/testReport)**
 for PR 22173 at commit 
[`574ba81`](https://github.com/apache/spark/commit/574ba81abf3e1d71bac84a83b80e4b67056f7442).
 * This patch **fails Java style tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22173: [SPARK-24355] Spark external shuffle server improvement ...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/22173
  
Merged build finished. Test FAILed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22173: [SPARK-24355] Spark external shuffle server improvement ...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/22173
  
**[Test build #96264 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96264/testReport)**
 for PR 22173 at commit 
[`e7b47e9`](https://github.com/apache/spark/commit/e7b47e9c37e42e8de251f9f91d9f85428ea7df73).
 * This patch **fails from timeout after a configured wait of `400m`**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22173: [SPARK-24355] Spark external shuffle server improvement ...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/22173
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/96264/
Test FAILed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22173: [SPARK-24355] Spark external shuffle server improvement ...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/22173
  
Merged build finished. Test FAILed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22470: [SPARK-25454][SQL] should not generate negative scale as...

[dilipbiswal]

Github user dilipbiswal commented on the issue:

https://github.com/apache/spark/pull/22470
  
@mgaido91 @cloud-fan On the other hand .. some use cases may work better 
:-) , for example
Before 
```
scala> spark.sql("create table dec as select (1e36 * 1) as col1")
org.apache.spark.SparkException: Cannot recognize hive type string: 
decimal(3,-36)
  at 
org.apache.spark.sql.hive.client.HiveClientImpl$.org$apache$spark$sql$hive$client$HiveClientImpl$$getSparkSQLDataType(HiveClientImpl.scala:883)
  at 
org.apache.spark.sql.hive.client.HiveClientImpl$$anonfun$org$apache$spark$sql$hive$client$HiveClientImpl$$verifyColumnDataType$1.apply(HiveClientImpl.scala:905)

with this pr
```
scala> spark.sql("create table dec as select (1e36 * 1) as col1")
18/09/19 14:29:29 WARN HiveMetaStore: Location: 
file:/user/hive/warehouse/dec specified for non-external table:dec
18/09/19 14:29:30 WARN ObjectStore: Failed to get database global_temp, 
returning NoSuchObjectException
res0: org.apache.spark.sql.DataFrame = []
scala> spark.sql("describe table dec").show
++-+---+
|col_name|data_type|comment|
++-+---+
|col1|decimal(38,0)|   null|
++-+---+
```
Perhaps we may have issues writing out data frames containing decimal with 
negative scale to file based datasources as well. I have not verified though. 


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22475: [SPARK-4502][SQL] spark.sql.optimizer.nestedSchemaPrunin...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/22475
  
**[Test build #96298 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96298/testReport)**
 for PR 22475 at commit 
[`5159883`](https://github.com/apache/spark/commit/5159883f5b4a65ac8ecec8b0368e172680aa6897).


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22476: [SPARK-24157] spark.sql.streaming.noDataMicroBatches.ena...

[rxin]

Github user rxin commented on the issue:

https://github.com/apache/spark/pull/22476
  
cc @tdas @marmbrus @jose-torres 


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #22477: [SPARK-25471][PYTHON][TEST] Fix pyspark-sql test ...

[BryanCutler]

GitHub user BryanCutler opened a pull request:

https://github.com/apache/spark/pull/22477

[SPARK-25471][PYTHON][TEST] Fix pyspark-sql test error when using Python 
3.6 and Pandas 0.23

## What changes were proposed in this pull request?

Fix test that constructs a Pandas DataFrame by specifying the column order. 
Previously this test assumed the columns would be sorted alphabetically, 
however when using Python 3.6 with Pandas 0.23 or higher, the original column 
order is maintained.

Manually tested with `python/run-tests` using Python 3.6.6 and Pandas 0.23.4


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/BryanCutler/spark 
pyspark-tests-py36-pd23-SPARK-25471

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/22477.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #22477


commit a268250b951a307f216ce4bb6bfd53aa5417bbfa
Author: Bryan Cutler 
Date:   2018-09-19T21:35:49Z

specify column ordering




---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #22476: [SPARK-24157] spark.sql.streaming.noDataMicroBatc...

[rxin]

GitHub user rxin opened a pull request:

https://github.com/apache/spark/pull/22476

[SPARK-24157] spark.sql.streaming.noDataMicroBatches.enabled

## What changes were proposed in this pull request?
This patch changes the config option 
`spark.sql.streaming.noDataMicroBatchesEnabled` to 
`spark.sql.streaming.noDataMicroBatches.enabled` to be more consistent with 
rest of the configs. Unfortunately there is one streaming config called 
`spark.sql.streaming.metricsEnabled`. For that one we should just use a 
fallback config and change it in a separate patch.

## How was this patch tested?
Made sure no other references to this config are in the code base:
```
> git grep "noDataMicro"
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala:
buildConf("spark.sql.streaming.noDataMicroBatches.enabled")
```

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/rxin/spark SPARK-24157

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/22476.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #22476


commit 37716c836a20a684ef7425addd5f43cf10cd857f
Author: Reynold Xin 
Date:   2018-09-19T21:40:33Z

[SPARK-24157] spark.sql.streaming.noDataMicroBatches.enabled




---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22475: [SPARK-4502][SQL] spark.sql.optimizer.nestedSchemaPrunin...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/22475
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 

https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/3262/
Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22475: [SPARK-4502][SQL] spark.sql.optimizer.nestedSchemaPrunin...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/22475
  
Merged build finished. Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22477: [SPARK-25471][PYTHON][TEST] Fix pyspark-sql test error w...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/22477
  
**[Test build #96299 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96299/testReport)**
 for PR 22477 at commit 
[`a268250`](https://github.com/apache/spark/commit/a268250b951a307f216ce4bb6bfd53aa5417bbfa).


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22476: [SPARK-24157] spark.sql.streaming.noDataMicroBatches.ena...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/22476
  
Merged build finished. Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22449: [SPARK-22666][ML][FOLLOW-UP] Improve testcase to tolerat...

[mengxr]

Github user mengxr commented on the issue:

https://github.com/apache/spark/pull/22449
  
LGTM. Merged into master and branch-2.4. Thanks!


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22376: [SPARK-25021][K8S][BACKPORT] Add spark.executor.pyspark....

[holdenk]

Github user holdenk commented on the issue:

https://github.com/apache/spark/pull/22376
  
For future reference the original PR is at 
https://github.com/apache/spark/pull/22298/files/fe8cc5aa6759cdf893e11c3d83814f8dffddce9c
 


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22376: [SPARK-25021][K8S][BACKPORT] Add spark.executor.pyspark....

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/22376
  
Kubernetes integration test starting
URL: 
https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/3265/



---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22477: [SPARK-25471][PYTHON][TEST] Fix pyspark-sql test error w...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/22477
  
**[Test build #96306 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96306/testReport)**
 for PR 22477 at commit 
[`a268250`](https://github.com/apache/spark/commit/a268250b951a307f216ce4bb6bfd53aa5417bbfa).


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22478: [SPARK-25472] Don't have legitimate stops of streams cau...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/22478
  
Merged build finished. Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22173: [SPARK-24355] Spark external shuffle server improvement ...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/22173
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/96273/
Test FAILed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22173: [SPARK-24355] Spark external shuffle server improvement ...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/22173
  
Merged build finished. Test FAILed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22173: [SPARK-24355] Spark external shuffle server improvement ...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/22173
  
**[Test build #96273 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96273/testReport)**
 for PR 22173 at commit 
[`1ea0655`](https://github.com/apache/spark/commit/1ea0655c9001fb8cd257ee5d677d4324658dfe28).
 * This patch **fails from timeout after a configured wait of `400m`**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22477: [SPARK-25471][PYTHON][TEST] Fix pyspark-sql test error w...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/22477
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/96306/
Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22478: [SPARK-25472] Don't have legitimate stops of streams cau...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/22478
  
**[Test build #96308 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96308/testReport)**
 for PR 22478 at commit 
[`3b8addb`](https://github.com/apache/spark/commit/3b8addb9cf02489978594505470fdd527a35c2a7).


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22477: [SPARK-25471][PYTHON][TEST] Fix pyspark-sql test error w...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/22477
  
Merged build finished. Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22477: [SPARK-25471][PYTHON][TEST] Fix pyspark-sql test error w...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/22477
  
**[Test build #96306 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96306/testReport)**
 for PR 22477 at commit 
[`a268250`](https://github.com/apache/spark/commit/a268250b951a307f216ce4bb6bfd53aa5417bbfa).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22478: [SPARK-25472] Don't have legitimate stops of streams cau...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/22478
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 

https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/3269/
Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22461: [SPARK-25453][SQL][TEST] OracleIntegrationSuite IllegalA...

[HyukjinKwon]

Github user HyukjinKwon commented on the issue:

https://github.com/apache/spark/pull/22461
  
Oh, I am not aware of it.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22325: [SPARK-25318]. Add exception handling when wrapping the ...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/22325
  
**[Test build #96285 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96285/testReport)**
 for PR 22325 at commit 
[`4983d69`](https://github.com/apache/spark/commit/4983d69abf48594e9876d7a40f3f531836333243).


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22173: [SPARK-24335] Spark external shuffle server improvement ...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/22173
  
Merged build finished. Test FAILed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22173: [SPARK-24335] Spark external shuffle server improvement ...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/22173
  
**[Test build #96255 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96255/testReport)**
 for PR 22173 at commit 
[`f05924a`](https://github.com/apache/spark/commit/f05924a52f8dcb0c8acc928108628d9ddf5c33d9).
 * This patch **fails from timeout after a configured wait of `400m`**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22173: [SPARK-24335] Spark external shuffle server improvement ...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/22173
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/96255/
Test FAILed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22173: [SPARK-24355] Spark external shuffle server improvement ...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/22173
  
**[Test build #96259 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96259/testReport)**
 for PR 22173 at commit 
[`573033c`](https://github.com/apache/spark/commit/573033c5b42abf9220b6bf656b4c2f04ea615ab7).
 * This patch **fails from timeout after a configured wait of `400m`**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22138: [SPARK-25151][SS] Apply Apache Commons Pool to KafkaData...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/22138
  
**[Test build #96295 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96295/testReport)**
 for PR 22138 at commit 
[`da5bd7f`](https://github.com/apache/spark/commit/da5bd7fd79a0ce41ca7301ecdc4e375c897e8036).


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22173: [SPARK-24355] Spark external shuffle server improvement ...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/22173
  
Merged build finished. Test FAILed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22173: [SPARK-24355] Spark external shuffle server improvement ...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/22173
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/96259/
Test FAILed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22460: DO NOT MERGE

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/22460
  
**[Test build #96296 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96296/testReport)**
 for PR 22460 at commit 
[`4106040`](https://github.com/apache/spark/commit/410604012cbd1c9e7c284a1e05f95b3827c728a5).


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22472: [SPARK-23173][SQL] Reverting of spark.sql.fromJsonForceN...

[dongjoon-hyun]

Github user dongjoon-hyun commented on the issue:

https://github.com/apache/spark/pull/22472
  
Thank you, @rxin . Then, I'll proceed it .


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #22402: [SPARK-25414][SS][TEST] make it clear that the nu...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/spark/pull/22402


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #22475: [SPARK-4502][SQL] spark.sql.optimizer.nestedSchem...

[rxin]

GitHub user rxin opened a pull request:

https://github.com/apache/spark/pull/22475

[SPARK-4502][SQL] spark.sql.optimizer.nestedSchemaPruning.enabled

## What changes were proposed in this pull request?
This patch adds an "optimizer" prefix to nested schema pruning.

## How was this patch tested?
Should be covered by existing tests.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/rxin/spark SPARK-4502

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/22475.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #22475


commit 5159883f5b4a65ac8ecec8b0368e172680aa6897
Author: Reynold Xin 
Date:   2018-09-19T21:37:08Z

[SPARK-4502][SQL] spark.sql.optimizer.nestedSchemaPruning.enabled




---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22475: [SPARK-4502][SQL] spark.sql.optimizer.nestedSchemaPrunin...

[rxin]

Github user rxin commented on the issue:

https://github.com/apache/spark/pull/22475
  
cc @cloud-fan 


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22138: [SPARK-25151][SS] Apply Apache Commons Pool to KafkaData...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/22138
  
**[Test build #96300 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96300/testReport)**
 for PR 22138 at commit 
[`ddd4f2f`](https://github.com/apache/spark/commit/ddd4f2fc38c42dd1b781b0e3df46432bb6829e7b).


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22376: [SPARK-25021][K8S][BACKPORT] Add spark.executor.pyspark....

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/22376
  
Merged build finished. Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22376: [SPARK-25021][K8S][BACKPORT] Add spark.executor.pyspark....

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/22376
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 

https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/3265/
Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22376: [SPARK-25021][K8S][BACKPORT] Add spark.executor.pyspark....

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/22376
  
Kubernetes integration test status success
URL: 
https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/3265/



---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22473: [SPARK-25449][CORE] Heartbeat shouldn't include accumula...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/22473
  
**[Test build #96303 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96303/testReport)**
 for PR 22473 at commit 
[`6bb91b0`](https://github.com/apache/spark/commit/6bb91b01571601e22566d3c4084dc16f1f3f8f61).


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22376: [SPARK-25021][K8S][BACKPORT] Add spark.executor.pyspark....

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/22376
  
**[Test build #96304 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96304/testReport)**
 for PR 22376 at commit 
[`7c20ed9`](https://github.com/apache/spark/commit/7c20ed94955a93c1586ff9c57981f2085fb7a8ee).


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22376: [SPARK-25021][K8S][BACKPORT] Add spark.executor.pyspark....

[holdenk]

Github user holdenk commented on the issue:

https://github.com/apache/spark/pull/22376
  
Jenkins retest this please


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22476: [SPARK-24157][SS][FOLLOWUP] Rename to spark.sql.streamin...

[tdas]

Github user tdas commented on the issue:

https://github.com/apache/spark/pull/22476
  
LGTM.
Please make sure to merge it to 2.4


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22478: [SPARK-25472] Don't have legitimate stops of streams cau...

[zsxwing]

Github user zsxwing commented on the issue:

https://github.com/apache/spark/pull/22478
  
LGTM pending tests. Could you add `[SS]` to your title?


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22478: [SPARK-25472][SS] Don't have legitimate stops of streams...

[jose-torres]

Github user jose-torres commented on the issue:

https://github.com/apache/spark/pull/22478
  
Lgtm pending tests

On Wed, Sep 19, 2018 at 5:16 PM Shixiong Zhu 
wrote:

> LGTM pending tests. Could you add [SS] to your title?
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> , or 
mute
> the thread
> 

> .
>



---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22465: [SPARK-25457][SQL] IntegralDivide returns data type of t...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/22465
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/96287/
Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22465: [SPARK-25457][SQL] IntegralDivide returns data type of t...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/22465
  
Merged build finished. Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22476: [SPARK-24157][SS][FOLLOWUP] Rename to spark.sql.streamin...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/22476
  
**[Test build #96297 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96297/testReport)**
 for PR 22476 at commit 
[`37716c8`](https://github.com/apache/spark/commit/37716c836a20a684ef7425addd5f43cf10cd857f).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22470: [SPARK-25454][SQL] should not generate negative scale as...

[cloud-fan]

Github user cloud-fan commented on the issue:

https://github.com/apache/spark/pull/22470
  
@mgaido91 you are right, this still has behavior changes if the intermedia 
result exceed the max precision. Since most of the storages don't support 
negative scale(hive, parquet, etc.), I think we should eventually forbid it 
too. Let's move the discussion to #22450


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22173: [SPARK-24355] Spark external shuffle server improvement ...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/22173
  
**[Test build #96291 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96291/testReport)**
 for PR 22173 at commit 
[`574ba81`](https://github.com/apache/spark/commit/574ba81abf3e1d71bac84a83b80e4b67056f7442).


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22447: [SPARK-25450][SQL] PushProjectThroughUnion rule uses the...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/22447
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/96280/
Test FAILed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22447: [SPARK-25450][SQL] PushProjectThroughUnion rule uses the...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/22447
  
**[Test build #96280 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96280/testReport)**
 for PR 22447 at commit 
[`c7756ed`](https://github.com/apache/spark/commit/c7756edc46d77ce8fc9b846e91b37a9fcef97600).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22460: DO NOT MERGE

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/22460
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 

https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/3261/
Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22460: DO NOT MERGE

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/22460
  
Merged build finished. Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22477: [SPARK-25471][PYTHON][TEST] Fix pyspark-sql test error w...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/22477
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/96299/
Test FAILed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22473: [SPARK-25449][CORE] Heartbeat shouldn't include accumula...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/22473
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/96301/
Test FAILed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22477: [SPARK-25471][PYTHON][TEST] Fix pyspark-sql test error w...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/22477
  
**[Test build #96299 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96299/testReport)**
 for PR 22477 at commit 
[`a268250`](https://github.com/apache/spark/commit/a268250b951a307f216ce4bb6bfd53aa5417bbfa).
 * This patch **fails PySpark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22477: [SPARK-25471][PYTHON][TEST] Fix pyspark-sql test error w...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/22477
  
Merged build finished. Test FAILed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22173: [SPARK-24355] Spark external shuffle server improvement ...

[redsanket]

Github user redsanket commented on the issue:

https://github.com/apache/spark/pull/22173
  
test this please


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22173: [SPARK-24355] Spark external shuffle server improvement ...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/22173
  
**[Test build #96305 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96305/testReport)**
 for PR 22173 at commit 
[`4c5d6f1`](https://github.com/apache/spark/commit/4c5d6f1c2e4f87e74de07435eb4bb196bd150cd0).


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22472: [SPARK-23173][SQL] Reverting of spark.sql.fromJsonForceN...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/22472
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/96278/
Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22472: [SPARK-23173][SQL] Reverting of spark.sql.fromJsonForceN...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/22472
  
Merged build finished. Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22474: [SPARK-25425][SQL][BACKPORT-2.4] Extra options should ov...

[dongjoon-hyun]

Github user dongjoon-hyun commented on the issue:

https://github.com/apache/spark/pull/22474
  
Since it's merged, could you close this PR?


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22325: [SPARK-25318]. Add exception handling when wrapping the ...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/22325
  
Merged build finished. Test FAILed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22325: [SPARK-25318]. Add exception handling when wrapping the ...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/22325
  
**[Test build #96285 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96285/testReport)**
 for PR 22325 at commit 
[`4983d69`](https://github.com/apache/spark/commit/4983d69abf48594e9876d7a40f3f531836333243).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22475: [SPARK-4502][SQL] Rename to spark.sql.optimizer.nestedSc...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/22475
  
**[Test build #96298 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96298/testReport)**
 for PR 22475 at commit 
[`5159883`](https://github.com/apache/spark/commit/5159883f5b4a65ac8ecec8b0368e172680aa6897).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22475: [SPARK-4502][SQL] Rename to spark.sql.optimizer.nestedSc...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/22475
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 

https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/3270/
Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22475: [SPARK-4502][SQL] Rename to spark.sql.optimizer.nestedSc...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/22475
  
Merged build finished. Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22460: DO NOT MERGE

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/22460
  
Merged build finished. Test FAILed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22460: DO NOT MERGE

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/22460
  
**[Test build #4344 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/4344/testReport)**
 for PR 22460 at commit 
[`4106040`](https://github.com/apache/spark/commit/410604012cbd1c9e7c284a1e05f95b3827c728a5).


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #22465: [SPARK-25457][SQL] IntegralDivide returns data ty...

[cloud-fan]

Github user cloud-fan commented on a diff in the pull request:

https://github.com/apache/spark/pull/22465#discussion_r219011641
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala ---
@@ -1561,6 +1561,13 @@ object SQLConf {
 "are performed before any UNION, EXCEPT and MINUS operations.")
   .booleanConf
   .createWithDefault(false)
+
+  val LEGACY_INTEGRALDIVIDE_RETURN_LONG = 
buildConf("spark.sql.legacy.integralDivide.returnBigint")
--- End diff --

`big int` is more commonly used in the SQL world


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22460: DO NOT MERGE

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/22460
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 

https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/3271/
Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22460: DO NOT MERGE

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/22460
  
Merged build finished. Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22476: [SPARK-24157][SS][FOLLOWUP] Rename to spark.sql.streamin...

[rxin]

Github user rxin commented on the issue:

https://github.com/apache/spark/pull/22476
  
Merged in master/2.4.



---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22358: [SPARK-25366][SQL]Zstd and brotli CompressionCodec are n...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/22358
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 

https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/3273/
Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22358: [SPARK-25366][SQL]Zstd and brotli CompressionCodec are n...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/22358
  
Merged build finished. Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #22358: [SPARK-25366][SQL]Zstd and brotli CompressionCode...

[HyukjinKwon]

Github user HyukjinKwon commented on a diff in the pull request:

https://github.com/apache/spark/pull/22358#discussion_r219015195
  
--- Diff: docs/sql-programming-guide.md ---
@@ -965,6 +965,8 @@ Configuration of Parquet can be done using the 
`setConf` method on `SparkSession
 `parquet.compression` is specified in the table-specific 
options/properties, the precedence would be
 `compression`, `parquet.compression`, 
`spark.sql.parquet.compression.codec`. Acceptable values include:
 none, uncompressed, snappy, gzip, lzo, brotli, lz4, zstd.
+Note that `zstd` requires `ZStandardCodec` to be installed before 
Hadoop 2.9.0, `brotli` requires
+`brotliCodec` to be installed.
--- End diff --

`brotliCodec` -> `BrotliCodec`


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22460: DO NOT MERGE

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/22460
  
Merged build finished. Test FAILed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22460: DO NOT MERGE

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/22460
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/96270/
Test FAILed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19045: [WIP][SPARK-20628][CORE] Keep track of nodes (/ spot ins...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/19045
  
**[Test build #96290 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96290/testReport)**
 for PR 19045 at commit 
[`4e6572f`](https://github.com/apache/spark/commit/4e6572f8a7798298fe4787fe5913ee94c2b97359).


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22465: [SPARK-25457][SQL] IntegralDivide returns data type of t...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/22465
  
Merged build finished. Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22465: [SPARK-25457][SQL] IntegralDivide returns data type of t...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/22465
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 

https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/3256/
Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22450: [SPARK-25454][SQL] Avoid precision loss in division with...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/22450
  
**[Test build #96289 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96289/testReport)**
 for PR 22450 at commit 
[`27a9ea6`](https://github.com/apache/spark/commit/27a9ea656428d4705c6323deda64b14eb7ced7a4).


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22469: [SPARK-24341][FOLLOWUP][DOCS] Add migration note for IN ...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/22469
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 

https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/3255/
Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22460: DO NOT MERGE

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/22460
  
**[Test build #96288 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96288/testReport)**
 for PR 22460 at commit 
[`7252653`](https://github.com/apache/spark/commit/7252653cb836780a12f26b82b5a27a0bc83ee171).


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22469: [SPARK-24341][FOLLOWUP][DOCS] Add migration note for IN ...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/22469
  
Merged build finished. Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22460: DO NOT MERGE

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/22460
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 

https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/3257/
Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #22138: [SPARK-25151][SS] Apply Apache Commons Pool to Ka...

[HeartSaVioR]

Github user HeartSaVioR commented on a diff in the pull request:

https://github.com/apache/spark/pull/22138#discussion_r218955883
  
--- Diff: 
external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/InternalKafkaConsumerPool.scala
 ---
@@ -0,0 +1,241 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.kafka010
+
+import java.{util => ju}
+import java.util.concurrent.ConcurrentHashMap
+
+import org.apache.commons.pool2.{BaseKeyedPooledObjectFactory, 
PooledObject, SwallowedExceptionListener}
+import org.apache.commons.pool2.impl.{DefaultEvictionPolicy, 
DefaultPooledObject, GenericKeyedObjectPool, GenericKeyedObjectPoolConfig}
+
+import org.apache.spark.SparkEnv
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql.kafka010.InternalKafkaConsumerPool._
+import org.apache.spark.sql.kafka010.KafkaDataConsumer.CacheKey
+
+/**
+ * Provides object pool for [[InternalKafkaConsumer]] which is grouped by 
[[CacheKey]].
+ *
+ * This class leverages [[GenericKeyedObjectPool]] internally, hence 
providing methods based on
+ * the class, and same contract applies: after using the borrowed object, 
you must either call
+ * returnObject() if the object is healthy to return to pool, or 
invalidateObject() if the object
+ * should be destroyed.
+ *
+ * The soft capacity of pool is determined by 
"spark.sql.kafkaConsumerCache.capacity" config value,
+ * and the pool will have reasonable default value if the value is not 
provided.
+ * (The instance will do its best effort to respect soft capacity but it 
can exceed when there's
+ * a borrowing request and there's neither free space nor idle object to 
clear.)
+ *
+ * This class guarantees that no caller will get pooled object once the 
object is borrowed and
+ * not yet returned, hence provide thread-safety usage of non-thread-safe 
[[InternalKafkaConsumer]]
+ * unless caller shares the object to multiple threads.
+ */
+private[kafka010] class InternalKafkaConsumerPool(
+objectFactory: ObjectFactory,
+poolConfig: PoolConfig) {
+
+  // the class is intended to have only soft capacity
+  assert(poolConfig.getMaxTotal < 0)
+
+  private lazy val pool = {
+val internalPool = new GenericKeyedObjectPool[CacheKey, 
InternalKafkaConsumer](
+  objectFactory, poolConfig)
+
internalPool.setSwallowedExceptionListener(CustomSwallowedExceptionListener)
+internalPool
+  }
+
+  /**
+   * Borrows [[InternalKafkaConsumer]] object from the pool. If there's no 
idle object for the key,
+   * the pool will create the [[InternalKafkaConsumer]] object.
+   *
+   * If the pool doesn't have idle object for the key and also exceeds the 
soft capacity,
+   * pool will try to clear some of idle objects.
+   *
+   * Borrowed object must be returned by either calling returnObject or 
invalidateObject, otherwise
+   * the object will be kept in pool as active object.
+   */
+  def borrowObject(key: CacheKey, kafkaParams: ju.Map[String, Object]): 
InternalKafkaConsumer = {
+updateKafkaParamForKey(key, kafkaParams)
+
+if (getTotal == poolConfig.getSoftMaxTotal()) {
+  pool.clearOldest()
+}
+
+pool.borrowObject(key)
+  }
+
+  /** Returns borrowed object to the pool. */
+  def returnObject(consumer: InternalKafkaConsumer): Unit = {
+pool.returnObject(extractCacheKey(consumer), consumer)
+  }
+
+  /** Invalidates (destroy) borrowed object to the pool. */
+  def invalidateObject(consumer: InternalKafkaConsumer): Unit = {
+pool.invalidateObject(extractCacheKey(consumer), consumer)
+  }
+
+  /** Invalidates all idle consumers for the key */
+  def invalidateKey(key: CacheKey): Unit = {
+pool.clear(key)
+  }
+
+  /**
+   * Closes the keyed object pool. Once the pool is closed,
+   * borrowObject will fail with [[IllegalStateException]], but