[GitHub] spark issue #21121: [SPARK-24042][SQL] Collection function: zip_with_index

2018-04-30 Thread Tagar
Github user Tagar commented on the issue: https://github.com/apache/spark/pull/21121 Would this cover https://issues.apache.org/jira/browse/SPARK-23074 as well? Thanks. --- - To unsubscribe, e-mail:

[GitHub] spark issue #21031: [SPARK-23923][SQL] Add cardinality function

2018-04-30 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/21031 retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #21031: [SPARK-23923][SQL] Add cardinality function

2018-04-30 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/21031 LGTM pending Jenkins --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #21187: [SPARK-24035][SQL] SQL syntax for Pivot

2018-04-30 Thread Tagar
Github user Tagar commented on the issue: https://github.com/apache/spark/pull/21187 Would be great to make `FOR` section optional. E.g. - make `FOR year IN (2012, 2013)` optional in one of your examples. Currently `pivot()` when called programmatically, doesn't require to have

[GitHub] spark pull request #21021: [SPARK-23921][SQL] Add array_sort function

2018-04-30 Thread ueshin
Github user ueshin commented on a diff in the pull request: https://github.com/apache/spark/pull/21021#discussion_r185166848 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala --- @@ -191,28 +161,205 @@ case class

[GitHub] spark pull request #21021: [SPARK-23921][SQL] Add array_sort function

2018-04-30 Thread ueshin
Github user ueshin commented on a diff in the pull request: https://github.com/apache/spark/pull/21021#discussion_r185163397 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala --- @@ -191,28 +161,205 @@ case class

[GitHub] spark pull request #21021: [SPARK-23921][SQL] Add array_sort function

2018-04-30 Thread ueshin
Github user ueshin commented on a diff in the pull request: https://github.com/apache/spark/pull/21021#discussion_r185163319 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala --- @@ -191,28 +161,205 @@ case class

[GitHub] spark pull request #21021: [SPARK-23921][SQL] Add array_sort function

2018-04-30 Thread ueshin
Github user ueshin commented on a diff in the pull request: https://github.com/apache/spark/pull/21021#discussion_r185166899 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala --- @@ -191,28 +161,205 @@ case class

[GitHub] spark pull request #21189: [SPARK-24117][SQL] Unified the getSizePerRow

2018-04-30 Thread jose-torres
Github user jose-torres commented on a diff in the pull request: https://github.com/apache/spark/pull/21189#discussion_r185167182 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/sources/memoryV2.scala --- @@ -178,7 +179,7 @@ class

[GitHub] spark pull request #21189: [SPARK-24117][SQL] Unified the getSizePerRow

2018-04-30 Thread jose-torres
Github user jose-torres commented on a diff in the pull request: https://github.com/apache/spark/pull/21189#discussion_r185167132 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/MemorySinkSuite.scala --- @@ -220,11 +220,11 @@ class MemorySinkSuite

[GitHub] spark issue #21193: [SPARK-24121][SQL] Add API for handling expression code ...

2018-04-30 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/21193 retest this please. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #21203: [SPARK-24131][PySpark] Add majorMinorVersion API to PySp...

2018-04-30 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/21203 retest this please. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #21141: [SPARK-23853][PYSPARK][TEST] Run Hive-related PySpark te...

2018-04-30 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/21141 Thank you, @HyukjinKwon! --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands,

[GitHub] spark issue #21185: [SPARK-23894][CORE][SQL] Defensively clear ActiveSession...

2018-04-30 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/21185 LGTM pending Jenkins --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark pull request #17086: [SPARK-24101][ML][MLLIB] ML Evaluators should use...

2018-04-30 Thread imatiach-msft
Github user imatiach-msft commented on a diff in the pull request: https://github.com/apache/spark/pull/17086#discussion_r185163922 --- Diff: mllib/src/test/scala/org/apache/spark/mllib/evaluation/MulticlassMetricsSuite.scala --- @@ -95,4 +95,95 @@ class MulticlassMetricsSuite

[GitHub] spark pull request #17086: [SPARK-24101][ML][MLLIB] ML Evaluators should use...

2018-04-30 Thread imatiach-msft
Github user imatiach-msft commented on a diff in the pull request: https://github.com/apache/spark/pull/17086#discussion_r185163674 --- Diff: mllib/src/test/scala/org/apache/spark/mllib/evaluation/MulticlassMetricsSuite.scala --- @@ -55,44 +60,128 @@ class MulticlassMetricsSuite

[GitHub] spark pull request #17086: [SPARK-24101][ML][MLLIB] ML Evaluators should use...

2018-04-30 Thread imatiach-msft
Github user imatiach-msft commented on a diff in the pull request: https://github.com/apache/spark/pull/17086#discussion_r185163483 --- Diff: mllib/src/test/scala/org/apache/spark/mllib/evaluation/MulticlassMetricsSuite.scala --- @@ -95,4 +95,95 @@ class MulticlassMetricsSuite

[GitHub] spark issue #21158: [SPARK-23850][sql] Add separate config for SQL options r...

2018-04-30 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/21158 Is the default `(?i)secret|password|url|user|username".r` merged to the released branches? If the default is not in the previously release, I am fine to change it back. ---

[GitHub] spark issue #21203: [SPARK-24131][PySpark] Add majorMinorVersion API to PySp...

2018-04-30 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/21203 This is moved from #21153 based on @jkbradley's suggestion. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org

[GitHub] spark issue #21203: [SPARK-24131][PySpark] Add majorMinorVersion API to PySp...

2018-04-30 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/21203 cc @jkbradley @HyukjinKwon @dbtsai --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands,

[GitHub] spark pull request #21203: [SPARK-24131][PySpark] Add majorMinorVersion API ...

2018-04-30 Thread viirya
GitHub user viirya opened a pull request: https://github.com/apache/spark/pull/21203 [SPARK-24131][PySpark] Add majorMinorVersion API to PySpark for determining Spark versions ## What changes were proposed in this pull request? We need to determine Spark major and minor

[GitHub] spark issue #21136: [SPARK-24061][SS]Add TypedFilter support for continuous ...

2018-04-30 Thread zsxwing
Github user zsxwing commented on the issue: https://github.com/apache/spark/pull/21136 retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #21136: [SPARK-24061][SS]Add TypedFilter support for continuous ...

2018-04-30 Thread zsxwing
Github user zsxwing commented on the issue: https://github.com/apache/spark/pull/21136 LGTM. Merging to master. Thanks! --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands,

[GitHub] spark issue #20929: [SPARK-23772][SQL][WIP] Provide an option to ignore colu...

2018-04-30 Thread mengxr
Github user mengxr commented on the issue: https://github.com/apache/spark/pull/20929 @maropu Any updates? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #21040: [SPARK-23930][SQL] Add slice function

2018-04-30 Thread ueshin
Github user ueshin commented on the issue: https://github.com/apache/spark/pull/21040 Jenkins, retest this please. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark pull request #21153: [SPARK-24058][ML][PySpark] Default Params in ML s...

2018-04-30 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/21153#discussion_r185159691 --- Diff: python/pyspark/util.py --- @@ -61,6 +62,26 @@ def _get_argspec(f): return argspec +def majorMinorVersion(version): ---

[GitHub] spark pull request #20604: [SPARK-23365][CORE] Do not adjust num executors w...

2018-04-30 Thread Ngone51
Github user Ngone51 commented on a diff in the pull request: https://github.com/apache/spark/pull/20604#discussion_r185159109 --- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala --- @@ -1643,7 +1646,10 @@ class SparkContext(config: SparkConf) extends Logging {

[GitHub] spark pull request #21198: [SPARK-24126][pyspark] Use build-specific temp di...

2018-04-30 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/21198#discussion_r185158876 --- Diff: python/pyspark/streaming/tests.py --- @@ -1549,7 +1549,9 @@ def search_kinesis_asl_assembly_jar(): kinesis_jar_present = True

[GitHub] spark pull request #21198: [SPARK-24126][pyspark] Use build-specific temp di...

2018-04-30 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/21198#discussion_r185158778 --- Diff: python/pyspark/streaming/tests.py --- @@ -1549,7 +1549,9 @@ def search_kinesis_asl_assembly_jar(): kinesis_jar_present = True

[GitHub] spark pull request #21198: [SPARK-24126][pyspark] Use build-specific temp di...

2018-04-30 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/21198#discussion_r185156339 --- Diff: python/pyspark/streaming/tests.py --- @@ -1549,7 +1549,9 @@ def search_kinesis_asl_assembly_jar(): kinesis_jar_present = True

[GitHub] spark issue #21158: [SPARK-23850][sql] Add separate config for SQL options r...

2018-04-30 Thread vanzin
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/21158 > We are making a behavior change here. We are *not*. That's the whole reason why I'm adding the SQL-specific option to extend the behavior of the core options. If I just wanted to

[GitHub] spark issue #21185: [SPARK-23894][CORE][SQL] Defensively clear ActiveSession...

2018-04-30 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21185 **[Test build #4161 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/4161/testReport)** for PR 21185 at commit

[GitHub] spark issue #21185: [SPARK-23894][CORE][SQL] Defensively clear ActiveSession...

2018-04-30 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/21185 Jenkins, retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #21200: [SPARK-24039][SS] Do continuous processing writes with m...

2018-04-30 Thread jose-torres
Github user jose-torres commented on the issue: https://github.com/apache/spark/pull/21200 @HeartSaVioR --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark pull request #21200: [SPARK-24039][SS] Do continuous processing writes...

2018-04-30 Thread jose-torres
Github user jose-torres commented on a diff in the pull request: https://github.com/apache/spark/pull/21200#discussion_r185153219 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/continuous/EpochPollRunnable.scala --- @@ -0,0 +1,61 @@ +/* + *

[GitHub] spark pull request #21200: [SPARK-24039][SS] Do continuous processing writes...

2018-04-30 Thread jose-torres
Github user jose-torres commented on a diff in the pull request: https://github.com/apache/spark/pull/21200#discussion_r185153197 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/continuous/ContinuousDataSourceRDD.scala --- @@ -0,0 +1,153 @@ +/*

[GitHub] spark pull request #21141: [SPARK-23853][PYSPARK][TEST] Run Hive-related PyS...

2018-04-30 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/21141 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #20973: [SPARK-20114][ML] spark.ml parity for sequential ...

2018-04-30 Thread WeichenXu123
Github user WeichenXu123 commented on a diff in the pull request: https://github.com/apache/spark/pull/20973#discussion_r185149879 --- Diff: mllib/src/main/scala/org/apache/spark/ml/fpm/PrefixSpan.scala --- @@ -44,26 +43,37 @@ object PrefixSpan { * * @param dataset

[GitHub] spark issue #21141: [SPARK-23853][PYSPARK][TEST] Run Hive-related PySpark te...

2018-04-30 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/21141 Merged to master and branch-2.3. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands,

[GitHub] spark pull request #21200: [SPARK-24039][SS] Do continuous processing writes...

2018-04-30 Thread arunmahadevan
Github user arunmahadevan commented on a diff in the pull request: https://github.com/apache/spark/pull/21200#discussion_r185147205 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/continuous/EpochPollRunnable.scala --- @@ -0,0 +1,61 @@ +/* + *

[GitHub] spark pull request #21200: [SPARK-24039][SS] Do continuous processing writes...

2018-04-30 Thread arunmahadevan
Github user arunmahadevan commented on a diff in the pull request: https://github.com/apache/spark/pull/21200#discussion_r185148972 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/continuous/ContinuousDataSourceRDD.scala --- @@ -0,0 +1,153 @@ +/*

[GitHub] spark pull request #21198: [SPARK-24126][pyspark] Use build-specific temp di...

2018-04-30 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/21198#discussion_r185148928 --- Diff: python/pyspark/streaming/tests.py --- @@ -1549,7 +1549,9 @@ def search_kinesis_asl_assembly_jar(): kinesis_jar_present = True

[GitHub] spark pull request #21202: [SPARK-24129] [K8S] Add option to pass --build-ar...

2018-04-30 Thread devaraj-kavali
GitHub user devaraj-kavali opened a pull request: https://github.com/apache/spark/pull/21202 [SPARK-24129] [K8S] Add option to pass --build-arg's to docker-image-tool.sh ## What changes were proposed in this pull request? Adding `-b arg` option to take `--build-arg`

[GitHub] spark issue #21187: [SPARK-24035][SQL] SQL syntax for Pivot

2018-04-30 Thread maryannxue
Github user maryannxue commented on the issue: https://github.com/apache/spark/pull/21187 Thank you, @aray! Thank you, @rxin, for the nice suggestion! Changes made accordingly in my latest commit. --- - To

[GitHub] spark pull request #20613: [SPARK-23368][SQL] Avoid unnecessary Exchange or ...

2018-04-30 Thread maryannxue
Github user maryannxue closed the pull request at: https://github.com/apache/spark/pull/20613 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark pull request #21201: [SPARK-24128][SQL] Mention configuration option i...

2018-04-30 Thread henryr
GitHub user henryr opened a pull request: https://github.com/apache/spark/pull/21201 [SPARK-24128][SQL] Mention configuration option in implicit CROSS JOIN error ## What changes were proposed in this pull request? Mention `spark.sql.crossJoin.enabled` in error message when

[GitHub] spark issue #21185: [SPARK-23894][CORE][SQL] Defensively clear ActiveSession...

2018-04-30 Thread ericl
Github user ericl commented on the issue: https://github.com/apache/spark/pull/21185 This makes sense to me. It would be slightly to clear it where where the session is getting leaked through threads, but if that's hard then this looks good. ---

[GitHub] spark issue #21095: [SPARK-23529][K8s] Support mounting hostPath volumes

2018-04-30 Thread foxish
Github user foxish commented on the issue: https://github.com/apache/spark/pull/21095 The testing is not a blocker for the review. When I said "tests for non-hostpath type volumes", I meant to say that we want to cover more than just hostpath mounts with the initial PR - because we

[GitHub] spark issue #21185: [SPARK-23894][CORE][SQL] Defensively clear ActiveSession...

2018-04-30 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/21185 cc @ericl too --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark pull request #21158: [SPARK-23850][sql] Add separate config for SQL op...

2018-04-30 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/21158#discussion_r185143980 --- Diff: core/src/main/scala/org/apache/spark/internal/config/package.scala --- @@ -342,7 +342,7 @@ package object config { "a property

[GitHub] spark issue #21185: [SPARK-23894][CORE][SQL] Defensively clear ActiveSession...

2018-04-30 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21185 **[Test build #4160 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/4160/testReport)** for PR 21185 at commit

[GitHub] spark issue #21187: [SPARK-24035][SQL] SQL syntax for Pivot

2018-04-30 Thread aray
Github user aray commented on the issue: https://github.com/apache/spark/pull/21187 LGTM thanks for doing this! --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark pull request #21158: [SPARK-23850][sql] Add separate config for SQL op...

2018-04-30 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/21158#discussion_r185142687 --- Diff: core/src/main/scala/org/apache/spark/internal/config/package.scala --- @@ -342,7 +342,7 @@ package object config { "a property

[GitHub] spark issue #21158: [SPARK-23850][sql] Add separate config for SQL options r...

2018-04-30 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/21158 The confs in the core modules are not part of the outputs of SQL statement `SET -v`, which only outputs the confs in Spark SQL. We are making a behavior change here. I am not confident

[GitHub] spark issue #21197: [SPARK-23971] [BACKPORT-2.3] Should not leak Spark sessi...

2018-04-30 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/21197 test this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #21197: [SPARK-23971] [BACKPORT-2.3] Should not leak Spark sessi...

2018-04-30 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/21197 ok to test --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #21095: [SPARK-23529][K8s] Support mounting hostPath volumes

2018-04-30 Thread madanadit
Github user madanadit commented on the issue: https://github.com/apache/spark/pull/21095 Hi @foxish, I don't see why the 2 testing concerns should block reviewing this PR. 1. This PR does not attempt to address non-hostpath volumes (both implementation and unit tests are hence

[GitHub] spark issue #21122: [SPARK-24017] [SQL] Refactor ExternalCatalog to be an in...

2018-04-30 Thread rdblue
Github user rdblue commented on the issue: https://github.com/apache/spark/pull/21122 Thanks for pointing this out, @henryr. This looks like a good change to support multiple catalogs. I think it looks fine, other than exposing `unwrapped` to get the Hive client. I think

[GitHub] spark pull request #21122: [SPARK-24017] [SQL] Refactor ExternalCatalog to b...

2018-04-30 Thread rdblue
Github user rdblue commented on a diff in the pull request: https://github.com/apache/spark/pull/21122#discussion_r185138677 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/ExternalCatalog.scala --- @@ -31,10 +30,16 @@ import

[GitHub] spark pull request #21153: [SPARK-24058][ML][PySpark] Default Params in ML s...

2018-04-30 Thread jkbradley
Github user jkbradley commented on a diff in the pull request: https://github.com/apache/spark/pull/21153#discussion_r185131834 --- Diff: python/pyspark/util.py --- @@ -61,6 +62,26 @@ def _get_argspec(f): return argspec +def majorMinorVersion(version):

[GitHub] spark issue #21109: [SPARK-24020][SQL] Sort-merge join inner range optimizat...

2018-04-30 Thread zecevicp
Github user zecevicp commented on the issue: https://github.com/apache/spark/pull/21109 The code path with the optimization but without wholegen code generation gives wrong results. And I haven't been able to figure out where is the bug. I spent several hours at this again today.

[GitHub] spark issue #21200: [SPARK-24039][SS] Do continuous processing writes with m...

2018-04-30 Thread jose-torres
Github user jose-torres commented on the issue: https://github.com/apache/spark/pull/21200 @tdas --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark pull request #21200: [SPARK-24039][SS] Do continuous processing writes...

2018-04-30 Thread jose-torres
GitHub user jose-torres opened a pull request: https://github.com/apache/spark/pull/21200 [SPARK-24039][SS] Do continuous processing writes with multiple compute() calls ## What changes were proposed in this pull request? Do continuous processing writes with multiple

[GitHub] spark issue #21195: [Spark-23975][ML] Add support of array input for all clu...

2018-04-30 Thread MrBago
Github user MrBago commented on the issue: https://github.com/apache/spark/pull/21195 Looking now. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark pull request #21199: [SPARK-24127][SS] Continuous text socket source

2018-04-30 Thread arunmahadevan
GitHub user arunmahadevan opened a pull request: https://github.com/apache/spark/pull/21199 [SPARK-24127][SS] Continuous text socket source ## What changes were proposed in this pull request? Support for text socket stream in spark structured streaming "continuous" mode.

[GitHub] spark issue #20940: [SPARK-23429][CORE] Add executor memory metrics to heart...

2018-04-30 Thread rezasafi
Github user rezasafi commented on the issue: https://github.com/apache/spark/pull/20940 This generally looks good to me. I have some concerns about piggy backing memory info on heart-beats, since basically this will be a uniform sampling of memory info and it may make the heart-beat

[GitHub] spark pull request #21068: [SPARK-16630][YARN] Blacklist a node if executors...

2018-04-30 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/21068#discussion_r185111225 --- Diff: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/config.scala --- @@ -328,4 +328,18 @@ package object config {

[GitHub] spark pull request #21196: [SPARK-24123][SQL] Fix precision issues in months...

2018-04-30 Thread mgaido91
Github user mgaido91 commented on a diff in the pull request: https://github.com/apache/spark/pull/21196#discussion_r185115238 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/DateTimeUtils.scala --- @@ -45,6 +45,7 @@ object DateTimeUtils { // it's

[GitHub] spark pull request #21196: [SPARK-24123][SQL] Fix precision issues in months...

2018-04-30 Thread dongjoon-hyun
Github user dongjoon-hyun commented on a diff in the pull request: https://github.com/apache/spark/pull/21196#discussion_r185114200 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/DateTimeUtils.scala --- @@ -45,6 +45,7 @@ object DateTimeUtils { //

[GitHub] spark pull request #21068: [SPARK-16630][YARN] Blacklist a node if executors...

2018-04-30 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/21068#discussion_r185106176 --- Diff: core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala --- @@ -170,8 +170,7 @@ class

[GitHub] spark pull request #21068: [SPARK-16630][YARN] Blacklist a node if executors...

2018-04-30 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/21068#discussion_r185110988 --- Diff: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocatorBlacklistTracker.scala --- @@ -0,0 +1,150 @@ +/* + *

[GitHub] spark pull request #21068: [SPARK-16630][YARN] Blacklist a node if executors...

2018-04-30 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/21068#discussion_r185110389 --- Diff: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocatorBlacklistTracker.scala --- @@ -0,0 +1,150 @@ +/* + *

[GitHub] spark pull request #21068: [SPARK-16630][YARN] Blacklist a node if executors...

2018-04-30 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/21068#discussion_r185106301 --- Diff: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/FailureTracker.scala --- @@ -0,0 +1,85 @@ +/* + * Licensed to the

[GitHub] spark pull request #21068: [SPARK-16630][YARN] Blacklist a node if executors...

2018-04-30 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/21068#discussion_r185105266 --- Diff: core/src/test/scala/org/apache/spark/HeartbeatReceiverSuite.scala --- @@ -73,6 +73,7 @@ class HeartbeatReceiverSuite sc = spy(new

[GitHub] spark pull request #21068: [SPARK-16630][YARN] Blacklist a node if executors...

2018-04-30 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/21068#discussion_r185110457 --- Diff: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocatorBlacklistTracker.scala --- @@ -0,0 +1,150 @@ +/* + *

[GitHub] spark pull request #21068: [SPARK-16630][YARN] Blacklist a node if executors...

2018-04-30 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/21068#discussion_r18531 --- Diff: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocatorBlacklistTracker.scala --- @@ -0,0 +1,150 @@ +/* + *

[GitHub] spark pull request #21068: [SPARK-16630][YARN] Blacklist a node if executors...

2018-04-30 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/21068#discussion_r185106409 --- Diff: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/FailureTracker.scala --- @@ -0,0 +1,85 @@ +/* + * Licensed to the

[GitHub] spark pull request #21068: [SPARK-16630][YARN] Blacklist a node if executors...

2018-04-30 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/21068#discussion_r185113461 --- Diff: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocatorBlacklistTracker.scala --- @@ -0,0 +1,150 @@ +/* + *

[GitHub] spark pull request #21068: [SPARK-16630][YARN] Blacklist a node if executors...

2018-04-30 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/21068#discussion_r185112100 --- Diff: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocatorBlacklistTracker.scala --- @@ -0,0 +1,150 @@ +/* + *

[GitHub] spark pull request #21068: [SPARK-16630][YARN] Blacklist a node if executors...

2018-04-30 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/21068#discussion_r185109185 --- Diff: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/FailureTracker.scala --- @@ -0,0 +1,85 @@ +/* + * Licensed to the

[GitHub] spark pull request #21068: [SPARK-16630][YARN] Blacklist a node if executors...

2018-04-30 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/21068#discussion_r185109769 --- Diff: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/FailureTracker.scala --- @@ -0,0 +1,85 @@ +/* + * Licensed to the

[GitHub] spark pull request #21068: [SPARK-16630][YARN] Blacklist a node if executors...

2018-04-30 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/21068#discussion_r185110730 --- Diff: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocatorBlacklistTracker.scala --- @@ -0,0 +1,150 @@ +/* + *

[GitHub] spark pull request #21068: [SPARK-16630][YARN] Blacklist a node if executors...

2018-04-30 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/21068#discussion_r185110240 --- Diff: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocator.scala --- @@ -204,35 +182,23 @@ private[yarn] class

[GitHub] spark pull request #21196: [SPARK-24123][SQL] Fix precision issues in months...

2018-04-30 Thread mgaido91
Github user mgaido91 commented on a diff in the pull request: https://github.com/apache/spark/pull/21196#discussion_r185106149 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/DateTimeUtils.scala --- @@ -45,6 +45,7 @@ object DateTimeUtils { // it's

[GitHub] spark pull request #21196: [SPARK-24123][SQL] Fix precision issues in months...

2018-04-30 Thread dongjoon-hyun
Github user dongjoon-hyun commented on a diff in the pull request: https://github.com/apache/spark/pull/21196#discussion_r185104899 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/DateTimeUtils.scala --- @@ -45,6 +45,7 @@ object DateTimeUtils { //

[GitHub] spark pull request #21088: [SPARK-24003][CORE] Add support to provide spark....

2018-04-30 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/21088 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21088: [SPARK-24003][CORE] Add support to provide spark.executo...

2018-04-30 Thread vanzin
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/21088 Merging to master. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #21141: [SPARK-23853][PYSPARK][TEST] Run Hive-related PySpark te...

2018-04-30 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/21141 Thank you, @bersprockets . --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands,

[GitHub] spark pull request #21198: [SPARK-24126][pyspark] Use build-specific temp di...

2018-04-30 Thread vanzin
GitHub user vanzin opened a pull request: https://github.com/apache/spark/pull/21198 [SPARK-24126][pyspark] Use build-specific temp directory for pyspark tests. This avoids polluting and leaving garbage behind in /tmp, and allows the usual build tools to clean up any leftover

[GitHub] spark issue #21141: [SPARK-23853][PYSPARK][TEST] Run Hive-related PySpark te...

2018-04-30 Thread bersprockets
Github user bersprockets commented on the issue: https://github.com/apache/spark/pull/21141 My experience here is limited. Still, it also looks good to me. --- - To unsubscribe, e-mail:

[GitHub] spark issue #21073: [SPARK-23936][SQL] Implement map_concat

2018-04-30 Thread bersprockets
Github user bersprockets commented on the issue: https://github.com/apache/spark/pull/21073 retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #21185: [SPARK-23894][CORE][SQL] Defensively clear ActiveSession...

2018-04-30 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21185 **[Test build #4160 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/4160/testReport)** for PR 21185 at commit

[GitHub] spark pull request #21197: [SPARK-23971] [BACKPORT-2.3] Should not leak Spar...

2018-04-30 Thread gatorsmile
GitHub user gatorsmile opened a pull request: https://github.com/apache/spark/pull/21197 [SPARK-23971] [BACKPORT-2.3] Should not leak Spark sessions across test suites This PR is to backport the PR https://github.com/apache/spark/pull/21058 to Apache 2.3. This should be the cause

[GitHub] spark issue #21073: [SPARK-23936][SQL] Implement map_concat

2018-04-30 Thread bersprockets
Github user bersprockets commented on the issue: https://github.com/apache/spark/pull/21073 @mn-mikke @kiszk Thanks for the review. I addressed the comments. Please take a look when you have a chance. --- - To

[GitHub] spark issue #21196: [SPARK-24123][SQL] Fix precision issues in monthsBetween...

2018-04-30 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/21196 cc @ueshin --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #21158: [SPARK-23850][sql] Add separate config for SQL options r...

2018-04-30 Thread vanzin
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/21158 Ping @gatorsmile --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #21185: [SPARK-23894][CORE][SQL] Defensively clear ActiveSession...

2018-04-30 Thread vanzin
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/21185 retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #21185: [SPARK-23894][CORE][SQL] Defensively clear ActiveSession...

2018-04-30 Thread vanzin
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/21185 Tests haven't triggered, weird. LGTM pending tests. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For

[GitHub] spark pull request #21187: [SPARK-24035][SQL] SQL syntax for Pivot

2018-04-30 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/21187#discussion_r185084802 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/PivotSuite.scala --- @@ -0,0 +1,197 @@ +/* + * Licensed to the Apache Software Foundation (ASF)

[GitHub] spark pull request #21196: [SPARK-24123][SQL] Fix precision issues in months...

2018-04-30 Thread mgaido91
GitHub user mgaido91 opened a pull request: https://github.com/apache/spark/pull/21196 [SPARK-24123][SQL] Fix precision issues in monthsBetween with more than 8 digits ## What changes were proposed in this pull request? SPARK-23902 introduced the ability to retrieve more

  1   2   >