[GitHub] HyukjinKwon commented on issue #23357: [SPARK-26451][SQL] Change lead/lag argument name from count to offset

2018-12-27 Thread GitBox
HyukjinKwon commented on issue #23357: [SPARK-26451][SQL] Change lead/lag argument name from count to offset URL: https://github.com/apache/spark/pull/23357#issuecomment-450178997 Merged to master. I think it's fine too but please let me know if anyone feels against. We can do

[GitHub] HyukjinKwon commented on issue #21732: [SPARK-24762][SQL] Enable Option of Product encoders

2018-12-27 Thread GitBox
HyukjinKwon commented on issue #21732: [SPARK-24762][SQL] Enable Option of Product encoders URL: https://github.com/apache/spark/pull/21732#issuecomment-450181891 Hm, for aggregators, I would consider this as non root level. Looks they use the same encoder but can't be the same.

[GitHub] SparkQA commented on issue #23386: [SPARK-26446][CORE] Improve docs on ExecutorAllocationManager

2018-12-27 Thread GitBox
SparkQA commented on issue #23386: [SPARK-26446][CORE] Improve docs on ExecutorAllocationManager URL: https://github.com/apache/spark/pull/23386#issuecomment-450201074 **[Test build #4487 has

[GitHub] SparkQA removed a comment on issue #23386: [SPARK-26446][CORE] Improve docs on ExecutorAllocationManager

2018-12-27 Thread GitBox
SparkQA removed a comment on issue #23386: [SPARK-26446][CORE] Improve docs on ExecutorAllocationManager URL: https://github.com/apache/spark/pull/23386#issuecomment-450160323 **[Test build #4487 has

[GitHub] dongjoon-hyun closed pull request #23389: [2.4] revert [SPARK-26021][SQL] replace minus zero with zero in Platform.putDouble/Float

2018-12-27 Thread GitBox
dongjoon-hyun closed pull request #23389: [2.4] revert [SPARK-26021][SQL] replace minus zero with zero in Platform.putDouble/Float URL: https://github.com/apache/spark/pull/23389 This is a PR merged from a forked repository. As GitHub hides the original diff on merge, it is displayed

[GitHub] srowen commented on a change in pull request #23401: Trigger GC on executor node idle

2018-12-31 Thread GitBox
srowen commented on a change in pull request #23401: Trigger GC on executor node idle URL: https://github.com/apache/spark/pull/23401#discussion_r244610240 ## File path: core/src/main/scala/org/apache/spark/executor/Executor.scala ## @@ -628,6 +628,24 @@ private[spark]

[GitHub] SandishKumarHN opened a new pull request #23401: Trigger GC on executor node idle

2018-12-31 Thread GitBox
SandishKumarHN opened a new pull request #23401: Trigger GC on executor node idle URL: https://github.com/apache/spark/pull/23401 ## What changes were proposed in this pull request? ### Trigger GC on executor node idle Correct me if I'm wrong. ### Stage On a large

[GitHub] SparkQA removed a comment on issue #23419: [SPARK-26507][CORE] Fix core tests for Java 11

2018-12-31 Thread GitBox
SparkQA removed a comment on issue #23419: [SPARK-26507][CORE] Fix core tests for Java 11 URL: https://github.com/apache/spark/pull/23419#issuecomment-450651496 **[Test build #4490 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/4490/testReport)** for

[GitHub] AmplabJenkins commented on issue #23401: Trigger GC on executor node idle

2018-12-31 Thread GitBox
AmplabJenkins commented on issue #23401: Trigger GC on executor node idle URL: https://github.com/apache/spark/pull/23401#issuecomment-450680325 Can one of the admins verify this patch? This is an automated message from the

[GitHub] AmplabJenkins removed a comment on issue #23401: Trigger GC on executor node idle

2018-12-31 Thread GitBox
AmplabJenkins removed a comment on issue #23401: Trigger GC on executor node idle URL: https://github.com/apache/spark/pull/23401#issuecomment-450680191 Can one of the admins verify this patch? This is an automated message

[GitHub] AmplabJenkins removed a comment on issue #23401: Trigger GC on executor node idle

2018-12-31 Thread GitBox
AmplabJenkins removed a comment on issue #23401: Trigger GC on executor node idle URL: https://github.com/apache/spark/pull/23401#issuecomment-450454120 Can one of the admins verify this patch? This is an automated message

[GitHub] AmplabJenkins commented on issue #23401: Trigger GC on executor node idle

2018-12-31 Thread GitBox
AmplabJenkins commented on issue #23401: Trigger GC on executor node idle URL: https://github.com/apache/spark/pull/23401#issuecomment-450680191 Can one of the admins verify this patch? This is an automated message from the

[GitHub] AmplabJenkins removed a comment on issue #23400: [SPARK-26499] [SQL] JdbcUtils.makeGetter does not handle ByteType

2018-12-31 Thread GitBox
AmplabJenkins removed a comment on issue #23400: [SPARK-26499] [SQL] JdbcUtils.makeGetter does not handle ByteType URL: https://github.com/apache/spark/pull/23400#issuecomment-450695809 Merged build finished. Test PASSed.

[GitHub] HeartSaVioR commented on a change in pull request #23418: [MINOR] Fix inconsistency log level among delegation token providers

2018-12-31 Thread GitBox
HeartSaVioR commented on a change in pull request #23418: [MINOR] Fix inconsistency log level among delegation token providers URL: https://github.com/apache/spark/pull/23418#discussion_r244621528 ## File path:

[GitHub] AmplabJenkins commented on issue #23400: [SPARK-26499] [SQL] JdbcUtils.makeGetter does not handle ByteType

2018-12-31 Thread GitBox
AmplabJenkins commented on issue #23400: [SPARK-26499] [SQL] JdbcUtils.makeGetter does not handle ByteType URL: https://github.com/apache/spark/pull/23400#issuecomment-450695809 Merged build finished. Test PASSed. This is an

[GitHub] AmplabJenkins removed a comment on issue #23400: [SPARK-26499] [SQL] JdbcUtils.makeGetter does not handle ByteType

2018-12-31 Thread GitBox
AmplabJenkins removed a comment on issue #23400: [SPARK-26499] [SQL] JdbcUtils.makeGetter does not handle ByteType URL: https://github.com/apache/spark/pull/23400#issuecomment-450695811 Test PASSed. Refer to this link for build results (access rights to CI server needed):

[GitHub] AmplabJenkins commented on issue #23400: [SPARK-26499] [SQL] JdbcUtils.makeGetter does not handle ByteType

2018-12-31 Thread GitBox
AmplabJenkins commented on issue #23400: [SPARK-26499] [SQL] JdbcUtils.makeGetter does not handle ByteType URL: https://github.com/apache/spark/pull/23400#issuecomment-450695811 Test PASSed. Refer to this link for build results (access rights to CI server needed):

[GitHub] holdenk commented on issue #23401: [SPARK-26513][Core] : Trigger GC on executor node idle

2018-12-31 Thread GitBox
holdenk commented on issue #23401: [SPARK-26513][Core] : Trigger GC on executor node idle URL: https://github.com/apache/spark/pull/23401#issuecomment-450700797 For "Dynamic allocation probably eliminates most of the benefit because of executor turn-over" I think we could still see some

[GitHub] holdenk commented on issue #20658: [SPARK-23488][python] Add missing catalog methods to python API

2018-12-31 Thread GitBox
holdenk commented on issue #20658: [SPARK-23488][python] Add missing catalog methods to python API URL: https://github.com/apache/spark/pull/20658#issuecomment-450701337 @HyukjinKwon I agree doctests would be awesome, but given the rest of the file doesn't have them I think it could be a

[GitHub] chanansh commented on a change in pull request #23414: [SPARK-26449][PYTHON] add a transform method to the Dataframe class

2019-01-01 Thread GitBox
chanansh commented on a change in pull request #23414: [SPARK-26449][PYTHON] add a transform method to the Dataframe class URL: https://github.com/apache/spark/pull/23414#discussion_r244631252 ## File path: python/pyspark/sql/dataframe.py ## @@ -2046,6 +2046,40 @@ def

[GitHub] chanansh commented on a change in pull request #23414: [SPARK-26449][PYTHON] add a transform method to the Dataframe class

2019-01-01 Thread GitBox
chanansh commented on a change in pull request #23414: [SPARK-26449][PYTHON] add a transform method to the Dataframe class URL: https://github.com/apache/spark/pull/23414#discussion_r244631269 ## File path: python/pyspark/sql/dataframe.py ## @@ -2046,6 +2046,40 @@ def

[GitHub] chanansh commented on a change in pull request #23414: [SPARK-26449][PYTHON] add a transform method to the Dataframe class

2019-01-01 Thread GitBox
chanansh commented on a change in pull request #23414: [SPARK-26449][PYTHON] add a transform method to the Dataframe class URL: https://github.com/apache/spark/pull/23414#discussion_r244631269 ## File path: python/pyspark/sql/dataframe.py ## @@ -2046,6 +2046,40 @@ def

[GitHub] yanlin-Lynn commented on a change in pull request #23369: [SPARK-26429][SS]add jdbc sink for Structured Streaming.

2019-01-01 Thread GitBox
yanlin-Lynn commented on a change in pull request #23369: [SPARK-26429][SS]add jdbc sink for Structured Streaming. URL: https://github.com/apache/spark/pull/23369#discussion_r244632164 ## File path:

[GitHub] SandishKumarHN commented on issue #23401: Trigger GC on executor node idle

2018-12-31 Thread GitBox
SandishKumarHN commented on issue #23401: Trigger GC on executor node idle URL: https://github.com/apache/spark/pull/23401#issuecomment-450682724 @srowen Yes we referred this article https://www.computer.org/csdl/proceedings/hipc/2016/5411/00/07839705.pdf which helped on us improve spark

[GitHub] HyukjinKwon commented on issue #23414: [Spark 26449][PYSPARK] add a transform method to the Dataframe class

2018-12-31 Thread GitBox
HyukjinKwon commented on issue #23414: [Spark 26449][PYSPARK] add a transform method to the Dataframe class URL: https://github.com/apache/spark/pull/23414#issuecomment-450682830 the build was failed due to the reason below: ``` pycodestyle checks failed:

[GitHub] MaxGekk commented on a change in pull request #23417: [SPARK-26374][TEST][SQL] Enable TimestampFormatter in HadoopFsRelationTest

2018-12-31 Thread GitBox
MaxGekk commented on a change in pull request #23417: [SPARK-26374][TEST][SQL] Enable TimestampFormatter in HadoopFsRelationTest URL: https://github.com/apache/spark/pull/23417#discussion_r244621072 ## File path:

[GitHub] srowen commented on a change in pull request #23420: [SPARK-26508][Core][SQL] Address warning messages in Java reported at lgtm.com

2018-12-31 Thread GitBox
srowen commented on a change in pull request #23420: [SPARK-26508][Core][SQL] Address warning messages in Java reported at lgtm.com URL: https://github.com/apache/spark/pull/23420#discussion_r244624369 ## File path:

[GitHub] holdenk commented on issue #20146: [SPARK-11215][ML] Add multiple columns support to StringIndexer

2018-12-31 Thread GitBox
holdenk commented on issue #20146: [SPARK-11215][ML] Add multiple columns support to StringIndexer URL: https://github.com/apache/spark/pull/20146#issuecomment-450701693 I like how you've addressed that with the handling of NaNs. I'll give this a once-over on Friday just to make sure

[GitHub] AmplabJenkins commented on issue #23400: [SPARK-26499] [SQL] JdbcUtils.makeGetter does not handle ByteType

2018-12-31 Thread GitBox
AmplabJenkins commented on issue #23400: [SPARK-26499] [SQL] JdbcUtils.makeGetter does not handle ByteType URL: https://github.com/apache/spark/pull/23400#issuecomment-450703836 Merged build finished. Test PASSed. This is an

[GitHub] AmplabJenkins commented on issue #23400: [SPARK-26499] [SQL] JdbcUtils.makeGetter does not handle ByteType

2018-12-31 Thread GitBox
AmplabJenkins commented on issue #23400: [SPARK-26499] [SQL] JdbcUtils.makeGetter does not handle ByteType URL: https://github.com/apache/spark/pull/23400#issuecomment-450703837 Test PASSed. Refer to this link for build results (access rights to CI server needed):

[GitHub] AmplabJenkins removed a comment on issue #23400: [SPARK-26499] [SQL] JdbcUtils.makeGetter does not handle ByteType

2018-12-31 Thread GitBox
AmplabJenkins removed a comment on issue #23400: [SPARK-26499] [SQL] JdbcUtils.makeGetter does not handle ByteType URL: https://github.com/apache/spark/pull/23400#issuecomment-450703836 Merged build finished. Test PASSed.

[GitHub] HyukjinKwon edited a comment on issue #23288: [SPARK-26339][SQL]Throws better exception when reading files that start with underscore

2018-12-31 Thread GitBox
HyukjinKwon edited a comment on issue #23288: [SPARK-26339][SQL]Throws better exception when reading files that start with underscore URL: https://github.com/apache/spark/pull/23288#issuecomment-450703754 @srowen, this didn't run the test! Looks some tests are being broken

[GitHub] HyukjinKwon commented on issue #23400: [SPARK-26499] [SQL] JdbcUtils.makeGetter does not handle ByteType

2018-12-31 Thread GitBox
HyukjinKwon commented on issue #23400: [SPARK-26499] [SQL] JdbcUtils.makeGetter does not handle ByteType URL: https://github.com/apache/spark/pull/23400#issuecomment-450703891 retest this please This is an automated message

[GitHub] AmplabJenkins removed a comment on issue #23400: [SPARK-26499] [SQL] JdbcUtils.makeGetter does not handle ByteType

2018-12-31 Thread GitBox
AmplabJenkins removed a comment on issue #23400: [SPARK-26499] [SQL] JdbcUtils.makeGetter does not handle ByteType URL: https://github.com/apache/spark/pull/23400#issuecomment-450703837 Test PASSed. Refer to this link for build results (access rights to CI server needed):

[GitHub] HyukjinKwon commented on issue #23184: [SPARK-26227][R] from_[csv|json] should accept schema_of_[csv|json] in R API

2018-12-31 Thread GitBox
HyukjinKwon commented on issue #23184: [SPARK-26227][R] from_[csv|json] should accept schema_of_[csv|json] in R API URL: https://github.com/apache/spark/pull/23184#issuecomment-450704499 Yea, Scala side change is already made about `from_[csv|json](schema_of_[csv|json]())`.

[GitHub] HeartSaVioR commented on issue #23418: [MINOR] Fix inconsistency log level among delegation token providers

2018-12-31 Thread GitBox
HeartSaVioR commented on issue #23418: [MINOR] Fix inconsistency log level among delegation token providers URL: https://github.com/apache/spark/pull/23418#issuecomment-450704550 Thanks @srowen and @HyukjinKwon for reviewing and merging!

[GitHub] AmplabJenkins removed a comment on issue #23400: [SPARK-26499] [SQL] JdbcUtils.makeGetter does not handle ByteType

2018-12-31 Thread GitBox
AmplabJenkins removed a comment on issue #23400: [SPARK-26499] [SQL] JdbcUtils.makeGetter does not handle ByteType URL: https://github.com/apache/spark/pull/23400#issuecomment-450710515 Test PASSed. Refer to this link for build results (access rights to CI server needed):

[GitHub] AmplabJenkins commented on issue #23400: [SPARK-26499] [SQL] JdbcUtils.makeGetter does not handle ByteType

2018-12-31 Thread GitBox
AmplabJenkins commented on issue #23400: [SPARK-26499] [SQL] JdbcUtils.makeGetter does not handle ByteType URL: https://github.com/apache/spark/pull/23400#issuecomment-450710515 Test PASSed. Refer to this link for build results (access rights to CI server needed):

[GitHub] AmplabJenkins removed a comment on issue #23400: [SPARK-26499] [SQL] JdbcUtils.makeGetter does not handle ByteType

2018-12-31 Thread GitBox
AmplabJenkins removed a comment on issue #23400: [SPARK-26499] [SQL] JdbcUtils.makeGetter does not handle ByteType URL: https://github.com/apache/spark/pull/23400#issuecomment-450710514 Merged build finished. Test PASSed.

[GitHub] AmplabJenkins commented on issue #23400: [SPARK-26499] [SQL] JdbcUtils.makeGetter does not handle ByteType

2018-12-31 Thread GitBox
AmplabJenkins commented on issue #23400: [SPARK-26499] [SQL] JdbcUtils.makeGetter does not handle ByteType URL: https://github.com/apache/spark/pull/23400#issuecomment-450710514 Merged build finished. Test PASSed. This is an

[GitHub] AmplabJenkins commented on issue #22617: [SPARK-25484][SQL][TEST] Refactor ExternalAppendOnlyUnsafeRowArrayBenchmark

2018-12-31 Thread GitBox
AmplabJenkins commented on issue #22617: [SPARK-25484][SQL][TEST] Refactor ExternalAppendOnlyUnsafeRowArrayBenchmark URL: https://github.com/apache/spark/pull/22617#issuecomment-450712071 Merged build finished. Test FAILed.

[GitHub] asfgit closed pull request #23400: [SPARK-26499] [SQL] JdbcUtils.makeGetter does not handle ByteType

2018-12-31 Thread GitBox
asfgit closed pull request #23400: [SPARK-26499] [SQL] JdbcUtils.makeGetter does not handle ByteType URL: https://github.com/apache/spark/pull/23400 This is a PR merged from a forked repository. As GitHub hides the original diff on merge, it is displayed below for the sake of provenance:

[GitHub] gatorsmile commented on issue #22617: [SPARK-25484][SQL][TEST] Refactor ExternalAppendOnlyUnsafeRowArrayBenchmark

2018-12-31 Thread GitBox
gatorsmile commented on issue #22617: [SPARK-25484][SQL][TEST] Refactor ExternalAppendOnlyUnsafeRowArrayBenchmark URL: https://github.com/apache/spark/pull/22617#issuecomment-450711900 Please help review it @dongjoon-hyun @kiszk @wangyum

[GitHub] SparkQA commented on issue #22617: [SPARK-25484][SQL][TEST] Refactor ExternalAppendOnlyUnsafeRowArrayBenchmark

2018-12-31 Thread GitBox
SparkQA commented on issue #22617: [SPARK-25484][SQL][TEST] Refactor ExternalAppendOnlyUnsafeRowArrayBenchmark URL: https://github.com/apache/spark/pull/22617#issuecomment-450711856 **[Test build #100609 has

[GitHub] gatorsmile commented on a change in pull request #20433: [SPARK-23264][SQL] Make INTERVAL keyword optional in INTERVAL clauses

2018-12-31 Thread GitBox
gatorsmile commented on a change in pull request #20433: [SPARK-23264][SQL] Make INTERVAL keyword optional in INTERVAL clauses URL: https://github.com/apache/spark/pull/20433#discussion_r244629240 ## File path: sql/core/src/test/resources/sql-tests/inputs/interval.sql ##

[GitHub] maropu commented on a change in pull request #20433: [SPARK-23264][SQL] Make INTERVAL keyword optional in INTERVAL clauses

2018-12-31 Thread GitBox
maropu commented on a change in pull request #20433: [SPARK-23264][SQL] Make INTERVAL keyword optional in INTERVAL clauses URL: https://github.com/apache/spark/pull/20433#discussion_r24463 ## File path: sql/core/src/test/resources/sql-tests/inputs/interval.sql ## @@

[GitHub] SparkQA commented on issue #23420: [SPARK-26508][Core][SQL] Address warning messages in Java reported at lgtm.com

2019-01-01 Thread GitBox
SparkQA commented on issue #23420: [SPARK-26508][Core][SQL] Address warning messages in Java reported at lgtm.com URL: https://github.com/apache/spark/pull/23420#issuecomment-450715697 **[Test build #100606 has

[GitHub] chanansh commented on a change in pull request #23414: [Spark 26449][PYSPARK] add a transform method to the Dataframe class

2019-01-01 Thread GitBox
chanansh commented on a change in pull request #23414: [Spark 26449][PYSPARK] add a transform method to the Dataframe class URL: https://github.com/apache/spark/pull/23414#discussion_r244631080 ## File path: python/pyspark/sql/dataframe.py ## @@ -2046,6 +2046,40 @@ def

[GitHub] chanansh commented on a change in pull request #23414: [Spark 26449][PYSPARK] add a transform method to the Dataframe class

2019-01-01 Thread GitBox
chanansh commented on a change in pull request #23414: [Spark 26449][PYSPARK] add a transform method to the Dataframe class URL: https://github.com/apache/spark/pull/23414#discussion_r244631053 ## File path: python/pyspark/sql/dataframe.py ## @@ -2046,6 +2046,40 @@ def

[GitHub] chanansh commented on a change in pull request #23414: [Spark 26449][PYSPARK] add a transform method to the Dataframe class

2019-01-01 Thread GitBox
chanansh commented on a change in pull request #23414: [Spark 26449][PYSPARK] add a transform method to the Dataframe class URL: https://github.com/apache/spark/pull/23414#discussion_r244631026 ## File path: python/pyspark/sql/dataframe.py ## @@ -2046,6 +2046,40 @@ def

[GitHub] chanansh commented on a change in pull request #23414: [Spark 26449][PYSPARK] add a transform method to the Dataframe class

2019-01-01 Thread GitBox
chanansh commented on a change in pull request #23414: [Spark 26449][PYSPARK] add a transform method to the Dataframe class URL: https://github.com/apache/spark/pull/23414#discussion_r244631035 ## File path: python/pyspark/sql/dataframe.py ## @@ -2046,6 +2046,40 @@ def

[GitHub] chanansh commented on a change in pull request #23414: [Spark 26449][PYSPARK] add a transform method to the Dataframe class

2019-01-01 Thread GitBox
chanansh commented on a change in pull request #23414: [Spark 26449][PYSPARK] add a transform method to the Dataframe class URL: https://github.com/apache/spark/pull/23414#discussion_r244631041 ## File path: python/pyspark/sql/dataframe.py ## @@ -2046,6 +2046,40 @@ def

[GitHub] SparkQA commented on issue #23414: [SPARK-26449][PYTHON] add a transform method to the Dataframe class

2019-01-01 Thread GitBox
SparkQA commented on issue #23414: [SPARK-26449][PYTHON] add a transform method to the Dataframe class URL: https://github.com/apache/spark/pull/23414#issuecomment-450717557 **[Test build #100610 has

[GitHub] chanansh commented on issue #23414: [SPARK-26449][PYTHON] add a transform method to the Dataframe class

2019-01-01 Thread GitBox
chanansh commented on issue #23414: [SPARK-26449][PYTHON] add a transform method to the Dataframe class URL: https://github.com/apache/spark/pull/23414#issuecomment-450718473 @HyukjinKwon, please review latest. This is an

[GitHub] AmplabJenkins commented on issue #23414: [SPARK-26449][PYTHON] add a transform method to the Dataframe class

2019-01-01 Thread GitBox
AmplabJenkins commented on issue #23414: [SPARK-26449][PYTHON] add a transform method to the Dataframe class URL: https://github.com/apache/spark/pull/23414#issuecomment-450719242 Test PASSed. Refer to this link for build results (access rights to CI server needed):

[GitHub] AmplabJenkins commented on issue #23414: [SPARK-26449][PYTHON] add a transform method to the Dataframe class

2019-01-01 Thread GitBox
AmplabJenkins commented on issue #23414: [SPARK-26449][PYTHON] add a transform method to the Dataframe class URL: https://github.com/apache/spark/pull/23414#issuecomment-450719241 Merged build finished. Test PASSed. This is

[GitHub] AmplabJenkins removed a comment on issue #23414: [SPARK-26449][PYTHON] add a transform method to the Dataframe class

2019-01-01 Thread GitBox
AmplabJenkins removed a comment on issue #23414: [SPARK-26449][PYTHON] add a transform method to the Dataframe class URL: https://github.com/apache/spark/pull/23414#issuecomment-450719242 Test PASSed. Refer to this link for build results (access rights to CI server needed):

[GitHub] AmplabJenkins removed a comment on issue #23414: [SPARK-26449][PYTHON] add a transform method to the Dataframe class

2019-01-01 Thread GitBox
AmplabJenkins removed a comment on issue #23414: [SPARK-26449][PYTHON] add a transform method to the Dataframe class URL: https://github.com/apache/spark/pull/23414#issuecomment-450719241 Merged build finished. Test PASSed.

[GitHub] SparkQA removed a comment on issue #23414: [SPARK-26449][PYTHON] add a transform method to the Dataframe class

2019-01-01 Thread GitBox
SparkQA removed a comment on issue #23414: [SPARK-26449][PYTHON] add a transform method to the Dataframe class URL: https://github.com/apache/spark/pull/23414#issuecomment-450718171 **[Test build #100612 has

[GitHub] SparkQA commented on issue #23414: [SPARK-26449][PYTHON] add a transform method to the Dataframe class

2019-01-01 Thread GitBox
SparkQA commented on issue #23414: [SPARK-26449][PYTHON] add a transform method to the Dataframe class URL: https://github.com/apache/spark/pull/23414#issuecomment-450719207 **[Test build #100612 has

[GitHub] SparkQA commented on issue #23391: [SPARK-26456][SQL] Cast date/timestamp to string by Date/TimestampFormatter

2019-01-01 Thread GitBox
SparkQA commented on issue #23391: [SPARK-26456][SQL] Cast date/timestamp to string by Date/TimestampFormatter URL: https://github.com/apache/spark/pull/23391#issuecomment-450719982 **[Test build #100614 has

[GitHub] SparkQA commented on issue #23393: [SPARK-26288][CORE]add initRegisteredExecutorsDB

2019-01-01 Thread GitBox
SparkQA commented on issue #23393: [SPARK-26288][CORE]add initRegisteredExecutorsDB URL: https://github.com/apache/spark/pull/23393#issuecomment-450721965 **[Test build #100608 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/100608/testReport)** for PR

[GitHub] AmplabJenkins commented on issue #23393: [SPARK-26288][CORE]add initRegisteredExecutorsDB

2019-01-01 Thread GitBox
AmplabJenkins commented on issue #23393: [SPARK-26288][CORE]add initRegisteredExecutorsDB URL: https://github.com/apache/spark/pull/23393#issuecomment-450722077 Merged build finished. Test PASSed. This is an automated

[GitHub] AmplabJenkins removed a comment on issue #23393: [SPARK-26288][CORE]add initRegisteredExecutorsDB

2019-01-01 Thread GitBox
AmplabJenkins removed a comment on issue #23393: [SPARK-26288][CORE]add initRegisteredExecutorsDB URL: https://github.com/apache/spark/pull/23393#issuecomment-450722077 Merged build finished. Test PASSed. This is an

[GitHub] AmplabJenkins commented on issue #23393: [SPARK-26288][CORE]add initRegisteredExecutorsDB

2019-01-01 Thread GitBox
AmplabJenkins commented on issue #23393: [SPARK-26288][CORE]add initRegisteredExecutorsDB URL: https://github.com/apache/spark/pull/23393#issuecomment-450722078 Test PASSed. Refer to this link for build results (access rights to CI server needed):

[GitHub] SparkQA removed a comment on issue #23393: [SPARK-26288][CORE]add initRegisteredExecutorsDB

2019-01-01 Thread GitBox
SparkQA removed a comment on issue #23393: [SPARK-26288][CORE]add initRegisteredExecutorsDB URL: https://github.com/apache/spark/pull/23393#issuecomment-450711701 **[Test build #100608 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/100608/testReport)**

[GitHub] srowen commented on a change in pull request #23419: [SPARK-26507][CORE] Fix core tests for Java 11

2018-12-31 Thread GitBox
srowen commented on a change in pull request #23419: [SPARK-26507][CORE] Fix core tests for Java 11 URL: https://github.com/apache/spark/pull/23419#discussion_r244610615 ## File path: pom.xml ## @@ -2113,7 +2115,7 @@ **/*Suite.java

[GitHub] AmplabJenkins removed a comment on issue #23401: Trigger GC on executor node idle

2018-12-31 Thread GitBox
AmplabJenkins removed a comment on issue #23401: Trigger GC on executor node idle URL: https://github.com/apache/spark/pull/23401#issuecomment-450680325 Can one of the admins verify this patch? This is an automated message

[GitHub] AmplabJenkins commented on issue #23401: Trigger GC on executor node idle

2018-12-31 Thread GitBox
AmplabJenkins commented on issue #23401: Trigger GC on executor node idle URL: https://github.com/apache/spark/pull/23401#issuecomment-450680671 Can one of the admins verify this patch? This is an automated message from the

[GitHub] SparkQA removed a comment on issue #23420: [SPARK-26508][Core][SQL] Address warning messages in Java reported at lgtm.com

2018-12-31 Thread GitBox
SparkQA removed a comment on issue #23420: [SPARK-26508][Core][SQL] Address warning messages in Java reported at lgtm.com URL: https://github.com/apache/spark/pull/23420#issuecomment-450664471 **[Test build #100598 has

[GitHub] AmplabJenkins commented on issue #23420: [SPARK-26508][Core][SQL] Address warning messages in Java reported at lgtm.com

2018-12-31 Thread GitBox
AmplabJenkins commented on issue #23420: [SPARK-26508][Core][SQL] Address warning messages in Java reported at lgtm.com URL: https://github.com/apache/spark/pull/23420#issuecomment-450686863 Test FAILed. Refer to this link for build results (access rights to CI server needed):

[GitHub] AmplabJenkins commented on issue #23420: [SPARK-26508][Core][SQL] Address warning messages in Java reported at lgtm.com

2018-12-31 Thread GitBox
AmplabJenkins commented on issue #23420: [SPARK-26508][Core][SQL] Address warning messages in Java reported at lgtm.com URL: https://github.com/apache/spark/pull/23420#issuecomment-450686862 Merged build finished. Test FAILed.

[GitHub] AmplabJenkins removed a comment on issue #23420: [SPARK-26508][Core][SQL] Address warning messages in Java reported at lgtm.com

2018-12-31 Thread GitBox
AmplabJenkins removed a comment on issue #23420: [SPARK-26508][Core][SQL] Address warning messages in Java reported at lgtm.com URL: https://github.com/apache/spark/pull/23420#issuecomment-450686862 Merged build finished. Test FAILed.

[GitHub] MaxGekk commented on a change in pull request #23417: [SPARK-26374][TEST][SQL] Enable TimestampFormatter in HadoopFsRelationTest

2018-12-31 Thread GitBox
MaxGekk commented on a change in pull request #23417: [SPARK-26374][TEST][SQL] Enable TimestampFormatter in HadoopFsRelationTest URL: https://github.com/apache/spark/pull/23417#discussion_r244620988 ## File path:

[GitHub] MaxGekk commented on a change in pull request #23417: [SPARK-26374][TEST][SQL] Enable TimestampFormatter in HadoopFsRelationTest

2018-12-31 Thread GitBox
MaxGekk commented on a change in pull request #23417: [SPARK-26374][TEST][SQL] Enable TimestampFormatter in HadoopFsRelationTest URL: https://github.com/apache/spark/pull/23417#discussion_r244620988 ## File path:

[GitHub] SandishKumarHN commented on issue #23401: [SPARK-26513][Core] : Trigger GC on executor node idle

2018-12-31 Thread GitBox
SandishKumarHN commented on issue #23401: [SPARK-26513][Core] : Trigger GC on executor node idle URL: https://github.com/apache/spark/pull/23401#issuecomment-450694795 @srowen we have 1TB data persisted on the cache and 100 GB data gets cached and unpersisted every 15 minutes. we found

[GitHub] srowen commented on issue #23401: [SPARK-26513][Core] : Trigger GC on executor node idle

2018-12-31 Thread GitBox
srowen commented on issue #23401: [SPARK-26513][Core] : Trigger GC on executor node idle URL: https://github.com/apache/spark/pull/23401#issuecomment-450697159 I wonder what happens if you just set something like `-XX:InitiatingHeapOccupancyPercent=25` to more aggressively GC along the

[GitHub] kiszk commented on a change in pull request #23420: [SPARK-26508][Core][SQL] Address warning messages in Java reported at lgtm.com

2018-12-31 Thread GitBox
kiszk commented on a change in pull request #23420: [SPARK-26508][Core][SQL] Address warning messages in Java reported at lgtm.com URL: https://github.com/apache/spark/pull/23420#discussion_r244623896 ## File path:

[GitHub] rdblue edited a comment on issue #23401: [SPARK-26513][Core] : Trigger GC on executor node idle

2018-12-31 Thread GitBox
rdblue edited a comment on issue #23401: [SPARK-26513][Core] : Trigger GC on executor node idle URL: https://github.com/apache/spark/pull/23401#issuecomment-450701012 > Dynamic scale down is often done fairly conservatively when combined with cached blocks I agree. We (Netflix)

[GitHub] rdblue commented on issue #23401: [SPARK-26513][Core] : Trigger GC on executor node idle

2018-12-31 Thread GitBox
rdblue commented on issue #23401: [SPARK-26513][Core] : Trigger GC on executor node idle URL: https://github.com/apache/spark/pull/23401#issuecomment-450701012 > Dynamic scale down is often done fairly conservatively when combined with cached blocks I agree. We actually don't

[GitHub] HyukjinKwon commented on a change in pull request #23184: [SPARK-26227][R] from_[csv|json] should accept schema_of_[csv|json] in R API

2018-12-31 Thread GitBox
HyukjinKwon commented on a change in pull request #23184: [SPARK-26227][R] from_[csv|json] should accept schema_of_[csv|json] in R API URL: https://github.com/apache/spark/pull/23184#discussion_r244626719 ## File path: R/pkg/R/functions.R ## @@ -2328,22 +2343,27 @@

[GitHub] felixcheung commented on a change in pull request #23419: [SPARK-26507][CORE] Fix core tests for Java 11

2018-12-31 Thread GitBox
felixcheung commented on a change in pull request #23419: [SPARK-26507][CORE] Fix core tests for Java 11 URL: https://github.com/apache/spark/pull/23419#discussion_r244626715 ## File path: pom.xml ## @@ -2113,7 +2115,7 @@ **/*Suite.java

[GitHub] SparkQA commented on issue #23400: [SPARK-26499] [SQL] JdbcUtils.makeGetter does not handle ByteType

2018-12-31 Thread GitBox
SparkQA commented on issue #23400: [SPARK-26499] [SQL] JdbcUtils.makeGetter does not handle ByteType URL: https://github.com/apache/spark/pull/23400#issuecomment-450706630 **[Test build #4491 has

[GitHub] gatorsmile commented on a change in pull request #23391: [SPARK-26456][SQL] Cast date/timestamp to string by Date/TimestampFormatter

2018-12-31 Thread GitBox
gatorsmile commented on a change in pull request #23391: [SPARK-26456][SQL] Cast date/timestamp to string by Date/TimestampFormatter URL: https://github.com/apache/spark/pull/23391#discussion_r244627486 ## File path:

[GitHub] singhpraveen2010 commented on issue #22758: [SPARK-25332][SQL] Instead of broadcast hash join , Sort merge join node is added in the plan for the join queries executed in a new spark session/

2018-12-31 Thread GitBox
singhpraveen2010 commented on issue #22758: [SPARK-25332][SQL] Instead of broadcast hash join ,Sort merge join node is added in the plan for the join queries executed in a new spark session/context URL: https://github.com/apache/spark/pull/22758#issuecomment-450707984 test

[GitHub] SparkQA commented on issue #23400: [SPARK-26499] [SQL] JdbcUtils.makeGetter does not handle ByteType

2018-12-31 Thread GitBox
SparkQA commented on issue #23400: [SPARK-26499] [SQL] JdbcUtils.makeGetter does not handle ByteType URL: https://github.com/apache/spark/pull/23400#issuecomment-450709144 **[Test build #100601 has

[GitHub] SandishKumarHN edited a comment on issue #23401: [SPARK-26513][Core] : Trigger GC on executor node idle

2018-12-31 Thread GitBox
SandishKumarHN edited a comment on issue #23401: [SPARK-26513][Core] : Trigger GC on executor node idle URL: https://github.com/apache/spark/pull/23401#issuecomment-450708903 @gatorsmile that's what we are doing right now. we have taken spark source code modified with a param

[GitHub] AmplabJenkins removed a comment on issue #23420: [SPARK-26508][Core][SQL] Address warning messages in Java reported at lgtm.com

2018-12-31 Thread GitBox
AmplabJenkins removed a comment on issue #23420: [SPARK-26508][Core][SQL] Address warning messages in Java reported at lgtm.com URL: https://github.com/apache/spark/pull/23420#issuecomment-450709693 Test PASSed. Refer to this link for build results (access rights to CI server needed):

[GitHub] AmplabJenkins commented on issue #23420: [SPARK-26508][Core][SQL] Address warning messages in Java reported at lgtm.com

2018-12-31 Thread GitBox
AmplabJenkins commented on issue #23420: [SPARK-26508][Core][SQL] Address warning messages in Java reported at lgtm.com URL: https://github.com/apache/spark/pull/23420#issuecomment-450709692 Merged build finished. Test PASSed.

[GitHub] AmplabJenkins removed a comment on issue #23420: [SPARK-26508][Core][SQL] Address warning messages in Java reported at lgtm.com

2018-12-31 Thread GitBox
AmplabJenkins removed a comment on issue #23420: [SPARK-26508][Core][SQL] Address warning messages in Java reported at lgtm.com URL: https://github.com/apache/spark/pull/23420#issuecomment-450709692 Merged build finished. Test PASSed.

[GitHub] AmplabJenkins removed a comment on issue #23421: [MINOR][R] Deduplicate RStudio setup documentation

2018-12-31 Thread GitBox
AmplabJenkins removed a comment on issue #23421: [MINOR][R] Deduplicate RStudio setup documentation URL: https://github.com/apache/spark/pull/23421#issuecomment-450709653 Merged build finished. Test PASSed. This is an

[GitHub] AmplabJenkins commented on issue #23420: [SPARK-26508][Core][SQL] Address warning messages in Java reported at lgtm.com

2018-12-31 Thread GitBox
AmplabJenkins commented on issue #23420: [SPARK-26508][Core][SQL] Address warning messages in Java reported at lgtm.com URL: https://github.com/apache/spark/pull/23420#issuecomment-450709693 Test PASSed. Refer to this link for build results (access rights to CI server needed):

[GitHub] SparkQA commented on issue #23422: [SPARK-26514][CORE] Support running multi tasks per cpu core

2018-12-31 Thread GitBox
SparkQA commented on issue #23422: [SPARK-26514][CORE] Support running multi tasks per cpu core URL: https://github.com/apache/spark/pull/23422#issuecomment-450710240 **[Test build #100607 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/100607/testReport)**

[GitHub] gatorsmile commented on issue #23259: [SPARK-26215][SQL][WIP] Define reserved/non-reserved keywords based on the ANSI SQL standard

2018-12-31 Thread GitBox
gatorsmile commented on issue #23259: [SPARK-26215][SQL][WIP] Define reserved/non-reserved keywords based on the ANSI SQL standard URL: https://github.com/apache/spark/pull/23259#issuecomment-450710217 ping @maropu Anything is blocking this PR?

[GitHub] caneGuy opened a new pull request #23422: [SPARK-26514][CORE] Support running multi tasks per cpu core

2018-12-31 Thread GitBox
caneGuy opened a new pull request #23422: [SPARK-26514][CORE] Support running multi tasks per cpu core URL: https://github.com/apache/spark/pull/23422 ## What changes were proposed in this pull request? Introduce `spark.cpu.tasks` to improve performance for some applications and

[GitHub] AmplabJenkins commented on issue #22617: [SPARK-25484][SQL][TEST] Refactor ExternalAppendOnlyUnsafeRowArrayBenchmark

2018-12-31 Thread GitBox
AmplabJenkins commented on issue #22617: [SPARK-25484][SQL][TEST] Refactor ExternalAppendOnlyUnsafeRowArrayBenchmark URL: https://github.com/apache/spark/pull/22617#issuecomment-450712080 Merged build finished. Test PASSed.

[GitHub] AmplabJenkins removed a comment on issue #22617: [SPARK-25484][SQL][TEST] Refactor ExternalAppendOnlyUnsafeRowArrayBenchmark

2018-12-31 Thread GitBox
AmplabJenkins removed a comment on issue #22617: [SPARK-25484][SQL][TEST] Refactor ExternalAppendOnlyUnsafeRowArrayBenchmark URL: https://github.com/apache/spark/pull/22617#issuecomment-450712071 Merged build finished. Test FAILed.

[GitHub] maropu commented on issue #20433: [SPARK-23264][SQL] Make INTERVAL keyword optional in INTERVAL clauses

2018-12-31 Thread GitBox
maropu commented on issue #20433: [SPARK-23264][SQL] Make INTERVAL keyword optional in INTERVAL clauses URL: https://github.com/apache/spark/pull/20433#issuecomment-450712099 I think #23259 might have much room for discussions, so how about taking this with the new ansi option first, and

[GitHub] AmplabJenkins commented on issue #22617: [SPARK-25484][SQL][TEST] Refactor ExternalAppendOnlyUnsafeRowArrayBenchmark

2018-12-31 Thread GitBox
AmplabJenkins commented on issue #22617: [SPARK-25484][SQL][TEST] Refactor ExternalAppendOnlyUnsafeRowArrayBenchmark URL: https://github.com/apache/spark/pull/22617#issuecomment-450712081 Test PASSed. Refer to this link for build results (access rights to CI server needed):

[GitHub] AmplabJenkins removed a comment on issue #22617: [SPARK-25484][SQL][TEST] Refactor ExternalAppendOnlyUnsafeRowArrayBenchmark

2018-12-31 Thread GitBox
AmplabJenkins removed a comment on issue #22617: [SPARK-25484][SQL][TEST] Refactor ExternalAppendOnlyUnsafeRowArrayBenchmark URL: https://github.com/apache/spark/pull/22617#issuecomment-450712081 Test PASSed. Refer to this link for build results (access rights to CI server needed):

<    1   2   3   4   5   6   7   8   9   10   >