[jira] [Updated] (SPARK-23890) Support DDL for adding nested columns to struct types

2023-12-21 Thread Andrew Otto (Jira)
[ https://issues.apache.org/jira/browse/SPARK-23890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Otto updated SPARK-23890: Description: As part of SPARK-14118, Spark SQL removed support for sending ALTER TABLE CHANGE

[jira] [Updated] (SPARK-23890) Support DDL for adding nested columns to struct types

2023-12-21 Thread Andrew Otto (Jira)
[ https://issues.apache.org/jira/browse/SPARK-23890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Otto updated SPARK-23890: Summary: Support DDL for adding nested columns to struct types (was: Hive ALTER TABLE CHANGE

[jira] [Reopened] (SPARK-23890) Hive ALTER TABLE CHANGE COLUMN for struct type no longer works

2023-12-21 Thread Andrew Otto (Jira)
[ https://issues.apache.org/jira/browse/SPARK-23890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Otto reopened SPARK-23890: - This is fixed for DataSource v2 via {{{}alter table add column nested.new_field0{}}}, but

[jira] [Resolved] (SPARK-23890) Hive ALTER TABLE CHANGE COLUMN for struct type no longer works

2023-12-15 Thread Andrew Otto (Jira)
[ https://issues.apache.org/jira/browse/SPARK-23890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Otto resolved SPARK-23890. - Fix Version/s: 3.0.0 Resolution: Fixed Ah! This is supported in DataSource v2 after all,

[jira] [Updated] (SPARK-23890) Hive ALTER TABLE CHANGE COLUMN for struct type no longer works

2023-12-14 Thread Andrew Otto (Jira)
[ https://issues.apache.org/jira/browse/SPARK-23890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Otto updated SPARK-23890: Shepherd: Max Gekk Affects Version/s: 3.0.0 Description: As part of 

[jira] [Reopened] (SPARK-23890) Hive ALTER TABLE CHANGE COLUMN for struct type no longer works

2023-12-14 Thread Andrew Otto (Jira)
[ https://issues.apache.org/jira/browse/SPARK-23890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Otto reopened SPARK-23890: - This was supposed to have been fixed in Spark 3 datasource v2, but the issue persists. > Hive

[jira] [Comment Edited] (SPARK-43273) Spark can't read parquet files with a newer LZ4_RAW compression

2023-04-25 Thread Andrew Grigorev (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43273?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17716135#comment-17716135 ] Andrew Grigorev edited comment on SPARK-43273 at 4/25/23 12:56 PM: ---

[jira] [Updated] (SPARK-43273) Spark can't read parquet files with a newer LZ4_RAW compression

2023-04-25 Thread Andrew Grigorev (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43273?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Grigorev updated SPARK-43273: Description: hadoop-parquet version should be updated to 1.3.0 (together with other

[jira] [Commented] (SPARK-43273) Spark can't read parquet files with a newer LZ4_RAW compression

2023-04-25 Thread Andrew Grigorev (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43273?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17716135#comment-17716135 ] Andrew Grigorev commented on SPARK-43273: - Just as a icing on the cake - Clickhouse accidently

[jira] [Updated] (SPARK-43273) Spark can't read parquet files with a newer LZ4_RAW compression

2023-04-25 Thread Andrew Grigorev (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43273?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Grigorev updated SPARK-43273: Description: hadoop-parquet version should be updated to 1.3.0   {code:java}

[jira] [Created] (SPARK-43273) Spark can't read parquet files with a newer LZ4_RAW compression

2023-04-25 Thread Andrew Grigorev (Jira)
Andrew Grigorev created SPARK-43273: --- Summary: Spark can't read parquet files with a newer LZ4_RAW compression Key: SPARK-43273 URL: https://issues.apache.org/jira/browse/SPARK-43273 Project: Spark

[jira] [Updated] (SPARK-43189) No overload variant of "pandas_udf" matches argument type "str"

2023-04-19 Thread Andrew Grigorev (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Grigorev updated SPARK-43189: Description: h2. Issue Users who have mypy enabled in their IDE or CI environment face

[jira] [Created] (SPARK-43189) No overload variant of "pandas_udf" matches argument type "str"

2023-04-19 Thread Andrew Grigorev (Jira)
Andrew Grigorev created SPARK-43189: --- Summary: No overload variant of "pandas_udf" matches argument type "str" Key: SPARK-43189 URL: https://issues.apache.org/jira/browse/SPARK-43189 Project: Spark

[jira] [Created] (SPARK-39897) StackOverflowError in TaskMemoryManager

2022-07-27 Thread Andrew Ray (Jira)
Andrew Ray created SPARK-39897: -- Summary: StackOverflowError in TaskMemoryManager Key: SPARK-39897 URL: https://issues.apache.org/jira/browse/SPARK-39897 Project: Spark Issue Type: Bug

[jira] [Created] (SPARK-39883) Add DataFrame function parity check

2022-07-26 Thread Andrew Ray (Jira)
Andrew Ray created SPARK-39883: -- Summary: Add DataFrame function parity check Key: SPARK-39883 URL: https://issues.apache.org/jira/browse/SPARK-39883 Project: Spark Issue Type: Improvement

[jira] [Created] (SPARK-39734) Add call_udf to pyspark.sql.functions

2022-07-10 Thread Andrew Ray (Jira)
Andrew Ray created SPARK-39734: -- Summary: Add call_udf to pyspark.sql.functions Key: SPARK-39734 URL: https://issues.apache.org/jira/browse/SPARK-39734 Project: Spark Issue Type: Improvement

[jira] [Created] (SPARK-39733) Add map_contains_key to pyspark.sql.functions

2022-07-10 Thread Andrew Ray (Jira)
Andrew Ray created SPARK-39733: -- Summary: Add map_contains_key to pyspark.sql.functions Key: SPARK-39733 URL: https://issues.apache.org/jira/browse/SPARK-39733 Project: Spark Issue Type:

[jira] [Updated] (SPARK-39728) Test for parity of SQL functions between Python and JVM DataFrame API's

2022-07-09 Thread Andrew Ray (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39728?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Ray updated SPARK-39728: --- Priority: Minor (was: Major) > Test for parity of SQL functions between Python and JVM DataFrame

[jira] [Created] (SPARK-39728) Test for parity of SQL functions between Python and JVM DataFrame API's

2022-07-09 Thread Andrew Ray (Jira)
Andrew Ray created SPARK-39728: -- Summary: Test for parity of SQL functions between Python and JVM DataFrame API's Key: SPARK-39728 URL: https://issues.apache.org/jira/browse/SPARK-39728 Project: Spark

[jira] [Created] (SPARK-38618) Implement JDBCDataSourceV2

2022-03-21 Thread Andrew Murphy (Jira)
Andrew Murphy created SPARK-38618: - Summary: Implement JDBCDataSourceV2 Key: SPARK-38618 URL: https://issues.apache.org/jira/browse/SPARK-38618 Project: Spark Issue Type: Sub-task

[jira] [Commented] (SPARK-38288) Aggregate push down doesnt work using Spark SQL jdbc datasource with postgresql

2022-02-27 Thread Andrew Murphy (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17498668#comment-17498668 ] Andrew Murphy commented on SPARK-38288: --- Hi [~llozano] I believe this is because JDBC DataSource

[jira] [Updated] (SPARK-38296) Error Classes AnalysisException is not propagated in FunctionRegistry

2022-02-22 Thread Andrew Murphy (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38296?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Murphy updated SPARK-38296: -- Description: While trying to migrate invalidLiteralForWindowDurationError to use Error

[jira] [Updated] (SPARK-38296) Error Classes AnalysisException is not propagated in FunctionRegistry

2022-02-22 Thread Andrew Murphy (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38296?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Murphy updated SPARK-38296: -- External issue ID: (was: SPARK-38110) External issue URL: (was:

[jira] [Created] (SPARK-38296) Error Classes AnalysisException is not propagated in FunctionRegistry

2022-02-22 Thread Andrew Murphy (Jira)
Andrew Murphy created SPARK-38296: - Summary: Error Classes AnalysisException is not propagated in FunctionRegistry Key: SPARK-38296 URL: https://issues.apache.org/jira/browse/SPARK-38296 Project:

[jira] [Commented] (SPARK-37954) old columns should not be available after select or drop

2022-02-10 Thread Andrew Murphy (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17490630#comment-17490630 ] Andrew Murphy commented on SPARK-37954: ---  Hi all, Super excited to report on this one, this is my

[jira] [Created] (SPARK-37295) illegal reflective access operation has occurred; Please consider reporting this to the maintainers

2021-11-11 Thread Andrew Davidson (Jira)
Andrew Davidson created SPARK-37295: --- Summary: illegal reflective access operation has occurred; Please consider reporting this to the maintainers Key: SPARK-37295 URL:

[jira] [Commented] (SPARK-36649) Support Trigger.AvailableNow on Kafka data source

2021-09-07 Thread Andrew Olson (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17411256#comment-17411256 ] Andrew Olson commented on SPARK-36649: -- [~kabhwan] I am not sure I fully understand the details of

[jira] [Updated] (SPARK-36576) Improve range split calculation for Kafka Source minPartitions option

2021-08-24 Thread Andrew Olson (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36576?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Olson updated SPARK-36576: - Description: While the

[jira] [Created] (SPARK-36576) Improve range split calculation for Kafka Source minPartitions option

2021-08-24 Thread Andrew Olson (Jira)
Andrew Olson created SPARK-36576: Summary: Improve range split calculation for Kafka Source minPartitions option Key: SPARK-36576 URL: https://issues.apache.org/jira/browse/SPARK-36576 Project: Spark

[jira] [Commented] (SPARK-29223) Kafka source: offset by timestamp - allow specifying timestamp for "all partitions"

2021-08-23 Thread Andrew Grigorev (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17403341#comment-17403341 ] Andrew Grigorev commented on SPARK-29223: - Just an idea - couldn't this be implemented in

[jira] [Created] (SPARK-35919) Support pathlib.PurePath-like objects in DataFrameReader / DataFrameWriter

2021-06-28 Thread Andrew Grigorev (Jira)
Andrew Grigorev created SPARK-35919: --- Summary: Support pathlib.PurePath-like objects in DataFrameReader / DataFrameWriter Key: SPARK-35919 URL: https://issues.apache.org/jira/browse/SPARK-35919

[jira] [Commented] (SPARK-34723) Correct parameter type for subexpression elimination under whole-stage

2021-06-24 Thread Andrew Olson (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17368894#comment-17368894 ] Andrew Olson commented on SPARK-34723: -- In case it might be helpful to anyone, the compilation

[jira] [Commented] (SPARK-22876) spark.yarn.am.attemptFailuresValidityInterval does not work correctly

2019-12-09 Thread Andrew White (Jira)
[ https://issues.apache.org/jira/browse/SPARK-22876?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16991752#comment-16991752 ] Andrew White commented on SPARK-22876: -- This issue is most certainly unresolved, and the

[jira] [Commented] (SPARK-28085) Spark Scala API documentation URLs not working properly in Chrome

2019-08-08 Thread Andrew Leverentz (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16903377#comment-16903377 ] Andrew Leverentz commented on SPARK-28085: -- In Chrome 76, this issue appears to be resolved. 

[jira] [Commented] (SPARK-28085) Spark Scala API documentation URLs not working properly in Chrome

2019-07-22 Thread Andrew Leverentz (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16890314#comment-16890314 ] Andrew Leverentz commented on SPARK-28085: -- This issue still remains, more than a month after

[jira] [Resolved] (SPARK-28225) Unexpected behavior for Window functions

2019-07-22 Thread Andrew Leverentz (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28225?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Leverentz resolved SPARK-28225. -- Resolution: Not A Problem > Unexpected behavior for Window functions >

[jira] [Commented] (SPARK-28225) Unexpected behavior for Window functions

2019-07-22 Thread Andrew Leverentz (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16890310#comment-16890310 ] Andrew Leverentz commented on SPARK-28225: -- Marco, thanks for the explanation.  In this case,

[jira] [Comment Edited] (SPARK-28225) Unexpected behavior for Window functions

2019-07-22 Thread Andrew Leverentz (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16890310#comment-16890310 ] Andrew Leverentz edited comment on SPARK-28225 at 7/22/19 4:54 PM: ---

[jira] [Created] (SPARK-28225) Unexpected behavior for Window functions

2019-07-01 Thread Andrew Leverentz (JIRA)
Andrew Leverentz created SPARK-28225: Summary: Unexpected behavior for Window functions Key: SPARK-28225 URL: https://issues.apache.org/jira/browse/SPARK-28225 Project: Spark Issue Type:

[jira] [Updated] (SPARK-28085) Spark Scala API documentation URLs not working properly in Chrome

2019-06-18 Thread Andrew Leverentz (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28085?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Leverentz updated SPARK-28085: - Description: In Chrome version 75, URLs in the Scala API documentation are not working

[jira] [Created] (SPARK-28085) Spark Scala API documentation URLs not working properly in Chrome

2019-06-17 Thread Andrew Leverentz (JIRA)
Andrew Leverentz created SPARK-28085: Summary: Spark Scala API documentation URLs not working properly in Chrome Key: SPARK-28085 URL: https://issues.apache.org/jira/browse/SPARK-28085 Project:

[jira] [Created] (SPARK-28062) HuberAggregator copies coefficients vector every time an instance is added

2019-06-15 Thread Andrew Crosby (JIRA)
Andrew Crosby created SPARK-28062: - Summary: HuberAggregator copies coefficients vector every time an instance is added Key: SPARK-28062 URL: https://issues.apache.org/jira/browse/SPARK-28062

[jira] [Created] (SPARK-27560) HashPartitioner uses Object.hashCode which is not seeded

2019-04-24 Thread Andrew McHarg (JIRA)
Andrew McHarg created SPARK-27560: - Summary: HashPartitioner uses Object.hashCode which is not seeded Key: SPARK-27560 URL: https://issues.apache.org/jira/browse/SPARK-27560 Project: Spark

[jira] [Commented] (SPARK-26970) Can't load PipelineModel that was created in Scala with Python due to missing Interaction transformer

2019-04-20 Thread Andrew Crosby (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16822548#comment-16822548 ] Andrew Crosby commented on SPARK-26970: --- The code changes required for this looked relatively

[jira] [Commented] (SPARK-27216) Kryo serialization with RoaringBitmap

2019-03-28 Thread Andrew Allen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27216?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16804447#comment-16804447 ] Andrew Allen commented on SPARK-27216: -- Thanks for this bug report, [~cltlfcjin]. We we running

[jira] [Updated] (SPARK-26970) Can't load PipelineModel that was created in Scala with Python due to missing Interaction transformer

2019-02-22 Thread Andrew Crosby (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26970?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Crosby updated SPARK-26970: -- Description: The Interaction transformer 

[jira] [Updated] (SPARK-26970) Can't load PipelineModel that was created in Scala with Python due to missing Interaction transformer

2019-02-22 Thread Andrew Crosby (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26970?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Crosby updated SPARK-26970: -- Description: The Interaction transformer 

[jira] [Updated] (SPARK-26970) Can't load PipelineModel that was created in Scala with Python due to missing Interaction transformer

2019-02-22 Thread Andrew Crosby (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26970?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Crosby updated SPARK-26970: -- Description: The Interaction transformer ( 

[jira] [Created] (SPARK-26970) Can't load PipelineModel that was created in Scala with Python due to missing Interaction transformer

2019-02-22 Thread Andrew Crosby (JIRA)
Andrew Crosby created SPARK-26970: - Summary: Can't load PipelineModel that was created in Scala with Python due to missing Interaction transformer Key: SPARK-26970 URL:

[jira] [Commented] (SPARK-22865) Publish Official Apache Spark Docker images

2018-11-21 Thread Andrew Korzhuev (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16695338#comment-16695338 ] Andrew Korzhuev commented on SPARK-22865: - I've added builds for vanilla 2.3.1, 2.3.2 and 2.4.0

[jira] [Comment Edited] (SPARK-22865) Publish Official Apache Spark Docker images

2018-11-21 Thread Andrew Korzhuev (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16695338#comment-16695338 ] Andrew Korzhuev edited comment on SPARK-22865 at 11/21/18 11:00 PM:

[jira] [Comment Edited] (SPARK-22865) Publish Official Apache Spark Docker images

2018-11-21 Thread Andrew Korzhuev (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16695338#comment-16695338 ] Andrew Korzhuev edited comment on SPARK-22865 at 11/21/18 11:00 PM:

[jira] [Comment Edited] (SPARK-22865) Publish Official Apache Spark Docker images

2018-11-21 Thread Andrew Korzhuev (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16695338#comment-16695338 ] Andrew Korzhuev edited comment on SPARK-22865 at 11/21/18 10:59 PM:

[jira] [Commented] (SPARK-14492) Spark SQL 1.6.0 does not work with external Hive metastore version lower than 1.2.0; its not backwards compatible with earlier version

2018-11-19 Thread Andrew Otto (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16691787#comment-16691787 ] Andrew Otto commented on SPARK-14492: - I found my issue: we were loading some Hive 1.1.0 jars

[jira] [Commented] (SPARK-14492) Spark SQL 1.6.0 does not work with external Hive metastore version lower than 1.2.0; its not backwards compatible with earlier version

2018-11-13 Thread Andrew Otto (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16685375#comment-16685375 ] Andrew Otto commented on SPARK-14492: - FWIW, I am experiencing the same problem with Spark 2.3.1 and

[jira] [Commented] (SPARK-25544) Slow/failed convergence in Spark ML models due to internal predictor scaling

2018-10-01 Thread Andrew Crosby (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25544?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16634268#comment-16634268 ] Andrew Crosby commented on SPARK-25544: --- SPARK-23537 contains what might be another occurrence of

[jira] [Commented] (SPARK-23537) Logistic Regression without standardization

2018-10-01 Thread Andrew Crosby (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16634263#comment-16634263 ] Andrew Crosby commented on SPARK-23537: --- The different results for standardization=True vs

[jira] [Updated] (SPARK-25544) Slow/failed convergence in Spark ML models due to internal predictor scaling

2018-09-26 Thread Andrew Crosby (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25544?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Crosby updated SPARK-25544: -- Description: The LinearRegression and LogisticRegression estimators in Spark ML can take a

[jira] [Updated] (SPARK-25544) Slow/failed convergence in Spark ML models due to internal predictor scaling

2018-09-26 Thread Andrew Crosby (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25544?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Crosby updated SPARK-25544: -- Description: The LinearRegression and LogisticRegression estimators in Spark ML can take a

[jira] [Updated] (SPARK-25544) Slow/failed convergence in Spark ML models due to internal predictor scaling

2018-09-26 Thread Andrew Crosby (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25544?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Crosby updated SPARK-25544: -- Description: The LinearRegression and LogisticRegression estimators in Spark ML can take a

[jira] [Created] (SPARK-25544) Slow/failed convergence in Spark ML models due to internal predictor scaling

2018-09-26 Thread Andrew Crosby (JIRA)
Andrew Crosby created SPARK-25544: - Summary: Slow/failed convergence in Spark ML models due to internal predictor scaling Key: SPARK-25544 URL: https://issues.apache.org/jira/browse/SPARK-25544

[jira] [Commented] (SPARK-21962) Distributed Tracing in Spark

2018-07-12 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16541265#comment-16541265 ] Andrew Ash commented on SPARK-21962: Note that HTrace is now being removed from Hadoop – 

[jira] [Commented] (SPARK-24422) Add JDK9+ in our Jenkins' build servers

2018-07-03 Thread Andrew Korzhuev (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16531241#comment-16531241 ] Andrew Korzhuev commented on SPARK-24422: - Also `.travis.yml` needs to be fixed in the following

[jira] [Comment Edited] (SPARK-24422) Add JDK9+ in our Jenkins' build servers

2018-07-03 Thread Andrew Korzhuev (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16531241#comment-16531241 ] Andrew Korzhuev edited comment on SPARK-24422 at 7/3/18 11:46 AM: -- Also

[jira] [Comment Edited] (SPARK-24421) sun.misc.Unsafe in JDK9+

2018-07-03 Thread Andrew Korzhuev (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16531235#comment-16531235 ] Andrew Korzhuev edited comment on SPARK-24421 at 7/3/18 11:44 AM: -- If I

[jira] [Commented] (SPARK-24421) sun.misc.Unsafe in JDK9+

2018-07-03 Thread Andrew Korzhuev (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16531235#comment-16531235 ] Andrew Korzhuev commented on SPARK-24421: - If I understand this correctly, then the only

[jira] [Commented] (SPARK-24481) GeneratedIteratorForCodegenStage1 grows beyond 64 KB

2018-06-07 Thread Andrew Conegliano (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16504965#comment-16504965 ] Andrew Conegliano commented on SPARK-24481: --- Thanks Marco. Forgot to mention, this error

[jira] [Updated] (SPARK-24481) GeneratedIteratorForCodegenStage1 grows beyond 64 KB

2018-06-06 Thread Andrew Conegliano (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24481?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Conegliano updated SPARK-24481: -- Description: Similar to other "grows beyond 64 KB" errors.  Happens with large case

[jira] [Updated] (SPARK-24481) GeneratedIteratorForCodegenStage1 grows beyond 64 KB

2018-06-06 Thread Andrew Conegliano (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24481?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Conegliano updated SPARK-24481: -- Description: Similar to other "grows beyond 64 KB" errors.  Happens with large case

[jira] [Updated] (SPARK-24481) GeneratedIteratorForCodegenStage1 grows beyond 64 KB

2018-06-06 Thread Andrew Conegliano (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24481?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Conegliano updated SPARK-24481: -- Environment: Emr 5.13.0 and Databricks Cloud 4.0 (was: Emr 5.13.0) >

[jira] [Updated] (SPARK-24481) GeneratedIteratorForCodegenStage1 grows beyond 64 KB

2018-06-06 Thread Andrew Conegliano (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24481?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Conegliano updated SPARK-24481: -- Description: Similar to other "grows beyond 64 KB" errors.  Happens with large case

[jira] [Updated] (SPARK-24481) GeneratedIteratorForCodegenStage1 grows beyond 64 KB

2018-06-06 Thread Andrew Conegliano (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24481?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Conegliano updated SPARK-24481: -- Description: Similar to other "grows beyond 64 KB" errors.  Happens with large case

[jira] [Updated] (SPARK-24481) GeneratedIteratorForCodegenStage1 grows beyond 64 KB

2018-06-06 Thread Andrew Conegliano (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24481?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Conegliano updated SPARK-24481: -- Attachment: log4j-active(1).log > GeneratedIteratorForCodegenStage1 grows beyond 64

[jira] [Created] (SPARK-24481) GeneratedIteratorForCodegenStage1 grows beyond 64 KB

2018-06-06 Thread Andrew Conegliano (JIRA)
Andrew Conegliano created SPARK-24481: - Summary: GeneratedIteratorForCodegenStage1 grows beyond 64 KB Key: SPARK-24481 URL: https://issues.apache.org/jira/browse/SPARK-24481 Project: Spark

[jira] [Comment Edited] (SPARK-22371) dag-scheduler-event-loop thread stopped with error Attempted to access garbage collected accumulator 5605982

2018-04-17 Thread Andrew Clegg (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16440683#comment-16440683 ] Andrew Clegg edited comment on SPARK-22371 at 4/17/18 9:53 AM: --- Another

[jira] [Commented] (SPARK-22371) dag-scheduler-event-loop thread stopped with error Attempted to access garbage collected accumulator 5605982

2018-04-17 Thread Andrew Clegg (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16440683#comment-16440683 ] Andrew Clegg commented on SPARK-22371: -- Another data point -- I've seen this happen (in 2.3.0)

[jira] [Comment Edited] (SPARK-23890) Hive ALTER TABLE CHANGE COLUMN for struct type no longer works

2018-04-09 Thread Andrew Otto (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16431099#comment-16431099 ] Andrew Otto edited comment on SPARK-23890 at 4/9/18 7:32 PM: - Hah! As a

[jira] [Commented] (SPARK-23890) Hive ALTER TABLE CHANGE COLUMN for struct type no longer works

2018-04-09 Thread Andrew Otto (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16431099#comment-16431099 ] Andrew Otto commented on SPARK-23890: - Hah! As a temporary workaround, we are [instantiating a JDBC

[jira] [Comment Edited] (SPARK-23890) Hive ALTER TABLE CHANGE COLUMN for struct type no longer works

2018-04-09 Thread Andrew Otto (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16431099#comment-16431099 ] Andrew Otto edited comment on SPARK-23890 at 4/9/18 7:31 PM: - Hah! As a

[jira] [Comment Edited] (SPARK-23890) Hive ALTER TABLE CHANGE COLUMN for struct type no longer works

2018-04-09 Thread Andrew Otto (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16431099#comment-16431099 ] Andrew Otto edited comment on SPARK-23890 at 4/9/18 7:31 PM: - Hah! As a

[jira] [Comment Edited] (SPARK-23890) Hive ALTER TABLE CHANGE COLUMN for struct type no longer works

2018-04-09 Thread Andrew Otto (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16431099#comment-16431099 ] Andrew Otto edited comment on SPARK-23890 at 4/9/18 7:31 PM: - Hah! As a

[jira] [Updated] (SPARK-23890) Hive ALTER TABLE CHANGE COLUMN for struct type no longer works

2018-04-09 Thread Andrew Otto (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Otto updated SPARK-23890: External issue URL: https://github.com/apache/spark/pull/21012 > Hive ALTER TABLE CHANGE COLUMN

[jira] [Updated] (SPARK-23890) Hive ALTER TABLE CHANGE COLUMN for struct type no longer works

2018-04-06 Thread Andrew Otto (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Otto updated SPARK-23890: Description: As part of SPARK-14118, Spark SQL removed support for sending ALTER TABLE CHANGE

[jira] [Updated] (SPARK-23890) Hive ALTER TABLE CHANGE COLUMN for struct type no longer works

2018-04-06 Thread Andrew Otto (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Otto updated SPARK-23890: Description: As part of SPARK-14118, Spark SQL removed support for sending ALTER TABLE CHANGE

[jira] [Created] (SPARK-23890) Hive ALTER TABLE CHANGE COLUMN for struct type no longer works

2018-04-06 Thread Andrew Otto (JIRA)
Andrew Otto created SPARK-23890: --- Summary: Hive ALTER TABLE CHANGE COLUMN for struct type no longer works Key: SPARK-23890 URL: https://issues.apache.org/jira/browse/SPARK-23890 Project: Spark

[jira] [Commented] (SPARK-23878) unable to import col() or lit()

2018-04-06 Thread Andrew Davidson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16428517#comment-16428517 ] Andrew Davidson commented on SPARK-23878: - yes please mark resolved   > unable to import col()

[jira] [Commented] (SPARK-23878) unable to import col() or lit()

2018-04-06 Thread Andrew Davidson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16428509#comment-16428509 ] Andrew Davidson commented on SPARK-23878: - added screen shot to 

[jira] [Commented] (SPARK-23878) unable to import col() or lit()

2018-04-06 Thread Andrew Davidson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16428494#comment-16428494 ] Andrew Davidson commented on SPARK-23878: - Hi Hyukjin many thanks! I am not a python expert. I

[jira] [Updated] (SPARK-23878) unable to import col() or lit()

2018-04-06 Thread Andrew Davidson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23878?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Davidson updated SPARK-23878: Attachment: eclipsePyDevPySparkConfig.png > unable to import col() or lit() >

[jira] [Commented] (SPARK-23878) unable to import col() or lit()

2018-04-05 Thread Andrew Davidson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16427883#comment-16427883 ] Andrew Davidson commented on SPARK-23878: - Hi Hyukjin you are correct. Most IDE's are primarily

[jira] [Created] (SPARK-23878) unable to import col() or lit()

2018-04-05 Thread Andrew Davidson (JIRA)
Andrew Davidson created SPARK-23878: --- Summary: unable to import col() or lit() Key: SPARK-23878 URL: https://issues.apache.org/jira/browse/SPARK-23878 Project: Spark Issue Type: Bug

[jira] [Comment Edited] (SPARK-23682) Memory issue with Spark structured streaming

2018-03-28 Thread Andrew Korzhuev (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16417474#comment-16417474 ] Andrew Korzhuev edited comment on SPARK-23682 at 3/28/18 3:00 PM: -- I can

[jira] [Updated] (SPARK-23682) Memory issue with Spark structured streaming

2018-03-28 Thread Andrew Korzhuev (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23682?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Korzhuev updated SPARK-23682: Attachment: screen_shot_2018-03-20_at_15.23.29.png > Memory issue with Spark structured

[jira] [Comment Edited] (SPARK-23682) Memory issue with Spark structured streaming

2018-03-28 Thread Andrew Korzhuev (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16417474#comment-16417474 ] Andrew Korzhuev edited comment on SPARK-23682 at 3/28/18 2:56 PM: -- I can

[jira] [Comment Edited] (SPARK-23682) Memory issue with Spark structured streaming

2018-03-28 Thread Andrew Korzhuev (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16417474#comment-16417474 ] Andrew Korzhuev edited comment on SPARK-23682 at 3/28/18 2:55 PM: -- I can

[jira] [Commented] (SPARK-23682) Memory issue with Spark structured streaming

2018-03-28 Thread Andrew Korzhuev (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16417474#comment-16417474 ] Andrew Korzhuev commented on SPARK-23682: - I can confirm that _HDFSBackedStateStoreProvider_ is

[jira] [Updated] (SPARK-23682) Memory issue with Spark structured streaming

2018-03-28 Thread Andrew Korzhuev (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23682?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Korzhuev updated SPARK-23682: Attachment: Screen Shot 2018-03-28 at 16.44.20.png > Memory issue with Spark structured

[jira] [Updated] (SPARK-23682) Memory issue with Spark structured streaming

2018-03-28 Thread Andrew Korzhuev (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23682?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Korzhuev updated SPARK-23682: Attachment: Screen Shot 2018-03-28 at 16.44.20.png > Memory issue with Spark structured

[jira] [Updated] (SPARK-23682) Memory issue with Spark structured streaming

2018-03-28 Thread Andrew Korzhuev (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23682?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Korzhuev updated SPARK-23682: Attachment: Screen Shot 2018-03-28 at 16.44.20.png > Memory issue with Spark structured

[jira] [Commented] (SPARK-22865) Publish Official Apache Spark Docker images

2018-03-22 Thread Andrew Korzhuev (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16409609#comment-16409609 ] Andrew Korzhuev commented on SPARK-22865: - Here is an example how this can be done with Travis CI

  1   2   3   4   5   6   7   8   9   10   >