[jira] [Created] (SPARK-47728) Document G1 Concurrent GC metrics

2024-04-04 Thread Luca Canali (Jira)
Luca Canali created SPARK-47728: --- Summary: Document G1 Concurrent GC metrics Key: SPARK-47728 URL: https://issues.apache.org/jira/browse/SPARK-47728 Project: Spark Issue Type: Documentation

[jira] [Created] (SPARK-47726) Document push-based shuffle metrics

2024-04-04 Thread Luca Canali (Jira)
Luca Canali created SPARK-47726: --- Summary: Document push-based shuffle metrics Key: SPARK-47726 URL: https://issues.apache.org/jira/browse/SPARK-47726 Project: Spark Issue Type: Documentation

[jira] [Updated] (SPARK-45336) Update the Oracle docker image version used for test and integration to use Oracle Database 23c Free

2023-09-26 Thread Luca Canali (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45336?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luca Canali updated SPARK-45336: Description: This proposes to update the Docker image used for integration tests and builds to

[jira] [Created] (SPARK-45336) Update the Oracle docker image version used for test and integration to use Oracle Database 23c Free

2023-09-26 Thread Luca Canali (Jira)
Luca Canali created SPARK-45336: --- Summary: Update the Oracle docker image version used for test and integration to use Oracle Database 23c Free Key: SPARK-45336 URL:

[jira] [Updated] (SPARK-41585) The Spark exclude node functionality for YARN should work independently of dynamic allocation

2023-01-26 Thread Luca Canali (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41585?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luca Canali updated SPARK-41585: Affects Version/s: 3.2.2 3.1.3 3.0.3 > The Spark

[jira] [Updated] (SPARK-41585) The Spark exclude node functionality for YARN should work independently of dynamic allocation

2023-01-26 Thread Luca Canali (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41585?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luca Canali updated SPARK-41585: Description: The Spark exclude node functionality for Spark on YARN, introduced in SPARK-26688,

[jira] [Updated] (SPARK-41585) The Spark exclude node functionality for YARN should work independently of dynamic allocation

2023-01-26 Thread Luca Canali (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41585?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luca Canali updated SPARK-41585: Issue Type: Improvement (was: Bug) > The Spark exclude node functionality for YARN should work

[jira] [Updated] (SPARK-41585) The Spark exclude node functionality for YARN should work independently of dynamic allocation

2023-01-26 Thread Luca Canali (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41585?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luca Canali updated SPARK-41585: Description: The Spark exclude node functionality for Spark on YARN, introduced in SPARK-26688,

[jira] [Commented] (SPARK-21829) Enable config to permanently blacklist a list of nodes

2022-12-19 Thread Luca Canali (Jira)
[ https://issues.apache.org/jira/browse/SPARK-21829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17649234#comment-17649234 ] Luca Canali commented on SPARK-21829: - Note: similar functionality was later implemented in

[jira] [Updated] (SPARK-41585) The Spark exclude node functionality for YARN should work independently of dynamic allocation

2022-12-19 Thread Luca Canali (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41585?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luca Canali updated SPARK-41585: Description: The Spark exclude node functionality for Spark on YARN, introduced in SPARK-26688,

[jira] [Updated] (SPARK-41585) The Spark exclude node functionality for YARN should work independently of dynamic allocation

2022-12-19 Thread Luca Canali (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41585?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luca Canali updated SPARK-41585: Description: The Spark exclude node functionality for YARN, introduced in SPARK-26688, allows

[jira] [Created] (SPARK-41585) The Spark exclude node functionality for YARN should work independently of dynamic allocation

2022-12-19 Thread Luca Canali (Jira)
Luca Canali created SPARK-41585: --- Summary: The Spark exclude node functionality for YARN should work independently of dynamic allocation Key: SPARK-41585 URL: https://issues.apache.org/jira/browse/SPARK-41585

[jira] [Created] (SPARK-40485) Extend the partitioning options of the JDBC data source

2022-09-19 Thread Luca Canali (Jira)
Luca Canali created SPARK-40485: --- Summary: Extend the partitioning options of the JDBC data source Key: SPARK-40485 URL: https://issues.apache.org/jira/browse/SPARK-40485 Project: Spark Issue

[jira] [Created] (SPARK-40279) Document spark.yarn.report.interval

2022-08-30 Thread Luca Canali (Jira)
Luca Canali created SPARK-40279: --- Summary: Document spark.yarn.report.interval Key: SPARK-40279 URL: https://issues.apache.org/jira/browse/SPARK-40279 Project: Spark Issue Type: Documentation

[jira] [Updated] (SPARK-34265) Instrument Python UDF execution using SQL Metrics

2022-08-26 Thread Luca Canali (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34265?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luca Canali updated SPARK-34265: Affects Version/s: 3.3.0 (was: 3.2.0) > Instrument Python UDF

[jira] [Updated] (SPARK-34265) Instrument Python UDF execution using SQL Metrics

2022-08-26 Thread Luca Canali (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34265?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luca Canali updated SPARK-34265: Description: This proposes to add SQLMetrics instrumentation for Python UDF. This is aimed at

[jira] [Created] (SPARK-39286) Documentation for the decode function has an incorrect reference

2022-05-25 Thread Luca Canali (Jira)
Luca Canali created SPARK-39286: --- Summary: Documentation for the decode function has an incorrect reference Key: SPARK-39286 URL: https://issues.apache.org/jira/browse/SPARK-39286 Project: Spark

[jira] [Updated] (SPARK-38757) Update the Oracle docker image version used for test and integration

2022-04-01 Thread Luca Canali (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luca Canali updated SPARK-38757: Summary: Update the Oracle docker image version used for test and integration (was: Update the

[jira] [Created] (SPARK-38757) Update the Oracle docker image

2022-04-01 Thread Luca Canali (Jira)
Luca Canali created SPARK-38757: --- Summary: Update the Oracle docker image Key: SPARK-38757 URL: https://issues.apache.org/jira/browse/SPARK-38757 Project: Spark Issue Type: Improvement

[jira] [Updated] (SPARK-38098) Add support for ArrayType of nested StructType to arrow-based conversion

2022-02-04 Thread Luca Canali (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luca Canali updated SPARK-38098: Summary: Add support for ArrayType of nested StructType to arrow-based conversion (was: Support

[jira] [Updated] (SPARK-38098) Add support for ArrayType of nested StructType to arrow-based conversion

2022-02-04 Thread Luca Canali (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luca Canali updated SPARK-38098: Description: This proposes to add support for ArrayType of nested StructType to arrow-based

[jira] [Created] (SPARK-38098) Support Array of Struct for Pandas UDFs

2022-02-03 Thread Luca Canali (Jira)
Luca Canali created SPARK-38098: --- Summary: Support Array of Struct for Pandas UDFs Key: SPARK-38098 URL: https://issues.apache.org/jira/browse/SPARK-38098 Project: Spark Issue Type:

[jira] [Updated] (SPARK-34265) Instrument Python UDF execution using SQL Metrics

2022-01-12 Thread Luca Canali (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34265?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luca Canali updated SPARK-34265: Affects Version/s: 3.2.0 (was: 3.1.1) > Instrument Python UDF

[jira] [Updated] (SPARK-34265) Instrument Python UDF execution using SQL Metrics

2022-01-12 Thread Luca Canali (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34265?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luca Canali updated SPARK-34265: Description: This proposes to add SQLMetrics instrumentation for Python UDF. This is aimed at

[jira] [Updated] (SPARK-34265) Instrument Python UDF execution using SQL Metrics

2022-01-12 Thread Luca Canali (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34265?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luca Canali updated SPARK-34265: Attachment: (was: Python_UDF_instrumentation_lite_BatchEvalPython.png) > Instrument Python

[jira] [Updated] (SPARK-34265) Instrument Python UDF execution using SQL Metrics

2022-01-12 Thread Luca Canali (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34265?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luca Canali updated SPARK-34265: Attachment: (was: Python_UDF_instrumentation_lite_ArrowEvalPython.png) > Instrument Python

[jira] [Updated] (SPARK-34265) Instrument Python UDF execution using SQL Metrics

2022-01-12 Thread Luca Canali (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34265?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luca Canali updated SPARK-34265: Attachment: PandasUDF_ArrowEvalPython_Metrics.png > Instrument Python UDF execution using SQL

[jira] [Created] (SPARK-36573) Add a default value to ORACLE_DOCKER_IMAGE

2021-08-24 Thread Luca Canali (Jira)
Luca Canali created SPARK-36573: --- Summary: Add a default value to ORACLE_DOCKER_IMAGE Key: SPARK-36573 URL: https://issues.apache.org/jira/browse/SPARK-36573 Project: Spark Issue Type:

[jira] [Updated] (SPARK-34265) Instrument Python UDF execution using SQL Metrics

2021-07-28 Thread Luca Canali (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34265?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luca Canali updated SPARK-34265: Attachment: Python_UDF_instrumentation_lite_BatchEvalPython.png > Instrument Python UDF execution

[jira] [Updated] (SPARK-34265) Instrument Python UDF execution using SQL Metrics

2021-07-28 Thread Luca Canali (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34265?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luca Canali updated SPARK-34265: Attachment: Python_UDF_instrumentation_lite_ArrowEvalPython.png > Instrument Python UDF execution

[jira] [Updated] (SPARK-35369) Document ExecutorAllocationManager metrics

2021-05-11 Thread Luca Canali (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35369?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luca Canali updated SPARK-35369: Environment: (was: The ExecutorAllocationManager is instrumented with metrics using the Spark

[jira] [Updated] (SPARK-35369) Document ExecutorAllocationManager metrics

2021-05-11 Thread Luca Canali (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35369?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luca Canali updated SPARK-35369: Description: The ExecutorAllocationManager is instrumented with metrics using the Spark metrics

[jira] [Created] (SPARK-35369) Document ExecutorAllocationManager metrics

2021-05-11 Thread Luca Canali (Jira)
Luca Canali created SPARK-35369: --- Summary: Document ExecutorAllocationManager metrics Key: SPARK-35369 URL: https://issues.apache.org/jira/browse/SPARK-35369 Project: Spark Issue Type:

[jira] [Commented] (SPARK-32082) Project Zen: Improving Python usability

2021-02-04 Thread Luca Canali (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17278691#comment-17278691 ] Luca Canali commented on SPARK-32082: - Would adding "Improvements to monitoring and instrumentation"

[jira] [Updated] (SPARK-34265) Instrument Python UDF execution using SQL Metrics

2021-02-04 Thread Luca Canali (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34265?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luca Canali updated SPARK-34265: Summary: Instrument Python UDF execution using SQL Metrics (was: Add SQLMetrics instrumentation

[jira] [Updated] (SPARK-34265) Add SQLMetrics instrumentation to Python UDF

2021-01-29 Thread Luca Canali (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34265?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luca Canali updated SPARK-34265: Attachment: proposed_Python_SQLmetrics_v20210128.png > Add SQLMetrics instrumentation to Python

[jira] [Updated] (SPARK-34265) Add SQLMetrics instrumentation to Python UDF

2021-01-27 Thread Luca Canali (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34265?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luca Canali updated SPARK-34265: Attachment: PythonSQLMetrics_Jira_Picture.png > Add SQLMetrics instrumentation to Python UDF >

[jira] [Created] (SPARK-34265) Add SQLMetrics instrumentation to Python UDF

2021-01-27 Thread Luca Canali (Jira)
Luca Canali created SPARK-34265: --- Summary: Add SQLMetrics instrumentation to Python UDF Key: SPARK-34265 URL: https://issues.apache.org/jira/browse/SPARK-34265 Project: Spark Issue Type:

[jira] [Commented] (SPARK-32119) ExecutorPlugin doesn't work with Standalone Cluster and Kubernetes with --jars

2020-08-31 Thread Luca Canali (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17187980#comment-17187980 ] Luca Canali commented on SPARK-32119: - I believe it would be good to have this fix also in the

[jira] [Created] (SPARK-32661) Spark executors on K8S do not request extra memory for off-heap allocations

2020-08-19 Thread Luca Canali (Jira)
Luca Canali created SPARK-32661: --- Summary: Spark executors on K8S do not request extra memory for off-heap allocations Key: SPARK-32661 URL: https://issues.apache.org/jira/browse/SPARK-32661 Project:

[jira] [Updated] (SPARK-32409) Document the dependency between spark.metrics.staticSources.enabled and JVMSource registration

2020-08-10 Thread Luca Canali (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luca Canali updated SPARK-32409: Summary: Document the dependency between spark.metrics.staticSources.enabled and JVMSource

[jira] [Updated] (SPARK-32409) Remove dependency between spark.metrics.staticSources.enabled and JVMSource registration

2020-07-24 Thread Luca Canali (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luca Canali updated SPARK-32409: Description: SPARK-29654 has introduced the configuration

[jira] [Updated] (SPARK-32409) Remove dependency between spark.metrics.staticSources.enabled and JVMSource registration

2020-07-23 Thread Luca Canali (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luca Canali updated SPARK-32409: Description: SPARK-29654 has introduced the configuration

[jira] [Updated] (SPARK-32409) Remove dependency between spark.metrics.staticSources.enabled and JVMSource registration

2020-07-23 Thread Luca Canali (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luca Canali updated SPARK-32409: Description: SPARK-29654 has introduced the configuration

[jira] [Created] (SPARK-32409) Remove dependency between spark.metrics.staticSources.enabled and JVMSource registration

2020-07-23 Thread Luca Canali (Jira)
Luca Canali created SPARK-32409: --- Summary: Remove dependency between spark.metrics.staticSources.enabled and JVMSource registration Key: SPARK-32409 URL: https://issues.apache.org/jira/browse/SPARK-32409

[jira] [Commented] (SPARK-32119) ExecutorPlugin doesn't work with Standalone Cluster

2020-06-30 Thread Luca Canali (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17148350#comment-17148350 ] Luca Canali commented on SPARK-32119: - No plans that I know of. CC [~vanzin] > ExecutorPlugin

[jira] [Comment Edited] (SPARK-32119) ExecutorPlugin doesn't work with Standalone Cluster

2020-06-29 Thread Luca Canali (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17147750#comment-17147750 ] Luca Canali edited comment on SPARK-32119 at 6/29/20, 8:09 PM: --- I am

[jira] [Commented] (SPARK-32119) ExecutorPlugin doesn't work with Standalone Cluster

2020-06-29 Thread Luca Canali (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17147750#comment-17147750 ] Luca Canali commented on SPARK-32119: - I am adding the comment that the limitation discussed in this

[jira] [Updated] (SPARK-31711) Register the executor source with the metrics system when running in local mode.

2020-05-14 Thread Luca Canali (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31711?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luca Canali updated SPARK-31711: Description: The Apache Spark metrics system provides many useful insights on the Spark

[jira] [Created] (SPARK-31711) Register the executor source with the metrics system when running in local mode.

2020-05-14 Thread Luca Canali (Jira)
Luca Canali created SPARK-31711: --- Summary: Register the executor source with the metrics system when running in local mode. Key: SPARK-31711 URL: https://issues.apache.org/jira/browse/SPARK-31711

[jira] [Created] (SPARK-30775) Improve DOC for executor metrics

2020-02-10 Thread Luca Canali (Jira)
Luca Canali created SPARK-30775: --- Summary: Improve DOC for executor metrics Key: SPARK-30775 URL: https://issues.apache.org/jira/browse/SPARK-30775 Project: Spark Issue Type: Improvement

[jira] [Updated] (SPARK-30306) Instrument Python UDF execution time and metrics using Spark Metrics system

2019-12-19 Thread Luca Canali (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30306?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luca Canali updated SPARK-30306: Attachment: PandasUDF_Time_Instrumentation_Annotated.png > Instrument Python UDF execution time

[jira] [Created] (SPARK-30306) Instrument Python UDF execution time and metrics using Spark Metrics system

2019-12-19 Thread Luca Canali (Jira)
Luca Canali created SPARK-30306: --- Summary: Instrument Python UDF execution time and metrics using Spark Metrics system Key: SPARK-30306 URL: https://issues.apache.org/jira/browse/SPARK-30306 Project:

[jira] [Updated] (SPARK-30153) Extend data exchange options for vectorized UDF functions with vanilla Arrow serialization

2019-12-06 Thread Luca Canali (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30153?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luca Canali updated SPARK-30153: Attachment: Flamegraph_test_pandas_udf_SCALAR.png > Extend data exchange options for vectorized

[jira] [Updated] (SPARK-30153) Extend data exchange options for vectorized UDF functions with vanilla Arrow serialization

2019-12-06 Thread Luca Canali (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30153?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luca Canali updated SPARK-30153: Attachment: Flamegraph_test_pandas_udf_SCALAR_ARROW.png > Extend data exchange options for

[jira] [Created] (SPARK-30153) Extend data exchange options for vectorized UDF functions with vanilla Arrow serialization

2019-12-06 Thread Luca Canali (Jira)
Luca Canali created SPARK-30153: --- Summary: Extend data exchange options for vectorized UDF functions with vanilla Arrow serialization Key: SPARK-30153 URL: https://issues.apache.org/jira/browse/SPARK-30153

[jira] [Updated] (SPARK-30041) Add Codegen Stage Id to Stage DAG visualization in Web UI

2019-12-04 Thread Luca Canali (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30041?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luca Canali updated SPARK-30041: Attachment: (was: Snippet_StagesDags_with_CodegenId.png) > Add Codegen Stage Id to Stage DAG

[jira] [Updated] (SPARK-30041) Add Codegen Stage Id to Stage DAG visualization in Web UI

2019-12-04 Thread Luca Canali (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30041?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luca Canali updated SPARK-30041: Attachment: Snippet_StagesDags_with_CodegenId _annotated.png > Add Codegen Stage Id to Stage DAG

[jira] [Created] (SPARK-30060) Uniform naming for Spark Metrics configuration parameters

2019-11-27 Thread Luca Canali (Jira)
Luca Canali created SPARK-30060: --- Summary: Uniform naming for Spark Metrics configuration parameters Key: SPARK-30060 URL: https://issues.apache.org/jira/browse/SPARK-30060 Project: Spark

[jira] [Updated] (SPARK-30060) Uniform naming for Spark Metrics configuration parameters

2019-11-27 Thread Luca Canali (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30060?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luca Canali updated SPARK-30060: Description: Currently Spark has a few parameters to enable/disable metrics reporting. Their

[jira] [Updated] (SPARK-30041) Add Codegen Stage Id to Stage DAG visualization in Web UI

2019-11-26 Thread Luca Canali (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30041?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luca Canali updated SPARK-30041: Description: SPARK-29894 provides information on the Codegen Stage Id in WEBUI for SQL Plan

[jira] [Updated] (SPARK-30041) Add Codegen Stage Id to Stage DAG visualization in Web UI

2019-11-26 Thread Luca Canali (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30041?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luca Canali updated SPARK-30041: Attachment: Snippet_StagesDags_with_CodegenId.png > Add Codegen Stage Id to Stage DAG

[jira] [Created] (SPARK-30041) Add Codegen Stage Id to Stage DAG visualization in Web UI

2019-11-26 Thread Luca Canali (Jira)
Luca Canali created SPARK-30041: --- Summary: Add Codegen Stage Id to Stage DAG visualization in Web UI Key: SPARK-30041 URL: https://issues.apache.org/jira/browse/SPARK-30041 Project: Spark

[jira] [Updated] (SPARK-29894) Add Codegen Stage Id to Spark plan graphs in Web UI SQL Tab

2019-11-15 Thread Luca Canali (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luca Canali updated SPARK-29894: Attachment: Physical_plan_Annotated.png > Add Codegen Stage Id to Spark plan graphs in Web UI SQL

[jira] [Updated] (SPARK-29894) Add Codegen Stage Id to Spark plan graphs in Web UI SQL Tab

2019-11-14 Thread Luca Canali (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luca Canali updated SPARK-29894: Attachment: snippet_plan_graph_before_patch.png > Add Codegen Stage Id to Spark plan graphs in

[jira] [Updated] (SPARK-29894) Add Codegen Stage Id to Spark plan graphs in Web UI SQL Tab

2019-11-14 Thread Luca Canali (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luca Canali updated SPARK-29894: Attachment: snippet__plan_graph_with_Codegen_Stage_Id_Annotated.png > Add Codegen Stage Id to

[jira] [Created] (SPARK-29894) Add Codegen Stage Id to Spark plan graphs in Web UI SQL Tab

2019-11-14 Thread Luca Canali (Jira)
Luca Canali created SPARK-29894: --- Summary: Add Codegen Stage Id to Spark plan graphs in Web UI SQL Tab Key: SPARK-29894 URL: https://issues.apache.org/jira/browse/SPARK-29894 Project: Spark

[jira] [Updated] (SPARK-27189) Add Executor metrics and memory usage instrumentation to the metrics system

2019-11-11 Thread Luca Canali (Jira)
[ https://issues.apache.org/jira/browse/SPARK-27189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luca Canali updated SPARK-27189: Summary: Add Executor metrics and memory usage instrumentation to the metrics system (was: Add

[jira] [Created] (SPARK-29654) Add configuration to allow disabling registration of static sources to the metrics system

2019-10-30 Thread Luca Canali (Jira)
Luca Canali created SPARK-29654: --- Summary: Add configuration to allow disabling registration of static sources to the metrics system Key: SPARK-29654 URL: https://issues.apache.org/jira/browse/SPARK-29654

[jira] [Commented] (SPARK-29557) Upgrade dropwizard metrics library to 3.2.6

2019-10-23 Thread Luca Canali (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16957945#comment-16957945 ] Luca Canali commented on SPARK-29557: - Upgrading Apache Spark to use dropwizard/codahale metrics

[jira] [Updated] (SPARK-29557) Upgrade dropwizard metrics library to 3.2.6

2019-10-23 Thread Luca Canali (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luca Canali updated SPARK-29557: Summary: Upgrade dropwizard metrics library to 3.2.6 (was: Upgrade dropwizard metrics library to

[jira] [Created] (SPARK-29557) Upgrade dropwizard metrics library to 4.1.1

2019-10-22 Thread Luca Canali (Jira)
Luca Canali created SPARK-29557: --- Summary: Upgrade dropwizard metrics library to 4.1.1 Key: SPARK-29557 URL: https://issues.apache.org/jira/browse/SPARK-29557 Project: Spark Issue Type:

[jira] [Commented] (SPARK-29396) Extend Spark plugin interface to driver

2019-10-11 Thread Luca Canali (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16949765#comment-16949765 ] Luca Canali commented on SPARK-29396: - This looks quite a good improvement proposal to me.

[jira] [Updated] (SPARK-28091) Extend Spark metrics system with user-defined metrics using executor plugins

2019-07-24 Thread Luca Canali (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28091?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luca Canali updated SPARK-28091: Summary: Extend Spark metrics system with user-defined metrics using executor plugins (was:

[jira] [Updated] (SPARK-28091) Extend Spark metrics system with executor plugin metrics

2019-07-24 Thread Luca Canali (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28091?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luca Canali updated SPARK-28091: Description: This proposes to improve Spark instrumentation by adding a hook for user-defined

[jira] [Commented] (SPARK-28091) Extend Spark metrics system with executor plugin metrics

2019-06-26 Thread Luca Canali (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16873041#comment-16873041 ] Luca Canali commented on SPARK-28091: - Indeed toString() is quite useful, thanks. + good proposal, I

[jira] [Commented] (SPARK-28091) Extend Spark metrics system with executor plugin metrics

2019-06-25 Thread Luca Canali (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16872075#comment-16872075 ] Luca Canali commented on SPARK-28091: - Thank you [~ste...@apache.org] for your comment and

[jira] [Commented] (SPARK-28091) Extend Spark metrics system with executor plugin metrics

2019-06-21 Thread Luca Canali (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16869528#comment-16869528 ] Luca Canali commented on SPARK-28091: - {{I have added some example code for executor metrics plugins

[jira] [Comment Edited] (SPARK-28091) Extend Spark metrics system with executor plugin metrics

2019-06-21 Thread Luca Canali (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16869528#comment-16869528 ] Luca Canali edited comment on SPARK-28091 at 6/21/19 2:36 PM: -- I have added

[jira] [Comment Edited] (SPARK-28091) Extend Spark metrics system with executor plugin metrics

2019-06-21 Thread Luca Canali (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16869528#comment-16869528 ] Luca Canali edited comment on SPARK-28091 at 6/21/19 2:34 PM: -- I have added

[jira] [Comment Edited] (SPARK-28091) Extend Spark metrics system with executor plugin metrics

2019-06-21 Thread Luca Canali (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16869528#comment-16869528 ] Luca Canali edited comment on SPARK-28091 at 6/21/19 2:31 PM: -- I have added

[jira] [Comment Edited] (SPARK-28091) Extend Spark metrics system with executor plugin metrics

2019-06-18 Thread Luca Canali (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16866336#comment-16866336 ] Luca Canali edited comment on SPARK-28091 at 6/18/19 8:07 AM: -- [~irashid]

[jira] [Commented] (SPARK-28091) Extend Spark metrics system with executor plugin metrics

2019-06-18 Thread Luca Canali (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16866336#comment-16866336 ] Luca Canali commented on SPARK-28091: - @irashid given your work on SPARK-24918 you may be interested

[jira] [Created] (SPARK-28091) Extend Spark metrics system with executor plugin metrics

2019-06-18 Thread Luca Canali (JIRA)
Luca Canali created SPARK-28091: --- Summary: Extend Spark metrics system with executor plugin metrics Key: SPARK-28091 URL: https://issues.apache.org/jira/browse/SPARK-28091 Project: Spark Issue

[jira] [Updated] (SPARK-27189) Add Executor level memory usage metrics to the metrics system

2019-03-18 Thread Luca Canali (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luca Canali updated SPARK-27189: Summary: Add Executor level memory usage metrics to the metrics system (was: Add Executor level

[jira] [Updated] (SPARK-27189) Add Executor level metrics to the metrics system

2019-03-18 Thread Luca Canali (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luca Canali updated SPARK-27189: Attachment: Example_dashboard_Spark_Memory_Metrics.PNG > Add Executor level metrics to the

[jira] [Created] (SPARK-27189) Add Executor level metrics to the metrics system

2019-03-18 Thread Luca Canali (JIRA)
Luca Canali created SPARK-27189: --- Summary: Add Executor level metrics to the metrics system Key: SPARK-27189 URL: https://issues.apache.org/jira/browse/SPARK-27189 Project: Spark Issue Type:

[jira] [Resolved] (SPARK-24486) Slow performance reading ArrayType columns

2019-03-12 Thread Luca Canali (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24486?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luca Canali resolved SPARK-24486. - Resolution: Fixed Fix Version/s: 3.0.0 > Slow performance reading ArrayType columns >

[jira] [Commented] (SPARK-24486) Slow performance reading ArrayType columns

2019-03-12 Thread Luca Canali (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16790409#comment-16790409 ] Luca Canali commented on SPARK-24486: - Thanks [~yumwang] for looking at this. Indeed I confirm that

[jira] [Created] (SPARK-26995) Running Spark in Docker image with Alpine Linux 3.9.0 throws errors when using snappy

2019-02-26 Thread Luca Canali (JIRA)
Luca Canali created SPARK-26995: --- Summary: Running Spark in Docker image with Alpine Linux 3.9.0 throws errors when using snappy Key: SPARK-26995 URL: https://issues.apache.org/jira/browse/SPARK-26995

[jira] [Created] (SPARK-26928) Add driver CPU Time to the metrics system

2019-02-19 Thread Luca Canali (JIRA)
Luca Canali created SPARK-26928: --- Summary: Add driver CPU Time to the metrics system Key: SPARK-26928 URL: https://issues.apache.org/jira/browse/SPARK-26928 Project: Spark Issue Type:

[jira] [Created] (SPARK-26890) Add Dropwizard metrics list and additional configuration details to the documentation

2019-02-15 Thread Luca Canali (JIRA)
Luca Canali created SPARK-26890: --- Summary: Add Dropwizard metrics list and additional configuration details to the documentation Key: SPARK-26890 URL: https://issues.apache.org/jira/browse/SPARK-26890

[jira] [Updated] (SPARK-26642) Add --num-executors option to spark-submit for Spark on K8S

2019-01-17 Thread Luca Canali (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luca Canali updated SPARK-26642: Description: Currently spark-submit supports the option "--num-executors NUM" only for Spark on

[jira] [Created] (SPARK-26642) Add --num-executor option to spark-submit for Spark on K8S

2019-01-17 Thread Luca Canali (JIRA)
Luca Canali created SPARK-26642: --- Summary: Add --num-executor option to spark-submit for Spark on K8S Key: SPARK-26642 URL: https://issues.apache.org/jira/browse/SPARK-26642 Project: Spark

[jira] [Updated] (SPARK-26642) Add --num-executor option to spark-submit for Spark on K8S

2019-01-17 Thread Luca Canali (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luca Canali updated SPARK-26642: Description: Currently spark-submit supports the option "--num-executors NUM" only for Spark on

[jira] [Updated] (SPARK-26642) Add --num-executors option to spark-submit for Spark on K8S

2019-01-17 Thread Luca Canali (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luca Canali updated SPARK-26642: Description: Currently spark-submit supports the option --num-executors NUM only for Spark on

[jira] [Updated] (SPARK-26642) Add --num-executors option to spark-submit for Spark on K8S

2019-01-17 Thread Luca Canali (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luca Canali updated SPARK-26642: Description: Currently spark-submit supports the option "--num-executors NUM" only for Spark on

[jira] [Updated] (SPARK-26642) Add --num-executors option to spark-submit for Spark on K8S

2019-01-17 Thread Luca Canali (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luca Canali updated SPARK-26642: Description: Currently spark-submit supports the option "--num-executors NUM" only for Spark on

[jira] [Updated] (SPARK-26642) Add --num-executors option to spark-submit for Spark on K8S

2019-01-17 Thread Luca Canali (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luca Canali updated SPARK-26642: Summary: Add --num-executors option to spark-submit for Spark on K8S (was: Add --num-executor

[jira] [Updated] (SPARK-26600) Update spark-submit usage message

2019-01-11 Thread Luca Canali (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26600?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luca Canali updated SPARK-26600: Description: Spark-submit usage message should be put in sync with recent changes in particular

  1   2   >