[jira] [Commented] (SPARK-18120) QueryExecutionListener method doesnt' get executed for DataFrameWriter methods
[ https://issues.apache.org/jira/browse/SPARK-18120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15870586#comment-15870586 ] Apache Spark commented on SPARK-18120: -- User 'cloud-fan' has created a pull request for this issue: https://github.com/apache/spark/pull/16962 > QueryExecutionListener method doesnt' get executed for DataFrameWriter methods > -- > > Key: SPARK-18120 > URL: https://issues.apache.org/jira/browse/SPARK-18120 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 2.0.1 >Reporter: Salil Surendran > > QueryExecutionListener is a class that has methods named onSuccess() and > onFailure() that gets called when a query is executed. Each of those methods > takes a QueryExecution object as a parameter which can be used for metrics > analysis. It gets called for several of the DataSet methods like take, head, > first, collect etc. but doesn't get called for any of the DataFrameWriter > methods like saveAsTable, save etc. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-18120) QueryExecutionListener method doesnt' get executed for DataFrameWriter methods
[ https://issues.apache.org/jira/browse/SPARK-18120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15832362#comment-15832362 ] Apache Spark commented on SPARK-18120: -- User 'salilsurendran' has created a pull request for this issue: https://github.com/apache/spark/pull/16664 > QueryExecutionListener method doesnt' get executed for DataFrameWriter methods > -- > > Key: SPARK-18120 > URL: https://issues.apache.org/jira/browse/SPARK-18120 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 2.0.1 >Reporter: Salil Surendran > > QueryExecutionListener is a class that has methods named onSuccess() and > onFailure() that gets called when a query is executed. Each of those methods > takes a QueryExecution object as a parameter which can be used for metrics > analysis. It gets called for several of the DataSet methods like take, head, > first, collect etc. but doesn't get called for any of the DataFrameWriter > methods like saveAsTable, save etc. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-18120) QueryExecutionListener method doesnt' get executed for DataFrameWriter methods
[ https://issues.apache.org/jira/browse/SPARK-18120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15831719#comment-15831719 ] Salil Surendran commented on SPARK-18120: - Sorry for the delay but ran into some unexpected unit test failures. Rerunning the whole test suite to make sure nothing is broken. > QueryExecutionListener method doesnt' get executed for DataFrameWriter methods > -- > > Key: SPARK-18120 > URL: https://issues.apache.org/jira/browse/SPARK-18120 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 2.0.1 >Reporter: Salil Surendran > > QueryExecutionListener is a class that has methods named onSuccess() and > onFailure() that gets called when a query is executed. Each of those methods > takes a QueryExecution object as a parameter which can be used for metrics > analysis. It gets called for several of the DataSet methods like take, head, > first, collect etc. but doesn't get called for any of the DataFrameWriter > methods like saveAsTable, save etc. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-18120) QueryExecutionListener method doesnt' get executed for DataFrameWriter methods
[ https://issues.apache.org/jira/browse/SPARK-18120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15830630#comment-15830630 ] Salil Surendran commented on SPARK-18120: - [~thomastechs]I will be making a PR today. > QueryExecutionListener method doesnt' get executed for DataFrameWriter methods > -- > > Key: SPARK-18120 > URL: https://issues.apache.org/jira/browse/SPARK-18120 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 2.0.1 >Reporter: Salil Surendran > > QueryExecutionListener is a class that has methods named onSuccess() and > onFailure() that gets called when a query is executed. Each of those methods > takes a QueryExecution object as a parameter which can be used for metrics > analysis. It gets called for several of the DataSet methods like take, head, > first, collect etc. but doesn't get called for any of the DataFrameWriter > methods like saveAsTable, save etc. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-18120) QueryExecutionListener method doesnt' get executed for DataFrameWriter methods
[ https://issues.apache.org/jira/browse/SPARK-18120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15830307#comment-15830307 ] Thomas Sebastian commented on SPARK-18120: -- Hi [~salilsurendran] Are you working on this, please let us know ? I can submit a PR. > QueryExecutionListener method doesnt' get executed for DataFrameWriter methods > -- > > Key: SPARK-18120 > URL: https://issues.apache.org/jira/browse/SPARK-18120 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 2.0.1 >Reporter: Salil Surendran > > QueryExecutionListener is a class that has methods named onSuccess() and > onFailure() that gets called when a query is executed. Each of those methods > takes a QueryExecution object as a parameter which can be used for metrics > analysis. It gets called for several of the DataSet methods like take, head, > first, collect etc. but doesn't get called for any of the DataFrameWriter > methods like saveAsTable, save etc. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-18120) QueryExecutionListener method doesnt' get executed for DataFrameWriter methods
[ https://issues.apache.org/jira/browse/SPARK-18120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15705756#comment-15705756 ] Salil Surendran commented on SPARK-18120: - [~r...@databricks.com]. No it doesn't get triggered. [~jayadevan.m] I already have the code written and ready. Will be making a PR soon. > QueryExecutionListener method doesnt' get executed for DataFrameWriter methods > -- > > Key: SPARK-18120 > URL: https://issues.apache.org/jira/browse/SPARK-18120 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 2.0.1 >Reporter: Salil Surendran > > QueryExecutionListener is a class that has methods named onSuccess() and > onFailure() that gets called when a query is executed. Each of those methods > takes a QueryExecution object as a parameter which can be used for metrics > analysis. It gets called for several of the DataSet methods like take, head, > first, collect etc. but doesn't get called for any of hte DataFrameWriter > methods like saveAsTable, save etc. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-18120) QueryExecutionListener method doesnt' get executed for DataFrameWriter methods
[ https://issues.apache.org/jira/browse/SPARK-18120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15705744#comment-15705744 ] Jayadevan M commented on SPARK-18120: - [~thomastechs],[~salilsurendran][~rxin] Hi All, I am working on fix for this issue Regards Jay > QueryExecutionListener method doesnt' get executed for DataFrameWriter methods > -- > > Key: SPARK-18120 > URL: https://issues.apache.org/jira/browse/SPARK-18120 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 2.0.1 >Reporter: Salil Surendran > > QueryExecutionListener is a class that has methods named onSuccess() and > onFailure() that gets called when a query is executed. Each of those methods > takes a QueryExecution object as a parameter which can be used for metrics > analysis. It gets called for several of the DataSet methods like take, head, > first, collect etc. but doesn't get called for any of hte DataFrameWriter > methods like saveAsTable, save etc. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-18120) QueryExecutionListener method doesnt' get executed for DataFrameWriter methods
[ https://issues.apache.org/jira/browse/SPARK-18120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15700108#comment-15700108 ] Reynold Xin commented on SPARK-18120: - They should get triggered. Are you sure they don't? > QueryExecutionListener method doesnt' get executed for DataFrameWriter methods > -- > > Key: SPARK-18120 > URL: https://issues.apache.org/jira/browse/SPARK-18120 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 2.0.1 >Reporter: Salil Surendran > > QueryExecutionListener is a class that has methods named onSuccess() and > onFailure() that gets called when a query is executed. Each of those methods > takes a QueryExecution object as a parameter which can be used for metrics > analysis. It gets called for several of the DataSet methods like take, head, > first, collect etc. but doesn't get called for any of hte DataFrameWriter > methods like saveAsTable, save etc. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-18120) QueryExecutionListener method doesnt' get executed for DataFrameWriter methods
[ https://issues.apache.org/jira/browse/SPARK-18120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15699523#comment-15699523 ] Thomas Sebastian commented on SPARK-18120: -- [~salilsurendran] [~rxin] [~jayadevan.m] Is it really expected to have the listener similar implementation in the DataFrameWriter? Just would like to clarify, Or it was not included in the DataFrameWriter due to any particular reason? > QueryExecutionListener method doesnt' get executed for DataFrameWriter methods > -- > > Key: SPARK-18120 > URL: https://issues.apache.org/jira/browse/SPARK-18120 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 2.0.1 >Reporter: Salil Surendran > > QueryExecutionListener is a class that has methods named onSuccess() and > onFailure() that gets called when a query is executed. Each of those methods > takes a QueryExecution object as a parameter which can be used for metrics > analysis. It gets called for several of the DataSet methods like take, head, > first, collect etc. but doesn't get called for any of hte DataFrameWriter > methods like saveAsTable, save etc. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
