[jira] [Resolved] (SPARK-30617) Is there any possible that spark no longer restrict enumerate types of spark.sql.catalogImplementation
[ https://issues.apache.org/jira/browse/SPARK-30617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] weiwenda resolved SPARK-30617. -- Resolution: Pending Closed > Is there any possible that spark no longer restrict enumerate types of > spark.sql.catalogImplementation > -- > > Key: SPARK-30617 > URL: https://issues.apache.org/jira/browse/SPARK-30617 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 3.0.0 >Reporter: weiwenda >Priority: Minor > > # We have implemented a complex ExternalCatalog which is used for retrieving > multi isomerism database's metadata(sush as elasticsearch、postgresql), so > that we can make a mixture query between hive and our online data. > # But as spark require that value of spark.sql.catalogImplementation must be > one of in-memory/hive, we have to modify SparkSession and rebuild spark to > make our project work. > # Finally, we hope spark removing above restriction, so that it's will be > much easier to let us keep pace with new spark version. Thanks! -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-30617) Is there any possible that spark no longer restrict enumerate types of spark.sql.catalogImplementation
[ https://issues.apache.org/jira/browse/SPARK-30617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17022869#comment-17022869 ] weiwenda commented on SPARK-30617: -- [~dongjoon] Thanks for your advise. I will write Fix versions / Affected Version carefully next time. > Is there any possible that spark no longer restrict enumerate types of > spark.sql.catalogImplementation > -- > > Key: SPARK-30617 > URL: https://issues.apache.org/jira/browse/SPARK-30617 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 3.0.0 >Reporter: weiwenda >Priority: Minor > > # We have implemented a complex ExternalCatalog which is used for retrieving > multi isomerism database's metadata(sush as elasticsearch、postgresql), so > that we can make a mixture query between hive and our online data. > # But as spark require that value of spark.sql.catalogImplementation must be > one of in-memory/hive, we have to modify SparkSession and rebuild spark to > make our project work. > # Finally, we hope spark removing above restriction, so that it's will be > much easier to let us keep pace with new spark version. Thanks! -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-30617) Is there any possible that spark no longer restrict enumerate types of spark.sql.catalogImplementation
[ https://issues.apache.org/jira/browse/SPARK-30617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17021899#comment-17021899 ] weiwenda commented on SPARK-30617: -- one solution at [https://github.com/apache/spark/pull/27338] > Is there any possible that spark no longer restrict enumerate types of > spark.sql.catalogImplementation > -- > > Key: SPARK-30617 > URL: https://issues.apache.org/jira/browse/SPARK-30617 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 2.4.4 >Reporter: weiwenda >Priority: Minor > Fix For: 3.1.0, 2.4.6 > > > # We have implemented a complex ExternalCatalog which is used for retrieving > multi isomerism database's metadata(sush as elasticsearch、postgresql), so > that we can make a mixture query between hive and our online data. > # But as spark require that value of spark.sql.catalogImplementation must be > one of in-memory/hive, we have to modify SparkSession and rebuild spark to > make our project work. > # Finally, we hope spark removing above restriction, so that it's will be > much easier to let us keep pace with new spark version. Thanks! -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-30617) Is there any possible that spark no longer restrict enumerate types of spark.sql.catalogImplementation
weiwenda created SPARK-30617: Summary: Is there any possible that spark no longer restrict enumerate types of spark.sql.catalogImplementation Key: SPARK-30617 URL: https://issues.apache.org/jira/browse/SPARK-30617 Project: Spark Issue Type: Improvement Components: SQL Affects Versions: 2.4.4 Reporter: weiwenda Fix For: 3.1.0, 2.4.6 # We have implemented a complex ExternalCatalog which is used for retrieving multi isomerism database's metadata(sush as elasticsearch、postgresql), so that we can make a mixture query between hive and our online data. # But as spark require that value of spark.sql.catalogImplementation must be one of in-memory/hive, we have to modify SparkSession and rebuild spark to make our project work. # Finally, we hope spark removing above restriction, so that it's will be much easier to let us keep pace with new spark version. Thanks! -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-26938) failed task's SparkListenerTaskEnd lost taskMetrics, which is important for problem diagnosis, especially when checking exists of data skew
[
https://issues.apache.org/jira/browse/SPARK-26938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16772685#comment-16772685
]
weiwenda edited comment on SPARK-26938 at 2/20/19 6:56 AM:
---
for now, SparkListenerTaskEnd will write out to log file as below, so that
historyserver is unable to show taskMetrics of failed task.
!image-2019-02-20-14-55-58-002.png!
was (Author: 994184...@qq.com):
for now, SparkListenerTaskEnd will write out to log file as below, so that
historyserver is unable to show taskMetrics of failed task.
{quote}{
"Event": "SparkListenerTaskEnd",
"Stage ID": 0,
"Stage Attempt ID": 0,
"Task Type": "ShuffleMapTask",
"Task End Reason": {
"Reason": "ExecutorLostFailure",
"Executor ID": "101",
"Exit Caused By App": true,
"Loss Reason": ""
},
"Task Info": {
"Task ID": 113,
"Index": 50,
"Attempt": 0,
"Launch Time": 1548895514836,
"Executor ID": "101",
"Host": "",
"Locality": "RACK_LOCAL",
"Speculative": false,
"Getting Result Time": 0,
"Finish Time": 1548895518542,
"Failed": true,
"Accumulables": []
}
}
{quote}
> failed task's SparkListenerTaskEnd lost taskMetrics, which is important for
> problem diagnosis, especially when checking exists of data skew
> ---
>
> Key: SPARK-26938
> URL: https://issues.apache.org/jira/browse/SPARK-26938
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
>Affects Versions: 2.3.3
>Reporter: weiwenda
>Priority: Minor
> Attachments: image-2019-02-20-14-55-58-002.png
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-26938) failed task's SparkListenerTaskEnd lost taskMetrics, which is important for problem diagnosis, especially when checking exists of data skew
[
https://issues.apache.org/jira/browse/SPARK-26938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16772685#comment-16772685
]
weiwenda edited comment on SPARK-26938 at 2/20/19 6:55 AM:
---
for now, SparkListenerTaskEnd will write out to log file as below, so that
historyserver is unable to show taskMetrics of failed task.
{quote}{
"Event": "SparkListenerTaskEnd",
"Stage ID": 0,
"Stage Attempt ID": 0,
"Task Type": "ShuffleMapTask",
"Task End Reason": {
"Reason": "ExecutorLostFailure",
"Executor ID": "101",
"Exit Caused By App": true,
"Loss Reason": ""
},
"Task Info": {
"Task ID": 113,
"Index": 50,
"Attempt": 0,
"Launch Time": 1548895514836,
"Executor ID": "101",
"Host": "",
"Locality": "RACK_LOCAL",
"Speculative": false,
"Getting Result Time": 0,
"Finish Time": 1548895518542,
"Failed": true,
"Accumulables": []
}
}
{quote}
was (Author: 994184...@qq.com):
for now, SparkListenerTaskEnd will write out to log file as below, so that
historyserver is unable to show taskMetrics of failed task.
{quote}{
"Event": "SparkListenerTaskEnd",
"Stage ID": 0,
"Stage Attempt ID": 0,
"Task Type": "ShuffleMapTask",
"Task End Reason": {
"Reason": "ExecutorLostFailure",
"Executor ID": "101",
"Exit Caused By App": true,
"Loss Reason": ""
},
"Task Info": {
"Task ID": 113,
"Index": 50,
"Attempt": 0,
"Launch Time": 1548895514836,
"Executor ID": "101",
"Host": "10.203.100.137",
"Locality": "RACK_LOCAL",
"Speculative": false,
"Getting Result Time": 0,
"Finish Time": 1548895518542,
"Failed": true,
"Accumulables": []
}
}
{quote}
> failed task's SparkListenerTaskEnd lost taskMetrics, which is important for
> problem diagnosis, especially when checking exists of data skew
> ---
>
> Key: SPARK-26938
> URL: https://issues.apache.org/jira/browse/SPARK-26938
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
>Affects Versions: 2.3.3
>Reporter: weiwenda
>Priority: Minor
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-26938) failed task's SparkListenerTaskEnd lost taskMetrics, which is important for problem diagnosis, especially when checking exists of data skew
[
https://issues.apache.org/jira/browse/SPARK-26938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16772685#comment-16772685
]
weiwenda commented on SPARK-26938:
--
for now, SparkListenerTaskEnd will write out to log file as below, so that
historyserver is unable to show taskMetrics of failed task.
{quote}{
"Event": "SparkListenerTaskEnd",
"Stage ID": 0,
"Stage Attempt ID": 0,
"Task Type": "ShuffleMapTask",
"Task End Reason": {
"Reason": "ExecutorLostFailure",
"Executor ID": "101",
"Exit Caused By App": true,
"Loss Reason": ""
},
"Task Info": {
"Task ID": 113,
"Index": 50,
"Attempt": 0,
"Launch Time": 1548895514836,
"Executor ID": "101",
"Host": "10.203.100.137",
"Locality": "RACK_LOCAL",
"Speculative": false,
"Getting Result Time": 0,
"Finish Time": 1548895518542,
"Failed": true,
"Accumulables": []
}
}
{quote}
> failed task's SparkListenerTaskEnd lost taskMetrics, which is important for
> problem diagnosis, especially when checking exists of data skew
> ---
>
> Key: SPARK-26938
> URL: https://issues.apache.org/jira/browse/SPARK-26938
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
>Affects Versions: 2.3.3
>Reporter: weiwenda
>Priority: Minor
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-26938) failed task's SparkListenerTaskEnd lost taskMetrics, which is important for problem diagnosis, especially when checking exists of data skew
weiwenda created SPARK-26938: Summary: failed task's SparkListenerTaskEnd lost taskMetrics, which is important for problem diagnosis, especially when checking exists of data skew Key: SPARK-26938 URL: https://issues.apache.org/jira/browse/SPARK-26938 Project: Spark Issue Type: Improvement Components: Spark Core Affects Versions: 2.3.3 Reporter: weiwenda -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
