[ https://issues.apache.org/jira/browse/SPARK-20248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16086903#comment-16086903 ]
readme_kylin commented on SPARK-20248: -------------------------------------- yes,i have the problem too.if somebody queries a big table with no limitation,the driver would crushed > Spark SQL add limit parameter to enhance the reliability. > --------------------------------------------------------- > > Key: SPARK-20248 > URL: https://issues.apache.org/jira/browse/SPARK-20248 > Project: Spark > Issue Type: Improvement > Components: SQL > Affects Versions: 2.1.0 > Environment: 2.1.0 > Reporter: shaolinliu > Priority: Minor > > When we using thrift server, it is difficult to constrain the user's sql > statement; > When the user query a large table without limit, this will lead to thrift > server process memory occupancy lead to service instability; > In general, the user is not used correctly, because if you really need to > return the whole table: > 1, if you use this data to compute , you can complete the computation in > the cluster and then return > 2, if you want obtain the data, you can store it in hdfs > For the above scene, it is recommended to add a > "spark.sql.thriftserver.retainedResults" parameter, > 1, when it is 0, we don not restrict user's operation > 2, when it is greater than 0, if user query with limit, we use user's > limit;if not we use this to limit query's result > Priority user's limit is because, if the user consider the limit, in > general, the user is aware of the exact meaning of this query -- This message was sent by Atlassian JIRA (v6.4.14#64029) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org