GitHub user gatorsmile opened a pull request:
https://github.com/apache/spark/pull/14034
[16355] [16354] [SQL] Fix Bugs When LIMIT/TABLESAMPLE is Zero or Negative
#### What changes were proposed in this pull request?
**Issue 1:** When a query containing LIMIT/TABLESAMPLE 0, the statistics
could be zero. Results are correct but it could cause a huge performance
regression. For example,
```Scala
Seq(("one", 1), ("two", 2), ("three", 3), ("four", 4)).toDF("k", "v")
.createOrReplaceTempView("test")
val df1 = spark.table("test")
val df2 = spark.table("test").limit(0)
val df = df1.join(df2, Seq("k"), "left")
```
The statistics of both `df` and `df2` are zero. The statistics values
should never be zero; otherwise `sizeInBytes` of `BinaryNode` will also be zero
(product of children). This PR is to increase it to `1` when the num of rows is
equal to 0.
**Issue 2:** When a query containing LIMIT/TABLESAMPLE is negative, we
should issue exceptions. Negative values could break the assumption of multiple
parts. For example, statistics calculation. Below is the example query.
```SQL
SELECT * FROM testData TABLESAMPLE (-1 rows)
SELECT * FROM testData LIMIT -1
```
This PR is to issue an appropriate exception in this case.
#### How was this patch tested?
Added test cases.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/gatorsmile/spark limit
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/14034.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #14034
----
commit 1255968908454bfb01b247567f796e10ca6e6d30
Author: gatorsmile <[email protected]>
Date: 2016-07-03T05:46:44Z
fix
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
