[GitHub] spark pull request #17051: [SPARK-17075][SQL] Follow up: fix file line endin...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/spark/pull/17051


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #17051: [SPARK-17075][SQL] Follow up: fix file line endin...

[lins05]

Github user lins05 commented on a diff in the pull request:

https://github.com/apache/spark/pull/17051#discussion_r10239
  
--- Diff: 
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/statsEstimation/FilterEstimationSuite.scala
 ---
@@ -398,6 +398,27 @@ class FilterEstimationSuite extends 
StatsEstimationTestBase {
 // For all other SQL types, we compare the entire object directly.
 assert(filteredStats.attributeStats(ar) == expectedColStats)
 }
-  }
 
+// If the filter has a binary operator (including those nested inside
+// AND/OR/NOT), swap the sides of the attribte and the literal, 
reverse the
+// operator, and then check again.
+val rewrittenFilter = filterNode transformExpressionsDown {
+  case op @ EqualTo(ar: AttributeReference, l: Literal) =>
--- End diff --

Emm, we not only switch the side of the attr and the literal, but also 
reversed the operator, e.g. `LessThan` would be changed to `GreaterThan`. So I 
guess we can't use `withNewChildren` here.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #17051: [SPARK-17075][SQL] Follow up: fix file line endin...

[lins05]

Github user lins05 commented on a diff in the pull request:

https://github.com/apache/spark/pull/17051#discussion_r102888024
  
--- Diff: 
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/statsEstimation/FilterEstimationSuite.scala
 ---
@@ -398,6 +398,27 @@ class FilterEstimationSuite extends 
StatsEstimationTestBase {
 // For all other SQL types, we compare the entire object directly.
 assert(filteredStats.attributeStats(ar) == expectedColStats)
 }
-  }
 
+// If the filter has a binary operator (including those nested inside
+// AND/OR/NOT), swap the sides of the attribte and the literal, 
reverse the
+// operator, and then check again.
+val rewrittenFilter = filterNode transformExpressionsDown {
+  case op @ EqualTo(ar: AttributeReference, l: Literal) =>
--- End diff --

👍 

I tried to find something like this but failed to, so I resorted to the 
current code.  Thanks for the tip!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #17051: [SPARK-17075][SQL] Follow up: fix file line endin...

[cloud-fan]

Github user cloud-fan commented on a diff in the pull request:

https://github.com/apache/spark/pull/17051#discussion_r102887733
  
--- Diff: 
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/statsEstimation/FilterEstimationSuite.scala
 ---
@@ -398,6 +398,27 @@ class FilterEstimationSuite extends 
StatsEstimationTestBase {
 // For all other SQL types, we compare the entire object directly.
 assert(filteredStats.attributeStats(ar) == expectedColStats)
 }
-  }
 
+// If the filter has a binary operator (including those nested inside
+// AND/OR/NOT), swap the sides of the attribte and the literal, 
reverse the
+// operator, and then check again.
+val rewrittenFilter = filterNode transformExpressionsDown {
+  case op @ EqualTo(ar: AttributeReference, l: Literal) =>
--- End diff --

nit: `case b @  BinaryComparison(ar: AttributeReference, l: Literal) => 
b.withNewChildren(l, ar)`


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #17051: [SPARK-17075][SQL] Follow up: fix file line endin...

[lins05]

Github user lins05 commented on a diff in the pull request:

https://github.com/apache/spark/pull/17051#discussion_r102887655
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/statsEstimation/FilterEstimation.scala
 ---
@@ -1,511 +1,509 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one or more
- * contributor license agreements.  See the NOTICE file distributed with
- * this work for additional information regarding copyright ownership.
- * The ASF licenses this file to You under the Apache License, Version 2.0
- * (the "License"); you may not use this file except in compliance with
- * the License.  You may obtain a copy of the License at
- *
- *http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package org.apache.spark.sql.catalyst.plans.logical.statsEstimation
-
-import java.sql.{Date, Timestamp}
-
-import scala.collection.immutable.{HashSet, Map}
-import scala.collection.mutable
-
-import org.apache.spark.internal.Logging
-import org.apache.spark.sql.catalyst.CatalystConf
-import org.apache.spark.sql.catalyst.expressions._
-import org.apache.spark.sql.catalyst.plans.logical._
-import org.apache.spark.sql.catalyst.util.DateTimeUtils
-import org.apache.spark.sql.types._
-
-case class FilterEstimation(plan: Filter, catalystConf: CatalystConf) 
extends Logging {
-
-  /**
-   * We use a mutable colStats because we need to update the corresponding 
ColumnStat
-   * for a column after we apply a predicate condition.  For example, 
column c has
-   * [min, max] value as [0, 100].  In a range condition such as (c > 40 
AND c <= 50),
-   * we need to set the column's [min, max] value to [40, 100] after we 
evaluate the
-   * first condition c > 40.  We need to set the column's [min, max] value 
to [40, 50]
-   * after we evaluate the second condition c <= 50.
-   */
-  private var mutableColStats: mutable.Map[ExprId, ColumnStat] = 
mutable.Map.empty
-
-  /**
-   * Returns an option of Statistics for a Filter logical plan node.
-   * For a given compound expression condition, this method computes 
filter selectivity
-   * (or the percentage of rows meeting the filter condition), which
-   * is used to compute row count, size in bytes, and the updated 
statistics after a given
-   * predicated is applied.
-   *
-   * @return Option[Statistics] When there is no statistics collected, it 
returns None.
-   */
-  def estimate: Option[Statistics] = {
-// We first copy child node's statistics and then modify it based on 
filter selectivity.
-val stats: Statistics = plan.child.stats(catalystConf)
-if (stats.rowCount.isEmpty) return None
-
-// save a mutable copy of colStats so that we can later change it 
recursively
-mutableColStats = mutable.Map(stats.attributeStats.map(kv => 
(kv._1.exprId, kv._2)).toSeq: _*)
-
-// estimate selectivity of this filter predicate
-val filterSelectivity: Double = 
calculateFilterSelectivity(plan.condition) match {
-  case Some(percent) => percent
-  // for not-supported condition, set filter selectivity to a 
conservative estimate 100%
-  case None => 1.0
-}
-
-// attributeStats has mapping Attribute-to-ColumnStat.
-// mutableColStats has mapping ExprId-to-ColumnStat.
-// We use an ExprId-to-Attribute map to facilitate the mapping 
Attribute-to-ColumnStat
-val expridToAttrMap: Map[ExprId, Attribute] =
-  stats.attributeStats.map(kv => (kv._1.exprId, kv._1))
-// copy mutableColStats contents to an immutable AttributeMap.
-val mutableAttributeStats: mutable.Map[Attribute, ColumnStat] =
-  mutableColStats.map(kv => expridToAttrMap(kv._1) -> kv._2)
-val newColStats = AttributeMap(mutableAttributeStats.toSeq)
-
-val filteredRowCount: BigInt =
-  EstimationUtils.ceil(BigDecimal(stats.rowCount.get) * 
filterSelectivity)
-val filteredSizeInBytes =
-  EstimationUtils.getOutputSize(plan.output, filteredRowCount, 
newColStats)
-
-Some(stats.copy(sizeInBytes = filteredSizeInBytes, rowCount = 
Some(filteredRowCount),
-  attributeStats = newColStats))
-  }
-
-  /**
-   * Returns a percentage of rows meeting a compound condition in Filter 
node.
-   * A compound condition is decomposed into multiple single conditions 
linked with AND, OR, NOT.
-   * For 

[GitHub] spark pull request #17051: [SPARK-17075][SQL] Follow up: fix file line endin...

[lins05]

Github user lins05 commented on a diff in the pull request:

https://github.com/apache/spark/pull/17051#discussion_r102887355
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/statsEstimation/FilterEstimation.scala
 ---
@@ -1,511 +1,509 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one or more
- * contributor license agreements.  See the NOTICE file distributed with
- * this work for additional information regarding copyright ownership.
- * The ASF licenses this file to You under the Apache License, Version 2.0
- * (the "License"); you may not use this file except in compliance with
- * the License.  You may obtain a copy of the License at
- *
- *http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package org.apache.spark.sql.catalyst.plans.logical.statsEstimation
-
-import java.sql.{Date, Timestamp}
-
-import scala.collection.immutable.{HashSet, Map}
-import scala.collection.mutable
-
-import org.apache.spark.internal.Logging
-import org.apache.spark.sql.catalyst.CatalystConf
-import org.apache.spark.sql.catalyst.expressions._
-import org.apache.spark.sql.catalyst.plans.logical._
-import org.apache.spark.sql.catalyst.util.DateTimeUtils
-import org.apache.spark.sql.types._
-
-case class FilterEstimation(plan: Filter, catalystConf: CatalystConf) 
extends Logging {
-
-  /**
-   * We use a mutable colStats because we need to update the corresponding 
ColumnStat
-   * for a column after we apply a predicate condition.  For example, 
column c has
-   * [min, max] value as [0, 100].  In a range condition such as (c > 40 
AND c <= 50),
-   * we need to set the column's [min, max] value to [40, 100] after we 
evaluate the
-   * first condition c > 40.  We need to set the column's [min, max] value 
to [40, 50]
-   * after we evaluate the second condition c <= 50.
-   */
-  private var mutableColStats: mutable.Map[ExprId, ColumnStat] = 
mutable.Map.empty
-
-  /**
-   * Returns an option of Statistics for a Filter logical plan node.
-   * For a given compound expression condition, this method computes 
filter selectivity
-   * (or the percentage of rows meeting the filter condition), which
-   * is used to compute row count, size in bytes, and the updated 
statistics after a given
-   * predicated is applied.
-   *
-   * @return Option[Statistics] When there is no statistics collected, it 
returns None.
-   */
-  def estimate: Option[Statistics] = {
-// We first copy child node's statistics and then modify it based on 
filter selectivity.
-val stats: Statistics = plan.child.stats(catalystConf)
-if (stats.rowCount.isEmpty) return None
-
-// save a mutable copy of colStats so that we can later change it 
recursively
-mutableColStats = mutable.Map(stats.attributeStats.map(kv => 
(kv._1.exprId, kv._2)).toSeq: _*)
-
-// estimate selectivity of this filter predicate
-val filterSelectivity: Double = 
calculateFilterSelectivity(plan.condition) match {
-  case Some(percent) => percent
-  // for not-supported condition, set filter selectivity to a 
conservative estimate 100%
-  case None => 1.0
-}
-
-// attributeStats has mapping Attribute-to-ColumnStat.
-// mutableColStats has mapping ExprId-to-ColumnStat.
-// We use an ExprId-to-Attribute map to facilitate the mapping 
Attribute-to-ColumnStat
-val expridToAttrMap: Map[ExprId, Attribute] =
-  stats.attributeStats.map(kv => (kv._1.exprId, kv._1))
-// copy mutableColStats contents to an immutable AttributeMap.
-val mutableAttributeStats: mutable.Map[Attribute, ColumnStat] =
-  mutableColStats.map(kv => expridToAttrMap(kv._1) -> kv._2)
-val newColStats = AttributeMap(mutableAttributeStats.toSeq)
-
-val filteredRowCount: BigInt =
-  EstimationUtils.ceil(BigDecimal(stats.rowCount.get) * 
filterSelectivity)
-val filteredSizeInBytes =
-  EstimationUtils.getOutputSize(plan.output, filteredRowCount, 
newColStats)
-
-Some(stats.copy(sizeInBytes = filteredSizeInBytes, rowCount = 
Some(filteredRowCount),
-  attributeStats = newColStats))
-  }
-
-  /**
-   * Returns a percentage of rows meeting a compound condition in Filter 
node.
-   * A compound condition is decomposed into multiple single conditions 
linked with AND, OR, NOT.
-   * For 

[GitHub] spark pull request #17051: [SPARK-17075][SQL] Follow up: fix file line endin...

[cloud-fan]

Github user cloud-fan commented on a diff in the pull request:

https://github.com/apache/spark/pull/17051#discussion_r102887105
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/statsEstimation/FilterEstimation.scala
 ---
@@ -1,511 +1,509 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one or more
- * contributor license agreements.  See the NOTICE file distributed with
- * this work for additional information regarding copyright ownership.
- * The ASF licenses this file to You under the Apache License, Version 2.0
- * (the "License"); you may not use this file except in compliance with
- * the License.  You may obtain a copy of the License at
- *
- *http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package org.apache.spark.sql.catalyst.plans.logical.statsEstimation
-
-import java.sql.{Date, Timestamp}
-
-import scala.collection.immutable.{HashSet, Map}
-import scala.collection.mutable
-
-import org.apache.spark.internal.Logging
-import org.apache.spark.sql.catalyst.CatalystConf
-import org.apache.spark.sql.catalyst.expressions._
-import org.apache.spark.sql.catalyst.plans.logical._
-import org.apache.spark.sql.catalyst.util.DateTimeUtils
-import org.apache.spark.sql.types._
-
-case class FilterEstimation(plan: Filter, catalystConf: CatalystConf) 
extends Logging {
-
-  /**
-   * We use a mutable colStats because we need to update the corresponding 
ColumnStat
-   * for a column after we apply a predicate condition.  For example, 
column c has
-   * [min, max] value as [0, 100].  In a range condition such as (c > 40 
AND c <= 50),
-   * we need to set the column's [min, max] value to [40, 100] after we 
evaluate the
-   * first condition c > 40.  We need to set the column's [min, max] value 
to [40, 50]
-   * after we evaluate the second condition c <= 50.
-   */
-  private var mutableColStats: mutable.Map[ExprId, ColumnStat] = 
mutable.Map.empty
-
-  /**
-   * Returns an option of Statistics for a Filter logical plan node.
-   * For a given compound expression condition, this method computes 
filter selectivity
-   * (or the percentage of rows meeting the filter condition), which
-   * is used to compute row count, size in bytes, and the updated 
statistics after a given
-   * predicated is applied.
-   *
-   * @return Option[Statistics] When there is no statistics collected, it 
returns None.
-   */
-  def estimate: Option[Statistics] = {
-// We first copy child node's statistics and then modify it based on 
filter selectivity.
-val stats: Statistics = plan.child.stats(catalystConf)
-if (stats.rowCount.isEmpty) return None
-
-// save a mutable copy of colStats so that we can later change it 
recursively
-mutableColStats = mutable.Map(stats.attributeStats.map(kv => 
(kv._1.exprId, kv._2)).toSeq: _*)
-
-// estimate selectivity of this filter predicate
-val filterSelectivity: Double = 
calculateFilterSelectivity(plan.condition) match {
-  case Some(percent) => percent
-  // for not-supported condition, set filter selectivity to a 
conservative estimate 100%
-  case None => 1.0
-}
-
-// attributeStats has mapping Attribute-to-ColumnStat.
-// mutableColStats has mapping ExprId-to-ColumnStat.
-// We use an ExprId-to-Attribute map to facilitate the mapping 
Attribute-to-ColumnStat
-val expridToAttrMap: Map[ExprId, Attribute] =
-  stats.attributeStats.map(kv => (kv._1.exprId, kv._1))
-// copy mutableColStats contents to an immutable AttributeMap.
-val mutableAttributeStats: mutable.Map[Attribute, ColumnStat] =
-  mutableColStats.map(kv => expridToAttrMap(kv._1) -> kv._2)
-val newColStats = AttributeMap(mutableAttributeStats.toSeq)
-
-val filteredRowCount: BigInt =
-  EstimationUtils.ceil(BigDecimal(stats.rowCount.get) * 
filterSelectivity)
-val filteredSizeInBytes =
-  EstimationUtils.getOutputSize(plan.output, filteredRowCount, 
newColStats)
-
-Some(stats.copy(sizeInBytes = filteredSizeInBytes, rowCount = 
Some(filteredRowCount),
-  attributeStats = newColStats))
-  }
-
-  /**
-   * Returns a percentage of rows meeting a compound condition in Filter 
node.
-   * A compound condition is decomposed into multiple single conditions 
linked with AND, OR, NOT.
-   * 

[GitHub] spark pull request #17051: [SPARK-17075][SQL] Follow up: fix file line endin...

[lins05]

GitHub user lins05 opened a pull request:

https://github.com/apache/spark/pull/17051

[SPARK-17075][SQL] Follow up: fix file line ending and improve the tests

## What changes were proposed in this pull request?

Fixed the line ending of `FilterEstimation.scala`. Also improved the tests 
to cover more cases.

## How was this patch tested?

Existing unit tests.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/lins05/spark fix-cbo-filter-file-encoding

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/17051.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #17051


commit ee6d9915b26254db176a5aa34c1d59e304e201e0
Author: Shuai Lin 
Date:   2017-02-24T05:59:41Z

[SPARK-17075][SQL] Follow up: fix file line ending and improve the tests.

commit 0f56d0f1003268e4945ec5a427bbcc4bb7061a49
Author: Shuai Lin 
Date:   2017-02-24T05:58:37Z

Use transformExpressionsDown to rewrite the filter.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org