[calcite] branch master updated: [CALCITE-4192] RelMdColumnOrigins gets the wrong index of group by columns after RelNode was optimized by AggregateProjectMergeRule rule (FangZheng Li)

2020-09-20 Thread chunwei 
This is an automated email from the ASF dual-hosted git repository.

chunwei pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/calcite.git


The following commit(s) were added to refs/heads/master by this push:
 new fb1835c  [CALCITE-4192] RelMdColumnOrigins gets the wrong index of 
group by columns after RelNode was optimized by AggregateProjectMergeRule rule 
(FangZheng Li)
fb1835c is described below

commit fb1835c3741018f805b61aae7888f0afe65761ee
Author: majiao 
AuthorDate: Wed Aug 26 14:52:04 2020 +0800

[CALCITE-4192] RelMdColumnOrigins gets the wrong index of group by columns 
after RelNode was optimized by AggregateProjectMergeRule rule (FangZheng Li)

close apache/calcite#2121
---
 .../calcite/rel/metadata/RelMdColumnOrigins.java   |  4 ++--
 .../org/apache/calcite/test/RelMetadataTest.java   | 23 ++
 .../org/apache/calcite/test/RelOptRulesTest.java   |  1 -
 3 files changed, 25 insertions(+), 3 deletions(-)

diff --git 
a/core/src/main/java/org/apache/calcite/rel/metadata/RelMdColumnOrigins.java 
b/core/src/main/java/org/apache/calcite/rel/metadata/RelMdColumnOrigins.java
index 2958c12..ed1b9cc 100644
--- a/core/src/main/java/org/apache/calcite/rel/metadata/RelMdColumnOrigins.java
+++ b/core/src/main/java/org/apache/calcite/rel/metadata/RelMdColumnOrigins.java
@@ -65,8 +65,8 @@ public class RelMdColumnOrigins
   public Set getColumnOrigins(Aggregate rel,
   RelMetadataQuery mq, int iOutputColumn) {
 if (iOutputColumn < rel.getGroupCount()) {
-  // Group columns pass through directly.
-  return mq.getColumnOrigins(rel.getInput(), iOutputColumn);
+  // get actual index of Group columns.
+  return mq.getColumnOrigins(rel.getInput(), 
rel.getGroupSet().asList().get(iOutputColumn));
 }
 
 // Aggregate columns are derived from input columns
diff --git a/core/src/test/java/org/apache/calcite/test/RelMetadataTest.java 
b/core/src/test/java/org/apache/calcite/test/RelMetadataTest.java
index 62485dc..7e9dd3e 100644
--- a/core/src/test/java/org/apache/calcite/test/RelMetadataTest.java
+++ b/core/src/test/java/org/apache/calcite/test/RelMetadataTest.java
@@ -3273,4 +3273,27 @@ public class RelMetadataTest extends SqlToRelTestBase {
   return this;
 }
   }
+
+  /** Test case for
+   * https://issues.apache.org/jira/browse/CALCITE-4192;>[CALCITE-4192]
+   * RelMdColumnOrigins get the wrong index of group by columns after RelNode 
was optimized by
+   * AggregateProjectMergeRule rule. */
+  @Test void testColumnOriginAfterAggProjectMergeRule() {
+final String sql = "select count(ename), SAL from emp group by SAL";
+final RelNode rel = tester.convertSqlToRel(sql).rel;
+final HepProgramBuilder programBuilder = HepProgram.builder();
+programBuilder.addRuleInstance(CoreRules.AGGREGATE_PROJECT_MERGE);
+final HepPlanner planner = new HepPlanner(programBuilder.build());
+planner.setRoot(rel);
+final RelNode optimizedRel = planner.findBestExp();
+
+Set origins = RelMetadataQuery.instance()
+.getColumnOrigins(optimizedRel, 1);
+assertThat(origins.size(), equalTo(1));
+
+RelColumnOrigin columnOrigin = origins.iterator().next();
+assertThat(columnOrigin.getOriginColumnOrdinal(), equalTo(5));
+
assertThat(columnOrigin.getOriginTable().getRowType().getFieldNames().get(5),
+equalTo("SAL"));
+  }
 }
diff --git a/core/src/test/java/org/apache/calcite/test/RelOptRulesTest.java 
b/core/src/test/java/org/apache/calcite/test/RelOptRulesTest.java
index b049154..bafe0a0 100644
--- a/core/src/test/java/org/apache/calcite/test/RelOptRulesTest.java
+++ b/core/src/test/java/org/apache/calcite/test/RelOptRulesTest.java
@@ -6812,5 +6812,4 @@ class RelOptRulesTest extends RelOptTestBase {
 .withRule(CoreRules.FILTER_REDUCE_EXPRESSIONS)
 .checkUnchanged();
   }
-
 }

[GitHub] [calcite] chunweilei closed pull request #2121: [CALCITE-4192] RelMdColumnOrigins should obtain the true column index of the group by column of the aggregation operator instead of passing thr

2020-09-20 Thread GitBox 


chunweilei closed pull request #2121:
URL: https://github.com/apache/calcite/pull/2121


   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[calcite] branch master updated: [CALCITE-4241] Some improvements to metadata query

2020-09-20 Thread chunwei 
This is an automated email from the ASF dual-hosted git repository.

chunwei pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/calcite.git


The following commit(s) were added to refs/heads/master by this push:
 new e7ac26d  [CALCITE-4241] Some improvements to metadata query
e7ac26d is described below

commit e7ac26d6ec083848531b456d2acf3f7f58a499b5
Author: liyafan82 
AuthorDate: Thu Sep 10 16:36:57 2020 +0800

[CALCITE-4241] Some improvements to metadata query
---
 .../rel/metadata/RelMdColumnUniqueness.java|  7 +++--
 .../rel/metadata/RelMdDistinctRowCount.java| 28 +---
 .../org/apache/calcite/rel/metadata/RelMdUtil.java | 10 ++--
 .../org/apache/calcite/test/RelMetadataTest.java   | 30 ++
 4 files changed, 59 insertions(+), 16 deletions(-)

diff --git 
a/core/src/main/java/org/apache/calcite/rel/metadata/RelMdColumnUniqueness.java 
b/core/src/main/java/org/apache/calcite/rel/metadata/RelMdColumnUniqueness.java
index 755964c..c75925b 100644
--- 
a/core/src/main/java/org/apache/calcite/rel/metadata/RelMdColumnUniqueness.java
+++ 
b/core/src/main/java/org/apache/calcite/rel/metadata/RelMdColumnUniqueness.java
@@ -364,7 +364,7 @@ public class RelMdColumnUniqueness
   return true;
 }
 final Set> set = new HashSet<>();
-final List values = new ArrayList<>();
+final List values = new ArrayList<>(columns.cardinality());
 for (ImmutableList tuple : rel.tuples) {
   for (int column : columns) {
 final RexLiteral literal = tuple.get(column);
@@ -395,7 +395,6 @@ public class RelMdColumnUniqueness
   public Boolean areColumnsUnique(RelSubset rel, RelMetadataQuery mq,
   ImmutableBitSet columns, boolean ignoreNulls) {
 columns = decorateWithConstantColumnsFromPredicates(columns, rel, mq);
-int nullCount = 0;
 for (RelNode rel2 : rel.getRels()) {
   if (rel2 instanceof Aggregate
   || rel2 instanceof Filter
@@ -410,7 +409,7 @@ public class RelMdColumnUniqueness
   return true;
 }
   } else {
-++nullCount;
+return null;
   }
 } catch (CyclicMetadataException e) {
   // Ignore this relational expression; there will be non-cyclic ones
@@ -418,7 +417,7 @@ public class RelMdColumnUniqueness
 }
   }
 }
-return nullCount == 0 ? false : null;
+return false;
   }
 
   private boolean simplyProjects(RelNode rel, ImmutableBitSet columns) {
diff --git 
a/core/src/main/java/org/apache/calcite/rel/metadata/RelMdDistinctRowCount.java 
b/core/src/main/java/org/apache/calcite/rel/metadata/RelMdDistinctRowCount.java
index 6a6b8e5..971f4f3 100644
--- 
a/core/src/main/java/org/apache/calcite/rel/metadata/RelMdDistinctRowCount.java
+++ 
b/core/src/main/java/org/apache/calcite/rel/metadata/RelMdDistinctRowCount.java
@@ -29,6 +29,7 @@ import org.apache.calcite.rel.core.TableModify;
 import org.apache.calcite.rel.core.Union;
 import org.apache.calcite.rel.core.Values;
 import org.apache.calcite.rex.RexBuilder;
+import org.apache.calcite.rex.RexLiteral;
 import org.apache.calcite.rex.RexNode;
 import org.apache.calcite.rex.RexUtil;
 import org.apache.calcite.util.Bug;
@@ -36,8 +37,12 @@ import org.apache.calcite.util.BuiltInMethod;
 import org.apache.calcite.util.ImmutableBitSet;
 import org.apache.calcite.util.NumberUtil;
 
+import com.google.common.collect.ImmutableList;
+
 import java.util.ArrayList;
+import java.util.HashSet;
 import java.util.List;
+import java.util.Set;
 
 /**
  * RelMdDistinctRowCount supplies a default implementation of
@@ -192,11 +197,26 @@ public class RelMdDistinctRowCount
 return 1D;
   }
 }
-double selectivity = RelMdUtil.guessSelectivity(predicate);
 
-// assume half the rows are duplicates
-double nRows = rel.estimateRowCount(mq) / 2;
-return RelMdUtil.numDistinctVals(nRows, nRows * selectivity);
+final Set> set = new HashSet<>();
+final List values = new ArrayList<>(groupKey.cardinality());
+for (ImmutableList tuple : rel.tuples) {
+  for (int column : groupKey) {
+final RexLiteral literal = tuple.get(column);
+values.add(literal.isNull()
+? NullSentinel.INSTANCE
+: literal.getValueAs(Comparable.class));
+  }
+  set.add(ImmutableList.copyOf(values));
+  values.clear();
+}
+double nRows = set.size();
+if ((predicate == null) || predicate.isAlwaysTrue()) {
+  return nRows;
+} else {
+  double selectivity = RelMdUtil.guessSelectivity(predicate);
+  return RelMdUtil.numDistinctVals(nRows, nRows * selectivity);
+}
   }
 
   public Double getDistinctRowCount(Project rel, RelMetadataQuery mq,
diff --git a/core/src/main/java/org/apache/calcite/rel/metadata/RelMdUtil.java 
b/core/src/main/java/org/apache/calcite/rel/metadata/RelMdUtil.java
index ae94a46..4f642a2 100644
---

[GitHub] [calcite] chunweilei merged pull request #2146: [CALCITE-4241] Some improvements to metadata query

2020-09-20 Thread GitBox 


chunweilei merged pull request #2146:
URL: https://github.com/apache/calcite/pull/2146


   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [calcite] Aaaaaaron commented on pull request #2161: [CALCITE-4240] Fix SqlTypeUtil#getMaxPrecisionScaleDecimal returns a decimal that with same precision and scale(Jiatao Tao)

2020-09-20 Thread GitBox 


Aaron commented on pull request #2161:
URL: https://github.com/apache/calcite/pull/2161#issuecomment-695883614


Still have some ut failed, will fix than, with review comments.



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [calcite] Aaaaaaron opened a new pull request #2161: [CALCITE-4240] Fix SqlTypeUtil#getMaxPrecisionScaleDecimal returns a decimal that with same precision and scale(Jiatao Tao)

2020-09-20 Thread GitBox 


Aaron opened a new pull request #2161:
URL: https://github.com/apache/calcite/pull/2161


   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [calcite] amaliujia commented on pull request #2160: [CALCITE-4269] Improvement on enumerable implementation for HOP and SESSION

2020-09-20 Thread GitBox 


amaliujia commented on pull request #2160:
URL: https://github.com/apache/calcite/pull/2160#issuecomment-695848881


   cc @vlsi 



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [calcite] amaliujia opened a new pull request #2160: [CALCITE-3780] Improve SESSION Table-valued Function

2020-09-20 Thread GitBox 


amaliujia opened a new pull request #2160:
URL: https://github.com/apache/calcite/pull/2160


   1. Use ArrayDeque to replace LinkedList for the sake of efficiency.
   2. Mark the list as "final".
   
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org