[jira] [Commented] (BEAM-2676) move BeamSqlRow and BeamSqlRowType to sdk/java/core

[Xu Mingmin (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16110323#comment-16110323
 ] 

Xu Mingmin commented on BEAM-2676:
--

[~robertwb] [~takidau], get your point. 

I will move {{BeamRecord}} to core, which contains a default 
{{BeamRecordType}}. A {{BeamRecordCoder}} is included in core as well, which 
wraps a list of Coders, one for each field.

In SQL module, there's a {{BeamSqlRecordType}} extends {{BeamRecordType}} to 
support SQL types.



> move BeamSqlRow and BeamSqlRowType to sdk/java/core
> ---
>
> Key: BEAM-2676
> URL: https://issues.apache.org/jira/browse/BEAM-2676
> Project: Beam
>  Issue Type: Test
>  Components: dsl-sql
>Reporter: Xu Mingmin
>Assignee: Xu Mingmin
>  Labels: dsl_sql_merge
>
> BeamSqlRow/BeamSqlRowType is the fundamental of structured data processing in 
> Beam, like joins, simple projections/expansions. It's more visible to move 
> them to sdk-java-core.
> It contains two parts:
> 1). remove SQL word in the name,
> BeamSqlRow --> BeamRow
> BeamSqlRowType --> BeamRowType
> 2). move from package {{org.apache.beam.dsls.sql.schema}} to 
> {{org.apache.beam.sdk.sd}} (sd stands for structure data), in module 
> {{beam-sdks-java-core}}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BEAM-2441) DSL SQL maven module location and name

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16110314#comment-16110314
 ] 

ASF GitHub Bot commented on BEAM-2441:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3666


> DSL SQL maven module location and name
> --
>
> Key: BEAM-2441
> URL: https://issues.apache.org/jira/browse/BEAM-2441
> Project: Beam
>  Issue Type: Improvement
>  Components: dsl-sql
>Reporter: Luke Cwik
>Assignee: James Xu
>  Labels: dsl_sql_merge
>
> The current maven module location is *dsl/sql*, unfortunately this occludes 
> the fact that this is for the Java SDK and also prevents alternative language 
> implementations.
> Some alternative locations could be:
> {code}
> sdks/java/extensions/sql
> sdks/java/dsls/sql
> dsls/sql/java
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] beam pull request #3666: [BEAM-2441] move dsls/sql to sdks/java/extensions/s...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3666


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[23/59] beam git commit: rename package org.apache.beam.dsls.sql to org.apache.beam.sdk.extensions.sql

[takidau]

http://git-wip-us.apache.org/repos/asf/beam/blob/c1b5482d/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/interpreter/operator/string/BeamSqlOverlayExpressionTest.java
--
diff --git 
a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/interpreter/operator/string/BeamSqlOverlayExpressionTest.java
 
b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/interpreter/operator/string/BeamSqlOverlayExpressionTest.java
new file mode 100644
index 000..2ca0a98
--- /dev/null
+++ 
b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/interpreter/operator/string/BeamSqlOverlayExpressionTest.java
@@ -0,0 +1,87 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.beam.sdk.extensions.sql.interpreter.operator.string;
+
+import static org.junit.Assert.assertTrue;
+
+import java.util.ArrayList;
+import java.util.List;
+import 
org.apache.beam.sdk.extensions.sql.interpreter.BeamSqlFnExecutorTestBase;
+import 
org.apache.beam.sdk.extensions.sql.interpreter.operator.BeamSqlExpression;
+import 
org.apache.beam.sdk.extensions.sql.interpreter.operator.BeamSqlPrimitive;
+import org.apache.calcite.sql.type.SqlTypeName;
+import org.junit.Assert;
+import org.junit.Test;
+
+/**
+ * Test for BeamSqlOverlayExpression.
+ */
+public class BeamSqlOverlayExpressionTest extends BeamSqlFnExecutorTestBase {
+
+  @Test public void accept() throws Exception {
+List operands = new ArrayList<>();
+
+operands.add(BeamSqlPrimitive.of(SqlTypeName.VARCHAR, "hello"));
+operands.add(BeamSqlPrimitive.of(SqlTypeName.VARCHAR, "hello"));
+operands.add(BeamSqlPrimitive.of(SqlTypeName.INTEGER, 1));
+assertTrue(new BeamSqlOverlayExpression(operands).accept());
+
+operands.clear();
+operands.add(BeamSqlPrimitive.of(SqlTypeName.VARCHAR, "hello"));
+operands.add(BeamSqlPrimitive.of(SqlTypeName.VARCHAR, "hello"));
+operands.add(BeamSqlPrimitive.of(SqlTypeName.INTEGER, 1));
+operands.add(BeamSqlPrimitive.of(SqlTypeName.INTEGER, 2));
+assertTrue(new BeamSqlOverlayExpression(operands).accept());
+  }
+
+  @Test public void evaluate() throws Exception {
+List operands = new ArrayList<>();
+
+operands.add(BeamSqlPrimitive.of(SqlTypeName.VARCHAR, "w333rce"));
+operands.add(BeamSqlPrimitive.of(SqlTypeName.VARCHAR, "resou"));
+operands.add(BeamSqlPrimitive.of(SqlTypeName.INTEGER, 3));
+Assert.assertEquals("w3resou3rce",
+new BeamSqlOverlayExpression(operands).evaluate(record).getValue());
+
+operands.clear();
+operands.add(BeamSqlPrimitive.of(SqlTypeName.VARCHAR, "w333rce"));
+operands.add(BeamSqlPrimitive.of(SqlTypeName.VARCHAR, "resou"));
+operands.add(BeamSqlPrimitive.of(SqlTypeName.INTEGER, 3));
+operands.add(BeamSqlPrimitive.of(SqlTypeName.INTEGER, 4));
+Assert.assertEquals("w3resou33rce",
+new BeamSqlOverlayExpression(operands).evaluate(record).getValue());
+
+operands.clear();
+operands.add(BeamSqlPrimitive.of(SqlTypeName.VARCHAR, "w333rce"));
+operands.add(BeamSqlPrimitive.of(SqlTypeName.VARCHAR, "resou"));
+operands.add(BeamSqlPrimitive.of(SqlTypeName.INTEGER, 3));
+operands.add(BeamSqlPrimitive.of(SqlTypeName.INTEGER, 5));
+Assert.assertEquals("w3resou3rce",
+new BeamSqlOverlayExpression(operands).evaluate(record).getValue());
+
+operands.clear();
+operands.add(BeamSqlPrimitive.of(SqlTypeName.VARCHAR, "w333rce"));
+operands.add(BeamSqlPrimitive.of(SqlTypeName.VARCHAR, "resou"));
+operands.add(BeamSqlPrimitive.of(SqlTypeName.INTEGER, 3));
+operands.add(BeamSqlPrimitive.of(SqlTypeName.INTEGER, 7));
+Assert.assertEquals("w3resouce",
+new BeamSqlOverlayExpression(operands).evaluate(record).getValue());
+  }
+
+}

http://git-wip-us.apache.org/repos/asf/beam/blob/c1b5482d/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/interpreter/operator/string/BeamSqlPositionExpressionTest.java
--
diff --git 

[08/59] beam git commit: move dsls/sql to sdks/java/extensions/sql

[takidau]

http://git-wip-us.apache.org/repos/asf/beam/blob/ba493f85/sdks/java/extensions/sql/src/main/java/org/apache/beam/dsls/sql/interpreter/operator/string/BeamSqlOverlayExpression.java
--
diff --git 
a/sdks/java/extensions/sql/src/main/java/org/apache/beam/dsls/sql/interpreter/operator/string/BeamSqlOverlayExpression.java
 
b/sdks/java/extensions/sql/src/main/java/org/apache/beam/dsls/sql/interpreter/operator/string/BeamSqlOverlayExpression.java
new file mode 100644
index 000..cb6a523
--- /dev/null
+++ 
b/sdks/java/extensions/sql/src/main/java/org/apache/beam/dsls/sql/interpreter/operator/string/BeamSqlOverlayExpression.java
@@ -0,0 +1,77 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.beam.dsls.sql.interpreter.operator.string;
+
+import java.util.List;
+
+import org.apache.beam.dsls.sql.interpreter.operator.BeamSqlExpression;
+import org.apache.beam.dsls.sql.interpreter.operator.BeamSqlPrimitive;
+import org.apache.beam.dsls.sql.schema.BeamSqlRow;
+import org.apache.calcite.sql.type.SqlTypeName;
+
+/**
+ * 'OVERLAY' operator.
+ *
+ * 
+ *   OVERLAY(string1 PLACING string2 FROM integer [ FOR integer2 ])
+ * 
+ */
+public class BeamSqlOverlayExpression extends BeamSqlExpression {
+  public BeamSqlOverlayExpression(List operands) {
+super(operands, SqlTypeName.VARCHAR);
+  }
+
+  @Override public boolean accept() {
+if (operands.size() < 3 || operands.size() > 4) {
+  return false;
+}
+
+if (!SqlTypeName.CHAR_TYPES.contains(opType(0))
+|| !SqlTypeName.CHAR_TYPES.contains(opType(1))
+|| !SqlTypeName.INT_TYPES.contains(opType(2))) {
+  return false;
+}
+
+if (operands.size() == 4 && !SqlTypeName.INT_TYPES.contains(opType(3))) {
+  return false;
+}
+
+return true;
+  }
+
+  @Override public BeamSqlPrimitive evaluate(BeamSqlRow inputRow) {
+String str = opValueEvaluated(0, inputRow);
+String replaceStr = opValueEvaluated(1, inputRow);
+int idx = opValueEvaluated(2, inputRow);
+// the index is 1 based.
+idx -= 1;
+int length = replaceStr.length();
+if (operands.size() == 4) {
+  length = opValueEvaluated(3, inputRow);
+}
+
+StringBuilder result = new StringBuilder(
+str.length() + replaceStr.length() - length);
+result.append(str.substring(0, idx))
+.append(replaceStr)
+.append(str.substring(idx + length));
+
+return BeamSqlPrimitive.of(SqlTypeName.VARCHAR, result.toString());
+  }
+}

http://git-wip-us.apache.org/repos/asf/beam/blob/ba493f85/sdks/java/extensions/sql/src/main/java/org/apache/beam/dsls/sql/interpreter/operator/string/BeamSqlPositionExpression.java
--
diff --git 
a/sdks/java/extensions/sql/src/main/java/org/apache/beam/dsls/sql/interpreter/operator/string/BeamSqlPositionExpression.java
 
b/sdks/java/extensions/sql/src/main/java/org/apache/beam/dsls/sql/interpreter/operator/string/BeamSqlPositionExpression.java
new file mode 100644
index 000..144acbf
--- /dev/null
+++ 
b/sdks/java/extensions/sql/src/main/java/org/apache/beam/dsls/sql/interpreter/operator/string/BeamSqlPositionExpression.java
@@ -0,0 +1,73 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.beam.dsls.sql.interpreter.operator.string;
+
+import java.util.List;
+
+import 

[50/59] beam git commit: move all implementation classes/packages into impl package

[takidau]

http://git-wip-us.apache.org/repos/asf/beam/blob/febd044a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/rel/BeamFilterRel.java
--
diff --git 
a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/rel/BeamFilterRel.java
 
b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/rel/BeamFilterRel.java
deleted file mode 100644
index f1da29f..000
--- 
a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/rel/BeamFilterRel.java
+++ /dev/null
@@ -1,70 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.sdk.extensions.sql.rel;
-
-import org.apache.beam.sdk.extensions.sql.BeamSqlEnv;
-import 
org.apache.beam.sdk.extensions.sql.interpreter.BeamSqlExpressionExecutor;
-import org.apache.beam.sdk.extensions.sql.interpreter.BeamSqlFnExecutor;
-import org.apache.beam.sdk.extensions.sql.schema.BeamSqlRow;
-import org.apache.beam.sdk.extensions.sql.schema.BeamSqlRowCoder;
-import org.apache.beam.sdk.extensions.sql.transform.BeamSqlFilterFn;
-import org.apache.beam.sdk.extensions.sql.utils.CalciteUtils;
-import org.apache.beam.sdk.transforms.ParDo;
-import org.apache.beam.sdk.values.PCollection;
-import org.apache.beam.sdk.values.PCollectionTuple;
-import org.apache.calcite.plan.RelOptCluster;
-import org.apache.calcite.plan.RelTraitSet;
-import org.apache.calcite.rel.RelNode;
-import org.apache.calcite.rel.core.Filter;
-import org.apache.calcite.rex.RexNode;
-
-/**
- * BeamRelNode to replace a {@code Filter} node.
- *
- */
-public class BeamFilterRel extends Filter implements BeamRelNode {
-
-  public BeamFilterRel(RelOptCluster cluster, RelTraitSet traits, RelNode 
child,
-  RexNode condition) {
-super(cluster, traits, child, condition);
-  }
-
-  @Override
-  public Filter copy(RelTraitSet traitSet, RelNode input, RexNode condition) {
-return new BeamFilterRel(getCluster(), traitSet, input, condition);
-  }
-
-  @Override
-  public PCollection buildBeamPipeline(PCollectionTuple 
inputPCollections
-  , BeamSqlEnv sqlEnv) throws Exception {
-RelNode input = getInput();
-String stageName = BeamSqlRelUtils.getStageName(this);
-
-PCollection upstream =
-
BeamSqlRelUtils.getBeamRelInput(input).buildBeamPipeline(inputPCollections, 
sqlEnv);
-
-BeamSqlExpressionExecutor executor = new BeamSqlFnExecutor(this);
-
-PCollection filterStream = upstream.apply(stageName,
-ParDo.of(new BeamSqlFilterFn(getRelTypeName(), executor)));
-filterStream.setCoder(new 
BeamSqlRowCoder(CalciteUtils.toBeamRowType(getRowType(;
-
-return filterStream;
-  }
-
-}

http://git-wip-us.apache.org/repos/asf/beam/blob/febd044a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/rel/BeamIOSinkRel.java
--
diff --git 
a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/rel/BeamIOSinkRel.java
 
b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/rel/BeamIOSinkRel.java
deleted file mode 100644
index ce941a0..000
--- 
a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/rel/BeamIOSinkRel.java
+++ /dev/null
@@ -1,75 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.sdk.extensions.sql.rel;
-
-import 

[41/59] beam git commit: rename package org.apache.beam.dsls.sql to org.apache.beam.sdk.extensions.sql

[takidau]

http://git-wip-us.apache.org/repos/asf/beam/blob/c1b5482d/sdks/java/extensions/sql/src/main/java/org/apache/beam/dsls/sql/interpreter/operator/BeamSqlPrimitive.java
--
diff --git 
a/sdks/java/extensions/sql/src/main/java/org/apache/beam/dsls/sql/interpreter/operator/BeamSqlPrimitive.java
 
b/sdks/java/extensions/sql/src/main/java/org/apache/beam/dsls/sql/interpreter/operator/BeamSqlPrimitive.java
deleted file mode 100644
index 51724bb..000
--- 
a/sdks/java/extensions/sql/src/main/java/org/apache/beam/dsls/sql/interpreter/operator/BeamSqlPrimitive.java
+++ /dev/null
@@ -1,152 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.dsls.sql.interpreter.operator;
-
-import java.math.BigDecimal;
-import java.util.Date;
-import java.util.GregorianCalendar;
-import java.util.List;
-import org.apache.beam.dsls.sql.schema.BeamSqlRow;
-import org.apache.calcite.sql.type.SqlTypeName;
-import org.apache.calcite.util.NlsString;
-
-/**
- * {@link BeamSqlPrimitive} is a special, self-reference {@link 
BeamSqlExpression}.
- * It holds the value, and return it directly during {@link 
#evaluate(BeamSqlRow)}.
- *
- */
-public class BeamSqlPrimitive extends BeamSqlExpression {
-  private T value;
-
-  private BeamSqlPrimitive() {
-  }
-
-  private BeamSqlPrimitive(List operands, SqlTypeName 
outputType) {
-super(operands, outputType);
-  }
-
-  /**
-   * A builder function to create from Type and value directly.
-   */
-  public static  BeamSqlPrimitive of(SqlTypeName outputType, T value){
-BeamSqlPrimitive exp = new BeamSqlPrimitive<>();
-exp.outputType = outputType;
-exp.value = value;
-if (!exp.accept()) {
-  throw new IllegalArgumentException(
-  String.format("value [%s] doesn't match type [%s].", value, 
outputType));
-}
-return exp;
-  }
-
-  public SqlTypeName getOutputType() {
-return outputType;
-  }
-
-  public T getValue() {
-return value;
-  }
-
-  public long getLong() {
-return (Long) getValue();
-  }
-
-  public double getDouble() {
-return (Double) getValue();
-  }
-
-  public float getFloat() {
-return (Float) getValue();
-  }
-
-  public int getInteger() {
-return (Integer) getValue();
-  }
-
-  public short getShort() {
-return (Short) getValue();
-  }
-
-  public byte getByte() {
-return (Byte) getValue();
-  }
-  public boolean getBoolean() {
-return (Boolean) getValue();
-  }
-
-  public String getString() {
-return (String) getValue();
-  }
-
-  public Date getDate() {
-return (Date) getValue();
-  }
-
-  public BigDecimal getDecimal() {
-return (BigDecimal) getValue();
-  }
-
-  @Override
-  public boolean accept() {
-if (value == null) {
-  return true;
-}
-
-switch (outputType) {
-case BIGINT:
-  return value instanceof Long;
-case DECIMAL:
-  return value instanceof BigDecimal;
-case DOUBLE:
-  return value instanceof Double;
-case FLOAT:
-  return value instanceof Float;
-case INTEGER:
-  return value instanceof Integer;
-case SMALLINT:
-  return value instanceof Short;
-case TINYINT:
-  return value instanceof Byte;
-case BOOLEAN:
-  return value instanceof Boolean;
-case CHAR:
-case VARCHAR:
-  return value instanceof String || value instanceof NlsString;
-case TIME:
-  return value instanceof GregorianCalendar;
-case TIMESTAMP:
-case DATE:
-  return value instanceof Date;
-case INTERVAL_HOUR:
-  return value instanceof BigDecimal;
-case INTERVAL_MINUTE:
-  return value instanceof BigDecimal;
-case SYMBOL:
-  // for SYMBOL, it supports anything...
-  return true;
-default:
-  throw new UnsupportedOperationException(outputType.name());
-}
-  }
-
-  @Override
-  public BeamSqlPrimitive evaluate(BeamSqlRow inputRow) {
-return this;
-  }
-
-}

http://git-wip-us.apache.org/repos/asf/beam/blob/c1b5482d/sdks/java/extensions/sql/src/main/java/org/apache/beam/dsls/sql/interpreter/operator/BeamSqlReinterpretExpression.java
--
diff --git 

[11/59] beam git commit: move dsls/sql to sdks/java/extensions/sql

[takidau]

http://git-wip-us.apache.org/repos/asf/beam/blob/ba493f85/dsls/sql/src/test/java/org/apache/beam/dsls/sql/schema/transform/BeamAggregationTransformTest.java
--
diff --git 
a/dsls/sql/src/test/java/org/apache/beam/dsls/sql/schema/transform/BeamAggregationTransformTest.java
 
b/dsls/sql/src/test/java/org/apache/beam/dsls/sql/schema/transform/BeamAggregationTransformTest.java
deleted file mode 100644
index 5d5d4fc..000
--- 
a/dsls/sql/src/test/java/org/apache/beam/dsls/sql/schema/transform/BeamAggregationTransformTest.java
+++ /dev/null
@@ -1,453 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- * 
- * http://www.apache.org/licenses/LICENSE-2.0
- * 
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.dsls.sql.schema.transform;
-
-import java.text.ParseException;
-import java.util.ArrayList;
-import java.util.Arrays;
-import java.util.List;
-import org.apache.beam.dsls.sql.planner.BeamQueryPlanner;
-import org.apache.beam.dsls.sql.schema.BeamSqlRow;
-import org.apache.beam.dsls.sql.schema.BeamSqlRowCoder;
-import org.apache.beam.dsls.sql.schema.BeamSqlRowType;
-import org.apache.beam.dsls.sql.transform.BeamAggregationTransforms;
-import org.apache.beam.dsls.sql.utils.CalciteUtils;
-import org.apache.beam.sdk.coders.IterableCoder;
-import org.apache.beam.sdk.coders.KvCoder;
-import org.apache.beam.sdk.testing.PAssert;
-import org.apache.beam.sdk.testing.TestPipeline;
-import org.apache.beam.sdk.transforms.Combine;
-import org.apache.beam.sdk.transforms.Create;
-import org.apache.beam.sdk.transforms.GroupByKey;
-import org.apache.beam.sdk.transforms.ParDo;
-import org.apache.beam.sdk.transforms.WithKeys;
-import org.apache.beam.sdk.values.KV;
-import org.apache.beam.sdk.values.PCollection;
-import org.apache.calcite.rel.core.AggregateCall;
-import org.apache.calcite.rel.type.RelDataTypeFactory.FieldInfoBuilder;
-import org.apache.calcite.rel.type.RelDataTypeSystem;
-import org.apache.calcite.sql.SqlKind;
-import org.apache.calcite.sql.fun.SqlAvgAggFunction;
-import org.apache.calcite.sql.fun.SqlCountAggFunction;
-import org.apache.calcite.sql.fun.SqlMinMaxAggFunction;
-import org.apache.calcite.sql.fun.SqlSumAggFunction;
-import org.apache.calcite.sql.type.BasicSqlType;
-import org.apache.calcite.sql.type.SqlTypeName;
-import org.apache.calcite.util.ImmutableBitSet;
-import org.junit.Rule;
-import org.junit.Test;
-
-/**
- * Unit tests for {@link BeamAggregationTransforms}.
- *
- */
-public class BeamAggregationTransformTest extends BeamTransformBaseTest{
-
-  @Rule
-  public TestPipeline p = TestPipeline.create();
-
-  private List aggCalls;
-
-  private BeamSqlRowType keyType;
-  private BeamSqlRowType aggPartType;
-  private BeamSqlRowType outputType;
-
-  private BeamSqlRowCoder inRecordCoder;
-  private BeamSqlRowCoder keyCoder;
-  private BeamSqlRowCoder aggCoder;
-  private BeamSqlRowCoder outRecordCoder;
-
-  /**
-   * This step equals to below query.
-   * 
-   * SELECT `f_int`
-   * , COUNT(*) AS `size`
-   * , SUM(`f_long`) AS `sum1`, AVG(`f_long`) AS `avg1`
-   * , MAX(`f_long`) AS `max1`, MIN(`f_long`) AS `min1`
-   * , SUM(`f_short`) AS `sum2`, AVG(`f_short`) AS `avg2`
-   * , MAX(`f_short`) AS `max2`, MIN(`f_short`) AS `min2`
-   * , SUM(`f_byte`) AS `sum3`, AVG(`f_byte`) AS `avg3`
-   * , MAX(`f_byte`) AS `max3`, MIN(`f_byte`) AS `min3`
-   * , SUM(`f_float`) AS `sum4`, AVG(`f_float`) AS `avg4`
-   * , MAX(`f_float`) AS `max4`, MIN(`f_float`) AS `min4`
-   * , SUM(`f_double`) AS `sum5`, AVG(`f_double`) AS `avg5`
-   * , MAX(`f_double`) AS `max5`, MIN(`f_double`) AS `min5`
-   * , MAX(`f_timestamp`) AS `max7`, MIN(`f_timestamp`) AS `min7`
-   * ,SUM(`f_int2`) AS `sum8`, AVG(`f_int2`) AS `avg8`
-   * , MAX(`f_int2`) AS `max8`, MIN(`f_int2`) AS `min8`
-   * FROM TABLE_NAME
-   * GROUP BY `f_int`
-   * 
-   * @throws ParseException
-   */
-  @Test
-  public void testCountPerElementBasic() throws ParseException {
-setupEnvironment();
-
-PCollection input = p.apply(Create.of(inputRows));
-
-//1. extract fields in group-by key part
-PCollection> exGroupByStream = 
input.apply("exGroupBy",
-WithKeys
-.of(new BeamAggregationTransforms.AggregationGroupByKeyFn(-1, 
ImmutableBitSet.of(0

[38/59] beam git commit: rename package org.apache.beam.dsls.sql to org.apache.beam.sdk.extensions.sql

[takidau]

http://git-wip-us.apache.org/repos/asf/beam/blob/c1b5482d/sdks/java/extensions/sql/src/main/java/org/apache/beam/dsls/sql/rel/BeamSortRel.java
--
diff --git 
a/sdks/java/extensions/sql/src/main/java/org/apache/beam/dsls/sql/rel/BeamSortRel.java
 
b/sdks/java/extensions/sql/src/main/java/org/apache/beam/dsls/sql/rel/BeamSortRel.java
deleted file mode 100644
index ba344df..000
--- 
a/sdks/java/extensions/sql/src/main/java/org/apache/beam/dsls/sql/rel/BeamSortRel.java
+++ /dev/null
@@ -1,247 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package org.apache.beam.dsls.sql.rel;
-
-import java.io.Serializable;
-import java.lang.reflect.Type;
-import java.math.BigDecimal;
-import java.util.ArrayList;
-import java.util.Comparator;
-import java.util.List;
-import org.apache.beam.dsls.sql.BeamSqlEnv;
-import org.apache.beam.dsls.sql.schema.BeamSqlRow;
-import org.apache.beam.dsls.sql.schema.BeamSqlRowCoder;
-import org.apache.beam.dsls.sql.utils.CalciteUtils;
-import org.apache.beam.sdk.coders.ListCoder;
-import org.apache.beam.sdk.transforms.DoFn;
-import org.apache.beam.sdk.transforms.Flatten;
-import org.apache.beam.sdk.transforms.ParDo;
-import org.apache.beam.sdk.transforms.Top;
-import org.apache.beam.sdk.transforms.windowing.GlobalWindow;
-import org.apache.beam.sdk.values.PCollection;
-import org.apache.beam.sdk.values.PCollectionTuple;
-import org.apache.calcite.plan.RelOptCluster;
-import org.apache.calcite.plan.RelTraitSet;
-import org.apache.calcite.rel.RelCollation;
-import org.apache.calcite.rel.RelCollationImpl;
-import org.apache.calcite.rel.RelFieldCollation;
-import org.apache.calcite.rel.RelNode;
-import org.apache.calcite.rel.core.Sort;
-import org.apache.calcite.rex.RexInputRef;
-import org.apache.calcite.rex.RexLiteral;
-import org.apache.calcite.rex.RexNode;
-import org.apache.calcite.sql.type.SqlTypeName;
-
-/**
- * {@code BeamRelNode} to replace a {@code Sort} node.
- *
- * Since Beam does not fully supported global sort we are using {@link Top} 
to implement
- * the {@code Sort} algebra. The following types of ORDER BY are supported:
-
- * {@code
- * select * from t order by id desc limit 10;
- * select * from t order by id desc limit 10, 5;
- * }
- *
- * but Order BY without a limit is NOT supported:
- *
- * {@code
- *   select * from t order by id desc
- * }
- *
- * Constraints
- * 
- *   Due to the constraints of {@link Top}, the result of a `ORDER BY 
LIMIT`
- *   must fit into the memory of a single machine.
- *   Since `WINDOW`(HOP, TUMBLE, SESSION etc) is always associated with 
`GroupBy`,
- *   it does not make much sense to use `ORDER BY` with `WINDOW`.
- *   
- * 
- */
-public class BeamSortRel extends Sort implements BeamRelNode {
-  private List fieldIndices = new ArrayList<>();
-  private List orientation = new ArrayList<>();
-  private List nullsFirst = new ArrayList<>();
-
-  private int startIndex = 0;
-  private int count;
-
-  public BeamSortRel(
-  RelOptCluster cluster,
-  RelTraitSet traits,
-  RelNode child,
-  RelCollation collation,
-  RexNode offset,
-  RexNode fetch) {
-super(cluster, traits, child, collation, offset, fetch);
-
-List fieldExps = getChildExps();
-RelCollationImpl collationImpl = (RelCollationImpl) collation;
-List collations = collationImpl.getFieldCollations();
-for (int i = 0; i < fieldExps.size(); i++) {
-  RexNode fieldExp = fieldExps.get(i);
-  RexInputRef inputRef = (RexInputRef) fieldExp;
-  fieldIndices.add(inputRef.getIndex());
-  orientation.add(collations.get(i).getDirection() == 
RelFieldCollation.Direction.ASCENDING);
-
-  RelFieldCollation.NullDirection rawNullDirection = 
collations.get(i).nullDirection;
-  if (rawNullDirection == RelFieldCollation.NullDirection.UNSPECIFIED) {
-rawNullDirection = 
collations.get(i).getDirection().defaultNullDirection();
-  }
-  nullsFirst.add(rawNullDirection == 
RelFieldCollation.NullDirection.FIRST);
-}
-
-if (fetch == null) {
-  throw new UnsupportedOperationException("ORDER BY without a LIMIT is not 
supported!");
-}
-
-RexLiteral fetchLiteral = 

[18/59] beam git commit: move dsls/sql to sdks/java/extensions/sql

[takidau]

http://git-wip-us.apache.org/repos/asf/beam/blob/ba493f85/dsls/sql/src/main/java/org/apache/beam/dsls/sql/interpreter/operator/string/BeamSqlConcatExpression.java
--
diff --git 
a/dsls/sql/src/main/java/org/apache/beam/dsls/sql/interpreter/operator/string/BeamSqlConcatExpression.java
 
b/dsls/sql/src/main/java/org/apache/beam/dsls/sql/interpreter/operator/string/BeamSqlConcatExpression.java
deleted file mode 100644
index 93e1f71..000
--- 
a/dsls/sql/src/main/java/org/apache/beam/dsls/sql/interpreter/operator/string/BeamSqlConcatExpression.java
+++ /dev/null
@@ -1,63 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package org.apache.beam.dsls.sql.interpreter.operator.string;
-
-import java.util.List;
-
-import org.apache.beam.dsls.sql.interpreter.operator.BeamSqlExpression;
-import org.apache.beam.dsls.sql.interpreter.operator.BeamSqlPrimitive;
-import org.apache.beam.dsls.sql.schema.BeamSqlRow;
-import org.apache.calcite.sql.type.SqlTypeName;
-
-/**
- * String concat operator.
- */
-public class BeamSqlConcatExpression extends BeamSqlExpression {
-
-  protected BeamSqlConcatExpression(List operands, 
SqlTypeName outputType) {
-super(operands, outputType);
-  }
-
-  public BeamSqlConcatExpression(List operands) {
-super(operands, SqlTypeName.VARCHAR);
-  }
-
-  @Override public boolean accept() {
-if (operands.size() != 2) {
-  return false;
-}
-
-for (BeamSqlExpression exp : getOperands()) {
-  if (!SqlTypeName.CHAR_TYPES.contains(exp.getOutputType())) {
-return false;
-  }
-}
-
-return true;
-  }
-
-  @Override public BeamSqlPrimitive evaluate(BeamSqlRow inputRow) {
-String left = opValueEvaluated(0, inputRow);
-String right = opValueEvaluated(1, inputRow);
-
-return BeamSqlPrimitive.of(SqlTypeName.VARCHAR,
-new StringBuilder(left.length() + right.length())
-.append(left).append(right).toString());
-  }
-}

http://git-wip-us.apache.org/repos/asf/beam/blob/ba493f85/dsls/sql/src/main/java/org/apache/beam/dsls/sql/interpreter/operator/string/BeamSqlInitCapExpression.java
--
diff --git 
a/dsls/sql/src/main/java/org/apache/beam/dsls/sql/interpreter/operator/string/BeamSqlInitCapExpression.java
 
b/dsls/sql/src/main/java/org/apache/beam/dsls/sql/interpreter/operator/string/BeamSqlInitCapExpression.java
deleted file mode 100644
index 7726e27..000
--- 
a/dsls/sql/src/main/java/org/apache/beam/dsls/sql/interpreter/operator/string/BeamSqlInitCapExpression.java
+++ /dev/null
@@ -1,56 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package org.apache.beam.dsls.sql.interpreter.operator.string;
-
-import java.util.List;
-
-import org.apache.beam.dsls.sql.interpreter.operator.BeamSqlExpression;
-import org.apache.beam.dsls.sql.interpreter.operator.BeamSqlPrimitive;
-import org.apache.beam.dsls.sql.schema.BeamSqlRow;
-import org.apache.calcite.sql.type.SqlTypeName;
-
-/**
- * 'INITCAP' operator.
- */
-public class BeamSqlInitCapExpression extends BeamSqlStringUnaryExpression {
-  public BeamSqlInitCapExpression(List operands) {
-super(operands, SqlTypeName.VARCHAR);
-  }
-
-  @Override public BeamSqlPrimitive evaluate(BeamSqlRow inputRow) {
-String str = opValueEvaluated(0, inputRow);
-
-StringBuilder ret = new StringBuilder(str);
-boolean 

[10/59] beam git commit: move dsls/sql to sdks/java/extensions/sql

[takidau]

http://git-wip-us.apache.org/repos/asf/beam/blob/ba493f85/sdks/java/extensions/sql/src/main/java/org/apache/beam/dsls/sql/interpreter/operator/BeamSqlExpression.java
--
diff --git 
a/sdks/java/extensions/sql/src/main/java/org/apache/beam/dsls/sql/interpreter/operator/BeamSqlExpression.java
 
b/sdks/java/extensions/sql/src/main/java/org/apache/beam/dsls/sql/interpreter/operator/BeamSqlExpression.java
new file mode 100644
index 000..9d2815c
--- /dev/null
+++ 
b/sdks/java/extensions/sql/src/main/java/org/apache/beam/dsls/sql/interpreter/operator/BeamSqlExpression.java
@@ -0,0 +1,78 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.dsls.sql.interpreter.operator;
+
+import java.io.Serializable;
+import java.util.List;
+import org.apache.beam.dsls.sql.schema.BeamSqlRow;
+import org.apache.calcite.rex.RexNode;
+import org.apache.calcite.sql.type.SqlTypeName;
+
+/**
+ * {@code BeamSqlExpression} is an equivalent expression in BeamSQL, of {@link 
RexNode} in Calcite.
+ *
+ * An implementation of {@link BeamSqlExpression} takes one or more {@code 
BeamSqlExpression}
+ * as its operands, and return a value with type {@link SqlTypeName}.
+ *
+ */
+public abstract class BeamSqlExpression implements Serializable {
+  protected List operands;
+  protected SqlTypeName outputType;
+
+  protected BeamSqlExpression(){}
+
+  public BeamSqlExpression(List operands, SqlTypeName 
outputType) {
+this.operands = operands;
+this.outputType = outputType;
+  }
+
+  public BeamSqlExpression op(int idx) {
+return operands.get(idx);
+  }
+
+  public SqlTypeName opType(int idx) {
+return op(idx).getOutputType();
+  }
+
+  public  T opValueEvaluated(int idx, BeamSqlRow row) {
+return (T) op(idx).evaluate(row).getValue();
+  }
+
+  /**
+   * assertion to make sure the input and output are supported in this 
expression.
+   */
+  public abstract boolean accept();
+
+  /**
+   * Apply input record {@link BeamSqlRow} to this expression,
+   * the output value is wrapped with {@link BeamSqlPrimitive}.
+   */
+  public abstract BeamSqlPrimitive evaluate(BeamSqlRow inputRow);
+
+  public List getOperands() {
+return operands;
+  }
+
+  public SqlTypeName getOutputType() {
+return outputType;
+  }
+
+  public int numberOfOperands() {
+return operands.size();
+  }
+}

http://git-wip-us.apache.org/repos/asf/beam/blob/ba493f85/sdks/java/extensions/sql/src/main/java/org/apache/beam/dsls/sql/interpreter/operator/BeamSqlInputRefExpression.java
--
diff --git 
a/sdks/java/extensions/sql/src/main/java/org/apache/beam/dsls/sql/interpreter/operator/BeamSqlInputRefExpression.java
 
b/sdks/java/extensions/sql/src/main/java/org/apache/beam/dsls/sql/interpreter/operator/BeamSqlInputRefExpression.java
new file mode 100644
index 000..710460b
--- /dev/null
+++ 
b/sdks/java/extensions/sql/src/main/java/org/apache/beam/dsls/sql/interpreter/operator/BeamSqlInputRefExpression.java
@@ -0,0 +1,43 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.dsls.sql.interpreter.operator;
+
+import org.apache.beam.dsls.sql.schema.BeamSqlRow;
+import org.apache.calcite.sql.type.SqlTypeName;
+
+/**
+ * An primitive operation for direct field extraction.
+ */
+public class BeamSqlInputRefExpression extends BeamSqlExpression {
+  private int 

[04/59] beam git commit: move dsls/sql to sdks/java/extensions/sql

[takidau]

http://git-wip-us.apache.org/repos/asf/beam/blob/ba493f85/sdks/java/extensions/sql/src/test/java/org/apache/beam/dsls/sql/integrationtest/BeamSqlDateFunctionsIntegrationTest.java
--
diff --git 
a/sdks/java/extensions/sql/src/test/java/org/apache/beam/dsls/sql/integrationtest/BeamSqlDateFunctionsIntegrationTest.java
 
b/sdks/java/extensions/sql/src/test/java/org/apache/beam/dsls/sql/integrationtest/BeamSqlDateFunctionsIntegrationTest.java
new file mode 100644
index 000..bd0d3ba
--- /dev/null
+++ 
b/sdks/java/extensions/sql/src/test/java/org/apache/beam/dsls/sql/integrationtest/BeamSqlDateFunctionsIntegrationTest.java
@@ -0,0 +1,88 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.beam.dsls.sql.integrationtest;
+
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertTrue;
+
+import java.util.Date;
+import java.util.Iterator;
+import org.apache.beam.dsls.sql.BeamSql;
+import org.apache.beam.dsls.sql.schema.BeamSqlRow;
+import org.apache.beam.sdk.testing.PAssert;
+import org.apache.beam.sdk.transforms.SerializableFunction;
+import org.apache.beam.sdk.values.PCollection;
+import org.junit.Test;
+
+/**
+ * Integration test for date functions.
+ */
+public class BeamSqlDateFunctionsIntegrationTest
+extends BeamSqlBuiltinFunctionsIntegrationTestBase {
+  @Test public void testDateTimeFunctions() throws Exception {
+ExpressionChecker checker = new ExpressionChecker()
+.addExpr("EXTRACT(YEAR FROM ts)", 1986L)
+.addExpr("YEAR(ts)", 1986L)
+.addExpr("QUARTER(ts)", 1L)
+.addExpr("MONTH(ts)", 2L)
+.addExpr("WEEK(ts)", 7L)
+.addExpr("DAYOFMONTH(ts)", 15L)
+.addExpr("DAYOFYEAR(ts)", 46L)
+.addExpr("DAYOFWEEK(ts)", 7L)
+.addExpr("HOUR(ts)", 11L)
+.addExpr("MINUTE(ts)", 35L)
+.addExpr("SECOND(ts)", 26L)
+.addExpr("FLOOR(ts TO YEAR)", parseDate("1986-01-01 00:00:00"))
+.addExpr("CEIL(ts TO YEAR)", parseDate("1987-01-01 00:00:00"))
+;
+checker.buildRunAndCheck();
+  }
+
+  @Test public void testDateTimeFunctions_currentTime() throws Exception {
+String sql = "SELECT "
++ "LOCALTIME as l,"
++ "LOCALTIMESTAMP as l1,"
++ "CURRENT_DATE as c1,"
++ "CURRENT_TIME as c2,"
++ "CURRENT_TIMESTAMP as c3"
++ " FROM PCOLLECTION"
+;
+PCollection rows = getTestPCollection().apply(
+BeamSql.simpleQuery(sql));
+PAssert.that(rows).satisfies(new Checker());
+pipeline.run();
+  }
+
+  private static class Checker implements 
SerializableFunction {
+@Override public Void apply(Iterable input) {
+  Iterator iter = input.iterator();
+  assertTrue(iter.hasNext());
+  BeamSqlRow row = iter.next();
+// LOCALTIME
+  Date date = new Date();
+  assertTrue(date.getTime() - 
row.getGregorianCalendar(0).getTime().getTime() < 1000);
+  assertTrue(date.getTime() - row.getDate(1).getTime() < 1000);
+  assertTrue(date.getTime() - row.getDate(2).getTime() < 1000);
+  assertTrue(date.getTime() - 
row.getGregorianCalendar(3).getTime().getTime() < 1000);
+  assertTrue(date.getTime() - row.getDate(4).getTime() < 1000);
+  assertFalse(iter.hasNext());
+  return null;
+}
+  }
+}

http://git-wip-us.apache.org/repos/asf/beam/blob/ba493f85/sdks/java/extensions/sql/src/test/java/org/apache/beam/dsls/sql/integrationtest/BeamSqlLogicalFunctionsIntegrationTest.java
--
diff --git 
a/sdks/java/extensions/sql/src/test/java/org/apache/beam/dsls/sql/integrationtest/BeamSqlLogicalFunctionsIntegrationTest.java
 
b/sdks/java/extensions/sql/src/test/java/org/apache/beam/dsls/sql/integrationtest/BeamSqlLogicalFunctionsIntegrationTest.java
new file mode 100644
index 000..4ed1f86
--- /dev/null
+++ 
b/sdks/java/extensions/sql/src/test/java/org/apache/beam/dsls/sql/integrationtest/BeamSqlLogicalFunctionsIntegrationTest.java
@@ -0,0 +1,43 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the 

[05/59] beam git commit: move dsls/sql to sdks/java/extensions/sql

[takidau]

http://git-wip-us.apache.org/repos/asf/beam/blob/ba493f85/sdks/java/extensions/sql/src/test/java/org/apache/beam/dsls/sql/BeamSqlApiSurfaceTest.java
--
diff --git 
a/sdks/java/extensions/sql/src/test/java/org/apache/beam/dsls/sql/BeamSqlApiSurfaceTest.java
 
b/sdks/java/extensions/sql/src/test/java/org/apache/beam/dsls/sql/BeamSqlApiSurfaceTest.java
new file mode 100644
index 000..922931c
--- /dev/null
+++ 
b/sdks/java/extensions/sql/src/test/java/org/apache/beam/dsls/sql/BeamSqlApiSurfaceTest.java
@@ -0,0 +1,59 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.beam.dsls.sql;
+
+import static org.apache.beam.sdk.util.ApiSurface.containsOnlyPackages;
+import static org.junit.Assert.assertThat;
+
+import com.google.common.collect.ImmutableSet;
+import java.util.Set;
+import org.apache.beam.sdk.util.ApiSurface;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.JUnit4;
+
+/**
+ * Surface test for BeamSql api.
+ */
+@RunWith(JUnit4.class)
+public class BeamSqlApiSurfaceTest {
+  @Test
+  public void testSdkApiSurface() throws Exception {
+
+@SuppressWarnings("unchecked")
+final Set allowed =
+ImmutableSet.of(
+"org.apache.beam",
+"org.joda.time",
+"org.apache.commons.csv");
+
+ApiSurface surface = ApiSurface
+.ofClass(BeamSqlCli.class)
+.includingClass(BeamSql.class)
+.includingClass(BeamSqlEnv.class)
+.includingPackage("org.apache.beam.dsls.sql.schema",
+getClass().getClassLoader())
+.pruningPrefix("java")
+.pruningPattern("org[.]apache[.]beam[.]dsls[.]sql[.].*Test")
+.pruningPattern("org[.]apache[.]beam[.]dsls[.]sql[.].*TestBase");
+
+assertThat(surface, containsOnlyPackages(allowed));
+  }
+
+}

http://git-wip-us.apache.org/repos/asf/beam/blob/ba493f85/sdks/java/extensions/sql/src/test/java/org/apache/beam/dsls/sql/BeamSqlDslAggregationTest.java
--
diff --git 
a/sdks/java/extensions/sql/src/test/java/org/apache/beam/dsls/sql/BeamSqlDslAggregationTest.java
 
b/sdks/java/extensions/sql/src/test/java/org/apache/beam/dsls/sql/BeamSqlDslAggregationTest.java
new file mode 100644
index 000..a142514
--- /dev/null
+++ 
b/sdks/java/extensions/sql/src/test/java/org/apache/beam/dsls/sql/BeamSqlDslAggregationTest.java
@@ -0,0 +1,380 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.dsls.sql;
+
+import java.sql.Types;
+import java.util.Arrays;
+import org.apache.beam.dsls.sql.schema.BeamSqlRow;
+import org.apache.beam.dsls.sql.schema.BeamSqlRowType;
+import org.apache.beam.sdk.testing.PAssert;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.PCollectionTuple;
+import org.apache.beam.sdk.values.TupleTag;
+import org.joda.time.Instant;
+import org.junit.Test;
+
+/**
+ * Tests for GROUP-BY/aggregation, with 
global_window/fix_time_window/sliding_window/session_window
+ * with BOUNDED PCollection.
+ */
+public class BeamSqlDslAggregationTest extends BeamSqlDslBase {
+  /**
+   * GROUP-BY with single aggregation function with bounded PCollection.
+   */
+  @Test
+  public void testAggregationWithoutWindowWithBounded() throws Exception {
+runAggregationWithoutWindow(boundedInput1);
+  }
+
+  /**
+   * GROUP-BY with single 

[57/59] beam git commit: move all implementation classes/packages into impl package

[takidau]

http://git-wip-us.apache.org/repos/asf/beam/blob/febd044a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/arithmetic/BeamSqlMinusExpression.java
--
diff --git 
a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/arithmetic/BeamSqlMinusExpression.java
 
b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/arithmetic/BeamSqlMinusExpression.java
new file mode 100644
index 000..4fc6a4b
--- /dev/null
+++ 
b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/arithmetic/BeamSqlMinusExpression.java
@@ -0,0 +1,36 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package 
org.apache.beam.sdk.extensions.sql.impl.interpreter.operator.arithmetic;
+
+import java.math.BigDecimal;
+import java.util.List;
+import 
org.apache.beam.sdk.extensions.sql.impl.interpreter.operator.BeamSqlExpression;
+
+/**
+ * '-' operator.
+ */
+public class BeamSqlMinusExpression extends BeamSqlArithmeticExpression {
+  public BeamSqlMinusExpression(List operands) {
+super(operands);
+  }
+
+  @Override protected BigDecimal calc(BigDecimal left, BigDecimal right) {
+return left.subtract(right);
+  }
+}

http://git-wip-us.apache.org/repos/asf/beam/blob/febd044a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/arithmetic/BeamSqlModExpression.java
--
diff --git 
a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/arithmetic/BeamSqlModExpression.java
 
b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/arithmetic/BeamSqlModExpression.java
new file mode 100644
index 000..5c55bf4
--- /dev/null
+++ 
b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/arithmetic/BeamSqlModExpression.java
@@ -0,0 +1,36 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package 
org.apache.beam.sdk.extensions.sql.impl.interpreter.operator.arithmetic;
+
+import java.math.BigDecimal;
+import java.util.List;
+import 
org.apache.beam.sdk.extensions.sql.impl.interpreter.operator.BeamSqlExpression;
+
+/**
+ * '%' operator.
+ */
+public class BeamSqlModExpression extends BeamSqlArithmeticExpression {
+  public BeamSqlModExpression(List operands) {
+super(operands, operands.get(1).getOutputType());
+  }
+
+  @Override protected BigDecimal calc(BigDecimal left, BigDecimal right) {
+return BigDecimal.valueOf(left.doubleValue() % right.doubleValue());
+  }
+}

http://git-wip-us.apache.org/repos/asf/beam/blob/febd044a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/arithmetic/BeamSqlMultiplyExpression.java
--
diff --git 
a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/arithmetic/BeamSqlMultiplyExpression.java
 
b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/arithmetic/BeamSqlMultiplyExpression.java
new file mode 100644
index 000..e6cd35d
--- /dev/null
+++ 

[27/59] beam git commit: rename package org.apache.beam.dsls.sql to org.apache.beam.sdk.extensions.sql

[takidau]

http://git-wip-us.apache.org/repos/asf/beam/blob/c1b5482d/sdks/java/extensions/sql/src/test/java/org/apache/beam/dsls/sql/mock/MockedUnboundedTable.java
--
diff --git 
a/sdks/java/extensions/sql/src/test/java/org/apache/beam/dsls/sql/mock/MockedUnboundedTable.java
 
b/sdks/java/extensions/sql/src/test/java/org/apache/beam/dsls/sql/mock/MockedUnboundedTable.java
deleted file mode 100644
index ee6eb22..000
--- 
a/sdks/java/extensions/sql/src/test/java/org/apache/beam/dsls/sql/mock/MockedUnboundedTable.java
+++ /dev/null
@@ -1,114 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package org.apache.beam.dsls.sql.mock;
-
-import static org.apache.beam.dsls.sql.TestUtils.buildBeamSqlRowType;
-import static org.apache.beam.dsls.sql.TestUtils.buildRows;
-
-import java.util.ArrayList;
-import java.util.Arrays;
-import java.util.List;
-import org.apache.beam.dsls.sql.schema.BeamIOType;
-import org.apache.beam.dsls.sql.schema.BeamSqlRow;
-import org.apache.beam.dsls.sql.schema.BeamSqlRowCoder;
-import org.apache.beam.dsls.sql.schema.BeamSqlRowType;
-import org.apache.beam.sdk.Pipeline;
-import org.apache.beam.sdk.testing.TestStream;
-import org.apache.beam.sdk.values.PCollection;
-import org.apache.beam.sdk.values.TimestampedValue;
-import org.apache.calcite.util.Pair;
-import org.joda.time.Duration;
-import org.joda.time.Instant;
-
-/**
- * A mocked unbounded table.
- */
-public class MockedUnboundedTable extends MockedTable {
-  /** rows flow out from this table with the specified watermark instant. */
-  private final List> timestampedRows = new 
ArrayList<>();
-  /** specify the index of column in the row which stands for the event time 
field. */
-  private int timestampField;
-  private MockedUnboundedTable(BeamSqlRowType beamSqlRowType) {
-super(beamSqlRowType);
-  }
-
-  /**
-   * Convenient way to build a mocked unbounded table.
-   *
-   * e.g.
-   *
-   * {@code
-   * MockedUnboundedTable
-   *   .of(Types.BIGINT, "order_id",
-   *   Types.INTEGER, "site_id",
-   *   Types.DOUBLE, "price",
-   *   Types.TIMESTAMP, "order_time")
-   * }
-   */
-  public static MockedUnboundedTable of(final Object... args){
-return new MockedUnboundedTable(buildBeamSqlRowType(args));
-  }
-
-  public MockedUnboundedTable timestampColumnIndex(int idx) {
-this.timestampField = idx;
-return this;
-  }
-
-  /**
-   * Add rows to the builder.
-   *
-   * Sample usage:
-   *
-   * {@code
-   * addRows(
-   *   duration,  -- duration which stands for the corresponding watermark 
instant
-   *   1, 3, "james", -- first row
-   *   2, 5, "bond"   -- second row
-   *   ...
-   * )
-   * }
-   */
-  public MockedUnboundedTable addRows(Duration duration, Object... args) {
-List rows = buildRows(getRowType(), Arrays.asList(args));
-// record the watermark + rows
-this.timestampedRows.add(Pair.of(duration, rows));
-return this;
-  }
-
-  @Override public BeamIOType getSourceType() {
-return BeamIOType.UNBOUNDED;
-  }
-
-  @Override public PCollection buildIOReader(Pipeline pipeline) {
-TestStream.Builder values = TestStream.create(
-new BeamSqlRowCoder(beamSqlRowType));
-
-for (Pair pair : timestampedRows) {
-  values = values.advanceWatermarkTo(new Instant(0).plus(pair.getKey()));
-  for (int i = 0; i < pair.getValue().size(); i++) {
-values = values.addElements(TimestampedValue.of(pair.getValue().get(i),
-new Instant(pair.getValue().get(i).getDate(timestampField;
-  }
-}
-
-return pipeline.begin().apply(
-"MockedUnboundedTable_" + COUNTER.incrementAndGet(),
-values.advanceWatermarkToInfinity());
-  }
-}

http://git-wip-us.apache.org/repos/asf/beam/blob/c1b5482d/sdks/java/extensions/sql/src/test/java/org/apache/beam/dsls/sql/rel/BeamIntersectRelTest.java
--
diff --git 
a/sdks/java/extensions/sql/src/test/java/org/apache/beam/dsls/sql/rel/BeamIntersectRelTest.java
 
b/sdks/java/extensions/sql/src/test/java/org/apache/beam/dsls/sql/rel/BeamIntersectRelTest.java
deleted 

[06/59] beam git commit: move dsls/sql to sdks/java/extensions/sql

[takidau]

http://git-wip-us.apache.org/repos/asf/beam/blob/ba493f85/sdks/java/extensions/sql/src/main/java/org/apache/beam/dsls/sql/schema/BeamSqlUdf.java
--
diff --git 
a/sdks/java/extensions/sql/src/main/java/org/apache/beam/dsls/sql/schema/BeamSqlUdf.java
 
b/sdks/java/extensions/sql/src/main/java/org/apache/beam/dsls/sql/schema/BeamSqlUdf.java
new file mode 100644
index 000..2066353
--- /dev/null
+++ 
b/sdks/java/extensions/sql/src/main/java/org/apache/beam/dsls/sql/schema/BeamSqlUdf.java
@@ -0,0 +1,41 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.dsls.sql.schema;
+
+import java.io.Serializable;
+
+/**
+ * Interface to create a UDF in Beam SQL.
+ *
+ * A static method {@code eval} is required. Here is an example:
+ *
+ * 
+ * public static class MyLeftFunction {
+ *   public String eval(
+ *   Parameter(name = "s") String s,
+ *   Parameter(name = "n", optional = true) Integer n) {
+ * return s.substring(0, n == null ? 1 : n);
+ *   }
+ * }
+ *
+ * The first parameter is named "s" and is mandatory,
+ * and the second parameter is named "n" and is optional.
+ */
+public interface BeamSqlUdf extends Serializable {
+  String UDF_METHOD = "eval";
+}

http://git-wip-us.apache.org/repos/asf/beam/blob/ba493f85/sdks/java/extensions/sql/src/main/java/org/apache/beam/dsls/sql/schema/BeamTableUtils.java
--
diff --git 
a/sdks/java/extensions/sql/src/main/java/org/apache/beam/dsls/sql/schema/BeamTableUtils.java
 
b/sdks/java/extensions/sql/src/main/java/org/apache/beam/dsls/sql/schema/BeamTableUtils.java
new file mode 100644
index 000..4b7e76b
--- /dev/null
+++ 
b/sdks/java/extensions/sql/src/main/java/org/apache/beam/dsls/sql/schema/BeamTableUtils.java
@@ -0,0 +1,122 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.beam.dsls.sql.schema;
+
+import java.io.IOException;
+import java.io.StringReader;
+import java.io.StringWriter;
+import java.math.BigDecimal;
+import org.apache.beam.dsls.sql.utils.CalciteUtils;
+import org.apache.calcite.sql.type.SqlTypeName;
+import org.apache.calcite.util.NlsString;
+import org.apache.commons.csv.CSVFormat;
+import org.apache.commons.csv.CSVParser;
+import org.apache.commons.csv.CSVPrinter;
+import org.apache.commons.csv.CSVRecord;
+
+/**
+ * Utility methods for working with {@code BeamTable}.
+ */
+public final class BeamTableUtils {
+  public static BeamSqlRow csvLine2BeamSqlRow(
+  CSVFormat csvFormat,
+  String line,
+  BeamSqlRowType beamSqlRowType) {
+BeamSqlRow row = new BeamSqlRow(beamSqlRowType);
+try (StringReader reader = new StringReader(line)) {
+  CSVParser parser = csvFormat.parse(reader);
+  CSVRecord rawRecord = parser.getRecords().get(0);
+
+  if (rawRecord.size() != beamSqlRowType.size()) {
+throw new IllegalArgumentException(String.format(
+"Expect %d fields, but actually %d",
+beamSqlRowType.size(), rawRecord.size()
+));
+  } else {
+for (int idx = 0; idx < beamSqlRowType.size(); idx++) {
+  String raw = rawRecord.get(idx);
+  addFieldWithAutoTypeCasting(row, idx, raw);
+}
+  }
+} catch (IOException e) {
+  throw new IllegalArgumentException("decodeRecord failed!", e);
+}
+return row;
+  }
+
+  public static String 

[19/59] beam git commit: move dsls/sql to sdks/java/extensions/sql

[takidau]

http://git-wip-us.apache.org/repos/asf/beam/blob/ba493f85/dsls/sql/src/main/java/org/apache/beam/dsls/sql/interpreter/operator/date/BeamSqlExtractExpression.java
--
diff --git 
a/dsls/sql/src/main/java/org/apache/beam/dsls/sql/interpreter/operator/date/BeamSqlExtractExpression.java
 
b/dsls/sql/src/main/java/org/apache/beam/dsls/sql/interpreter/operator/date/BeamSqlExtractExpression.java
deleted file mode 100644
index d41a249..000
--- 
a/dsls/sql/src/main/java/org/apache/beam/dsls/sql/interpreter/operator/date/BeamSqlExtractExpression.java
+++ /dev/null
@@ -1,101 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package org.apache.beam.dsls.sql.interpreter.operator.date;
-
-import java.util.Calendar;
-import java.util.Date;
-import java.util.HashMap;
-import java.util.List;
-import java.util.Map;
-import org.apache.beam.dsls.sql.interpreter.operator.BeamSqlExpression;
-import org.apache.beam.dsls.sql.interpreter.operator.BeamSqlPrimitive;
-import org.apache.beam.dsls.sql.schema.BeamSqlRow;
-import org.apache.calcite.avatica.util.DateTimeUtils;
-import org.apache.calcite.avatica.util.TimeUnitRange;
-import org.apache.calcite.sql.type.SqlTypeName;
-
-/**
- * {@code BeamSqlExpression} for EXTRACT.
- *
- * The following date functions also implicitly converted to {@code 
EXTRACT}:
- * 
- *   YEAR(date) = EXTRACT(YEAR FROM date)
- *   MONTH(date) = EXTRACT(MONTH FROM date)
- *   DAY(date) = EXTRACT(DAY FROM date)
- *   QUARTER(date) = EXTRACT(QUARTER FROM date)
- *   WEEK(date) = EXTRACT(WEEK FROM date)
- *   DAYOFYEAR(date) = EXTRACT(DOY FROM date)
- *   DAYOFMONTH(date) = EXTRACT(DAY FROM date)
- *   DAYOFWEEK(date) = EXTRACT(DOW FROM date)
- * 
- */
-public class BeamSqlExtractExpression extends BeamSqlExpression {
-  private static final Map typeMapping = new 
HashMap<>();
-  static {
-typeMapping.put(TimeUnitRange.DOW, Calendar.DAY_OF_WEEK);
-typeMapping.put(TimeUnitRange.DOY, Calendar.DAY_OF_YEAR);
-typeMapping.put(TimeUnitRange.WEEK, Calendar.WEEK_OF_YEAR);
-  }
-
-  public BeamSqlExtractExpression(List operands) {
-super(operands, SqlTypeName.BIGINT);
-  }
-  @Override public boolean accept() {
-return operands.size() == 2
-&& opType(1) == SqlTypeName.BIGINT;
-  }
-
-  @Override public BeamSqlPrimitive evaluate(BeamSqlRow inputRow) {
-Long time = opValueEvaluated(1, inputRow);
-
-TimeUnitRange unit = ((BeamSqlPrimitive) op(0)).getValue();
-
-switch (unit) {
-  case YEAR:
-  case MONTH:
-  case DAY:
-Long timeByDay = time / 1000 / 3600 / 24;
-Long extracted = DateTimeUtils.unixDateExtract(
-unit,
-timeByDay
-);
-return BeamSqlPrimitive.of(outputType, extracted);
-
-  case DOY:
-  case DOW:
-  case WEEK:
-Calendar calendar = Calendar.getInstance();
-calendar.setTime(new Date(time));
-return BeamSqlPrimitive.of(outputType, (long) 
calendar.get(typeMapping.get(unit)));
-
-  case QUARTER:
-calendar = Calendar.getInstance();
-calendar.setTime(new Date(time));
-long ret = calendar.get(Calendar.MONTH) / 3;
-if (ret * 3 < calendar.get(Calendar.MONTH)) {
-  ret += 1;
-}
-return BeamSqlPrimitive.of(outputType, ret);
-
-  default:
-throw new UnsupportedOperationException(
-"Extract for time unit: " + unit + " not supported!");
-}
-  }
-}

http://git-wip-us.apache.org/repos/asf/beam/blob/ba493f85/dsls/sql/src/main/java/org/apache/beam/dsls/sql/interpreter/operator/date/package-info.java
--
diff --git 
a/dsls/sql/src/main/java/org/apache/beam/dsls/sql/interpreter/operator/date/package-info.java
 
b/dsls/sql/src/main/java/org/apache/beam/dsls/sql/interpreter/operator/date/package-info.java
deleted file mode 100644
index d3cc98f..000
--- 
a/dsls/sql/src/main/java/org/apache/beam/dsls/sql/interpreter/operator/date/package-info.java
+++ /dev/null
@@ -1,22 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor 

[43/59] beam git commit: move all implementation classes/packages into impl package

[takidau]

http://git-wip-us.apache.org/repos/asf/beam/blob/febd044a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/schema/transform/BeamTransformBaseTest.java
--
diff --git 
a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/schema/transform/BeamTransformBaseTest.java
 
b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/schema/transform/BeamTransformBaseTest.java
index b2aa6c4..af7ec23 100644
--- 
a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/schema/transform/BeamTransformBaseTest.java
+++ 
b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/schema/transform/BeamTransformBaseTest.java
@@ -22,10 +22,10 @@ import java.text.ParseException;
 import java.text.SimpleDateFormat;
 import java.util.Arrays;
 import java.util.List;
-import org.apache.beam.sdk.extensions.sql.planner.BeamQueryPlanner;
+import org.apache.beam.sdk.extensions.sql.impl.planner.BeamQueryPlanner;
+import org.apache.beam.sdk.extensions.sql.impl.utils.CalciteUtils;
 import org.apache.beam.sdk.extensions.sql.schema.BeamSqlRow;
 import org.apache.beam.sdk.extensions.sql.schema.BeamSqlRowType;
-import org.apache.beam.sdk.extensions.sql.utils.CalciteUtils;
 import org.apache.beam.sdk.values.KV;
 import org.apache.calcite.rel.type.RelDataTypeFactory.FieldInfoBuilder;
 import org.apache.calcite.sql.type.SqlTypeName;

[58/59] beam git commit: move all implementation classes/packages into impl package

[takidau]

move all implementation classes/packages into impl package


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/febd044a
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/febd044a
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/febd044a

Branch: refs/heads/DSL_SQL
Commit: febd044ae306a28fa3797a1663e54c1d7fbe43ce
Parents: c1b5482
Author: James Xu 
Authored: Mon Jul 31 17:11:53 2017 +0800
Committer: James Xu 
Committed: Mon Jul 31 17:11:53 2017 +0800

--
 .../apache/beam/sdk/extensions/sql/BeamSql.java |   2 +-
 .../beam/sdk/extensions/sql/BeamSqlCli.java |   2 +-
 .../beam/sdk/extensions/sql/BeamSqlEnv.java |   4 +-
 .../interpreter/BeamSqlExpressionExecutor.java  |  43 ++
 .../sql/impl/interpreter/BeamSqlFnExecutor.java | 442 +++
 .../operator/BeamSqlCaseExpression.java |  63 +++
 .../operator/BeamSqlCastExpression.java | 131 ++
 .../interpreter/operator/BeamSqlExpression.java |  78 
 .../operator/BeamSqlInputRefExpression.java |  43 ++
 .../interpreter/operator/BeamSqlPrimitive.java  | 152 +++
 .../operator/BeamSqlReinterpretExpression.java  |  54 +++
 .../operator/BeamSqlUdfExpression.java  |  86 
 .../operator/BeamSqlWindowEndExpression.java|  42 ++
 .../operator/BeamSqlWindowExpression.java   |  50 +++
 .../operator/BeamSqlWindowStartExpression.java  |  43 ++
 .../arithmetic/BeamSqlArithmeticExpression.java | 122 +
 .../arithmetic/BeamSqlDivideExpression.java |  37 ++
 .../arithmetic/BeamSqlMinusExpression.java  |  36 ++
 .../arithmetic/BeamSqlModExpression.java|  36 ++
 .../arithmetic/BeamSqlMultiplyExpression.java   |  36 ++
 .../arithmetic/BeamSqlPlusExpression.java   |  36 ++
 .../operator/arithmetic/package-info.java   |  22 +
 .../comparison/BeamSqlCompareExpression.java|  96 
 .../comparison/BeamSqlEqualsExpression.java |  49 ++
 .../BeamSqlGreaterThanExpression.java   |  49 ++
 .../BeamSqlGreaterThanOrEqualsExpression.java   |  49 ++
 .../comparison/BeamSqlIsNotNullExpression.java  |  53 +++
 .../comparison/BeamSqlIsNullExpression.java |  53 +++
 .../comparison/BeamSqlLessThanExpression.java   |  49 ++
 .../BeamSqlLessThanOrEqualsExpression.java  |  49 ++
 .../comparison/BeamSqlNotEqualsExpression.java  |  49 ++
 .../operator/comparison/package-info.java   |  22 +
 .../date/BeamSqlCurrentDateExpression.java  |  44 ++
 .../date/BeamSqlCurrentTimeExpression.java  |  52 +++
 .../date/BeamSqlCurrentTimestampExpression.java |  48 ++
 .../date/BeamSqlDateCeilExpression.java |  54 +++
 .../date/BeamSqlDateFloorExpression.java|  54 +++
 .../operator/date/BeamSqlExtractExpression.java | 101 +
 .../interpreter/operator/date/package-info.java |  22 +
 .../operator/logical/BeamSqlAndExpression.java  |  47 ++
 .../logical/BeamSqlLogicalExpression.java   |  46 ++
 .../operator/logical/BeamSqlNotExpression.java  |  53 +++
 .../operator/logical/BeamSqlOrExpression.java   |  47 ++
 .../operator/logical/package-info.java  |  22 +
 .../operator/math/BeamSqlAbsExpression.java |  74 
 .../operator/math/BeamSqlAcosExpression.java|  40 ++
 .../operator/math/BeamSqlAsinExpression.java|  40 ++
 .../operator/math/BeamSqlAtan2Expression.java   |  42 ++
 .../operator/math/BeamSqlAtanExpression.java|  40 ++
 .../operator/math/BeamSqlCeilExpression.java|  45 ++
 .../operator/math/BeamSqlCosExpression.java |  40 ++
 .../operator/math/BeamSqlCotExpression.java |  40 ++
 .../operator/math/BeamSqlDegreesExpression.java |  40 ++
 .../operator/math/BeamSqlExpExpression.java |  40 ++
 .../operator/math/BeamSqlFloorExpression.java   |  45 ++
 .../operator/math/BeamSqlLnExpression.java  |  40 ++
 .../operator/math/BeamSqlLogExpression.java |  40 ++
 .../math/BeamSqlMathBinaryExpression.java   |  63 +++
 .../math/BeamSqlMathUnaryExpression.java|  58 +++
 .../operator/math/BeamSqlPiExpression.java  |  42 ++
 .../operator/math/BeamSqlPowerExpression.java   |  44 ++
 .../operator/math/BeamSqlRadiansExpression.java |  40 ++
 .../operator/math/BeamSqlRandExpression.java|  54 +++
 .../math/BeamSqlRandIntegerExpression.java  |  58 +++
 .../operator/math/BeamSqlRoundExpression.java   | 107 +
 .../operator/math/BeamSqlSignExpression.java|  72 +++
 .../operator/math/BeamSqlSinExpression.java |  40 ++
 .../operator/math/BeamSqlTanExpression.java |  40 ++
 .../math/BeamSqlTruncateExpression.java |  75 
 .../interpreter/operator/math/package-info.java |  22 +
 .../impl/interpreter/operator/package-info.java |  22 +
 .../string/BeamSqlCharLengthExpression.java |  39 ++
 .../string/BeamSqlConcatExpression.java |  62 +++
 .../string/BeamSqlInitCapExpression.java|  55 +++
 

[52/59] beam git commit: move all implementation classes/packages into impl package

[takidau]

http://git-wip-us.apache.org/repos/asf/beam/blob/febd044a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/interpreter/operator/comparison/BeamSqlGreaterThanOrEqualsExpression.java
--
diff --git 
a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/interpreter/operator/comparison/BeamSqlGreaterThanOrEqualsExpression.java
 
b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/interpreter/operator/comparison/BeamSqlGreaterThanOrEqualsExpression.java
deleted file mode 100644
index ae22054..000
--- 
a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/interpreter/operator/comparison/BeamSqlGreaterThanOrEqualsExpression.java
+++ /dev/null
@@ -1,49 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.sdk.extensions.sql.interpreter.operator.comparison;
-
-import java.util.List;
-import 
org.apache.beam.sdk.extensions.sql.interpreter.operator.BeamSqlExpression;
-
-/**
- * {@code BeamSqlExpression} for {@code >=} operation.
- */
-public class BeamSqlGreaterThanOrEqualsExpression extends 
BeamSqlCompareExpression {
-
-  public BeamSqlGreaterThanOrEqualsExpression(List 
operands) {
-super(operands);
-  }
-
-  @Override
-  public Boolean compare(CharSequence leftValue, CharSequence rightValue) {
-return String.valueOf(leftValue).compareTo(String.valueOf(rightValue)) >= 
0;
-  }
-
-  @Override
-  public Boolean compare(Boolean leftValue, Boolean rightValue) {
-throw new IllegalArgumentException(">= is not supported for Boolean.");
-  }
-
-  @Override
-  public Boolean compare(Number leftValue, Number rightValue) {
-return (leftValue == null && rightValue == null)
-|| (leftValue != null && rightValue != null
-  && leftValue.floatValue() >= (rightValue).floatValue());
-  }
-
-}

http://git-wip-us.apache.org/repos/asf/beam/blob/febd044a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/interpreter/operator/comparison/BeamSqlIsNotNullExpression.java
--
diff --git 
a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/interpreter/operator/comparison/BeamSqlIsNotNullExpression.java
 
b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/interpreter/operator/comparison/BeamSqlIsNotNullExpression.java
deleted file mode 100644
index 78660cb..000
--- 
a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/interpreter/operator/comparison/BeamSqlIsNotNullExpression.java
+++ /dev/null
@@ -1,53 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.sdk.extensions.sql.interpreter.operator.comparison;
-
-import java.util.Arrays;
-import java.util.List;
-import 
org.apache.beam.sdk.extensions.sql.interpreter.operator.BeamSqlExpression;
-import 
org.apache.beam.sdk.extensions.sql.interpreter.operator.BeamSqlPrimitive;
-import org.apache.beam.sdk.extensions.sql.schema.BeamSqlRow;
-import org.apache.calcite.sql.type.SqlTypeName;
-
-/**
- * {@code BeamSqlExpression} for 'IS NOT NULL' operation.
- */
-public class BeamSqlIsNotNullExpression extends BeamSqlExpression {
-
-  private BeamSqlIsNotNullExpression(List operands, 
SqlTypeName outputType) {
-super(operands, outputType);
-  }
-
-  public 

[45/59] beam git commit: move all implementation classes/packages into impl package

[takidau]

http://git-wip-us.apache.org/repos/asf/beam/blob/febd044a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/interpreter/operator/BeamSqlCompareExpressionTest.java
--
diff --git 
a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/interpreter/operator/BeamSqlCompareExpressionTest.java
 
b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/interpreter/operator/BeamSqlCompareExpressionTest.java
deleted file mode 100644
index ed77ffb..000
--- 
a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/interpreter/operator/BeamSqlCompareExpressionTest.java
+++ /dev/null
@@ -1,115 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.sdk.extensions.sql.interpreter.operator;
-
-import java.util.Arrays;
-import 
org.apache.beam.sdk.extensions.sql.interpreter.BeamSqlFnExecutorTestBase;
-import 
org.apache.beam.sdk.extensions.sql.interpreter.operator.comparison.BeamSqlCompareExpression;
-import 
org.apache.beam.sdk.extensions.sql.interpreter.operator.comparison.BeamSqlEqualsExpression;
-import 
org.apache.beam.sdk.extensions.sql.interpreter.operator.comparison.BeamSqlGreaterThanExpression;
-import 
org.apache.beam.sdk.extensions.sql.interpreter.operator.comparison.BeamSqlGreaterThanOrEqualsExpression;
-import 
org.apache.beam.sdk.extensions.sql.interpreter.operator.comparison.BeamSqlLessThanExpression;
-import 
org.apache.beam.sdk.extensions.sql.interpreter.operator.comparison.BeamSqlLessThanOrEqualsExpression;
-import 
org.apache.beam.sdk.extensions.sql.interpreter.operator.comparison.BeamSqlNotEqualsExpression;
-import org.apache.calcite.sql.type.SqlTypeName;
-import org.junit.Assert;
-import org.junit.Test;
-
-/**
- * Test cases for the collections of {@link BeamSqlCompareExpression}.
- */
-public class BeamSqlCompareExpressionTest extends BeamSqlFnExecutorTestBase {
-
-  @Test
-  public void testEqual() {
-BeamSqlEqualsExpression exp1 = new BeamSqlEqualsExpression(
-Arrays.asList(new BeamSqlInputRefExpression(SqlTypeName.BIGINT, 0),
-BeamSqlPrimitive.of(SqlTypeName.BIGINT, 100L)));
-Assert.assertEquals(false, exp1.evaluate(record).getValue());
-
-BeamSqlEqualsExpression exp2 = new BeamSqlEqualsExpression(
-Arrays.asList(new BeamSqlInputRefExpression(SqlTypeName.BIGINT, 0),
-BeamSqlPrimitive.of(SqlTypeName.BIGINT, 1234567L)));
-Assert.assertEquals(true, exp2.evaluate(record).getValue());
-  }
-
-  @Test
-  public void testLargerThan(){
-BeamSqlGreaterThanExpression exp1 = new BeamSqlGreaterThanExpression(
-Arrays.asList(new BeamSqlInputRefExpression(SqlTypeName.BIGINT, 0),
-BeamSqlPrimitive.of(SqlTypeName.BIGINT, 1234567L)));
-Assert.assertEquals(false, exp1.evaluate(record).getValue());
-
-BeamSqlGreaterThanExpression exp2 = new BeamSqlGreaterThanExpression(
-Arrays.asList(new BeamSqlInputRefExpression(SqlTypeName.BIGINT, 0),
-BeamSqlPrimitive.of(SqlTypeName.BIGINT, 1234566L)));
-Assert.assertEquals(true, exp2.evaluate(record).getValue());
-  }
-
-  @Test
-  public void testLargerThanEqual(){
-BeamSqlGreaterThanOrEqualsExpression exp1 = new 
BeamSqlGreaterThanOrEqualsExpression(
-Arrays.asList(new BeamSqlInputRefExpression(SqlTypeName.BIGINT, 0),
-BeamSqlPrimitive.of(SqlTypeName.BIGINT, 1234567L)));
-Assert.assertEquals(true, exp1.evaluate(record).getValue());
-
-BeamSqlGreaterThanOrEqualsExpression exp2 = new 
BeamSqlGreaterThanOrEqualsExpression(
-Arrays.asList(new BeamSqlInputRefExpression(SqlTypeName.BIGINT, 0),
-BeamSqlPrimitive.of(SqlTypeName.BIGINT, 1234568L)));
-Assert.assertEquals(false, exp2.evaluate(record).getValue());
-  }
-
-  @Test
-  public void testLessThan(){
-BeamSqlLessThanExpression exp1 = new BeamSqlLessThanExpression(
-Arrays.asList(new BeamSqlInputRefExpression(SqlTypeName.INTEGER, 1),
-BeamSqlPrimitive.of(SqlTypeName.INTEGER, 1)));
-Assert.assertEquals(true, exp1.evaluate(record).getValue());
-
-BeamSqlLessThanExpression exp2 = new 

[26/59] beam git commit: rename package org.apache.beam.dsls.sql to org.apache.beam.sdk.extensions.sql

[takidau]

http://git-wip-us.apache.org/repos/asf/beam/blob/c1b5482d/sdks/java/extensions/sql/src/test/java/org/apache/beam/dsls/sql/schema/transform/BeamAggregationTransformTest.java
--
diff --git 
a/sdks/java/extensions/sql/src/test/java/org/apache/beam/dsls/sql/schema/transform/BeamAggregationTransformTest.java
 
b/sdks/java/extensions/sql/src/test/java/org/apache/beam/dsls/sql/schema/transform/BeamAggregationTransformTest.java
deleted file mode 100644
index 5d5d4fc..000
--- 
a/sdks/java/extensions/sql/src/test/java/org/apache/beam/dsls/sql/schema/transform/BeamAggregationTransformTest.java
+++ /dev/null
@@ -1,453 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- * 
- * http://www.apache.org/licenses/LICENSE-2.0
- * 
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.dsls.sql.schema.transform;
-
-import java.text.ParseException;
-import java.util.ArrayList;
-import java.util.Arrays;
-import java.util.List;
-import org.apache.beam.dsls.sql.planner.BeamQueryPlanner;
-import org.apache.beam.dsls.sql.schema.BeamSqlRow;
-import org.apache.beam.dsls.sql.schema.BeamSqlRowCoder;
-import org.apache.beam.dsls.sql.schema.BeamSqlRowType;
-import org.apache.beam.dsls.sql.transform.BeamAggregationTransforms;
-import org.apache.beam.dsls.sql.utils.CalciteUtils;
-import org.apache.beam.sdk.coders.IterableCoder;
-import org.apache.beam.sdk.coders.KvCoder;
-import org.apache.beam.sdk.testing.PAssert;
-import org.apache.beam.sdk.testing.TestPipeline;
-import org.apache.beam.sdk.transforms.Combine;
-import org.apache.beam.sdk.transforms.Create;
-import org.apache.beam.sdk.transforms.GroupByKey;
-import org.apache.beam.sdk.transforms.ParDo;
-import org.apache.beam.sdk.transforms.WithKeys;
-import org.apache.beam.sdk.values.KV;
-import org.apache.beam.sdk.values.PCollection;
-import org.apache.calcite.rel.core.AggregateCall;
-import org.apache.calcite.rel.type.RelDataTypeFactory.FieldInfoBuilder;
-import org.apache.calcite.rel.type.RelDataTypeSystem;
-import org.apache.calcite.sql.SqlKind;
-import org.apache.calcite.sql.fun.SqlAvgAggFunction;
-import org.apache.calcite.sql.fun.SqlCountAggFunction;
-import org.apache.calcite.sql.fun.SqlMinMaxAggFunction;
-import org.apache.calcite.sql.fun.SqlSumAggFunction;
-import org.apache.calcite.sql.type.BasicSqlType;
-import org.apache.calcite.sql.type.SqlTypeName;
-import org.apache.calcite.util.ImmutableBitSet;
-import org.junit.Rule;
-import org.junit.Test;
-
-/**
- * Unit tests for {@link BeamAggregationTransforms}.
- *
- */
-public class BeamAggregationTransformTest extends BeamTransformBaseTest{
-
-  @Rule
-  public TestPipeline p = TestPipeline.create();
-
-  private List aggCalls;
-
-  private BeamSqlRowType keyType;
-  private BeamSqlRowType aggPartType;
-  private BeamSqlRowType outputType;
-
-  private BeamSqlRowCoder inRecordCoder;
-  private BeamSqlRowCoder keyCoder;
-  private BeamSqlRowCoder aggCoder;
-  private BeamSqlRowCoder outRecordCoder;
-
-  /**
-   * This step equals to below query.
-   * 
-   * SELECT `f_int`
-   * , COUNT(*) AS `size`
-   * , SUM(`f_long`) AS `sum1`, AVG(`f_long`) AS `avg1`
-   * , MAX(`f_long`) AS `max1`, MIN(`f_long`) AS `min1`
-   * , SUM(`f_short`) AS `sum2`, AVG(`f_short`) AS `avg2`
-   * , MAX(`f_short`) AS `max2`, MIN(`f_short`) AS `min2`
-   * , SUM(`f_byte`) AS `sum3`, AVG(`f_byte`) AS `avg3`
-   * , MAX(`f_byte`) AS `max3`, MIN(`f_byte`) AS `min3`
-   * , SUM(`f_float`) AS `sum4`, AVG(`f_float`) AS `avg4`
-   * , MAX(`f_float`) AS `max4`, MIN(`f_float`) AS `min4`
-   * , SUM(`f_double`) AS `sum5`, AVG(`f_double`) AS `avg5`
-   * , MAX(`f_double`) AS `max5`, MIN(`f_double`) AS `min5`
-   * , MAX(`f_timestamp`) AS `max7`, MIN(`f_timestamp`) AS `min7`
-   * ,SUM(`f_int2`) AS `sum8`, AVG(`f_int2`) AS `avg8`
-   * , MAX(`f_int2`) AS `max8`, MIN(`f_int2`) AS `min8`
-   * FROM TABLE_NAME
-   * GROUP BY `f_int`
-   * 
-   * @throws ParseException
-   */
-  @Test
-  public void testCountPerElementBasic() throws ParseException {
-setupEnvironment();
-
-PCollection input = p.apply(Create.of(inputRows));
-
-//1. extract fields in group-by key part
-PCollection> exGroupByStream = 
input.apply("exGroupBy",
-WithKeys
-.of(new 

[37/59] beam git commit: rename package org.apache.beam.dsls.sql to org.apache.beam.sdk.extensions.sql

[takidau]

http://git-wip-us.apache.org/repos/asf/beam/blob/c1b5482d/sdks/java/extensions/sql/src/main/java/org/apache/beam/dsls/sql/schema/BeamSqlUdaf.java
--
diff --git 
a/sdks/java/extensions/sql/src/main/java/org/apache/beam/dsls/sql/schema/BeamSqlUdaf.java
 
b/sdks/java/extensions/sql/src/main/java/org/apache/beam/dsls/sql/schema/BeamSqlUdaf.java
deleted file mode 100644
index 9582ffa..000
--- 
a/sdks/java/extensions/sql/src/main/java/org/apache/beam/dsls/sql/schema/BeamSqlUdaf.java
+++ /dev/null
@@ -1,72 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.dsls.sql.schema;
-
-import java.io.Serializable;
-import java.lang.reflect.ParameterizedType;
-import org.apache.beam.sdk.coders.CannotProvideCoderException;
-import org.apache.beam.sdk.coders.Coder;
-import org.apache.beam.sdk.coders.CoderRegistry;
-import org.apache.beam.sdk.transforms.Combine.CombineFn;
-
-/**
- * abstract class of aggregation functions in Beam SQL.
- *
- * There're several constrains for a UDAF:
- * 1. A constructor with an empty argument list is required;
- * 2. The type of {@code InputT} and {@code OutputT} can only be 
Interger/Long/Short/Byte/Double
- * /Float/Date/BigDecimal, mapping as SQL type 
INTEGER/BIGINT/SMALLINT/TINYINE/DOUBLE/FLOAT
- * /TIMESTAMP/DECIMAL;
- * 3. Keep intermediate data in {@code AccumT}, and do not rely on elements in 
class;
- */
-public abstract class BeamSqlUdaf implements 
Serializable {
-  public BeamSqlUdaf(){}
-
-  /**
-   * create an initial aggregation object, equals to {@link 
CombineFn#createAccumulator()}.
-   */
-  public abstract AccumT init();
-
-  /**
-   * add an input value, equals to {@link CombineFn#addInput(Object, Object)}.
-   */
-  public abstract AccumT add(AccumT accumulator, InputT input);
-
-  /**
-   * merge aggregation objects from parallel tasks, equals to
-   *  {@link CombineFn#mergeAccumulators(Iterable)}.
-   */
-  public abstract AccumT merge(Iterable accumulators);
-
-  /**
-   * extract output value from aggregation object, equals to
-   * {@link CombineFn#extractOutput(Object)}.
-   */
-  public abstract OutputT result(AccumT accumulator);
-
-  /**
-   * get the coder for AccumT which stores the intermediate result.
-   * By default it's fetched from {@link CoderRegistry}.
-   */
-  public Coder getAccumulatorCoder(CoderRegistry registry)
-  throws CannotProvideCoderException {
-return registry.getCoder(
-(Class) ((ParameterizedType) getClass()
-.getGenericSuperclass()).getActualTypeArguments()[1]);
-  }
-}

http://git-wip-us.apache.org/repos/asf/beam/blob/c1b5482d/sdks/java/extensions/sql/src/main/java/org/apache/beam/dsls/sql/schema/BeamSqlUdf.java
--
diff --git 
a/sdks/java/extensions/sql/src/main/java/org/apache/beam/dsls/sql/schema/BeamSqlUdf.java
 
b/sdks/java/extensions/sql/src/main/java/org/apache/beam/dsls/sql/schema/BeamSqlUdf.java
deleted file mode 100644
index 2066353..000
--- 
a/sdks/java/extensions/sql/src/main/java/org/apache/beam/dsls/sql/schema/BeamSqlUdf.java
+++ /dev/null
@@ -1,41 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.dsls.sql.schema;
-
-import java.io.Serializable;
-
-/**
- * Interface to create a UDF in Beam SQL.
- *
- * A static method {@code eval} is required. Here is 

[29/59] beam git commit: rename package org.apache.beam.dsls.sql to org.apache.beam.sdk.extensions.sql

[takidau]

http://git-wip-us.apache.org/repos/asf/beam/blob/c1b5482d/sdks/java/extensions/sql/src/test/java/org/apache/beam/dsls/sql/integrationtest/BeamSqlMathFunctionsIntegrationTest.java
--
diff --git 
a/sdks/java/extensions/sql/src/test/java/org/apache/beam/dsls/sql/integrationtest/BeamSqlMathFunctionsIntegrationTest.java
 
b/sdks/java/extensions/sql/src/test/java/org/apache/beam/dsls/sql/integrationtest/BeamSqlMathFunctionsIntegrationTest.java
deleted file mode 100644
index 9f7d917..000
--- 
a/sdks/java/extensions/sql/src/test/java/org/apache/beam/dsls/sql/integrationtest/BeamSqlMathFunctionsIntegrationTest.java
+++ /dev/null
@@ -1,351 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.dsls.sql.integrationtest;
-
-import java.math.BigDecimal;
-import java.util.Random;
-import org.apache.calcite.runtime.SqlFunctions;
-import org.junit.Test;
-
-/**
- * Integration test for built-in MATH functions.
- */
-public class BeamSqlMathFunctionsIntegrationTest
-extends BeamSqlBuiltinFunctionsIntegrationTestBase {
-  private static final int INTEGER_VALUE = 1;
-  private static final long LONG_VALUE = 1L;
-  private static final short SHORT_VALUE = 1;
-  private static final byte BYTE_VALUE = 1;
-  private static final double DOUBLE_VALUE = 1.0;
-  private static final float FLOAT_VALUE = 1.0f;
-  private static final BigDecimal DECIMAL_VALUE = new BigDecimal(1);
-
-  @Test
-  public void testAbs() throws Exception{
-ExpressionChecker checker = new ExpressionChecker()
-.addExpr("ABS(c_integer)", Math.abs(INTEGER_VALUE))
-.addExpr("ABS(c_bigint)", Math.abs(LONG_VALUE))
-.addExpr("ABS(c_smallint)", (short) Math.abs(SHORT_VALUE))
-.addExpr("ABS(c_tinyint)", (byte) Math.abs(BYTE_VALUE))
-.addExpr("ABS(c_double)", Math.abs(DOUBLE_VALUE))
-.addExpr("ABS(c_float)", Math.abs(FLOAT_VALUE))
-.addExpr("ABS(c_decimal)", new 
BigDecimal(Math.abs(DECIMAL_VALUE.doubleValue(
-;
-
-checker.buildRunAndCheck();
-  }
-
-  @Test
-  public void testSqrt() throws Exception{
-ExpressionChecker checker = new ExpressionChecker()
-.addExpr("SQRT(c_integer)", Math.sqrt(INTEGER_VALUE))
-.addExpr("SQRT(c_bigint)", Math.sqrt(LONG_VALUE))
-.addExpr("SQRT(c_smallint)", Math.sqrt(SHORT_VALUE))
-.addExpr("SQRT(c_tinyint)", Math.sqrt(BYTE_VALUE))
-.addExpr("SQRT(c_double)", Math.sqrt(DOUBLE_VALUE))
-.addExpr("SQRT(c_float)", Math.sqrt(FLOAT_VALUE))
-.addExpr("SQRT(c_decimal)", Math.sqrt(DECIMAL_VALUE.doubleValue()))
-;
-
-checker.buildRunAndCheck();
-  }
-
-  @Test
-  public void testRound() throws Exception{
-ExpressionChecker checker = new ExpressionChecker()
-.addExpr("ROUND(c_integer, 0)", SqlFunctions.sround(INTEGER_VALUE, 0))
-.addExpr("ROUND(c_bigint, 0)", SqlFunctions.sround(LONG_VALUE, 0))
-.addExpr("ROUND(c_smallint, 0)", (short) 
SqlFunctions.sround(SHORT_VALUE, 0))
-.addExpr("ROUND(c_tinyint, 0)", (byte) SqlFunctions.sround(BYTE_VALUE, 
0))
-.addExpr("ROUND(c_double, 0)", SqlFunctions.sround(DOUBLE_VALUE, 0))
-.addExpr("ROUND(c_float, 0)", (float) SqlFunctions.sround(FLOAT_VALUE, 
0))
-.addExpr("ROUND(c_decimal, 0)",
-new BigDecimal(SqlFunctions.sround(DECIMAL_VALUE.doubleValue(), 
0)))
-;
-
-checker.buildRunAndCheck();
-  }
-
-  @Test
-  public void testLn() throws Exception{
-ExpressionChecker checker = new ExpressionChecker()
-.addExpr("LN(c_integer)", Math.log(INTEGER_VALUE))
-.addExpr("LN(c_bigint)", Math.log(LONG_VALUE))
-.addExpr("LN(c_smallint)", Math.log(SHORT_VALUE))
-.addExpr("LN(c_tinyint)", Math.log(BYTE_VALUE))
-.addExpr("LN(c_double)", Math.log(DOUBLE_VALUE))
-.addExpr("LN(c_float)", Math.log(FLOAT_VALUE))
-.addExpr("LN(c_decimal)", Math.log(DECIMAL_VALUE.doubleValue()))
-;
-
-checker.buildRunAndCheck();
-  }
-
-  @Test
-  public void testLog10() throws Exception{
-ExpressionChecker checker = new ExpressionChecker()
-.addExpr("LOG10(c_integer)", 

[12/59] beam git commit: move dsls/sql to sdks/java/extensions/sql

[takidau]

http://git-wip-us.apache.org/repos/asf/beam/blob/ba493f85/dsls/sql/src/test/java/org/apache/beam/dsls/sql/interpreter/operator/string/BeamSqlUpperExpressionTest.java
--
diff --git 
a/dsls/sql/src/test/java/org/apache/beam/dsls/sql/interpreter/operator/string/BeamSqlUpperExpressionTest.java
 
b/dsls/sql/src/test/java/org/apache/beam/dsls/sql/interpreter/operator/string/BeamSqlUpperExpressionTest.java
deleted file mode 100644
index 1a734bc..000
--- 
a/dsls/sql/src/test/java/org/apache/beam/dsls/sql/interpreter/operator/string/BeamSqlUpperExpressionTest.java
+++ /dev/null
@@ -1,45 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package org.apache.beam.dsls.sql.interpreter.operator.string;
-
-import static org.junit.Assert.assertEquals;
-
-import java.util.ArrayList;
-import java.util.List;
-
-import org.apache.beam.dsls.sql.interpreter.BeamSqlFnExecutorTestBase;
-import org.apache.beam.dsls.sql.interpreter.operator.BeamSqlExpression;
-import org.apache.beam.dsls.sql.interpreter.operator.BeamSqlPrimitive;
-import org.apache.calcite.sql.type.SqlTypeName;
-import org.junit.Test;
-
-/**
- * Test of BeamSqlUpperExpression.
- */
-public class BeamSqlUpperExpressionTest extends BeamSqlFnExecutorTestBase {
-
-  @Test public void evaluate() throws Exception {
-List operands = new ArrayList<>();
-
-operands.add(BeamSqlPrimitive.of(SqlTypeName.VARCHAR, "hello"));
-assertEquals("HELLO",
-new BeamSqlUpperExpression(operands).evaluate(record).getValue());
-  }
-
-}

http://git-wip-us.apache.org/repos/asf/beam/blob/ba493f85/dsls/sql/src/test/java/org/apache/beam/dsls/sql/mock/MockedBoundedTable.java
--
diff --git 
a/dsls/sql/src/test/java/org/apache/beam/dsls/sql/mock/MockedBoundedTable.java 
b/dsls/sql/src/test/java/org/apache/beam/dsls/sql/mock/MockedBoundedTable.java
deleted file mode 100644
index 6c1dcb2..000
--- 
a/dsls/sql/src/test/java/org/apache/beam/dsls/sql/mock/MockedBoundedTable.java
+++ /dev/null
@@ -1,134 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.dsls.sql.mock;
-
-import static org.apache.beam.dsls.sql.TestUtils.buildBeamSqlRowType;
-import static org.apache.beam.dsls.sql.TestUtils.buildRows;
-
-import java.util.ArrayList;
-import java.util.Arrays;
-import java.util.List;
-import java.util.concurrent.ConcurrentLinkedQueue;
-import org.apache.beam.dsls.sql.schema.BeamIOType;
-import org.apache.beam.dsls.sql.schema.BeamSqlRow;
-import org.apache.beam.dsls.sql.schema.BeamSqlRowType;
-import org.apache.beam.sdk.Pipeline;
-import org.apache.beam.sdk.transforms.Create;
-import org.apache.beam.sdk.transforms.DoFn;
-import org.apache.beam.sdk.transforms.PTransform;
-import org.apache.beam.sdk.transforms.ParDo;
-import org.apache.beam.sdk.values.PBegin;
-import org.apache.beam.sdk.values.PCollection;
-import org.apache.beam.sdk.values.PDone;
-
-/**
- * Mocked table for bounded data sources.
- */
-public class MockedBoundedTable extends MockedTable {
-  /** rows written to this table. */
-  private static final ConcurrentLinkedQueue CONTENT = new 
ConcurrentLinkedQueue<>();
-  /** rows flow out from this table. */
-  private final List rows = new ArrayList<>();
-
-  public MockedBoundedTable(BeamSqlRowType beamSqlRowType) {
-super(beamSqlRowType);
-  }
-
-  /**
-   * Convenient way to 

[30/59] beam git commit: rename package org.apache.beam.dsls.sql to org.apache.beam.sdk.extensions.sql

[takidau]

http://git-wip-us.apache.org/repos/asf/beam/blob/c1b5482d/sdks/java/extensions/sql/src/test/java/org/apache/beam/dsls/sql/BeamSqlDslBase.java
--
diff --git 
a/sdks/java/extensions/sql/src/test/java/org/apache/beam/dsls/sql/BeamSqlDslBase.java
 
b/sdks/java/extensions/sql/src/test/java/org/apache/beam/dsls/sql/BeamSqlDslBase.java
deleted file mode 100644
index a5d92e7..000
--- 
a/sdks/java/extensions/sql/src/test/java/org/apache/beam/dsls/sql/BeamSqlDslBase.java
+++ /dev/null
@@ -1,170 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.dsls.sql;
-
-import java.math.BigDecimal;
-import java.sql.Types;
-import java.text.DateFormat;
-import java.text.ParseException;
-import java.text.SimpleDateFormat;
-import java.util.ArrayList;
-import java.util.Arrays;
-import java.util.List;
-import org.apache.beam.dsls.sql.schema.BeamSqlRow;
-import org.apache.beam.dsls.sql.schema.BeamSqlRowCoder;
-import org.apache.beam.dsls.sql.schema.BeamSqlRowType;
-import org.apache.beam.sdk.testing.TestPipeline;
-import org.apache.beam.sdk.testing.TestStream;
-import org.apache.beam.sdk.transforms.Create;
-import org.apache.beam.sdk.values.PBegin;
-import org.apache.beam.sdk.values.PCollection;
-import org.joda.time.Instant;
-import org.junit.Before;
-import org.junit.BeforeClass;
-import org.junit.Rule;
-import org.junit.rules.ExpectedException;
-
-/**
- * prepare input records to test {@link BeamSql}.
- *
- * Note that, any change in these records would impact tests in this 
package.
- *
- */
-public class BeamSqlDslBase {
-  public static final DateFormat FORMAT = new SimpleDateFormat("-MM-dd 
HH:mm:ss");
-
-  @Rule
-  public final TestPipeline pipeline = TestPipeline.create();
-  @Rule
-  public ExpectedException exceptions = ExpectedException.none();
-
-  public static BeamSqlRowType rowTypeInTableA;
-  public static List recordsInTableA;
-
-  //bounded PCollections
-  public PCollection boundedInput1;
-  public PCollection boundedInput2;
-
-  //unbounded PCollections
-  public PCollection unboundedInput1;
-  public PCollection unboundedInput2;
-
-  @BeforeClass
-  public static void prepareClass() throws ParseException {
-rowTypeInTableA = BeamSqlRowType.create(
-Arrays.asList("f_int", "f_long", "f_short", "f_byte", "f_float", 
"f_double", "f_string",
-"f_timestamp", "f_int2", "f_decimal"),
-Arrays.asList(Types.INTEGER, Types.BIGINT, Types.SMALLINT, 
Types.TINYINT, Types.FLOAT,
-Types.DOUBLE, Types.VARCHAR, Types.TIMESTAMP, Types.INTEGER, 
Types.DECIMAL));
-
-recordsInTableA = prepareInputRowsInTableA();
-  }
-
-  @Before
-  public void preparePCollections(){
-boundedInput1 = PBegin.in(pipeline).apply("boundedInput1",
-Create.of(recordsInTableA).withCoder(new 
BeamSqlRowCoder(rowTypeInTableA)));
-
-boundedInput2 = PBegin.in(pipeline).apply("boundedInput2",
-Create.of(recordsInTableA.get(0)).withCoder(new 
BeamSqlRowCoder(rowTypeInTableA)));
-
-unboundedInput1 = prepareUnboundedPCollection1();
-unboundedInput2 = prepareUnboundedPCollection2();
-  }
-
-  private PCollection prepareUnboundedPCollection1() {
-TestStream.Builder values = TestStream
-.create(new BeamSqlRowCoder(rowTypeInTableA));
-
-for (BeamSqlRow row : recordsInTableA) {
-  values = values.advanceWatermarkTo(new 
Instant(row.getDate("f_timestamp")));
-  values = values.addElements(row);
-}
-
-return PBegin.in(pipeline).apply("unboundedInput1", 
values.advanceWatermarkToInfinity());
-  }
-
-  private PCollection prepareUnboundedPCollection2() {
-TestStream.Builder values = TestStream
-.create(new BeamSqlRowCoder(rowTypeInTableA));
-
-BeamSqlRow row = recordsInTableA.get(0);
-values = values.advanceWatermarkTo(new 
Instant(row.getDate("f_timestamp")));
-values = values.addElements(row);
-
-return PBegin.in(pipeline).apply("unboundedInput2", 
values.advanceWatermarkToInfinity());
-  }
-
-  private static List prepareInputRowsInTableA() throws 
ParseException{
-List rows = new ArrayList<>();
-
-BeamSqlRow row1 = new BeamSqlRow(rowTypeInTableA);
-

[32/59] beam git commit: rename package org.apache.beam.dsls.sql to org.apache.beam.sdk.extensions.sql

[takidau]

http://git-wip-us.apache.org/repos/asf/beam/blob/c1b5482d/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/rel/package-info.java
--
diff --git 
a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/rel/package-info.java
 
b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/rel/package-info.java
new file mode 100644
index 000..fb0a8e2
--- /dev/null
+++ 
b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/rel/package-info.java
@@ -0,0 +1,23 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+/**
+ * BeamSQL specified nodes, to replace {@link org.apache.calcite.rel.RelNode}.
+ *
+ */
+package org.apache.beam.sdk.extensions.sql.rel;

http://git-wip-us.apache.org/repos/asf/beam/blob/c1b5482d/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/rule/BeamAggregationRule.java
--
diff --git 
a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/rule/BeamAggregationRule.java
 
b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/rule/BeamAggregationRule.java
new file mode 100644
index 000..17e3f80
--- /dev/null
+++ 
b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/rule/BeamAggregationRule.java
@@ -0,0 +1,162 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.rule;
+
+import com.google.common.collect.ImmutableList;
+import java.util.GregorianCalendar;
+import java.util.List;
+import org.apache.beam.sdk.extensions.sql.rel.BeamAggregationRel;
+import org.apache.beam.sdk.extensions.sql.rel.BeamLogicalConvention;
+import org.apache.beam.sdk.transforms.windowing.AfterProcessingTime;
+import org.apache.beam.sdk.transforms.windowing.AfterWatermark;
+import org.apache.beam.sdk.transforms.windowing.FixedWindows;
+import org.apache.beam.sdk.transforms.windowing.GlobalWindows;
+import org.apache.beam.sdk.transforms.windowing.Repeatedly;
+import org.apache.beam.sdk.transforms.windowing.Sessions;
+import org.apache.beam.sdk.transforms.windowing.SlidingWindows;
+import org.apache.beam.sdk.transforms.windowing.Trigger;
+import org.apache.beam.sdk.transforms.windowing.WindowFn;
+import org.apache.calcite.plan.RelOptRule;
+import org.apache.calcite.plan.RelOptRuleCall;
+import org.apache.calcite.plan.RelOptRuleOperand;
+import org.apache.calcite.rel.core.Aggregate;
+import org.apache.calcite.rel.core.Project;
+import org.apache.calcite.rel.core.RelFactories;
+import org.apache.calcite.rex.RexCall;
+import org.apache.calcite.rex.RexLiteral;
+import org.apache.calcite.rex.RexNode;
+import org.apache.calcite.sql.SqlOperator;
+import org.apache.calcite.tools.RelBuilderFactory;
+import org.apache.calcite.util.ImmutableBitSet;
+import org.joda.time.Duration;
+
+/**
+ * Rule to detect the window/trigger settings.
+ *
+ */
+public class BeamAggregationRule extends RelOptRule {
+  public static final BeamAggregationRule INSTANCE =
+  new BeamAggregationRule(Aggregate.class, Project.class, 
RelFactories.LOGICAL_BUILDER);
+
+  public BeamAggregationRule(
+  Class aggregateClass,
+  Class projectClass,
+  RelBuilderFactory relBuilderFactory) {
+super(
+operand(aggregateClass,
+operand(projectClass, any())),
+   

[36/59] beam git commit: rename package org.apache.beam.dsls.sql to org.apache.beam.sdk.extensions.sql

[takidau]

http://git-wip-us.apache.org/repos/asf/beam/blob/c1b5482d/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/BeamSql.java
--
diff --git 
a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/BeamSql.java
 
b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/BeamSql.java
new file mode 100644
index 000..d64ae41
--- /dev/null
+++ 
b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/BeamSql.java
@@ -0,0 +1,244 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql;
+
+import com.google.auto.value.AutoValue;
+import org.apache.beam.sdk.annotations.Experimental;
+import org.apache.beam.sdk.extensions.sql.rel.BeamRelNode;
+import org.apache.beam.sdk.extensions.sql.schema.BeamPCollectionTable;
+import org.apache.beam.sdk.extensions.sql.schema.BeamSqlRow;
+import org.apache.beam.sdk.extensions.sql.schema.BeamSqlRowCoder;
+import org.apache.beam.sdk.extensions.sql.schema.BeamSqlUdaf;
+import org.apache.beam.sdk.extensions.sql.schema.BeamSqlUdf;
+import org.apache.beam.sdk.transforms.PTransform;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.PCollectionTuple;
+import org.apache.beam.sdk.values.TupleTag;
+import org.apache.calcite.sql.SqlNode;
+import org.apache.calcite.sql.SqlSelect;
+import org.apache.calcite.sql.parser.SqlParseException;
+import org.apache.calcite.tools.RelConversionException;
+import org.apache.calcite.tools.ValidationException;
+
+/**
+ * {@code BeamSql} is the DSL interface of BeamSQL. It translates a SQL query 
as a
+ * {@link PTransform}, so developers can use standard SQL queries in a Beam 
pipeline.
+ *
+ * Beam SQL DSL usage:
+ * A typical pipeline with Beam SQL DSL is:
+ * 
+ *{@code
+PipelineOptions options = PipelineOptionsFactory.create();
+Pipeline p = Pipeline.create(options);
+
+//create table from TextIO;
+PCollection inputTableA = 
p.apply(TextIO.read().from("/my/input/patha"))
+.apply(...);
+PCollection inputTableB = 
p.apply(TextIO.read().from("/my/input/pathb"))
+.apply(...);
+
+//run a simple query, and register the output as a table in BeamSql;
+String sql1 = "select MY_FUNC(c1), c2 from PCOLLECTION";
+PCollection outputTableA = inputTableA.apply(
+BeamSql.simpleQuery(sql1)
+.withUdf("MY_FUNC", MY_FUNC.class, "FUNC"));
+
+//run a JOIN with one table from TextIO, and one table from another query
+PCollection outputTableB = PCollectionTuple.of(
+new TupleTag("TABLE_O_A"), outputTableA)
+.and(new TupleTag("TABLE_B"), inputTableB)
+.apply(BeamSql.query("select * from TABLE_O_A JOIN TABLE_B where ..."));
+
+//output the final result with TextIO
+outputTableB.apply(...).apply(TextIO.write().to("/my/output/path"));
+
+p.run().waitUntilFinish();
+ * }
+ * 
+ */
+@Experimental
+public class BeamSql {
+  /**
+   * Transforms a SQL query into a {@link PTransform} representing an 
equivalent execution plan.
+   *
+   * The returned {@link PTransform} can be applied to a {@link 
PCollectionTuple} representing
+   * all the input tables and results in a {@code PCollection} 
representing the output
+   * table. The {@link PCollectionTuple} contains the mapping from {@code 
table names} to
+   * {@code PCollection}, each representing an input table.
+   *
+   * It is an error to apply a {@link PCollectionTuple} missing any {@code 
table names}
+   * referenced within the query.
+   */
+  public static QueryTransform query(String sqlQuery) {
+return QueryTransform.builder()
+.setSqlEnv(new BeamSqlEnv())
+.setSqlQuery(sqlQuery)
+.build();
+  }
+
+  /**
+   * Transforms a SQL query into a {@link PTransform} representing an 
equivalent execution plan.
+   *
+   * This is a simplified form of {@link #query(String)} where the query 
must reference
+   * a single input table.
+   *
+   * Make sure to query it from a static table name PCOLLECTION.
+   */
+  public static SimpleQueryTransform simpleQuery(String sqlQuery) throws 
Exception {
+return SimpleQueryTransform.builder()
+.setSqlEnv(new BeamSqlEnv())
+

[55/59] beam git commit: move all implementation classes/packages into impl package

[takidau]

http://git-wip-us.apache.org/repos/asf/beam/blob/febd044a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/planner/BeamQueryPlanner.java
--
diff --git 
a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/planner/BeamQueryPlanner.java
 
b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/planner/BeamQueryPlanner.java
new file mode 100644
index 000..dd01a87
--- /dev/null
+++ 
b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/planner/BeamQueryPlanner.java
@@ -0,0 +1,167 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.impl.planner;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import org.apache.beam.sdk.Pipeline;
+import org.apache.beam.sdk.extensions.sql.BeamSqlEnv;
+import org.apache.beam.sdk.extensions.sql.impl.rel.BeamLogicalConvention;
+import org.apache.beam.sdk.extensions.sql.impl.rel.BeamRelNode;
+import org.apache.beam.sdk.extensions.sql.schema.BaseBeamTable;
+import org.apache.beam.sdk.extensions.sql.schema.BeamSqlRow;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.PCollectionTuple;
+import org.apache.calcite.adapter.java.JavaTypeFactory;
+import org.apache.calcite.config.Lex;
+import org.apache.calcite.jdbc.CalciteSchema;
+import org.apache.calcite.jdbc.JavaTypeFactoryImpl;
+import org.apache.calcite.plan.Contexts;
+import org.apache.calcite.plan.ConventionTraitDef;
+import org.apache.calcite.plan.RelOptUtil;
+import org.apache.calcite.plan.RelTraitDef;
+import org.apache.calcite.plan.RelTraitSet;
+import org.apache.calcite.prepare.CalciteCatalogReader;
+import org.apache.calcite.rel.RelCollationTraitDef;
+import org.apache.calcite.rel.RelNode;
+import org.apache.calcite.rel.type.RelDataTypeSystem;
+import org.apache.calcite.schema.SchemaPlus;
+import org.apache.calcite.sql.SqlNode;
+import org.apache.calcite.sql.SqlOperatorTable;
+import org.apache.calcite.sql.fun.SqlStdOperatorTable;
+import org.apache.calcite.sql.parser.SqlParseException;
+import org.apache.calcite.sql.parser.SqlParser;
+import org.apache.calcite.sql.util.ChainedSqlOperatorTable;
+import org.apache.calcite.tools.FrameworkConfig;
+import org.apache.calcite.tools.Frameworks;
+import org.apache.calcite.tools.Planner;
+import org.apache.calcite.tools.RelConversionException;
+import org.apache.calcite.tools.ValidationException;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * The core component to handle through a SQL statement, from explain 
execution plan,
+ * to generate a Beam pipeline.
+ *
+ */
+public class BeamQueryPlanner {
+  private static final Logger LOG = 
LoggerFactory.getLogger(BeamQueryPlanner.class);
+
+  protected final Planner planner;
+  private Map sourceTables = new HashMap<>();
+
+  public static final JavaTypeFactory TYPE_FACTORY = new JavaTypeFactoryImpl(
+  RelDataTypeSystem.DEFAULT);
+
+  public BeamQueryPlanner(SchemaPlus schema) {
+final List traitDefs = new ArrayList<>();
+traitDefs.add(ConventionTraitDef.INSTANCE);
+traitDefs.add(RelCollationTraitDef.INSTANCE);
+
+List sqlOperatorTables = new ArrayList<>();
+sqlOperatorTables.add(SqlStdOperatorTable.instance());
+sqlOperatorTables.add(new CalciteCatalogReader(CalciteSchema.from(schema), 
false,
+Collections.emptyList(), TYPE_FACTORY));
+
+FrameworkConfig config = Frameworks.newConfigBuilder()
+
.parserConfig(SqlParser.configBuilder().setLex(Lex.MYSQL).build()).defaultSchema(schema)
+
.traitDefs(traitDefs).context(Contexts.EMPTY_CONTEXT).ruleSets(BeamRuleSets.getRuleSets())
+
.costFactory(null).typeSystem(BeamRelDataTypeSystem.BEAM_REL_DATATYPE_SYSTEM)
+.operatorTable(new ChainedSqlOperatorTable(sqlOperatorTables))
+.build();
+this.planner = Frameworks.getPlanner(config);
+
+for (String t : schema.getTableNames()) {
+  sourceTables.put(t, (BaseBeamTable) schema.getTable(t));
+}
+  }
+
+  /**
+   * 

[20/59] beam git commit: move dsls/sql to sdks/java/extensions/sql

[takidau]

http://git-wip-us.apache.org/repos/asf/beam/blob/ba493f85/dsls/sql/src/main/java/org/apache/beam/dsls/sql/interpreter/operator/BeamSqlCastExpression.java
--
diff --git 
a/dsls/sql/src/main/java/org/apache/beam/dsls/sql/interpreter/operator/BeamSqlCastExpression.java
 
b/dsls/sql/src/main/java/org/apache/beam/dsls/sql/interpreter/operator/BeamSqlCastExpression.java
deleted file mode 100644
index 524d1df..000
--- 
a/dsls/sql/src/main/java/org/apache/beam/dsls/sql/interpreter/operator/BeamSqlCastExpression.java
+++ /dev/null
@@ -1,132 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package org.apache.beam.dsls.sql.interpreter.operator;
-
-import java.sql.Date;
-import java.sql.Timestamp;
-import java.util.List;
-
-import org.apache.beam.dsls.sql.schema.BeamSqlRow;
-import org.apache.calcite.runtime.SqlFunctions;
-import org.apache.calcite.sql.type.SqlTypeName;
-import org.joda.time.format.DateTimeFormat;
-import org.joda.time.format.DateTimeFormatter;
-import org.joda.time.format.DateTimeFormatterBuilder;
-import org.joda.time.format.DateTimeParser;
-
-/**
- * Base class to support 'CAST' operations for all {@link SqlTypeName}.
- */
-public class BeamSqlCastExpression extends BeamSqlExpression {
-
-  private static final int index = 0;
-  private static final String outputTimestampFormat = "-MM-dd HH:mm:ss";
-  private static final String outputDateFormat = "-MM-dd";
-  /**
-   * Date and Timestamp formats used to parse
-   * {@link SqlTypeName#DATE}, {@link SqlTypeName#TIMESTAMP}.
-   */
-  private static final DateTimeFormatter dateTimeFormatter = new 
DateTimeFormatterBuilder()
-  .append(null/*printer*/, new DateTimeParser[] {
-  // date formats
-  DateTimeFormat.forPattern("yy-MM-dd").getParser(),
-  DateTimeFormat.forPattern("yy/MM/dd").getParser(),
-  DateTimeFormat.forPattern("yy.MM.dd").getParser(),
-  DateTimeFormat.forPattern("yyMMdd").getParser(),
-  DateTimeFormat.forPattern("MMdd").getParser(),
-  DateTimeFormat.forPattern("-MM-dd").getParser(),
-  DateTimeFormat.forPattern("/MM/dd").getParser(),
-  DateTimeFormat.forPattern(".MM.dd").getParser(),
-  // datetime formats
-  DateTimeFormat.forPattern("-MM-dd HH:mm:ss").getParser(),
-  DateTimeFormat.forPattern("-MM-dd HH:mm:ssz").getParser(),
-  DateTimeFormat.forPattern("-MM-dd HH:mm:ss z").getParser(),
-  DateTimeFormat.forPattern("-MM-dd 
HH:mm:ss.S").getParser(),
-  DateTimeFormat.forPattern("-MM-dd 
HH:mm:ss.Sz").getParser(),
-  DateTimeFormat.forPattern("-MM-dd HH:mm:ss.S 
z").getParser() }).toFormatter()
-  .withPivotYear(2020);
-
-  public BeamSqlCastExpression(List operands, SqlTypeName 
castType) {
-super(operands, castType);
-  }
-
-  @Override
-  public boolean accept() {
-return numberOfOperands() == 1;
-  }
-
-  @Override
-  public BeamSqlPrimitive evaluate(BeamSqlRow inputRow) {
-SqlTypeName castOutputType = getOutputType();
-switch (castOutputType) {
-  case INTEGER:
-return BeamSqlPrimitive
-.of(SqlTypeName.INTEGER, 
SqlFunctions.toInt(opValueEvaluated(index, inputRow)));
-  case DOUBLE:
-return BeamSqlPrimitive
-.of(SqlTypeName.DOUBLE, 
SqlFunctions.toDouble(opValueEvaluated(index, inputRow)));
-  case SMALLINT:
-return BeamSqlPrimitive
-.of(SqlTypeName.SMALLINT, 
SqlFunctions.toShort(opValueEvaluated(index, inputRow)));
-  case TINYINT:
-return BeamSqlPrimitive
-.of(SqlTypeName.TINYINT, 
SqlFunctions.toByte(opValueEvaluated(index, inputRow)));
-  case BIGINT:
-return BeamSqlPrimitive
-.of(SqlTypeName.BIGINT, 
SqlFunctions.toLong(opValueEvaluated(index, inputRow)));
-  case DECIMAL:
-return BeamSqlPrimitive.of(SqlTypeName.DECIMAL,
-SqlFunctions.toBigDecimal(opValueEvaluated(index, inputRow)));
-  case FLOAT:
-return BeamSqlPrimitive
-.of(SqlTypeName.FLOAT, 
SqlFunctions.toFloat(opValueEvaluated(index, 

[51/59] beam git commit: move all implementation classes/packages into impl package

[takidau]

http://git-wip-us.apache.org/repos/asf/beam/blob/febd044a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/interpreter/operator/math/BeamSqlPiExpression.java
--
diff --git 
a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/interpreter/operator/math/BeamSqlPiExpression.java
 
b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/interpreter/operator/math/BeamSqlPiExpression.java
deleted file mode 100644
index ed89c49..000
--- 
a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/interpreter/operator/math/BeamSqlPiExpression.java
+++ /dev/null
@@ -1,42 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package org.apache.beam.sdk.extensions.sql.interpreter.operator.math;
-
-import 
org.apache.beam.sdk.extensions.sql.interpreter.operator.BeamSqlExpression;
-import 
org.apache.beam.sdk.extensions.sql.interpreter.operator.BeamSqlPrimitive;
-import org.apache.beam.sdk.extensions.sql.schema.BeamSqlRow;
-import org.apache.calcite.sql.type.SqlTypeName;
-
-/**
- * Base class for the PI function.
- */
-public class BeamSqlPiExpression extends BeamSqlExpression {
-
-  public BeamSqlPiExpression() {
-this.outputType = SqlTypeName.DOUBLE;
-  }
-
-  @Override public boolean accept() {
-return true;
-  }
-
-  @Override public BeamSqlPrimitive evaluate(BeamSqlRow inputRow) {
-return BeamSqlPrimitive.of(SqlTypeName.DOUBLE, Math.PI);
-  }
-}

http://git-wip-us.apache.org/repos/asf/beam/blob/febd044a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/interpreter/operator/math/BeamSqlPowerExpression.java
--
diff --git 
a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/interpreter/operator/math/BeamSqlPowerExpression.java
 
b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/interpreter/operator/math/BeamSqlPowerExpression.java
deleted file mode 100644
index e2bdd05..000
--- 
a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/interpreter/operator/math/BeamSqlPowerExpression.java
+++ /dev/null
@@ -1,44 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package org.apache.beam.sdk.extensions.sql.interpreter.operator.math;
-
-import java.util.List;
-import 
org.apache.beam.sdk.extensions.sql.interpreter.operator.BeamSqlExpression;
-import 
org.apache.beam.sdk.extensions.sql.interpreter.operator.BeamSqlPrimitive;
-import org.apache.calcite.runtime.SqlFunctions;
-import org.apache.calcite.sql.type.SqlTypeName;
-
-/**
- * {@code BeamSqlMathBinaryExpression} for 'POWER' function.
- */
-public class BeamSqlPowerExpression extends BeamSqlMathBinaryExpression {
-
-  public BeamSqlPowerExpression(List operands) {
-super(operands, SqlTypeName.DOUBLE);
-  }
-
-  @Override
-  public BeamSqlPrimitive calculate(BeamSqlPrimitive leftOp,
-  BeamSqlPrimitive rightOp) {
-return BeamSqlPrimitive.of(SqlTypeName.DOUBLE, SqlFunctions
-.power(SqlFunctions.toDouble(leftOp.getValue()),
-SqlFunctions.toDouble(rightOp.getValue(;
-  }
-
-}

http://git-wip-us.apache.org/repos/asf/beam/blob/febd044a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/interpreter/operator/math/BeamSqlRadiansExpression.java

[31/59] beam git commit: rename package org.apache.beam.dsls.sql to org.apache.beam.sdk.extensions.sql

[takidau]

http://git-wip-us.apache.org/repos/asf/beam/blob/c1b5482d/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/schema/package-info.java
--
diff --git 
a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/schema/package-info.java
 
b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/schema/package-info.java
new file mode 100644
index 000..9655ebd
--- /dev/null
+++ 
b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/schema/package-info.java
@@ -0,0 +1,22 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+/**
+ * define table schema, to map with Beam IO components.
+ *
+ */
+package org.apache.beam.sdk.extensions.sql.schema;

http://git-wip-us.apache.org/repos/asf/beam/blob/c1b5482d/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/schema/text/BeamTextCSVTable.java
--
diff --git 
a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/schema/text/BeamTextCSVTable.java
 
b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/schema/text/BeamTextCSVTable.java
new file mode 100644
index 000..c44faab
--- /dev/null
+++ 
b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/schema/text/BeamTextCSVTable.java
@@ -0,0 +1,70 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.beam.sdk.extensions.sql.schema.text;
+
+import org.apache.beam.sdk.Pipeline;
+import org.apache.beam.sdk.extensions.sql.schema.BeamSqlRow;
+import org.apache.beam.sdk.extensions.sql.schema.BeamSqlRowType;
+import org.apache.beam.sdk.io.TextIO;
+import org.apache.beam.sdk.transforms.PTransform;
+import org.apache.beam.sdk.values.PBegin;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.PDone;
+import org.apache.commons.csv.CSVFormat;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * {@code BeamTextCSVTable} is a {@code BeamTextTable} which formatted in CSV.
+ *
+ * 
+ * {@link CSVFormat} itself has many dialects, check its javadoc for more info.
+ * 
+ */
+public class BeamTextCSVTable extends BeamTextTable {
+  private static final Logger LOG = LoggerFactory
+  .getLogger(BeamTextCSVTable.class);
+
+  private CSVFormat csvFormat;
+
+  /**
+   * CSV table with {@link CSVFormat#DEFAULT DEFAULT} format.
+   */
+  public BeamTextCSVTable(BeamSqlRowType beamSqlRowType, String filePattern)  {
+this(beamSqlRowType, filePattern, CSVFormat.DEFAULT);
+  }
+
+  public BeamTextCSVTable(BeamSqlRowType beamSqlRowType, String filePattern,
+  CSVFormat csvFormat) {
+super(beamSqlRowType, filePattern);
+this.csvFormat = csvFormat;
+  }
+
+  @Override
+  public PCollection buildIOReader(Pipeline pipeline) {
+return PBegin.in(pipeline).apply("decodeRecord", 
TextIO.read().from(filePattern))
+.apply("parseCSVLine",
+new BeamTextCSVTableIOReader(beamSqlRowType, filePattern, 
csvFormat));
+  }
+
+  @Override
+  public PTransform, PDone> buildIOWriter() {
+return new BeamTextCSVTableIOWriter(beamSqlRowType, filePattern, 
csvFormat);
+  }
+}

[28/59] beam git commit: rename package org.apache.beam.dsls.sql to org.apache.beam.sdk.extensions.sql

[takidau]

http://git-wip-us.apache.org/repos/asf/beam/blob/c1b5482d/sdks/java/extensions/sql/src/test/java/org/apache/beam/dsls/sql/interpreter/operator/date/BeamSqlCurrentTimeExpressionTest.java
--
diff --git 
a/sdks/java/extensions/sql/src/test/java/org/apache/beam/dsls/sql/interpreter/operator/date/BeamSqlCurrentTimeExpressionTest.java
 
b/sdks/java/extensions/sql/src/test/java/org/apache/beam/dsls/sql/interpreter/operator/date/BeamSqlCurrentTimeExpressionTest.java
deleted file mode 100644
index ddf0a22..000
--- 
a/sdks/java/extensions/sql/src/test/java/org/apache/beam/dsls/sql/interpreter/operator/date/BeamSqlCurrentTimeExpressionTest.java
+++ /dev/null
@@ -1,40 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package org.apache.beam.dsls.sql.interpreter.operator.date;
-
-import static org.junit.Assert.assertEquals;
-
-import java.util.ArrayList;
-import java.util.List;
-
-import org.apache.beam.dsls.sql.interpreter.operator.BeamSqlExpression;
-import org.apache.calcite.sql.type.SqlTypeName;
-import org.junit.Test;
-
-/**
- * Test for BeamSqlLocalTimeExpression.
- */
-public class BeamSqlCurrentTimeExpressionTest extends 
BeamSqlDateExpressionTestBase {
-  @Test
-  public void test() {
-List operands = new ArrayList<>();
-assertEquals(SqlTypeName.TIME,
-new 
BeamSqlCurrentTimeExpression(operands).evaluate(record).getOutputType());
-  }
-}

http://git-wip-us.apache.org/repos/asf/beam/blob/c1b5482d/sdks/java/extensions/sql/src/test/java/org/apache/beam/dsls/sql/interpreter/operator/date/BeamSqlCurrentTimestampExpressionTest.java
--
diff --git 
a/sdks/java/extensions/sql/src/test/java/org/apache/beam/dsls/sql/interpreter/operator/date/BeamSqlCurrentTimestampExpressionTest.java
 
b/sdks/java/extensions/sql/src/test/java/org/apache/beam/dsls/sql/interpreter/operator/date/BeamSqlCurrentTimestampExpressionTest.java
deleted file mode 100644
index a1554f1..000
--- 
a/sdks/java/extensions/sql/src/test/java/org/apache/beam/dsls/sql/interpreter/operator/date/BeamSqlCurrentTimestampExpressionTest.java
+++ /dev/null
@@ -1,40 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package org.apache.beam.dsls.sql.interpreter.operator.date;
-
-import static org.junit.Assert.assertEquals;
-
-import java.util.ArrayList;
-import java.util.List;
-
-import org.apache.beam.dsls.sql.interpreter.operator.BeamSqlExpression;
-import org.apache.calcite.sql.type.SqlTypeName;
-import org.junit.Test;
-
-/**
- * Test for BeamSqlLocalTimestampExpression.
- */
-public class BeamSqlCurrentTimestampExpressionTest extends 
BeamSqlDateExpressionTestBase {
-  @Test
-  public void test() {
-List operands = new ArrayList<>();
-assertEquals(SqlTypeName.TIMESTAMP,
-new 
BeamSqlCurrentTimestampExpression(operands).evaluate(record).getOutputType());
-  }
-}

http://git-wip-us.apache.org/repos/asf/beam/blob/c1b5482d/sdks/java/extensions/sql/src/test/java/org/apache/beam/dsls/sql/interpreter/operator/date/BeamSqlDateCeilExpressionTest.java
--
diff --git 
a/sdks/java/extensions/sql/src/test/java/org/apache/beam/dsls/sql/interpreter/operator/date/BeamSqlDateCeilExpressionTest.java
 

[33/59] beam git commit: rename package org.apache.beam.dsls.sql to org.apache.beam.sdk.extensions.sql

[takidau]

http://git-wip-us.apache.org/repos/asf/beam/blob/c1b5482d/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/planner/BeamQueryPlanner.java
--
diff --git 
a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/planner/BeamQueryPlanner.java
 
b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/planner/BeamQueryPlanner.java
new file mode 100644
index 000..ba6235f
--- /dev/null
+++ 
b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/planner/BeamQueryPlanner.java
@@ -0,0 +1,167 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.planner;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import org.apache.beam.sdk.Pipeline;
+import org.apache.beam.sdk.extensions.sql.BeamSqlEnv;
+import org.apache.beam.sdk.extensions.sql.rel.BeamLogicalConvention;
+import org.apache.beam.sdk.extensions.sql.rel.BeamRelNode;
+import org.apache.beam.sdk.extensions.sql.schema.BaseBeamTable;
+import org.apache.beam.sdk.extensions.sql.schema.BeamSqlRow;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.PCollectionTuple;
+import org.apache.calcite.adapter.java.JavaTypeFactory;
+import org.apache.calcite.config.Lex;
+import org.apache.calcite.jdbc.CalciteSchema;
+import org.apache.calcite.jdbc.JavaTypeFactoryImpl;
+import org.apache.calcite.plan.Contexts;
+import org.apache.calcite.plan.ConventionTraitDef;
+import org.apache.calcite.plan.RelOptUtil;
+import org.apache.calcite.plan.RelTraitDef;
+import org.apache.calcite.plan.RelTraitSet;
+import org.apache.calcite.prepare.CalciteCatalogReader;
+import org.apache.calcite.rel.RelCollationTraitDef;
+import org.apache.calcite.rel.RelNode;
+import org.apache.calcite.rel.type.RelDataTypeSystem;
+import org.apache.calcite.schema.SchemaPlus;
+import org.apache.calcite.sql.SqlNode;
+import org.apache.calcite.sql.SqlOperatorTable;
+import org.apache.calcite.sql.fun.SqlStdOperatorTable;
+import org.apache.calcite.sql.parser.SqlParseException;
+import org.apache.calcite.sql.parser.SqlParser;
+import org.apache.calcite.sql.util.ChainedSqlOperatorTable;
+import org.apache.calcite.tools.FrameworkConfig;
+import org.apache.calcite.tools.Frameworks;
+import org.apache.calcite.tools.Planner;
+import org.apache.calcite.tools.RelConversionException;
+import org.apache.calcite.tools.ValidationException;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * The core component to handle through a SQL statement, from explain 
execution plan,
+ * to generate a Beam pipeline.
+ *
+ */
+public class BeamQueryPlanner {
+  private static final Logger LOG = 
LoggerFactory.getLogger(BeamQueryPlanner.class);
+
+  protected final Planner planner;
+  private Map sourceTables = new HashMap<>();
+
+  public static final JavaTypeFactory TYPE_FACTORY = new JavaTypeFactoryImpl(
+  RelDataTypeSystem.DEFAULT);
+
+  public BeamQueryPlanner(SchemaPlus schema) {
+final List traitDefs = new ArrayList<>();
+traitDefs.add(ConventionTraitDef.INSTANCE);
+traitDefs.add(RelCollationTraitDef.INSTANCE);
+
+List sqlOperatorTables = new ArrayList<>();
+sqlOperatorTables.add(SqlStdOperatorTable.instance());
+sqlOperatorTables.add(new CalciteCatalogReader(CalciteSchema.from(schema), 
false,
+Collections.emptyList(), TYPE_FACTORY));
+
+FrameworkConfig config = Frameworks.newConfigBuilder()
+
.parserConfig(SqlParser.configBuilder().setLex(Lex.MYSQL).build()).defaultSchema(schema)
+
.traitDefs(traitDefs).context(Contexts.EMPTY_CONTEXT).ruleSets(BeamRuleSets.getRuleSets())
+
.costFactory(null).typeSystem(BeamRelDataTypeSystem.BEAM_REL_DATATYPE_SYSTEM)
+.operatorTable(new ChainedSqlOperatorTable(sqlOperatorTables))
+.build();
+this.planner = Frameworks.getPlanner(config);
+
+for (String t : schema.getTableNames()) {
+  sourceTables.put(t, (BaseBeamTable) schema.getTable(t));
+}
+  }
+
+  /**
+   * Parse input SQL query, and return a 

[34/59] beam git commit: rename package org.apache.beam.dsls.sql to org.apache.beam.sdk.extensions.sql

[takidau]

http://git-wip-us.apache.org/repos/asf/beam/blob/c1b5482d/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/interpreter/operator/math/BeamSqlDegreesExpression.java
--
diff --git 
a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/interpreter/operator/math/BeamSqlDegreesExpression.java
 
b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/interpreter/operator/math/BeamSqlDegreesExpression.java
new file mode 100644
index 000..b41f090
--- /dev/null
+++ 
b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/interpreter/operator/math/BeamSqlDegreesExpression.java
@@ -0,0 +1,40 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.beam.sdk.extensions.sql.interpreter.operator.math;
+
+import java.util.List;
+import 
org.apache.beam.sdk.extensions.sql.interpreter.operator.BeamSqlExpression;
+import 
org.apache.beam.sdk.extensions.sql.interpreter.operator.BeamSqlPrimitive;
+import org.apache.calcite.runtime.SqlFunctions;
+import org.apache.calcite.sql.type.SqlTypeName;
+
+/**
+ * {@code BeamSqlMathUnaryExpression} for 'DEGREES' function.
+ */
+public class BeamSqlDegreesExpression extends BeamSqlMathUnaryExpression {
+
+  public BeamSqlDegreesExpression(List operands) {
+super(operands, SqlTypeName.DOUBLE);
+  }
+
+  @Override public BeamSqlPrimitive calculate(BeamSqlPrimitive op) {
+return BeamSqlPrimitive
+.of(SqlTypeName.DOUBLE, 
SqlFunctions.degrees(SqlFunctions.toDouble(op.getValue(;
+  }
+}

http://git-wip-us.apache.org/repos/asf/beam/blob/c1b5482d/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/interpreter/operator/math/BeamSqlExpExpression.java
--
diff --git 
a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/interpreter/operator/math/BeamSqlExpExpression.java
 
b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/interpreter/operator/math/BeamSqlExpExpression.java
new file mode 100644
index 000..f7a8f11
--- /dev/null
+++ 
b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/interpreter/operator/math/BeamSqlExpExpression.java
@@ -0,0 +1,40 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.beam.sdk.extensions.sql.interpreter.operator.math;
+
+import java.util.List;
+import 
org.apache.beam.sdk.extensions.sql.interpreter.operator.BeamSqlExpression;
+import 
org.apache.beam.sdk.extensions.sql.interpreter.operator.BeamSqlPrimitive;
+import org.apache.calcite.runtime.SqlFunctions;
+import org.apache.calcite.sql.type.SqlTypeName;
+
+/**
+ * {@code BeamSqlMathUnaryExpression} for 'EXP' function.
+ */
+public class BeamSqlExpExpression extends BeamSqlMathUnaryExpression {
+
+  public BeamSqlExpExpression(List operands) {
+super(operands, SqlTypeName.DOUBLE);
+  }
+
+  @Override public BeamSqlPrimitive calculate(BeamSqlPrimitive op) {
+return BeamSqlPrimitive
+.of(SqlTypeName.DOUBLE, 
SqlFunctions.exp(SqlFunctions.toDouble(op.getValue(;
+  }
+}

http://git-wip-us.apache.org/repos/asf/beam/blob/c1b5482d/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/interpreter/operator/math/BeamSqlFloorExpression.java

[21/59] beam git commit: move dsls/sql to sdks/java/extensions/sql

[takidau]

move dsls/sql to sdks/java/extensions/sql


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/ba493f85
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/ba493f85
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/ba493f85

Branch: refs/heads/DSL_SQL
Commit: ba493f85a5a18665cd6ae4eb61e0a86fab1a6c07
Parents: d32aea9
Author: James Xu 
Authored: Sun Jul 30 23:39:37 2017 +0800
Committer: James Xu 
Committed: Sun Jul 30 23:39:37 2017 +0800

--
 dsls/pom.xml|  60 ---
 dsls/sql/pom.xml| 226 -
 .../java/org/apache/beam/dsls/sql/BeamSql.java  | 244 --
 .../org/apache/beam/dsls/sql/BeamSqlCli.java|  65 ---
 .../org/apache/beam/dsls/sql/BeamSqlEnv.java| 120 -
 .../beam/dsls/sql/example/BeamSqlExample.java   |  97 
 .../beam/dsls/sql/example/package-info.java |  23 -
 .../interpreter/BeamSqlExpressionExecutor.java  |  43 --
 .../dsls/sql/interpreter/BeamSqlFnExecutor.java | 442 --
 .../operator/BeamSqlCaseExpression.java |  64 ---
 .../operator/BeamSqlCastExpression.java | 132 --
 .../interpreter/operator/BeamSqlExpression.java |  78 
 .../operator/BeamSqlInputRefExpression.java |  43 --
 .../interpreter/operator/BeamSqlPrimitive.java  | 152 ---
 .../operator/BeamSqlReinterpretExpression.java  |  55 ---
 .../operator/BeamSqlUdfExpression.java  |  86 
 .../operator/BeamSqlWindowEndExpression.java|  42 --
 .../operator/BeamSqlWindowExpression.java   |  50 --
 .../operator/BeamSqlWindowStartExpression.java  |  43 --
 .../arithmetic/BeamSqlArithmeticExpression.java | 122 -
 .../arithmetic/BeamSqlDivideExpression.java |  37 --
 .../arithmetic/BeamSqlMinusExpression.java  |  36 --
 .../arithmetic/BeamSqlModExpression.java|  36 --
 .../arithmetic/BeamSqlMultiplyExpression.java   |  36 --
 .../arithmetic/BeamSqlPlusExpression.java   |  36 --
 .../operator/arithmetic/package-info.java   |  22 -
 .../comparison/BeamSqlCompareExpression.java|  96 
 .../comparison/BeamSqlEqualsExpression.java |  49 --
 .../BeamSqlGreaterThanExpression.java   |  49 --
 .../BeamSqlGreaterThanOrEqualsExpression.java   |  49 --
 .../comparison/BeamSqlIsNotNullExpression.java  |  53 ---
 .../comparison/BeamSqlIsNullExpression.java |  53 ---
 .../comparison/BeamSqlLessThanExpression.java   |  49 --
 .../BeamSqlLessThanOrEqualsExpression.java  |  49 --
 .../comparison/BeamSqlNotEqualsExpression.java  |  49 --
 .../operator/comparison/package-info.java   |  22 -
 .../date/BeamSqlCurrentDateExpression.java  |  45 --
 .../date/BeamSqlCurrentTimeExpression.java  |  53 ---
 .../date/BeamSqlCurrentTimestampExpression.java |  49 --
 .../date/BeamSqlDateCeilExpression.java |  55 ---
 .../date/BeamSqlDateFloorExpression.java|  55 ---
 .../operator/date/BeamSqlExtractExpression.java | 101 -
 .../interpreter/operator/date/package-info.java |  22 -
 .../operator/logical/BeamSqlAndExpression.java  |  48 --
 .../logical/BeamSqlLogicalExpression.java   |  47 --
 .../operator/logical/BeamSqlNotExpression.java  |  54 ---
 .../operator/logical/BeamSqlOrExpression.java   |  48 --
 .../operator/logical/package-info.java  |  22 -
 .../operator/math/BeamSqlAbsExpression.java |  74 ---
 .../operator/math/BeamSqlAcosExpression.java|  41 --
 .../operator/math/BeamSqlAsinExpression.java|  41 --
 .../operator/math/BeamSqlAtan2Expression.java   |  43 --
 .../operator/math/BeamSqlAtanExpression.java|  41 --
 .../operator/math/BeamSqlCeilExpression.java|  46 --
 .../operator/math/BeamSqlCosExpression.java |  41 --
 .../operator/math/BeamSqlCotExpression.java |  41 --
 .../operator/math/BeamSqlDegreesExpression.java |  41 --
 .../operator/math/BeamSqlExpExpression.java |  41 --
 .../operator/math/BeamSqlFloorExpression.java   |  46 --
 .../operator/math/BeamSqlLnExpression.java  |  41 --
 .../operator/math/BeamSqlLogExpression.java |  41 --
 .../math/BeamSqlMathBinaryExpression.java   |  64 ---
 .../math/BeamSqlMathUnaryExpression.java|  58 ---
 .../operator/math/BeamSqlPiExpression.java  |  42 --
 .../operator/math/BeamSqlPowerExpression.java   |  45 --
 .../operator/math/BeamSqlRadiansExpression.java |  41 --
 .../operator/math/BeamSqlRandExpression.java|  54 ---
 .../math/BeamSqlRandIntegerExpression.java  |  58 ---
 .../operator/math/BeamSqlRoundExpression.java   | 108 -
 .../operator/math/BeamSqlSignExpression.java|  72 ---
 .../operator/math/BeamSqlSinExpression.java |  41 --
 .../operator/math/BeamSqlTanExpression.java |  41 --
 .../math/BeamSqlTruncateExpression.java |  76 
 .../interpreter/operator/math/package-info.java |  22 -
 

[24/59] beam git commit: rename package org.apache.beam.dsls.sql to org.apache.beam.sdk.extensions.sql

[takidau]

http://git-wip-us.apache.org/repos/asf/beam/blob/c1b5482d/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/interpreter/operator/BeamSqlCompareExpressionTest.java
--
diff --git 
a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/interpreter/operator/BeamSqlCompareExpressionTest.java
 
b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/interpreter/operator/BeamSqlCompareExpressionTest.java
new file mode 100644
index 000..ed77ffb
--- /dev/null
+++ 
b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/interpreter/operator/BeamSqlCompareExpressionTest.java
@@ -0,0 +1,115 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.interpreter.operator;
+
+import java.util.Arrays;
+import 
org.apache.beam.sdk.extensions.sql.interpreter.BeamSqlFnExecutorTestBase;
+import 
org.apache.beam.sdk.extensions.sql.interpreter.operator.comparison.BeamSqlCompareExpression;
+import 
org.apache.beam.sdk.extensions.sql.interpreter.operator.comparison.BeamSqlEqualsExpression;
+import 
org.apache.beam.sdk.extensions.sql.interpreter.operator.comparison.BeamSqlGreaterThanExpression;
+import 
org.apache.beam.sdk.extensions.sql.interpreter.operator.comparison.BeamSqlGreaterThanOrEqualsExpression;
+import 
org.apache.beam.sdk.extensions.sql.interpreter.operator.comparison.BeamSqlLessThanExpression;
+import 
org.apache.beam.sdk.extensions.sql.interpreter.operator.comparison.BeamSqlLessThanOrEqualsExpression;
+import 
org.apache.beam.sdk.extensions.sql.interpreter.operator.comparison.BeamSqlNotEqualsExpression;
+import org.apache.calcite.sql.type.SqlTypeName;
+import org.junit.Assert;
+import org.junit.Test;
+
+/**
+ * Test cases for the collections of {@link BeamSqlCompareExpression}.
+ */
+public class BeamSqlCompareExpressionTest extends BeamSqlFnExecutorTestBase {
+
+  @Test
+  public void testEqual() {
+BeamSqlEqualsExpression exp1 = new BeamSqlEqualsExpression(
+Arrays.asList(new BeamSqlInputRefExpression(SqlTypeName.BIGINT, 0),
+BeamSqlPrimitive.of(SqlTypeName.BIGINT, 100L)));
+Assert.assertEquals(false, exp1.evaluate(record).getValue());
+
+BeamSqlEqualsExpression exp2 = new BeamSqlEqualsExpression(
+Arrays.asList(new BeamSqlInputRefExpression(SqlTypeName.BIGINT, 0),
+BeamSqlPrimitive.of(SqlTypeName.BIGINT, 1234567L)));
+Assert.assertEquals(true, exp2.evaluate(record).getValue());
+  }
+
+  @Test
+  public void testLargerThan(){
+BeamSqlGreaterThanExpression exp1 = new BeamSqlGreaterThanExpression(
+Arrays.asList(new BeamSqlInputRefExpression(SqlTypeName.BIGINT, 0),
+BeamSqlPrimitive.of(SqlTypeName.BIGINT, 1234567L)));
+Assert.assertEquals(false, exp1.evaluate(record).getValue());
+
+BeamSqlGreaterThanExpression exp2 = new BeamSqlGreaterThanExpression(
+Arrays.asList(new BeamSqlInputRefExpression(SqlTypeName.BIGINT, 0),
+BeamSqlPrimitive.of(SqlTypeName.BIGINT, 1234566L)));
+Assert.assertEquals(true, exp2.evaluate(record).getValue());
+  }
+
+  @Test
+  public void testLargerThanEqual(){
+BeamSqlGreaterThanOrEqualsExpression exp1 = new 
BeamSqlGreaterThanOrEqualsExpression(
+Arrays.asList(new BeamSqlInputRefExpression(SqlTypeName.BIGINT, 0),
+BeamSqlPrimitive.of(SqlTypeName.BIGINT, 1234567L)));
+Assert.assertEquals(true, exp1.evaluate(record).getValue());
+
+BeamSqlGreaterThanOrEqualsExpression exp2 = new 
BeamSqlGreaterThanOrEqualsExpression(
+Arrays.asList(new BeamSqlInputRefExpression(SqlTypeName.BIGINT, 0),
+BeamSqlPrimitive.of(SqlTypeName.BIGINT, 1234568L)));
+Assert.assertEquals(false, exp2.evaluate(record).getValue());
+  }
+
+  @Test
+  public void testLessThan(){
+BeamSqlLessThanExpression exp1 = new BeamSqlLessThanExpression(
+Arrays.asList(new BeamSqlInputRefExpression(SqlTypeName.INTEGER, 1),
+BeamSqlPrimitive.of(SqlTypeName.INTEGER, 1)));
+Assert.assertEquals(true, exp1.evaluate(record).getValue());
+
+BeamSqlLessThanExpression exp2 = new 

[25/59] beam git commit: rename package org.apache.beam.dsls.sql to org.apache.beam.sdk.extensions.sql

[takidau]

http://git-wip-us.apache.org/repos/asf/beam/blob/c1b5482d/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/integrationtest/BeamSqlBuiltinFunctionsIntegrationTestBase.java
--
diff --git 
a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/integrationtest/BeamSqlBuiltinFunctionsIntegrationTestBase.java
 
b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/integrationtest/BeamSqlBuiltinFunctionsIntegrationTestBase.java
new file mode 100644
index 000..ffc6833
--- /dev/null
+++ 
b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/integrationtest/BeamSqlBuiltinFunctionsIntegrationTestBase.java
@@ -0,0 +1,169 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.beam.sdk.extensions.sql.integrationtest;
+
+import com.google.common.base.Joiner;
+import java.math.BigDecimal;
+import java.sql.Types;
+import java.text.SimpleDateFormat;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Date;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.TimeZone;
+import org.apache.beam.sdk.extensions.sql.BeamSql;
+import org.apache.beam.sdk.extensions.sql.TestUtils;
+import org.apache.beam.sdk.extensions.sql.mock.MockedBoundedTable;
+import org.apache.beam.sdk.extensions.sql.schema.BeamSqlRow;
+import org.apache.beam.sdk.extensions.sql.schema.BeamSqlRowCoder;
+import org.apache.beam.sdk.extensions.sql.schema.BeamSqlRowType;
+import org.apache.beam.sdk.testing.PAssert;
+import org.apache.beam.sdk.testing.TestPipeline;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.calcite.util.Pair;
+import org.junit.Rule;
+
+/**
+ * Base class for all built-in functions integration tests.
+ */
+public class BeamSqlBuiltinFunctionsIntegrationTestBase {
+  private static final Map JAVA_CLASS_TO_SQL_TYPE = new 
HashMap<>();
+  static {
+JAVA_CLASS_TO_SQL_TYPE.put(Byte.class, Types.TINYINT);
+JAVA_CLASS_TO_SQL_TYPE.put(Short.class, Types.SMALLINT);
+JAVA_CLASS_TO_SQL_TYPE.put(Integer.class, Types.INTEGER);
+JAVA_CLASS_TO_SQL_TYPE.put(Long.class, Types.BIGINT);
+JAVA_CLASS_TO_SQL_TYPE.put(Float.class, Types.FLOAT);
+JAVA_CLASS_TO_SQL_TYPE.put(Double.class, Types.DOUBLE);
+JAVA_CLASS_TO_SQL_TYPE.put(BigDecimal.class, Types.DECIMAL);
+JAVA_CLASS_TO_SQL_TYPE.put(String.class, Types.VARCHAR);
+JAVA_CLASS_TO_SQL_TYPE.put(Date.class, Types.DATE);
+JAVA_CLASS_TO_SQL_TYPE.put(Boolean.class, Types.BOOLEAN);
+  }
+
+  @Rule
+  public final TestPipeline pipeline = TestPipeline.create();
+
+  protected PCollection getTestPCollection() {
+BeamSqlRowType type = BeamSqlRowType.create(
+Arrays.asList("ts", "c_tinyint", "c_smallint",
+"c_integer", "c_bigint", "c_float", "c_double", "c_decimal",
+"c_tinyint_max", "c_smallint_max", "c_integer_max", 
"c_bigint_max"),
+Arrays.asList(Types.DATE, Types.TINYINT, Types.SMALLINT,
+Types.INTEGER, Types.BIGINT, Types.FLOAT, Types.DOUBLE, 
Types.DECIMAL,
+Types.TINYINT, Types.SMALLINT, Types.INTEGER, Types.BIGINT)
+);
+try {
+  return MockedBoundedTable
+  .of(type)
+  .addRows(
+  parseDate("1986-02-15 11:35:26"),
+  (byte) 1,
+  (short) 1,
+  1,
+  1L,
+  1.0f,
+  1.0,
+  BigDecimal.ONE,
+  (byte) 127,
+  (short) 32767,
+  2147483647,
+  9223372036854775807L
+  )
+  .buildIOReader(pipeline)
+  .setCoder(new BeamSqlRowCoder(type));
+} catch (Exception e) {
+  throw new RuntimeException(e);
+}
+  }
+
+  protected static Date parseDate(String str) {
+try {
+  SimpleDateFormat sdf = new SimpleDateFormat("-MM-dd HH:mm:ss");
+  sdf.setTimeZone(TimeZone.getTimeZone("GMT"));
+  return sdf.parse(str);
+} catch (Exception e) {
+  throw new RuntimeException(e);
+}
+  }
+
+
+  /**
+   * Helper class to make write integration test for built-in functions 

[53/59] beam git commit: move all implementation classes/packages into impl package

[takidau]

http://git-wip-us.apache.org/repos/asf/beam/blob/febd044a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/utils/CalciteUtils.java
--
diff --git 
a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/utils/CalciteUtils.java
 
b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/utils/CalciteUtils.java
new file mode 100644
index 000..b80e045
--- /dev/null
+++ 
b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/utils/CalciteUtils.java
@@ -0,0 +1,113 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.beam.sdk.extensions.sql.impl.utils;
+
+import java.sql.Types;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import org.apache.beam.sdk.extensions.sql.schema.BeamSqlRowType;
+import org.apache.calcite.rel.type.RelDataType;
+import org.apache.calcite.rel.type.RelDataTypeFactory;
+import org.apache.calcite.rel.type.RelDataTypeField;
+import org.apache.calcite.rel.type.RelProtoDataType;
+import org.apache.calcite.sql.type.SqlTypeName;
+
+/**
+ * Utility methods for Calcite related operations.
+ */
+public class CalciteUtils {
+  private static final Map JAVA_TO_CALCITE_MAPPING = new 
HashMap<>();
+  private static final Map CALCITE_TO_JAVA_MAPPING = new 
HashMap<>();
+  static {
+JAVA_TO_CALCITE_MAPPING.put(Types.TINYINT, SqlTypeName.TINYINT);
+JAVA_TO_CALCITE_MAPPING.put(Types.SMALLINT, SqlTypeName.SMALLINT);
+JAVA_TO_CALCITE_MAPPING.put(Types.INTEGER, SqlTypeName.INTEGER);
+JAVA_TO_CALCITE_MAPPING.put(Types.BIGINT, SqlTypeName.BIGINT);
+
+JAVA_TO_CALCITE_MAPPING.put(Types.FLOAT, SqlTypeName.FLOAT);
+JAVA_TO_CALCITE_MAPPING.put(Types.DOUBLE, SqlTypeName.DOUBLE);
+
+JAVA_TO_CALCITE_MAPPING.put(Types.DECIMAL, SqlTypeName.DECIMAL);
+
+JAVA_TO_CALCITE_MAPPING.put(Types.CHAR, SqlTypeName.CHAR);
+JAVA_TO_CALCITE_MAPPING.put(Types.VARCHAR, SqlTypeName.VARCHAR);
+
+JAVA_TO_CALCITE_MAPPING.put(Types.DATE, SqlTypeName.DATE);
+JAVA_TO_CALCITE_MAPPING.put(Types.TIME, SqlTypeName.TIME);
+JAVA_TO_CALCITE_MAPPING.put(Types.TIMESTAMP, SqlTypeName.TIMESTAMP);
+
+JAVA_TO_CALCITE_MAPPING.put(Types.BOOLEAN, SqlTypeName.BOOLEAN);
+
+for (Map.Entry pair : 
JAVA_TO_CALCITE_MAPPING.entrySet()) {
+  CALCITE_TO_JAVA_MAPPING.put(pair.getValue(), pair.getKey());
+}
+  }
+
+  /**
+   * Get the corresponding {@code SqlTypeName} for an integer sql type.
+   */
+  public static SqlTypeName toCalciteType(int type) {
+return JAVA_TO_CALCITE_MAPPING.get(type);
+  }
+
+  /**
+   * Get the integer sql type from Calcite {@code SqlTypeName}.
+   */
+  public static Integer toJavaType(SqlTypeName typeName) {
+return CALCITE_TO_JAVA_MAPPING.get(typeName);
+  }
+
+  /**
+   * Get the {@code SqlTypeName} for the specified column of a table.
+   */
+  public static SqlTypeName getFieldType(BeamSqlRowType schema, int index) {
+return toCalciteType(schema.getFieldsType().get(index));
+  }
+
+  /**
+   * Generate {@code BeamSqlRowType} from {@code RelDataType} which is used to 
create table.
+   */
+  public static BeamSqlRowType toBeamRowType(RelDataType tableInfo) {
+List fieldNames = new ArrayList<>();
+List fieldTypes = new ArrayList<>();
+for (RelDataTypeField f : tableInfo.getFieldList()) {
+  fieldNames.add(f.getName());
+  fieldTypes.add(toJavaType(f.getType().getSqlTypeName()));
+}
+return BeamSqlRowType.create(fieldNames, fieldTypes);
+  }
+
+  /**
+   * Create an instance of {@code RelDataType} so it can be used to create a 
table.
+   */
+  public static RelProtoDataType toCalciteRowType(final BeamSqlRowType that) {
+return new RelProtoDataType() {
+  @Override
+  public RelDataType apply(RelDataTypeFactory a) {
+RelDataTypeFactory.FieldInfoBuilder builder = a.builder();
+for (int idx = 0; idx < that.getFieldsName().size(); ++idx) {
+  builder.add(that.getFieldsName().get(idx), 
toCalciteType(that.getFieldsType().get(idx)));

[49/59] beam git commit: move all implementation classes/packages into impl package

[takidau]

http://git-wip-us.apache.org/repos/asf/beam/blob/febd044a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/rule/BeamMinusRule.java
--
diff --git 
a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/rule/BeamMinusRule.java
 
b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/rule/BeamMinusRule.java
deleted file mode 100644
index 363cf3b..000
--- 
a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/rule/BeamMinusRule.java
+++ /dev/null
@@ -1,50 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package org.apache.beam.sdk.extensions.sql.rule;
-
-import java.util.List;
-import org.apache.beam.sdk.extensions.sql.rel.BeamLogicalConvention;
-import org.apache.beam.sdk.extensions.sql.rel.BeamMinusRel;
-import org.apache.calcite.plan.Convention;
-import org.apache.calcite.rel.RelNode;
-import org.apache.calcite.rel.convert.ConverterRule;
-import org.apache.calcite.rel.core.Minus;
-import org.apache.calcite.rel.logical.LogicalMinus;
-
-/**
- * {@code ConverterRule} to replace {@code Minus} with {@code BeamMinusRel}.
- */
-public class BeamMinusRule extends ConverterRule {
-  public static final BeamMinusRule INSTANCE = new BeamMinusRule();
-  private BeamMinusRule() {
-super(LogicalMinus.class, Convention.NONE,
-BeamLogicalConvention.INSTANCE, "BeamMinusRule");
-  }
-
-  @Override public RelNode convert(RelNode rel) {
-Minus minus = (Minus) rel;
-final List inputs = minus.getInputs();
-return new BeamMinusRel(
-minus.getCluster(),
-minus.getTraitSet().replace(BeamLogicalConvention.INSTANCE),
-convertList(inputs, BeamLogicalConvention.INSTANCE),
-minus.all
-);
-  }
-}

http://git-wip-us.apache.org/repos/asf/beam/blob/febd044a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/rule/BeamProjectRule.java
--
diff --git 
a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/rule/BeamProjectRule.java
 
b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/rule/BeamProjectRule.java
deleted file mode 100644
index 4f2f8c9..000
--- 
a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/rule/BeamProjectRule.java
+++ /dev/null
@@ -1,50 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- * 
- * http://www.apache.org/licenses/LICENSE-2.0
- * 
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.sdk.extensions.sql.rule;
-
-import org.apache.beam.sdk.extensions.sql.rel.BeamLogicalConvention;
-import org.apache.beam.sdk.extensions.sql.rel.BeamProjectRel;
-import org.apache.calcite.plan.Convention;
-import org.apache.calcite.rel.RelNode;
-import org.apache.calcite.rel.convert.ConverterRule;
-import org.apache.calcite.rel.core.Project;
-import org.apache.calcite.rel.logical.LogicalProject;
-
-/**
- * A {@code ConverterRule} to replace {@link Project} with
- * {@link BeamProjectRel}.
- *
- */
-public class BeamProjectRule extends ConverterRule {
-  public static final BeamProjectRule INSTANCE = new BeamProjectRule();
-
-  private BeamProjectRule() {
-super(LogicalProject.class, Convention.NONE, 
BeamLogicalConvention.INSTANCE, "BeamProjectRule");
-  }
-
-  @Override
-  public RelNode convert(RelNode rel) {
-final Project project = (Project) rel;
-final RelNode input = project.getInput();
-

[54/59] beam git commit: move all implementation classes/packages into impl package

[takidau]

http://git-wip-us.apache.org/repos/asf/beam/blob/febd044a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/package-info.java
--
diff --git 
a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/package-info.java
 
b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/package-info.java
new file mode 100644
index 000..76b335d
--- /dev/null
+++ 
b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/package-info.java
@@ -0,0 +1,23 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+/**
+ * BeamSQL specified nodes, to replace {@link org.apache.calcite.rel.RelNode}.
+ *
+ */
+package org.apache.beam.sdk.extensions.sql.impl.rel;

http://git-wip-us.apache.org/repos/asf/beam/blob/febd044a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rule/BeamAggregationRule.java
--
diff --git 
a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rule/BeamAggregationRule.java
 
b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rule/BeamAggregationRule.java
new file mode 100644
index 000..cdf6712
--- /dev/null
+++ 
b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rule/BeamAggregationRule.java
@@ -0,0 +1,162 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.impl.rule;
+
+import com.google.common.collect.ImmutableList;
+import java.util.GregorianCalendar;
+import java.util.List;
+import org.apache.beam.sdk.extensions.sql.impl.rel.BeamAggregationRel;
+import org.apache.beam.sdk.extensions.sql.impl.rel.BeamLogicalConvention;
+import org.apache.beam.sdk.transforms.windowing.AfterProcessingTime;
+import org.apache.beam.sdk.transforms.windowing.AfterWatermark;
+import org.apache.beam.sdk.transforms.windowing.FixedWindows;
+import org.apache.beam.sdk.transforms.windowing.GlobalWindows;
+import org.apache.beam.sdk.transforms.windowing.Repeatedly;
+import org.apache.beam.sdk.transforms.windowing.Sessions;
+import org.apache.beam.sdk.transforms.windowing.SlidingWindows;
+import org.apache.beam.sdk.transforms.windowing.Trigger;
+import org.apache.beam.sdk.transforms.windowing.WindowFn;
+import org.apache.calcite.plan.RelOptRule;
+import org.apache.calcite.plan.RelOptRuleCall;
+import org.apache.calcite.plan.RelOptRuleOperand;
+import org.apache.calcite.rel.core.Aggregate;
+import org.apache.calcite.rel.core.Project;
+import org.apache.calcite.rel.core.RelFactories;
+import org.apache.calcite.rex.RexCall;
+import org.apache.calcite.rex.RexLiteral;
+import org.apache.calcite.rex.RexNode;
+import org.apache.calcite.sql.SqlOperator;
+import org.apache.calcite.tools.RelBuilderFactory;
+import org.apache.calcite.util.ImmutableBitSet;
+import org.joda.time.Duration;
+
+/**
+ * Rule to detect the window/trigger settings.
+ *
+ */
+public class BeamAggregationRule extends RelOptRule {
+  public static final BeamAggregationRule INSTANCE =
+  new BeamAggregationRule(Aggregate.class, Project.class, 
RelFactories.LOGICAL_BUILDER);
+
+  public BeamAggregationRule(
+  Class aggregateClass,
+  Class projectClass,
+  RelBuilderFactory relBuilderFactory) {
+super(
+

[46/59] beam git commit: move all implementation classes/packages into impl package

[takidau]

http://git-wip-us.apache.org/repos/asf/beam/blob/febd044a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamJoinRelUnboundedVsBoundedTest.java
--
diff --git 
a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamJoinRelUnboundedVsBoundedTest.java
 
b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamJoinRelUnboundedVsBoundedTest.java
new file mode 100644
index 000..1dbd8b4
--- /dev/null
+++ 
b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamJoinRelUnboundedVsBoundedTest.java
@@ -0,0 +1,241 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.beam.sdk.extensions.sql.impl.rel;
+
+import java.sql.Types;
+import java.util.Date;
+import org.apache.beam.sdk.extensions.sql.BeamSqlCli;
+import org.apache.beam.sdk.extensions.sql.BeamSqlEnv;
+import org.apache.beam.sdk.extensions.sql.TestUtils;
+import 
org.apache.beam.sdk.extensions.sql.impl.transform.BeamSqlOutputToConsoleFn;
+import org.apache.beam.sdk.extensions.sql.mock.MockedBoundedTable;
+import org.apache.beam.sdk.extensions.sql.mock.MockedUnboundedTable;
+import org.apache.beam.sdk.extensions.sql.schema.BeamSqlRow;
+import org.apache.beam.sdk.testing.PAssert;
+import org.apache.beam.sdk.testing.TestPipeline;
+import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.values.PCollection;
+import org.joda.time.Duration;
+import org.junit.BeforeClass;
+import org.junit.Rule;
+import org.junit.Test;
+
+/**
+ * Unbounded + Unbounded Test for {@code BeamJoinRel}.
+ */
+public class BeamJoinRelUnboundedVsBoundedTest {
+  @Rule
+  public final TestPipeline pipeline = TestPipeline.create();
+  private static final BeamSqlEnv beamSqlEnv = new BeamSqlEnv();
+  public static final Date FIRST_DATE = new Date(1);
+  public static final Date SECOND_DATE = new Date(1 + 3600 * 1000);
+  public static final Date THIRD_DATE = new Date(1 + 3600 * 1000 + 3600 * 1000 
+ 1);
+  private static final Duration WINDOW_SIZE = Duration.standardHours(1);
+
+  @BeforeClass
+  public static void prepare() {
+beamSqlEnv.registerTable("ORDER_DETAILS", MockedUnboundedTable
+.of(
+Types.INTEGER, "order_id",
+Types.INTEGER, "site_id",
+Types.INTEGER, "price",
+Types.TIMESTAMP, "order_time"
+)
+.timestampColumnIndex(3)
+.addRows(
+Duration.ZERO,
+1, 1, 1, FIRST_DATE,
+1, 2, 2, FIRST_DATE
+)
+.addRows(
+WINDOW_SIZE.plus(Duration.standardSeconds(1)),
+2, 2, 3, SECOND_DATE,
+2, 3, 3, SECOND_DATE,
+// this late data is omitted
+1, 2, 3, FIRST_DATE
+)
+.addRows(
+WINDOW_SIZE.plus(WINDOW_SIZE).plus(Duration.standardSeconds(1)),
+3, 3, 3, THIRD_DATE,
+// this late data is omitted
+2, 2, 3, SECOND_DATE
+)
+);
+
+beamSqlEnv.registerTable("ORDER_DETAILS1", MockedBoundedTable
+.of(Types.INTEGER, "order_id",
+Types.VARCHAR, "buyer"
+).addRows(
+1, "james",
+2, "bond"
+));
+  }
+
+  @Test
+  public void testInnerJoin_unboundedTableOnTheLeftSide() throws Exception {
+String sql = "SELECT o1.order_id, o1.sum_site_id, o2.buyer FROM "
++ "(select order_id, sum(site_id) as sum_site_id FROM ORDER_DETAILS "
++ "  GROUP BY order_id, TUMBLE(order_time, INTERVAL '1' HOUR)) 
o1 "
++ " JOIN "
++ " ORDER_DETAILS1 o2 "
++ " on "
++ " o1.order_id=o2.order_id"
+;
+
+PCollection rows = BeamSqlCli.compilePipeline(sql, pipeline, 
beamSqlEnv);
+PAssert.that(rows.apply(ParDo.of(new TestUtils.BeamSqlRow2StringDoFn(
+.containsInAnyOrder(
+TestUtils.RowsBuilder.of(
+Types.INTEGER, "order_id",
+Types.INTEGER, "sum_site_id",
+Types.VARCHAR, "buyer"
+).addRows(
+1, 3, "james",
+2, 5, "bond"
+

[48/59] beam git commit: move all implementation classes/packages into impl package

[takidau]

http://git-wip-us.apache.org/repos/asf/beam/blob/febd044a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/BeamSqlFnExecutorTest.java
--
diff --git 
a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/BeamSqlFnExecutorTest.java
 
b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/BeamSqlFnExecutorTest.java
new file mode 100644
index 000..f350087
--- /dev/null
+++ 
b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/BeamSqlFnExecutorTest.java
@@ -0,0 +1,416 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.impl.interpreter;
+
+import static org.junit.Assert.assertTrue;
+
+import java.math.BigDecimal;
+import java.util.Arrays;
+import java.util.Calendar;
+import java.util.Date;
+import java.util.TimeZone;
+import 
org.apache.beam.sdk.extensions.sql.impl.interpreter.operator.BeamSqlCaseExpression;
+import 
org.apache.beam.sdk.extensions.sql.impl.interpreter.operator.BeamSqlExpression;
+import 
org.apache.beam.sdk.extensions.sql.impl.interpreter.operator.BeamSqlInputRefExpression;
+import 
org.apache.beam.sdk.extensions.sql.impl.interpreter.operator.BeamSqlPrimitive;
+import 
org.apache.beam.sdk.extensions.sql.impl.interpreter.operator.arithmetic.BeamSqlDivideExpression;
+import 
org.apache.beam.sdk.extensions.sql.impl.interpreter.operator.arithmetic.BeamSqlMinusExpression;
+import 
org.apache.beam.sdk.extensions.sql.impl.interpreter.operator.arithmetic.BeamSqlModExpression;
+import 
org.apache.beam.sdk.extensions.sql.impl.interpreter.operator.arithmetic.BeamSqlMultiplyExpression;
+import 
org.apache.beam.sdk.extensions.sql.impl.interpreter.operator.arithmetic.BeamSqlPlusExpression;
+import 
org.apache.beam.sdk.extensions.sql.impl.interpreter.operator.comparison.BeamSqlEqualsExpression;
+import 
org.apache.beam.sdk.extensions.sql.impl.interpreter.operator.comparison.BeamSqlLessThanOrEqualsExpression;
+import 
org.apache.beam.sdk.extensions.sql.impl.interpreter.operator.date.BeamSqlCurrentDateExpression;
+import 
org.apache.beam.sdk.extensions.sql.impl.interpreter.operator.date.BeamSqlCurrentTimeExpression;
+import 
org.apache.beam.sdk.extensions.sql.impl.interpreter.operator.date.BeamSqlCurrentTimestampExpression;
+import 
org.apache.beam.sdk.extensions.sql.impl.interpreter.operator.date.BeamSqlDateCeilExpression;
+import 
org.apache.beam.sdk.extensions.sql.impl.interpreter.operator.date.BeamSqlDateFloorExpression;
+import 
org.apache.beam.sdk.extensions.sql.impl.interpreter.operator.date.BeamSqlExtractExpression;
+import 
org.apache.beam.sdk.extensions.sql.impl.interpreter.operator.logical.BeamSqlAndExpression;
+import 
org.apache.beam.sdk.extensions.sql.impl.interpreter.operator.logical.BeamSqlNotExpression;
+import 
org.apache.beam.sdk.extensions.sql.impl.interpreter.operator.logical.BeamSqlOrExpression;
+import 
org.apache.beam.sdk.extensions.sql.impl.interpreter.operator.string.BeamSqlCharLengthExpression;
+import 
org.apache.beam.sdk.extensions.sql.impl.interpreter.operator.string.BeamSqlConcatExpression;
+import 
org.apache.beam.sdk.extensions.sql.impl.interpreter.operator.string.BeamSqlInitCapExpression;
+import 
org.apache.beam.sdk.extensions.sql.impl.interpreter.operator.string.BeamSqlLowerExpression;
+import 
org.apache.beam.sdk.extensions.sql.impl.interpreter.operator.string.BeamSqlOverlayExpression;
+import 
org.apache.beam.sdk.extensions.sql.impl.interpreter.operator.string.BeamSqlPositionExpression;
+import 
org.apache.beam.sdk.extensions.sql.impl.interpreter.operator.string.BeamSqlSubstringExpression;
+import 
org.apache.beam.sdk.extensions.sql.impl.interpreter.operator.string.BeamSqlTrimExpression;
+import 
org.apache.beam.sdk.extensions.sql.impl.interpreter.operator.string.BeamSqlUpperExpression;
+import org.apache.beam.sdk.extensions.sql.impl.planner.BeamQueryPlanner;
+import org.apache.beam.sdk.extensions.sql.impl.rel.BeamFilterRel;
+import org.apache.beam.sdk.extensions.sql.impl.rel.BeamProjectRel;
+import 

[42/59] beam git commit: rename package org.apache.beam.dsls.sql to org.apache.beam.sdk.extensions.sql

[takidau]

rename package org.apache.beam.dsls.sql to org.apache.beam.sdk.extensions.sql


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/c1b5482d
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/c1b5482d
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/c1b5482d

Branch: refs/heads/DSL_SQL
Commit: c1b5482d3fb13a14926f0ffc23d0810b3105ed24
Parents: ba493f8
Author: James Xu 
Authored: Sun Jul 30 23:58:02 2017 +0800
Committer: James Xu 
Committed: Sun Jul 30 23:58:02 2017 +0800

--
 .../java/org/apache/beam/dsls/sql/BeamSql.java  | 244 --
 .../org/apache/beam/dsls/sql/BeamSqlCli.java|  65 ---
 .../org/apache/beam/dsls/sql/BeamSqlEnv.java| 120 -
 .../beam/dsls/sql/example/BeamSqlExample.java   |  97 
 .../beam/dsls/sql/example/package-info.java |  23 -
 .../interpreter/BeamSqlExpressionExecutor.java  |  43 --
 .../dsls/sql/interpreter/BeamSqlFnExecutor.java | 442 --
 .../operator/BeamSqlCaseExpression.java |  64 ---
 .../operator/BeamSqlCastExpression.java | 132 --
 .../interpreter/operator/BeamSqlExpression.java |  78 
 .../operator/BeamSqlInputRefExpression.java |  43 --
 .../interpreter/operator/BeamSqlPrimitive.java  | 152 ---
 .../operator/BeamSqlReinterpretExpression.java  |  55 ---
 .../operator/BeamSqlUdfExpression.java  |  86 
 .../operator/BeamSqlWindowEndExpression.java|  42 --
 .../operator/BeamSqlWindowExpression.java   |  50 --
 .../operator/BeamSqlWindowStartExpression.java  |  43 --
 .../arithmetic/BeamSqlArithmeticExpression.java | 122 -
 .../arithmetic/BeamSqlDivideExpression.java |  37 --
 .../arithmetic/BeamSqlMinusExpression.java  |  36 --
 .../arithmetic/BeamSqlModExpression.java|  36 --
 .../arithmetic/BeamSqlMultiplyExpression.java   |  36 --
 .../arithmetic/BeamSqlPlusExpression.java   |  36 --
 .../operator/arithmetic/package-info.java   |  22 -
 .../comparison/BeamSqlCompareExpression.java|  96 
 .../comparison/BeamSqlEqualsExpression.java |  49 --
 .../BeamSqlGreaterThanExpression.java   |  49 --
 .../BeamSqlGreaterThanOrEqualsExpression.java   |  49 --
 .../comparison/BeamSqlIsNotNullExpression.java  |  53 ---
 .../comparison/BeamSqlIsNullExpression.java |  53 ---
 .../comparison/BeamSqlLessThanExpression.java   |  49 --
 .../BeamSqlLessThanOrEqualsExpression.java  |  49 --
 .../comparison/BeamSqlNotEqualsExpression.java  |  49 --
 .../operator/comparison/package-info.java   |  22 -
 .../date/BeamSqlCurrentDateExpression.java  |  45 --
 .../date/BeamSqlCurrentTimeExpression.java  |  53 ---
 .../date/BeamSqlCurrentTimestampExpression.java |  49 --
 .../date/BeamSqlDateCeilExpression.java |  55 ---
 .../date/BeamSqlDateFloorExpression.java|  55 ---
 .../operator/date/BeamSqlExtractExpression.java | 101 -
 .../interpreter/operator/date/package-info.java |  22 -
 .../operator/logical/BeamSqlAndExpression.java  |  48 --
 .../logical/BeamSqlLogicalExpression.java   |  47 --
 .../operator/logical/BeamSqlNotExpression.java  |  54 ---
 .../operator/logical/BeamSqlOrExpression.java   |  48 --
 .../operator/logical/package-info.java  |  22 -
 .../operator/math/BeamSqlAbsExpression.java |  74 ---
 .../operator/math/BeamSqlAcosExpression.java|  41 --
 .../operator/math/BeamSqlAsinExpression.java|  41 --
 .../operator/math/BeamSqlAtan2Expression.java   |  43 --
 .../operator/math/BeamSqlAtanExpression.java|  41 --
 .../operator/math/BeamSqlCeilExpression.java|  46 --
 .../operator/math/BeamSqlCosExpression.java |  41 --
 .../operator/math/BeamSqlCotExpression.java |  41 --
 .../operator/math/BeamSqlDegreesExpression.java |  41 --
 .../operator/math/BeamSqlExpExpression.java |  41 --
 .../operator/math/BeamSqlFloorExpression.java   |  46 --
 .../operator/math/BeamSqlLnExpression.java  |  41 --
 .../operator/math/BeamSqlLogExpression.java |  41 --
 .../math/BeamSqlMathBinaryExpression.java   |  64 ---
 .../math/BeamSqlMathUnaryExpression.java|  58 ---
 .../operator/math/BeamSqlPiExpression.java  |  42 --
 .../operator/math/BeamSqlPowerExpression.java   |  45 --
 .../operator/math/BeamSqlRadiansExpression.java |  41 --
 .../operator/math/BeamSqlRandExpression.java|  54 ---
 .../math/BeamSqlRandIntegerExpression.java  |  58 ---
 .../operator/math/BeamSqlRoundExpression.java   | 108 -
 .../operator/math/BeamSqlSignExpression.java|  72 ---
 .../operator/math/BeamSqlSinExpression.java |  41 --
 .../operator/math/BeamSqlTanExpression.java |  41 --
 .../math/BeamSqlTruncateExpression.java |  76 
 .../interpreter/operator/math/package-info.java |  22 -
 .../sql/interpreter/operator/package-info.java  |  22 -
 

[40/59] beam git commit: rename package org.apache.beam.dsls.sql to org.apache.beam.sdk.extensions.sql

[takidau]

http://git-wip-us.apache.org/repos/asf/beam/blob/c1b5482d/sdks/java/extensions/sql/src/main/java/org/apache/beam/dsls/sql/interpreter/operator/logical/BeamSqlOrExpression.java
--
diff --git 
a/sdks/java/extensions/sql/src/main/java/org/apache/beam/dsls/sql/interpreter/operator/logical/BeamSqlOrExpression.java
 
b/sdks/java/extensions/sql/src/main/java/org/apache/beam/dsls/sql/interpreter/operator/logical/BeamSqlOrExpression.java
deleted file mode 100644
index 450638c..000
--- 
a/sdks/java/extensions/sql/src/main/java/org/apache/beam/dsls/sql/interpreter/operator/logical/BeamSqlOrExpression.java
+++ /dev/null
@@ -1,48 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.dsls.sql.interpreter.operator.logical;
-
-import java.util.List;
-
-import org.apache.beam.dsls.sql.interpreter.operator.BeamSqlExpression;
-import org.apache.beam.dsls.sql.interpreter.operator.BeamSqlPrimitive;
-import org.apache.beam.dsls.sql.schema.BeamSqlRow;
-import org.apache.calcite.sql.type.SqlTypeName;
-
-/**
- * {@code BeamSqlExpression} for 'OR' operation.
- */
-public class BeamSqlOrExpression extends BeamSqlLogicalExpression {
-  public BeamSqlOrExpression(List operands) {
-super(operands);
-  }
-
-  @Override
-  public BeamSqlPrimitive evaluate(BeamSqlRow inputRow) {
-boolean result = false;
-for (BeamSqlExpression exp : operands) {
-  BeamSqlPrimitive expOut = exp.evaluate(inputRow);
-result = result || expOut.getValue();
-if (result) {
-  break;
-}
-}
-return BeamSqlPrimitive.of(SqlTypeName.BOOLEAN, result);
-  }
-
-}

http://git-wip-us.apache.org/repos/asf/beam/blob/c1b5482d/sdks/java/extensions/sql/src/main/java/org/apache/beam/dsls/sql/interpreter/operator/logical/package-info.java
--
diff --git 
a/sdks/java/extensions/sql/src/main/java/org/apache/beam/dsls/sql/interpreter/operator/logical/package-info.java
 
b/sdks/java/extensions/sql/src/main/java/org/apache/beam/dsls/sql/interpreter/operator/logical/package-info.java
deleted file mode 100644
index 7862045..000
--- 
a/sdks/java/extensions/sql/src/main/java/org/apache/beam/dsls/sql/interpreter/operator/logical/package-info.java
+++ /dev/null
@@ -1,22 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-/**
- * Logical operators.
- */
-package org.apache.beam.dsls.sql.interpreter.operator.logical;

http://git-wip-us.apache.org/repos/asf/beam/blob/c1b5482d/sdks/java/extensions/sql/src/main/java/org/apache/beam/dsls/sql/interpreter/operator/math/BeamSqlAbsExpression.java
--
diff --git 
a/sdks/java/extensions/sql/src/main/java/org/apache/beam/dsls/sql/interpreter/operator/math/BeamSqlAbsExpression.java
 
b/sdks/java/extensions/sql/src/main/java/org/apache/beam/dsls/sql/interpreter/operator/math/BeamSqlAbsExpression.java
deleted file mode 100644
index e563634..000
--- 
a/sdks/java/extensions/sql/src/main/java/org/apache/beam/dsls/sql/interpreter/operator/math/BeamSqlAbsExpression.java
+++ /dev/null
@@ -1,74 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses 

[07/59] beam git commit: move dsls/sql to sdks/java/extensions/sql

[takidau]

http://git-wip-us.apache.org/repos/asf/beam/blob/ba493f85/sdks/java/extensions/sql/src/main/java/org/apache/beam/dsls/sql/rel/BeamSortRel.java
--
diff --git 
a/sdks/java/extensions/sql/src/main/java/org/apache/beam/dsls/sql/rel/BeamSortRel.java
 
b/sdks/java/extensions/sql/src/main/java/org/apache/beam/dsls/sql/rel/BeamSortRel.java
new file mode 100644
index 000..ba344df
--- /dev/null
+++ 
b/sdks/java/extensions/sql/src/main/java/org/apache/beam/dsls/sql/rel/BeamSortRel.java
@@ -0,0 +1,247 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.beam.dsls.sql.rel;
+
+import java.io.Serializable;
+import java.lang.reflect.Type;
+import java.math.BigDecimal;
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.List;
+import org.apache.beam.dsls.sql.BeamSqlEnv;
+import org.apache.beam.dsls.sql.schema.BeamSqlRow;
+import org.apache.beam.dsls.sql.schema.BeamSqlRowCoder;
+import org.apache.beam.dsls.sql.utils.CalciteUtils;
+import org.apache.beam.sdk.coders.ListCoder;
+import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.transforms.Flatten;
+import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.transforms.Top;
+import org.apache.beam.sdk.transforms.windowing.GlobalWindow;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.PCollectionTuple;
+import org.apache.calcite.plan.RelOptCluster;
+import org.apache.calcite.plan.RelTraitSet;
+import org.apache.calcite.rel.RelCollation;
+import org.apache.calcite.rel.RelCollationImpl;
+import org.apache.calcite.rel.RelFieldCollation;
+import org.apache.calcite.rel.RelNode;
+import org.apache.calcite.rel.core.Sort;
+import org.apache.calcite.rex.RexInputRef;
+import org.apache.calcite.rex.RexLiteral;
+import org.apache.calcite.rex.RexNode;
+import org.apache.calcite.sql.type.SqlTypeName;
+
+/**
+ * {@code BeamRelNode} to replace a {@code Sort} node.
+ *
+ * Since Beam does not fully supported global sort we are using {@link Top} 
to implement
+ * the {@code Sort} algebra. The following types of ORDER BY are supported:
+
+ * {@code
+ * select * from t order by id desc limit 10;
+ * select * from t order by id desc limit 10, 5;
+ * }
+ *
+ * but Order BY without a limit is NOT supported:
+ *
+ * {@code
+ *   select * from t order by id desc
+ * }
+ *
+ * Constraints
+ * 
+ *   Due to the constraints of {@link Top}, the result of a `ORDER BY 
LIMIT`
+ *   must fit into the memory of a single machine.
+ *   Since `WINDOW`(HOP, TUMBLE, SESSION etc) is always associated with 
`GroupBy`,
+ *   it does not make much sense to use `ORDER BY` with `WINDOW`.
+ *   
+ * 
+ */
+public class BeamSortRel extends Sort implements BeamRelNode {
+  private List fieldIndices = new ArrayList<>();
+  private List orientation = new ArrayList<>();
+  private List nullsFirst = new ArrayList<>();
+
+  private int startIndex = 0;
+  private int count;
+
+  public BeamSortRel(
+  RelOptCluster cluster,
+  RelTraitSet traits,
+  RelNode child,
+  RelCollation collation,
+  RexNode offset,
+  RexNode fetch) {
+super(cluster, traits, child, collation, offset, fetch);
+
+List fieldExps = getChildExps();
+RelCollationImpl collationImpl = (RelCollationImpl) collation;
+List collations = collationImpl.getFieldCollations();
+for (int i = 0; i < fieldExps.size(); i++) {
+  RexNode fieldExp = fieldExps.get(i);
+  RexInputRef inputRef = (RexInputRef) fieldExp;
+  fieldIndices.add(inputRef.getIndex());
+  orientation.add(collations.get(i).getDirection() == 
RelFieldCollation.Direction.ASCENDING);
+
+  RelFieldCollation.NullDirection rawNullDirection = 
collations.get(i).nullDirection;
+  if (rawNullDirection == RelFieldCollation.NullDirection.UNSPECIFIED) {
+rawNullDirection = 
collations.get(i).getDirection().defaultNullDirection();
+  }
+  nullsFirst.add(rawNullDirection == 
RelFieldCollation.NullDirection.FIRST);
+}
+
+if (fetch == null) {
+  throw new UnsupportedOperationException("ORDER BY without a LIMIT is not 
supported!");
+}
+
+RexLiteral fetchLiteral = 

[16/59] beam git commit: move dsls/sql to sdks/java/extensions/sql

[takidau]

http://git-wip-us.apache.org/repos/asf/beam/blob/ba493f85/dsls/sql/src/main/java/org/apache/beam/dsls/sql/schema/BeamSqlTable.java
--
diff --git 
a/dsls/sql/src/main/java/org/apache/beam/dsls/sql/schema/BeamSqlTable.java 
b/dsls/sql/src/main/java/org/apache/beam/dsls/sql/schema/BeamSqlTable.java
deleted file mode 100644
index d419473..000
--- a/dsls/sql/src/main/java/org/apache/beam/dsls/sql/schema/BeamSqlTable.java
+++ /dev/null
@@ -1,52 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package org.apache.beam.dsls.sql.schema;
-
-import org.apache.beam.sdk.Pipeline;
-import org.apache.beam.sdk.transforms.PTransform;
-import org.apache.beam.sdk.values.PCollection;
-import org.apache.beam.sdk.values.PDone;
-
-/**
- * This interface defines a Beam Sql Table.
- */
-public interface BeamSqlTable {
-  /**
-   * In Beam SQL, there's no difference between a batch query and a streaming
-   * query. {@link BeamIOType} is used to validate the sources.
-   */
-  BeamIOType getSourceType();
-
-  /**
-   * create a {@code PCollection} from source.
-   *
-   */
-  PCollection buildIOReader(Pipeline pipeline);
-
-  /**
-   * create a {@code IO.write()} instance to write to target.
-   *
-   */
-   PTransform, PDone> buildIOWriter();
-
-  /**
-   * Get the schema info of the table.
-   */
-   BeamSqlRowType getRowType();
-}

http://git-wip-us.apache.org/repos/asf/beam/blob/ba493f85/dsls/sql/src/main/java/org/apache/beam/dsls/sql/schema/BeamSqlUdaf.java
--
diff --git 
a/dsls/sql/src/main/java/org/apache/beam/dsls/sql/schema/BeamSqlUdaf.java 
b/dsls/sql/src/main/java/org/apache/beam/dsls/sql/schema/BeamSqlUdaf.java
deleted file mode 100644
index 9582ffa..000
--- a/dsls/sql/src/main/java/org/apache/beam/dsls/sql/schema/BeamSqlUdaf.java
+++ /dev/null
@@ -1,72 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.dsls.sql.schema;
-
-import java.io.Serializable;
-import java.lang.reflect.ParameterizedType;
-import org.apache.beam.sdk.coders.CannotProvideCoderException;
-import org.apache.beam.sdk.coders.Coder;
-import org.apache.beam.sdk.coders.CoderRegistry;
-import org.apache.beam.sdk.transforms.Combine.CombineFn;
-
-/**
- * abstract class of aggregation functions in Beam SQL.
- *
- * There're several constrains for a UDAF:
- * 1. A constructor with an empty argument list is required;
- * 2. The type of {@code InputT} and {@code OutputT} can only be 
Interger/Long/Short/Byte/Double
- * /Float/Date/BigDecimal, mapping as SQL type 
INTEGER/BIGINT/SMALLINT/TINYINE/DOUBLE/FLOAT
- * /TIMESTAMP/DECIMAL;
- * 3. Keep intermediate data in {@code AccumT}, and do not rely on elements in 
class;
- */
-public abstract class BeamSqlUdaf implements 
Serializable {
-  public BeamSqlUdaf(){}
-
-  /**
-   * create an initial aggregation object, equals to {@link 
CombineFn#createAccumulator()}.
-   */
-  public abstract AccumT init();
-
-  /**
-   * add an input value, equals to {@link CombineFn#addInput(Object, Object)}.
-   */
-  public abstract AccumT add(AccumT accumulator, InputT input);
-
-  /**
-   * merge aggregation objects from parallel tasks, equals to
-   *  {@link CombineFn#mergeAccumulators(Iterable)}.
-   */
-  public abstract AccumT merge(Iterable accumulators);
-
-  /**
-   * extract output value from 

[14/59] beam git commit: move dsls/sql to sdks/java/extensions/sql

[takidau]

http://git-wip-us.apache.org/repos/asf/beam/blob/ba493f85/dsls/sql/src/test/java/org/apache/beam/dsls/sql/integrationtest/BeamSqlConditionalFunctionsIntegrationTest.java
--
diff --git 
a/dsls/sql/src/test/java/org/apache/beam/dsls/sql/integrationtest/BeamSqlConditionalFunctionsIntegrationTest.java
 
b/dsls/sql/src/test/java/org/apache/beam/dsls/sql/integrationtest/BeamSqlConditionalFunctionsIntegrationTest.java
deleted file mode 100644
index 6233aeb..000
--- 
a/dsls/sql/src/test/java/org/apache/beam/dsls/sql/integrationtest/BeamSqlConditionalFunctionsIntegrationTest.java
+++ /dev/null
@@ -1,60 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.dsls.sql.integrationtest;
-
-import org.junit.Test;
-
-/**
- * Integration test for conditional functions.
- */
-public class BeamSqlConditionalFunctionsIntegrationTest
-extends BeamSqlBuiltinFunctionsIntegrationTestBase {
-@Test
-public void testConditionalFunctions() throws Exception {
-  ExpressionChecker checker = new ExpressionChecker()
-  .addExpr(
-  "CASE 1 WHEN 1 THEN 'hello' ELSE 'world' END",
-  "hello"
-  )
-  .addExpr(
-  "CASE 2 "
-  + "WHEN 1 THEN 'hello' "
-  + "WHEN 3 THEN 'bond' "
-  + "ELSE 'world' END",
-  "world"
-  )
-  .addExpr(
-  "CASE "
-  + "WHEN 1 = 1 THEN 'hello' "
-  + "ELSE 'world' END",
-  "hello"
-  )
-  .addExpr(
-  "CASE "
-  + "WHEN 1 > 1 THEN 'hello' "
-  + "ELSE 'world' END",
-  "world"
-  )
-  .addExpr("NULLIF(5, 4) ", 5)
-  .addExpr("COALESCE(1, 5) ", 1)
-  .addExpr("COALESCE(NULL, 5) ", 5)
-  ;
-
-  checker.buildRunAndCheck();
-}
-}

http://git-wip-us.apache.org/repos/asf/beam/blob/ba493f85/dsls/sql/src/test/java/org/apache/beam/dsls/sql/integrationtest/BeamSqlDateFunctionsIntegrationTest.java
--
diff --git 
a/dsls/sql/src/test/java/org/apache/beam/dsls/sql/integrationtest/BeamSqlDateFunctionsIntegrationTest.java
 
b/dsls/sql/src/test/java/org/apache/beam/dsls/sql/integrationtest/BeamSqlDateFunctionsIntegrationTest.java
deleted file mode 100644
index bd0d3ba..000
--- 
a/dsls/sql/src/test/java/org/apache/beam/dsls/sql/integrationtest/BeamSqlDateFunctionsIntegrationTest.java
+++ /dev/null
@@ -1,88 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package org.apache.beam.dsls.sql.integrationtest;
-
-import static org.junit.Assert.assertFalse;
-import static org.junit.Assert.assertTrue;
-
-import java.util.Date;
-import java.util.Iterator;
-import org.apache.beam.dsls.sql.BeamSql;
-import org.apache.beam.dsls.sql.schema.BeamSqlRow;
-import org.apache.beam.sdk.testing.PAssert;
-import org.apache.beam.sdk.transforms.SerializableFunction;
-import org.apache.beam.sdk.values.PCollection;
-import org.junit.Test;
-
-/**
- * Integration test for date functions.
- */
-public class BeamSqlDateFunctionsIntegrationTest
-extends BeamSqlBuiltinFunctionsIntegrationTestBase {
-  @Test public void testDateTimeFunctions() throws Exception {
-ExpressionChecker checker = new ExpressionChecker()
-

[56/59] beam git commit: move all implementation classes/packages into impl package

[takidau]

http://git-wip-us.apache.org/repos/asf/beam/blob/febd044a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/math/BeamSqlCotExpression.java
--
diff --git 
a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/math/BeamSqlCotExpression.java
 
b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/math/BeamSqlCotExpression.java
new file mode 100644
index 000..68d56b5
--- /dev/null
+++ 
b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/math/BeamSqlCotExpression.java
@@ -0,0 +1,40 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.beam.sdk.extensions.sql.impl.interpreter.operator.math;
+
+import java.util.List;
+import 
org.apache.beam.sdk.extensions.sql.impl.interpreter.operator.BeamSqlExpression;
+import 
org.apache.beam.sdk.extensions.sql.impl.interpreter.operator.BeamSqlPrimitive;
+import org.apache.calcite.runtime.SqlFunctions;
+import org.apache.calcite.sql.type.SqlTypeName;
+
+/**
+ * {@code BeamSqlMathUnaryExpression} for 'COT' function.
+ */
+public class BeamSqlCotExpression extends BeamSqlMathUnaryExpression {
+
+  public BeamSqlCotExpression(List operands) {
+super(operands, SqlTypeName.DOUBLE);
+  }
+
+  @Override public BeamSqlPrimitive calculate(BeamSqlPrimitive op) {
+return BeamSqlPrimitive
+.of(SqlTypeName.DOUBLE, 
SqlFunctions.cot(SqlFunctions.toDouble(op.getValue(;
+  }
+}

http://git-wip-us.apache.org/repos/asf/beam/blob/febd044a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/math/BeamSqlDegreesExpression.java
--
diff --git 
a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/math/BeamSqlDegreesExpression.java
 
b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/math/BeamSqlDegreesExpression.java
new file mode 100644
index 000..de4eac2
--- /dev/null
+++ 
b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/math/BeamSqlDegreesExpression.java
@@ -0,0 +1,40 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.beam.sdk.extensions.sql.impl.interpreter.operator.math;
+
+import java.util.List;
+import 
org.apache.beam.sdk.extensions.sql.impl.interpreter.operator.BeamSqlExpression;
+import 
org.apache.beam.sdk.extensions.sql.impl.interpreter.operator.BeamSqlPrimitive;
+import org.apache.calcite.runtime.SqlFunctions;
+import org.apache.calcite.sql.type.SqlTypeName;
+
+/**
+ * {@code BeamSqlMathUnaryExpression} for 'DEGREES' function.
+ */
+public class BeamSqlDegreesExpression extends BeamSqlMathUnaryExpression {
+
+  public BeamSqlDegreesExpression(List operands) {
+super(operands, SqlTypeName.DOUBLE);
+  }
+
+  @Override public BeamSqlPrimitive calculate(BeamSqlPrimitive op) {
+return BeamSqlPrimitive
+.of(SqlTypeName.DOUBLE, 
SqlFunctions.degrees(SqlFunctions.toDouble(op.getValue(;
+  }
+}

[59/59] beam git commit: [BEAM-2441] This closes #3666

[takidau]

[BEAM-2441] This closes #3666


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/10962a34
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/10962a34
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/10962a34

Branch: refs/heads/DSL_SQL
Commit: 10962a34d3727f5ee2ee8df030847765e8eaab63
Parents: d32aea9 febd044
Author: Tyler Akidau 
Authored: Tue Aug 1 22:06:25 2017 -0700
Committer: Tyler Akidau 
Committed: Tue Aug 1 22:06:25 2017 -0700

--
 dsls/pom.xml|  60 ---
 dsls/sql/pom.xml| 226 -
 .../java/org/apache/beam/dsls/sql/BeamSql.java  | 244 --
 .../org/apache/beam/dsls/sql/BeamSqlCli.java|  65 ---
 .../org/apache/beam/dsls/sql/BeamSqlEnv.java| 120 -
 .../beam/dsls/sql/example/BeamSqlExample.java   |  97 
 .../beam/dsls/sql/example/package-info.java |  23 -
 .../interpreter/BeamSqlExpressionExecutor.java  |  43 --
 .../dsls/sql/interpreter/BeamSqlFnExecutor.java | 442 --
 .../operator/BeamSqlCaseExpression.java |  64 ---
 .../operator/BeamSqlCastExpression.java | 132 --
 .../interpreter/operator/BeamSqlExpression.java |  78 
 .../operator/BeamSqlInputRefExpression.java |  43 --
 .../interpreter/operator/BeamSqlPrimitive.java  | 152 ---
 .../operator/BeamSqlReinterpretExpression.java  |  55 ---
 .../operator/BeamSqlUdfExpression.java  |  86 
 .../operator/BeamSqlWindowEndExpression.java|  42 --
 .../operator/BeamSqlWindowExpression.java   |  50 --
 .../operator/BeamSqlWindowStartExpression.java  |  43 --
 .../arithmetic/BeamSqlArithmeticExpression.java | 122 -
 .../arithmetic/BeamSqlDivideExpression.java |  37 --
 .../arithmetic/BeamSqlMinusExpression.java  |  36 --
 .../arithmetic/BeamSqlModExpression.java|  36 --
 .../arithmetic/BeamSqlMultiplyExpression.java   |  36 --
 .../arithmetic/BeamSqlPlusExpression.java   |  36 --
 .../operator/arithmetic/package-info.java   |  22 -
 .../comparison/BeamSqlCompareExpression.java|  96 
 .../comparison/BeamSqlEqualsExpression.java |  49 --
 .../BeamSqlGreaterThanExpression.java   |  49 --
 .../BeamSqlGreaterThanOrEqualsExpression.java   |  49 --
 .../comparison/BeamSqlIsNotNullExpression.java  |  53 ---
 .../comparison/BeamSqlIsNullExpression.java |  53 ---
 .../comparison/BeamSqlLessThanExpression.java   |  49 --
 .../BeamSqlLessThanOrEqualsExpression.java  |  49 --
 .../comparison/BeamSqlNotEqualsExpression.java  |  49 --
 .../operator/comparison/package-info.java   |  22 -
 .../date/BeamSqlCurrentDateExpression.java  |  45 --
 .../date/BeamSqlCurrentTimeExpression.java  |  53 ---
 .../date/BeamSqlCurrentTimestampExpression.java |  49 --
 .../date/BeamSqlDateCeilExpression.java |  55 ---
 .../date/BeamSqlDateFloorExpression.java|  55 ---
 .../operator/date/BeamSqlExtractExpression.java | 101 -
 .../interpreter/operator/date/package-info.java |  22 -
 .../operator/logical/BeamSqlAndExpression.java  |  48 --
 .../logical/BeamSqlLogicalExpression.java   |  47 --
 .../operator/logical/BeamSqlNotExpression.java  |  54 ---
 .../operator/logical/BeamSqlOrExpression.java   |  48 --
 .../operator/logical/package-info.java  |  22 -
 .../operator/math/BeamSqlAbsExpression.java |  74 ---
 .../operator/math/BeamSqlAcosExpression.java|  41 --
 .../operator/math/BeamSqlAsinExpression.java|  41 --
 .../operator/math/BeamSqlAtan2Expression.java   |  43 --
 .../operator/math/BeamSqlAtanExpression.java|  41 --
 .../operator/math/BeamSqlCeilExpression.java|  46 --
 .../operator/math/BeamSqlCosExpression.java |  41 --
 .../operator/math/BeamSqlCotExpression.java |  41 --
 .../operator/math/BeamSqlDegreesExpression.java |  41 --
 .../operator/math/BeamSqlExpExpression.java |  41 --
 .../operator/math/BeamSqlFloorExpression.java   |  46 --
 .../operator/math/BeamSqlLnExpression.java  |  41 --
 .../operator/math/BeamSqlLogExpression.java |  41 --
 .../math/BeamSqlMathBinaryExpression.java   |  64 ---
 .../math/BeamSqlMathUnaryExpression.java|  58 ---
 .../operator/math/BeamSqlPiExpression.java  |  42 --
 .../operator/math/BeamSqlPowerExpression.java   |  45 --
 .../operator/math/BeamSqlRadiansExpression.java |  41 --
 .../operator/math/BeamSqlRandExpression.java|  54 ---
 .../math/BeamSqlRandIntegerExpression.java  |  58 ---
 .../operator/math/BeamSqlRoundExpression.java   | 108 -
 .../operator/math/BeamSqlSignExpression.java|  72 ---
 .../operator/math/BeamSqlSinExpression.java |  41 --
 .../operator/math/BeamSqlTanExpression.java |  41 --
 .../math/BeamSqlTruncateExpression.java |  76 
 .../interpreter/operator/math/package-info.java |  22 -
 

[47/59] beam git commit: move all implementation classes/packages into impl package

[takidau]

http://git-wip-us.apache.org/repos/asf/beam/blob/febd044a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/date/BeamSqlExtractExpressionTest.java
--
diff --git 
a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/date/BeamSqlExtractExpressionTest.java
 
b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/date/BeamSqlExtractExpressionTest.java
new file mode 100644
index 000..0ca7e3e
--- /dev/null
+++ 
b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/date/BeamSqlExtractExpressionTest.java
@@ -0,0 +1,103 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.beam.sdk.extensions.sql.impl.interpreter.operator.date;
+
+import static org.junit.Assert.assertEquals;
+
+import java.util.ArrayList;
+import java.util.List;
+import 
org.apache.beam.sdk.extensions.sql.impl.interpreter.BeamSqlFnExecutorTestBase;
+import 
org.apache.beam.sdk.extensions.sql.impl.interpreter.operator.BeamSqlExpression;
+import 
org.apache.beam.sdk.extensions.sql.impl.interpreter.operator.BeamSqlPrimitive;
+import org.apache.calcite.avatica.util.TimeUnitRange;
+import org.apache.calcite.sql.type.SqlTypeName;
+import org.junit.Test;
+
+/**
+ * Test for {@code BeamSqlExtractExpression}.
+ */
+public class BeamSqlExtractExpressionTest extends 
BeamSqlDateExpressionTestBase {
+  @Test public void evaluate() throws Exception {
+List operands = new ArrayList<>();
+long time = str2LongTime("2017-05-22 16:17:18");
+
+// YEAR
+operands.add(BeamSqlPrimitive.of(SqlTypeName.SYMBOL, TimeUnitRange.YEAR));
+operands.add(BeamSqlPrimitive.of(SqlTypeName.BIGINT,
+time));
+assertEquals(2017L,
+new BeamSqlExtractExpression(operands)
+.evaluate(BeamSqlFnExecutorTestBase.record).getValue());
+
+// MONTH
+operands.clear();
+operands.add(BeamSqlPrimitive.of(SqlTypeName.SYMBOL, TimeUnitRange.MONTH));
+operands.add(BeamSqlPrimitive.of(SqlTypeName.BIGINT,
+time));
+assertEquals(5L,
+new BeamSqlExtractExpression(operands)
+.evaluate(BeamSqlFnExecutorTestBase.record).getValue());
+
+// DAY
+operands.clear();
+operands.add(BeamSqlPrimitive.of(SqlTypeName.SYMBOL, TimeUnitRange.DAY));
+operands.add(BeamSqlPrimitive.of(SqlTypeName.BIGINT,
+time));
+assertEquals(22L,
+new BeamSqlExtractExpression(operands)
+.evaluate(BeamSqlFnExecutorTestBase.record).getValue());
+
+// DAY_OF_WEEK
+operands.clear();
+operands.add(BeamSqlPrimitive.of(SqlTypeName.SYMBOL, TimeUnitRange.DOW));
+operands.add(BeamSqlPrimitive.of(SqlTypeName.BIGINT,
+time));
+assertEquals(2L,
+new BeamSqlExtractExpression(operands)
+.evaluate(BeamSqlFnExecutorTestBase.record).getValue());
+
+// DAY_OF_YEAR
+operands.clear();
+operands.add(BeamSqlPrimitive.of(SqlTypeName.SYMBOL, TimeUnitRange.DOY));
+operands.add(BeamSqlPrimitive.of(SqlTypeName.BIGINT,
+time));
+assertEquals(142L,
+new BeamSqlExtractExpression(operands)
+.evaluate(BeamSqlFnExecutorTestBase.record).getValue());
+
+// WEEK
+operands.clear();
+operands.add(BeamSqlPrimitive.of(SqlTypeName.SYMBOL, TimeUnitRange.WEEK));
+operands.add(BeamSqlPrimitive.of(SqlTypeName.BIGINT,
+time));
+assertEquals(21L,
+new BeamSqlExtractExpression(operands)
+.evaluate(BeamSqlFnExecutorTestBase.record).getValue());
+
+// QUARTER
+operands.clear();
+operands.add(BeamSqlPrimitive.of(SqlTypeName.SYMBOL, 
TimeUnitRange.QUARTER));
+operands.add(BeamSqlPrimitive.of(SqlTypeName.BIGINT,
+time));
+assertEquals(2L,
+new BeamSqlExtractExpression(operands)
+.evaluate(BeamSqlFnExecutorTestBase.record).getValue());
+
+  }
+}

[22/59] beam git commit: rename package org.apache.beam.dsls.sql to org.apache.beam.sdk.extensions.sql

[takidau]

http://git-wip-us.apache.org/repos/asf/beam/blob/c1b5482d/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/rel/BeamValuesRelTest.java
--
diff --git 
a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/rel/BeamValuesRelTest.java
 
b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/rel/BeamValuesRelTest.java
new file mode 100644
index 000..ace1a3e
--- /dev/null
+++ 
b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/rel/BeamValuesRelTest.java
@@ -0,0 +1,105 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.beam.sdk.extensions.sql.rel;
+
+import java.sql.Types;
+import org.apache.beam.sdk.extensions.sql.BeamSqlCli;
+import org.apache.beam.sdk.extensions.sql.BeamSqlEnv;
+import org.apache.beam.sdk.extensions.sql.TestUtils;
+import org.apache.beam.sdk.extensions.sql.mock.MockedBoundedTable;
+import org.apache.beam.sdk.extensions.sql.schema.BeamSqlRow;
+import org.apache.beam.sdk.testing.PAssert;
+import org.apache.beam.sdk.testing.TestPipeline;
+import org.apache.beam.sdk.values.PCollection;
+import org.junit.BeforeClass;
+import org.junit.Rule;
+import org.junit.Test;
+
+/**
+ * Test for {@code BeamValuesRel}.
+ */
+public class BeamValuesRelTest {
+  static BeamSqlEnv sqlEnv = new BeamSqlEnv();
+
+  @Rule
+  public final TestPipeline pipeline = TestPipeline.create();
+
+  @BeforeClass
+  public static void prepare() {
+sqlEnv.registerTable("string_table",
+MockedBoundedTable.of(
+Types.VARCHAR, "name",
+Types.VARCHAR, "description"
+)
+);
+sqlEnv.registerTable("int_table",
+MockedBoundedTable.of(
+Types.INTEGER, "c0",
+Types.INTEGER, "c1"
+)
+);
+  }
+
+  @Test
+  public void testValues() throws Exception {
+String sql = "insert into string_table(name, description) values "
++ "('hello', 'world'), ('james', 'bond')";
+PCollection rows = BeamSqlCli.compilePipeline(sql, pipeline, 
sqlEnv);
+PAssert.that(rows).containsInAnyOrder(
+TestUtils.RowsBuilder.of(
+Types.VARCHAR, "name",
+Types.VARCHAR, "description"
+).addRows(
+"hello", "world",
+"james", "bond"
+).getRows()
+);
+pipeline.run();
+  }
+
+  @Test
+  public void testValues_castInt() throws Exception {
+String sql = "insert into int_table (c0, c1) values(cast(1 as int), cast(2 
as int))";
+PCollection rows = BeamSqlCli.compilePipeline(sql, pipeline, 
sqlEnv);
+PAssert.that(rows).containsInAnyOrder(
+TestUtils.RowsBuilder.of(
+Types.INTEGER, "c0",
+Types.INTEGER, "c1"
+).addRows(
+1, 2
+).getRows()
+);
+pipeline.run();
+  }
+
+  @Test
+  public void testValues_onlySelect() throws Exception {
+String sql = "select 1, '1'";
+PCollection rows = BeamSqlCli.compilePipeline(sql, pipeline, 
sqlEnv);
+PAssert.that(rows).containsInAnyOrder(
+TestUtils.RowsBuilder.of(
+Types.INTEGER, "EXPR$0",
+Types.CHAR, "EXPR$1"
+).addRows(
+1, "1"
+).getRows()
+);
+pipeline.run();
+  }
+}

http://git-wip-us.apache.org/repos/asf/beam/blob/c1b5482d/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/rel/CheckSize.java
--
diff --git 
a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/rel/CheckSize.java
 
b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/rel/CheckSize.java
new file mode 100644
index 000..f369076
--- /dev/null
+++ 
b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/rel/CheckSize.java
@@ -0,0 +1,41 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 

[15/59] beam git commit: move dsls/sql to sdks/java/extensions/sql

[takidau]

http://git-wip-us.apache.org/repos/asf/beam/blob/ba493f85/dsls/sql/src/main/java/org/apache/beam/dsls/sql/utils/package-info.java
--
diff --git 
a/dsls/sql/src/main/java/org/apache/beam/dsls/sql/utils/package-info.java 
b/dsls/sql/src/main/java/org/apache/beam/dsls/sql/utils/package-info.java
deleted file mode 100644
index b5c861a..000
--- a/dsls/sql/src/main/java/org/apache/beam/dsls/sql/utils/package-info.java
+++ /dev/null
@@ -1,22 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-/**
- * Utility classes.
- */
-package org.apache.beam.dsls.sql.utils;

http://git-wip-us.apache.org/repos/asf/beam/blob/ba493f85/dsls/sql/src/main/resources/log4j.properties
--
diff --git a/dsls/sql/src/main/resources/log4j.properties 
b/dsls/sql/src/main/resources/log4j.properties
deleted file mode 100644
index 709484b..000
--- a/dsls/sql/src/main/resources/log4j.properties
+++ /dev/null
@@ -1,23 +0,0 @@
-
-#  Licensed to the Apache Software Foundation (ASF) under one
-#  or more contributor license agreements.  See the NOTICE file
-#  distributed with this work for additional information
-#  regarding copyright ownership.  The ASF licenses this file
-#  to you under the Apache License, Version 2.0 (the
-#  "License"); you may not use this file except in compliance
-#  with the License.  You may obtain a copy of the License at
-#
-#  http://www.apache.org/licenses/LICENSE-2.0
-#
-#  Unless required by applicable law or agreed to in writing, software
-#  distributed under the License is distributed on an "AS IS" BASIS,
-#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-#  See the License for the specific language governing permissions and
-# limitations under the License.
-
-
-log4j.rootLogger=ERROR,console
-log4j.appender.console=org.apache.log4j.ConsoleAppender
-log4j.appender.console.target=System.err
-log4j.appender.console.layout=org.apache.log4j.PatternLayout
-log4j.appender.console.layout.ConversionPattern=%d{yy/MM/dd HH:mm:ss} %p 
%c{2}: %m%n
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/beam/blob/ba493f85/dsls/sql/src/test/java/org/apache/beam/dsls/sql/BeamSqlApiSurfaceTest.java
--
diff --git 
a/dsls/sql/src/test/java/org/apache/beam/dsls/sql/BeamSqlApiSurfaceTest.java 
b/dsls/sql/src/test/java/org/apache/beam/dsls/sql/BeamSqlApiSurfaceTest.java
deleted file mode 100644
index 922931c..000
--- a/dsls/sql/src/test/java/org/apache/beam/dsls/sql/BeamSqlApiSurfaceTest.java
+++ /dev/null
@@ -1,59 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package org.apache.beam.dsls.sql;
-
-import static org.apache.beam.sdk.util.ApiSurface.containsOnlyPackages;
-import static org.junit.Assert.assertThat;
-
-import com.google.common.collect.ImmutableSet;
-import java.util.Set;
-import org.apache.beam.sdk.util.ApiSurface;
-import org.junit.Test;
-import org.junit.runner.RunWith;
-import org.junit.runners.JUnit4;
-
-/**
- * Surface test for BeamSql api.
- */
-@RunWith(JUnit4.class)
-public class BeamSqlApiSurfaceTest {
-  @Test
-  public void testSdkApiSurface() throws 

[17/59] beam git commit: move dsls/sql to sdks/java/extensions/sql

[takidau]

http://git-wip-us.apache.org/repos/asf/beam/blob/ba493f85/dsls/sql/src/main/java/org/apache/beam/dsls/sql/rel/BeamRelNode.java
--
diff --git 
a/dsls/sql/src/main/java/org/apache/beam/dsls/sql/rel/BeamRelNode.java 
b/dsls/sql/src/main/java/org/apache/beam/dsls/sql/rel/BeamRelNode.java
deleted file mode 100644
index d4c98a3..000
--- a/dsls/sql/src/main/java/org/apache/beam/dsls/sql/rel/BeamRelNode.java
+++ /dev/null
@@ -1,38 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.dsls.sql.rel;
-
-import org.apache.beam.dsls.sql.BeamSqlEnv;
-import org.apache.beam.dsls.sql.schema.BeamSqlRow;
-import org.apache.beam.sdk.values.PCollection;
-import org.apache.beam.sdk.values.PCollectionTuple;
-import org.apache.calcite.rel.RelNode;
-
-/**
- * A new method {@link #buildBeamPipeline(PCollectionTuple, BeamSqlEnv)} is 
added.
- */
-public interface BeamRelNode extends RelNode {
-
-  /**
-   * A {@link BeamRelNode} is a recursive structure, the
-   * {@code BeamQueryPlanner} visits it with a DFS(Depth-First-Search)
-   * algorithm.
-   */
-  PCollection buildBeamPipeline(PCollectionTuple 
inputPCollections, BeamSqlEnv sqlEnv)
-  throws Exception;
-}

http://git-wip-us.apache.org/repos/asf/beam/blob/ba493f85/dsls/sql/src/main/java/org/apache/beam/dsls/sql/rel/BeamSetOperatorRelBase.java
--
diff --git 
a/dsls/sql/src/main/java/org/apache/beam/dsls/sql/rel/BeamSetOperatorRelBase.java
 
b/dsls/sql/src/main/java/org/apache/beam/dsls/sql/rel/BeamSetOperatorRelBase.java
deleted file mode 100644
index 939c9c8..000
--- 
a/dsls/sql/src/main/java/org/apache/beam/dsls/sql/rel/BeamSetOperatorRelBase.java
+++ /dev/null
@@ -1,98 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package org.apache.beam.dsls.sql.rel;
-
-import java.io.Serializable;
-import java.util.List;
-import org.apache.beam.dsls.sql.BeamSqlEnv;
-import org.apache.beam.dsls.sql.schema.BeamSqlRow;
-import org.apache.beam.dsls.sql.transform.BeamSetOperatorsTransforms;
-import org.apache.beam.sdk.transforms.MapElements;
-import org.apache.beam.sdk.transforms.ParDo;
-import org.apache.beam.sdk.transforms.join.CoGbkResult;
-import org.apache.beam.sdk.transforms.join.CoGroupByKey;
-import org.apache.beam.sdk.transforms.join.KeyedPCollectionTuple;
-import org.apache.beam.sdk.transforms.windowing.WindowFn;
-import org.apache.beam.sdk.values.KV;
-import org.apache.beam.sdk.values.PCollection;
-import org.apache.beam.sdk.values.PCollectionTuple;
-import org.apache.beam.sdk.values.TupleTag;
-import org.apache.calcite.rel.RelNode;
-
-/**
- * Delegate for Set operators: {@code BeamUnionRel}, {@code BeamIntersectRel}
- * and {@code BeamMinusRel}.
- */
-public class BeamSetOperatorRelBase {
-  /**
-   * Set operator type.
-   */
-  public enum OpType implements Serializable {
-UNION,
-INTERSECT,
-MINUS
-  }
-
-  private BeamRelNode beamRelNode;
-  private List inputs;
-  private boolean all;
-  private OpType opType;
-
-  public BeamSetOperatorRelBase(BeamRelNode beamRelNode, OpType opType,
-  List inputs, boolean all) {
-this.beamRelNode = beamRelNode;
-this.opType = opType;
-this.inputs = inputs;
-this.all = all;
-  }
-
-  public PCollection buildBeamPipeline(PCollectionTuple 
inputPCollections
-  , BeamSqlEnv sqlEnv) throws Exception {
-   

[09/59] beam git commit: move dsls/sql to sdks/java/extensions/sql

[takidau]

http://git-wip-us.apache.org/repos/asf/beam/blob/ba493f85/sdks/java/extensions/sql/src/main/java/org/apache/beam/dsls/sql/interpreter/operator/logical/BeamSqlLogicalExpression.java
--
diff --git 
a/sdks/java/extensions/sql/src/main/java/org/apache/beam/dsls/sql/interpreter/operator/logical/BeamSqlLogicalExpression.java
 
b/sdks/java/extensions/sql/src/main/java/org/apache/beam/dsls/sql/interpreter/operator/logical/BeamSqlLogicalExpression.java
new file mode 100644
index 000..c9ff186
--- /dev/null
+++ 
b/sdks/java/extensions/sql/src/main/java/org/apache/beam/dsls/sql/interpreter/operator/logical/BeamSqlLogicalExpression.java
@@ -0,0 +1,47 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.beam.dsls.sql.interpreter.operator.logical;
+
+import java.util.List;
+
+import org.apache.beam.dsls.sql.interpreter.operator.BeamSqlExpression;
+import org.apache.calcite.sql.type.SqlTypeName;
+
+/**
+ * {@code BeamSqlExpression} for Logical operators.
+ */
+public abstract class BeamSqlLogicalExpression extends BeamSqlExpression {
+  private BeamSqlLogicalExpression(List operands, 
SqlTypeName outputType) {
+super(operands, outputType);
+  }
+  public BeamSqlLogicalExpression(List operands) {
+this(operands, SqlTypeName.BOOLEAN);
+  }
+
+  @Override
+  public boolean accept() {
+for (BeamSqlExpression exp : operands) {
+  // only accept BOOLEAN expression as operand
+  if (!exp.getOutputType().equals(SqlTypeName.BOOLEAN)) {
+return false;
+  }
+}
+return true;
+  }
+}

http://git-wip-us.apache.org/repos/asf/beam/blob/ba493f85/sdks/java/extensions/sql/src/main/java/org/apache/beam/dsls/sql/interpreter/operator/logical/BeamSqlNotExpression.java
--
diff --git 
a/sdks/java/extensions/sql/src/main/java/org/apache/beam/dsls/sql/interpreter/operator/logical/BeamSqlNotExpression.java
 
b/sdks/java/extensions/sql/src/main/java/org/apache/beam/dsls/sql/interpreter/operator/logical/BeamSqlNotExpression.java
new file mode 100644
index 000..6df52aa
--- /dev/null
+++ 
b/sdks/java/extensions/sql/src/main/java/org/apache/beam/dsls/sql/interpreter/operator/logical/BeamSqlNotExpression.java
@@ -0,0 +1,54 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.beam.dsls.sql.interpreter.operator.logical;
+
+import java.util.List;
+
+import org.apache.beam.dsls.sql.interpreter.operator.BeamSqlExpression;
+import org.apache.beam.dsls.sql.interpreter.operator.BeamSqlPrimitive;
+import org.apache.beam.dsls.sql.schema.BeamSqlRow;
+import org.apache.calcite.sql.type.SqlTypeName;
+
+/**
+ * {@code BeamSqlExpression} for logical operator: NOT.
+ *
+ * Whether boolean is not TRUE; returns UNKNOWN if boolean is UNKNOWN.
+ */
+public class BeamSqlNotExpression extends BeamSqlLogicalExpression {
+  public BeamSqlNotExpression(List operands) {
+super(operands);
+  }
+
+  @Override
+  public boolean accept() {
+if (numberOfOperands() != 1) {
+  return false;
+}
+return super.accept();
+  }
+
+  @Override public BeamSqlPrimitive evaluate(BeamSqlRow inputRow) {
+Boolean value = opValueEvaluated(0, inputRow);
+if (value == null) {
+  return BeamSqlPrimitive.of(SqlTypeName.BOOLEAN, null);
+} else {
+  return BeamSqlPrimitive.of(SqlTypeName.BOOLEAN, !value);
+}
+  }
+}

[35/59] beam git commit: rename package org.apache.beam.dsls.sql to org.apache.beam.sdk.extensions.sql

[takidau]

http://git-wip-us.apache.org/repos/asf/beam/blob/c1b5482d/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/interpreter/operator/arithmetic/BeamSqlModExpression.java
--
diff --git 
a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/interpreter/operator/arithmetic/BeamSqlModExpression.java
 
b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/interpreter/operator/arithmetic/BeamSqlModExpression.java
new file mode 100644
index 000..fc137da
--- /dev/null
+++ 
b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/interpreter/operator/arithmetic/BeamSqlModExpression.java
@@ -0,0 +1,36 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.beam.sdk.extensions.sql.interpreter.operator.arithmetic;
+
+import java.math.BigDecimal;
+import java.util.List;
+import 
org.apache.beam.sdk.extensions.sql.interpreter.operator.BeamSqlExpression;
+
+/**
+ * '%' operator.
+ */
+public class BeamSqlModExpression extends BeamSqlArithmeticExpression {
+  public BeamSqlModExpression(List operands) {
+super(operands, operands.get(1).getOutputType());
+  }
+
+  @Override protected BigDecimal calc(BigDecimal left, BigDecimal right) {
+return BigDecimal.valueOf(left.doubleValue() % right.doubleValue());
+  }
+}

http://git-wip-us.apache.org/repos/asf/beam/blob/c1b5482d/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/interpreter/operator/arithmetic/BeamSqlMultiplyExpression.java
--
diff --git 
a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/interpreter/operator/arithmetic/BeamSqlMultiplyExpression.java
 
b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/interpreter/operator/arithmetic/BeamSqlMultiplyExpression.java
new file mode 100644
index 000..7ea974c
--- /dev/null
+++ 
b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/interpreter/operator/arithmetic/BeamSqlMultiplyExpression.java
@@ -0,0 +1,36 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.beam.sdk.extensions.sql.interpreter.operator.arithmetic;
+
+import java.math.BigDecimal;
+import java.util.List;
+import 
org.apache.beam.sdk.extensions.sql.interpreter.operator.BeamSqlExpression;
+
+/**
+ * '*' operator.
+ */
+public class BeamSqlMultiplyExpression extends BeamSqlArithmeticExpression {
+  public BeamSqlMultiplyExpression(List operands) {
+super(operands);
+  }
+
+  @Override protected BigDecimal calc(BigDecimal left, BigDecimal right) {
+return left.multiply(right);
+  }
+}

http://git-wip-us.apache.org/repos/asf/beam/blob/c1b5482d/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/interpreter/operator/arithmetic/BeamSqlPlusExpression.java
--
diff --git 
a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/interpreter/operator/arithmetic/BeamSqlPlusExpression.java
 
b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/interpreter/operator/arithmetic/BeamSqlPlusExpression.java
new file mode 100644
index 000..3ce806f
--- /dev/null
+++ 

[02/59] beam git commit: move dsls/sql to sdks/java/extensions/sql

[takidau]

http://git-wip-us.apache.org/repos/asf/beam/blob/ba493f85/sdks/java/extensions/sql/src/test/java/org/apache/beam/dsls/sql/interpreter/operator/string/BeamSqlTrimExpressionTest.java
--
diff --git 
a/sdks/java/extensions/sql/src/test/java/org/apache/beam/dsls/sql/interpreter/operator/string/BeamSqlTrimExpressionTest.java
 
b/sdks/java/extensions/sql/src/test/java/org/apache/beam/dsls/sql/interpreter/operator/string/BeamSqlTrimExpressionTest.java
new file mode 100644
index 000..9ae9212
--- /dev/null
+++ 
b/sdks/java/extensions/sql/src/test/java/org/apache/beam/dsls/sql/interpreter/operator/string/BeamSqlTrimExpressionTest.java
@@ -0,0 +1,103 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.beam.dsls.sql.interpreter.operator.string;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertTrue;
+
+import java.util.ArrayList;
+import java.util.List;
+
+import org.apache.beam.dsls.sql.interpreter.BeamSqlFnExecutorTestBase;
+import org.apache.beam.dsls.sql.interpreter.operator.BeamSqlExpression;
+import org.apache.beam.dsls.sql.interpreter.operator.BeamSqlPrimitive;
+import org.apache.calcite.sql.fun.SqlTrimFunction;
+import org.apache.calcite.sql.type.SqlTypeName;
+import org.junit.Test;
+
+/**
+ * Test for BeamSqlTrimExpression.
+ */
+public class BeamSqlTrimExpressionTest extends BeamSqlFnExecutorTestBase {
+
+  @Test public void accept() throws Exception {
+List operands = new ArrayList<>();
+
+operands.add(BeamSqlPrimitive.of(SqlTypeName.VARCHAR, " hello "));
+assertTrue(new BeamSqlTrimExpression(operands).accept());
+
+operands.clear();
+operands.add(BeamSqlPrimitive.of(SqlTypeName.SYMBOL, 
SqlTrimFunction.Flag.BOTH));
+operands.add(BeamSqlPrimitive.of(SqlTypeName.VARCHAR, "he"));
+operands.add(BeamSqlPrimitive.of(SqlTypeName.VARCHAR, "hehe__hehe"));
+assertTrue(new BeamSqlTrimExpression(operands).accept());
+
+operands.clear();
+operands.add(BeamSqlPrimitive.of(SqlTypeName.VARCHAR, "he"));
+operands.add(BeamSqlPrimitive.of(SqlTypeName.VARCHAR, "hehe__hehe"));
+assertFalse(new BeamSqlTrimExpression(operands).accept());
+  }
+
+  @Test public void evaluate() throws Exception {
+List operands = new ArrayList<>();
+
+operands.add(BeamSqlPrimitive.of(SqlTypeName.SYMBOL, 
SqlTrimFunction.Flag.LEADING));
+operands.add(BeamSqlPrimitive.of(SqlTypeName.VARCHAR, "he"));
+operands.add(BeamSqlPrimitive.of(SqlTypeName.VARCHAR, "hehe__hehe"));
+assertEquals("__hehe",
+new BeamSqlTrimExpression(operands).evaluate(record).getValue());
+
+operands.clear();
+operands.add(BeamSqlPrimitive.of(SqlTypeName.SYMBOL, 
SqlTrimFunction.Flag.TRAILING));
+operands.add(BeamSqlPrimitive.of(SqlTypeName.VARCHAR, "he"));
+operands.add(BeamSqlPrimitive.of(SqlTypeName.VARCHAR, "hehe__hehe"));
+assertEquals("hehe__",
+new BeamSqlTrimExpression(operands).evaluate(record).getValue());
+
+operands.clear();
+operands.add(BeamSqlPrimitive.of(SqlTypeName.SYMBOL, 
SqlTrimFunction.Flag.BOTH));
+operands.add(BeamSqlPrimitive.of(SqlTypeName.VARCHAR, "he"));
+operands.add(BeamSqlPrimitive.of(SqlTypeName.VARCHAR, "__"));
+assertEquals("__",
+new BeamSqlTrimExpression(operands).evaluate(record).getValue());
+
+operands.clear();
+operands.add(BeamSqlPrimitive.of(SqlTypeName.VARCHAR, " hello "));
+assertEquals("hello",
+new BeamSqlTrimExpression(operands).evaluate(record).getValue());
+  }
+
+  @Test public void leadingTrim() throws Exception {
+assertEquals("__hehe",
+BeamSqlTrimExpression.leadingTrim("hehe__hehe", "he"));
+  }
+
+  @Test public void trailingTrim() throws Exception {
+assertEquals("hehe__",
+BeamSqlTrimExpression.trailingTrim("hehe__hehe", "he"));
+  }
+
+  @Test public void trim() throws Exception {
+assertEquals("__",
+BeamSqlTrimExpression.leadingTrim(
+BeamSqlTrimExpression.trailingTrim("hehe__hehe", "he"), "he"
+));
+  }
+}

[13/59] beam git commit: move dsls/sql to sdks/java/extensions/sql

[takidau]

http://git-wip-us.apache.org/repos/asf/beam/blob/ba493f85/dsls/sql/src/test/java/org/apache/beam/dsls/sql/interpreter/operator/arithmetic/BeamSqlArithmeticExpressionTest.java
--
diff --git 
a/dsls/sql/src/test/java/org/apache/beam/dsls/sql/interpreter/operator/arithmetic/BeamSqlArithmeticExpressionTest.java
 
b/dsls/sql/src/test/java/org/apache/beam/dsls/sql/interpreter/operator/arithmetic/BeamSqlArithmeticExpressionTest.java
deleted file mode 100644
index a34f109..000
--- 
a/dsls/sql/src/test/java/org/apache/beam/dsls/sql/interpreter/operator/arithmetic/BeamSqlArithmeticExpressionTest.java
+++ /dev/null
@@ -1,237 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package org.apache.beam.dsls.sql.interpreter.operator.arithmetic;
-
-import static org.junit.Assert.assertEquals;
-import static org.junit.Assert.assertFalse;
-import static org.junit.Assert.assertTrue;
-
-import java.util.ArrayList;
-import java.util.List;
-import org.apache.beam.dsls.sql.interpreter.BeamSqlFnExecutorTestBase;
-import org.apache.beam.dsls.sql.interpreter.operator.BeamSqlExpression;
-import org.apache.beam.dsls.sql.interpreter.operator.BeamSqlPrimitive;
-import org.apache.calcite.sql.type.SqlTypeName;
-import org.junit.Test;
-
-/**
- * Tests for {@code BeamSqlArithmeticExpression}.
- */
-public class BeamSqlArithmeticExpressionTest extends BeamSqlFnExecutorTestBase 
{
-
-  @Test public void testAccept_normal() {
-List operands = new ArrayList<>();
-
-// byte, short
-operands.add(BeamSqlPrimitive.of(SqlTypeName.TINYINT, Byte.valueOf("1")));
-operands.add(BeamSqlPrimitive.of(SqlTypeName.SMALLINT, Short.MAX_VALUE));
-assertTrue(new BeamSqlPlusExpression(operands).accept());
-
-// integer, long
-operands.clear();
-operands.add(BeamSqlPrimitive.of(SqlTypeName.INTEGER, 1));
-operands.add(BeamSqlPrimitive.of(SqlTypeName.BIGINT, 1L));
-assertTrue(new BeamSqlPlusExpression(operands).accept());
-
-// float, double
-operands.clear();
-operands.add(BeamSqlPrimitive.of(SqlTypeName.FLOAT, 1.1F));
-operands.add(BeamSqlPrimitive.of(SqlTypeName.DOUBLE, 1.1));
-assertTrue(new BeamSqlPlusExpression(operands).accept());
-
-// varchar
-operands.clear();
-operands.add(BeamSqlPrimitive.of(SqlTypeName.FLOAT, 1.1F));
-operands.add(BeamSqlPrimitive.of(SqlTypeName.VARCHAR, "1"));
-assertFalse(new BeamSqlPlusExpression(operands).accept());
-  }
-
-  @Test public void testAccept_exception() {
-List operands = new ArrayList<>();
-
-// more than 2 operands
-operands.add(BeamSqlPrimitive.of(SqlTypeName.TINYINT, Byte.valueOf("1")));
-operands.add(BeamSqlPrimitive.of(SqlTypeName.SMALLINT, Short.MAX_VALUE));
-operands.add(BeamSqlPrimitive.of(SqlTypeName.SMALLINT, Short.MAX_VALUE));
-assertFalse(new BeamSqlPlusExpression(operands).accept());
-
-// boolean
-operands.clear();
-operands.add(BeamSqlPrimitive.of(SqlTypeName.TINYINT, Byte.valueOf("1")));
-operands.add(BeamSqlPrimitive.of(SqlTypeName.BOOLEAN, true));
-assertFalse(new BeamSqlPlusExpression(operands).accept());
-  }
-
-  @Test public void testPlus() {
-List operands = new ArrayList<>();
-
-// integer + integer => integer
-operands.add(BeamSqlPrimitive.of(SqlTypeName.INTEGER, 1));
-operands.add(BeamSqlPrimitive.of(SqlTypeName.INTEGER, 1));
-assertEquals(2, new 
BeamSqlPlusExpression(operands).evaluate(record).getValue());
-
-// integer + long => long
-operands.clear();
-operands.add(BeamSqlPrimitive.of(SqlTypeName.INTEGER, 1));
-operands.add(BeamSqlPrimitive.of(SqlTypeName.BIGINT, 1L));
-assertEquals(2L, new 
BeamSqlPlusExpression(operands).evaluate(record).getValue());
-
-// long + long => long
-operands.clear();
-operands.add(BeamSqlPrimitive.of(SqlTypeName.BIGINT, 1L));
-operands.add(BeamSqlPrimitive.of(SqlTypeName.BIGINT, 1L));
-assertEquals(2L, new 
BeamSqlPlusExpression(operands).evaluate(record).getValue());
-
-// float + long => float
-operands.clear();
-operands.add(BeamSqlPrimitive.of(SqlTypeName.FLOAT, 1.1F));
-

[44/59] beam git commit: move all implementation classes/packages into impl package

[takidau]

http://git-wip-us.apache.org/repos/asf/beam/blob/febd044a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/interpreter/operator/string/BeamSqlOverlayExpressionTest.java
--
diff --git 
a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/interpreter/operator/string/BeamSqlOverlayExpressionTest.java
 
b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/interpreter/operator/string/BeamSqlOverlayExpressionTest.java
deleted file mode 100644
index 2ca0a98..000
--- 
a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/interpreter/operator/string/BeamSqlOverlayExpressionTest.java
+++ /dev/null
@@ -1,87 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package org.apache.beam.sdk.extensions.sql.interpreter.operator.string;
-
-import static org.junit.Assert.assertTrue;
-
-import java.util.ArrayList;
-import java.util.List;
-import 
org.apache.beam.sdk.extensions.sql.interpreter.BeamSqlFnExecutorTestBase;
-import 
org.apache.beam.sdk.extensions.sql.interpreter.operator.BeamSqlExpression;
-import 
org.apache.beam.sdk.extensions.sql.interpreter.operator.BeamSqlPrimitive;
-import org.apache.calcite.sql.type.SqlTypeName;
-import org.junit.Assert;
-import org.junit.Test;
-
-/**
- * Test for BeamSqlOverlayExpression.
- */
-public class BeamSqlOverlayExpressionTest extends BeamSqlFnExecutorTestBase {
-
-  @Test public void accept() throws Exception {
-List operands = new ArrayList<>();
-
-operands.add(BeamSqlPrimitive.of(SqlTypeName.VARCHAR, "hello"));
-operands.add(BeamSqlPrimitive.of(SqlTypeName.VARCHAR, "hello"));
-operands.add(BeamSqlPrimitive.of(SqlTypeName.INTEGER, 1));
-assertTrue(new BeamSqlOverlayExpression(operands).accept());
-
-operands.clear();
-operands.add(BeamSqlPrimitive.of(SqlTypeName.VARCHAR, "hello"));
-operands.add(BeamSqlPrimitive.of(SqlTypeName.VARCHAR, "hello"));
-operands.add(BeamSqlPrimitive.of(SqlTypeName.INTEGER, 1));
-operands.add(BeamSqlPrimitive.of(SqlTypeName.INTEGER, 2));
-assertTrue(new BeamSqlOverlayExpression(operands).accept());
-  }
-
-  @Test public void evaluate() throws Exception {
-List operands = new ArrayList<>();
-
-operands.add(BeamSqlPrimitive.of(SqlTypeName.VARCHAR, "w333rce"));
-operands.add(BeamSqlPrimitive.of(SqlTypeName.VARCHAR, "resou"));
-operands.add(BeamSqlPrimitive.of(SqlTypeName.INTEGER, 3));
-Assert.assertEquals("w3resou3rce",
-new BeamSqlOverlayExpression(operands).evaluate(record).getValue());
-
-operands.clear();
-operands.add(BeamSqlPrimitive.of(SqlTypeName.VARCHAR, "w333rce"));
-operands.add(BeamSqlPrimitive.of(SqlTypeName.VARCHAR, "resou"));
-operands.add(BeamSqlPrimitive.of(SqlTypeName.INTEGER, 3));
-operands.add(BeamSqlPrimitive.of(SqlTypeName.INTEGER, 4));
-Assert.assertEquals("w3resou33rce",
-new BeamSqlOverlayExpression(operands).evaluate(record).getValue());
-
-operands.clear();
-operands.add(BeamSqlPrimitive.of(SqlTypeName.VARCHAR, "w333rce"));
-operands.add(BeamSqlPrimitive.of(SqlTypeName.VARCHAR, "resou"));
-operands.add(BeamSqlPrimitive.of(SqlTypeName.INTEGER, 3));
-operands.add(BeamSqlPrimitive.of(SqlTypeName.INTEGER, 5));
-Assert.assertEquals("w3resou3rce",
-new BeamSqlOverlayExpression(operands).evaluate(record).getValue());
-
-operands.clear();
-operands.add(BeamSqlPrimitive.of(SqlTypeName.VARCHAR, "w333rce"));
-operands.add(BeamSqlPrimitive.of(SqlTypeName.VARCHAR, "resou"));
-operands.add(BeamSqlPrimitive.of(SqlTypeName.INTEGER, 3));
-operands.add(BeamSqlPrimitive.of(SqlTypeName.INTEGER, 7));
-Assert.assertEquals("w3resouce",
-new BeamSqlOverlayExpression(operands).evaluate(record).getValue());
-  }
-
-}

http://git-wip-us.apache.org/repos/asf/beam/blob/febd044a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/interpreter/operator/string/BeamSqlPositionExpressionTest.java
--
diff --git 

[03/59] beam git commit: move dsls/sql to sdks/java/extensions/sql

[takidau]

http://git-wip-us.apache.org/repos/asf/beam/blob/ba493f85/sdks/java/extensions/sql/src/test/java/org/apache/beam/dsls/sql/interpreter/operator/arithmetic/BeamSqlArithmeticExpressionTest.java
--
diff --git 
a/sdks/java/extensions/sql/src/test/java/org/apache/beam/dsls/sql/interpreter/operator/arithmetic/BeamSqlArithmeticExpressionTest.java
 
b/sdks/java/extensions/sql/src/test/java/org/apache/beam/dsls/sql/interpreter/operator/arithmetic/BeamSqlArithmeticExpressionTest.java
new file mode 100644
index 000..a34f109
--- /dev/null
+++ 
b/sdks/java/extensions/sql/src/test/java/org/apache/beam/dsls/sql/interpreter/operator/arithmetic/BeamSqlArithmeticExpressionTest.java
@@ -0,0 +1,237 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.beam.dsls.sql.interpreter.operator.arithmetic;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertTrue;
+
+import java.util.ArrayList;
+import java.util.List;
+import org.apache.beam.dsls.sql.interpreter.BeamSqlFnExecutorTestBase;
+import org.apache.beam.dsls.sql.interpreter.operator.BeamSqlExpression;
+import org.apache.beam.dsls.sql.interpreter.operator.BeamSqlPrimitive;
+import org.apache.calcite.sql.type.SqlTypeName;
+import org.junit.Test;
+
+/**
+ * Tests for {@code BeamSqlArithmeticExpression}.
+ */
+public class BeamSqlArithmeticExpressionTest extends BeamSqlFnExecutorTestBase 
{
+
+  @Test public void testAccept_normal() {
+List operands = new ArrayList<>();
+
+// byte, short
+operands.add(BeamSqlPrimitive.of(SqlTypeName.TINYINT, Byte.valueOf("1")));
+operands.add(BeamSqlPrimitive.of(SqlTypeName.SMALLINT, Short.MAX_VALUE));
+assertTrue(new BeamSqlPlusExpression(operands).accept());
+
+// integer, long
+operands.clear();
+operands.add(BeamSqlPrimitive.of(SqlTypeName.INTEGER, 1));
+operands.add(BeamSqlPrimitive.of(SqlTypeName.BIGINT, 1L));
+assertTrue(new BeamSqlPlusExpression(operands).accept());
+
+// float, double
+operands.clear();
+operands.add(BeamSqlPrimitive.of(SqlTypeName.FLOAT, 1.1F));
+operands.add(BeamSqlPrimitive.of(SqlTypeName.DOUBLE, 1.1));
+assertTrue(new BeamSqlPlusExpression(operands).accept());
+
+// varchar
+operands.clear();
+operands.add(BeamSqlPrimitive.of(SqlTypeName.FLOAT, 1.1F));
+operands.add(BeamSqlPrimitive.of(SqlTypeName.VARCHAR, "1"));
+assertFalse(new BeamSqlPlusExpression(operands).accept());
+  }
+
+  @Test public void testAccept_exception() {
+List operands = new ArrayList<>();
+
+// more than 2 operands
+operands.add(BeamSqlPrimitive.of(SqlTypeName.TINYINT, Byte.valueOf("1")));
+operands.add(BeamSqlPrimitive.of(SqlTypeName.SMALLINT, Short.MAX_VALUE));
+operands.add(BeamSqlPrimitive.of(SqlTypeName.SMALLINT, Short.MAX_VALUE));
+assertFalse(new BeamSqlPlusExpression(operands).accept());
+
+// boolean
+operands.clear();
+operands.add(BeamSqlPrimitive.of(SqlTypeName.TINYINT, Byte.valueOf("1")));
+operands.add(BeamSqlPrimitive.of(SqlTypeName.BOOLEAN, true));
+assertFalse(new BeamSqlPlusExpression(operands).accept());
+  }
+
+  @Test public void testPlus() {
+List operands = new ArrayList<>();
+
+// integer + integer => integer
+operands.add(BeamSqlPrimitive.of(SqlTypeName.INTEGER, 1));
+operands.add(BeamSqlPrimitive.of(SqlTypeName.INTEGER, 1));
+assertEquals(2, new 
BeamSqlPlusExpression(operands).evaluate(record).getValue());
+
+// integer + long => long
+operands.clear();
+operands.add(BeamSqlPrimitive.of(SqlTypeName.INTEGER, 1));
+operands.add(BeamSqlPrimitive.of(SqlTypeName.BIGINT, 1L));
+assertEquals(2L, new 
BeamSqlPlusExpression(operands).evaluate(record).getValue());
+
+// long + long => long
+operands.clear();
+operands.add(BeamSqlPrimitive.of(SqlTypeName.BIGINT, 1L));
+operands.add(BeamSqlPrimitive.of(SqlTypeName.BIGINT, 1L));
+assertEquals(2L, new 
BeamSqlPlusExpression(operands).evaluate(record).getValue());
+
+// float + long => float
+operands.clear();
+operands.add(BeamSqlPrimitive.of(SqlTypeName.FLOAT, 

[01/59] beam git commit: move dsls/sql to sdks/java/extensions/sql

[takidau]

Repository: beam
Updated Branches:
  refs/heads/DSL_SQL d32aea969 -> 10962a34d


http://git-wip-us.apache.org/repos/asf/beam/blob/ba493f85/sdks/java/extensions/sql/src/test/java/org/apache/beam/dsls/sql/schema/text/BeamTextCSVTableTest.java
--
diff --git 
a/sdks/java/extensions/sql/src/test/java/org/apache/beam/dsls/sql/schema/text/BeamTextCSVTableTest.java
 
b/sdks/java/extensions/sql/src/test/java/org/apache/beam/dsls/sql/schema/text/BeamTextCSVTableTest.java
new file mode 100644
index 000..b6e11e5
--- /dev/null
+++ 
b/sdks/java/extensions/sql/src/test/java/org/apache/beam/dsls/sql/schema/text/BeamTextCSVTableTest.java
@@ -0,0 +1,176 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.beam.dsls.sql.schema.text;
+
+import java.io.File;
+import java.io.FileOutputStream;
+import java.io.IOException;
+import java.io.OutputStream;
+import java.io.PrintStream;
+import java.nio.file.FileVisitResult;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.nio.file.SimpleFileVisitor;
+import java.nio.file.attribute.BasicFileAttributes;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import org.apache.beam.dsls.sql.planner.BeamQueryPlanner;
+import org.apache.beam.dsls.sql.schema.BeamSqlRow;
+import org.apache.beam.dsls.sql.schema.BeamSqlRowType;
+import org.apache.beam.dsls.sql.utils.CalciteUtils;
+import org.apache.beam.sdk.testing.PAssert;
+import org.apache.beam.sdk.testing.TestPipeline;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.calcite.rel.type.RelDataType;
+import org.apache.calcite.rel.type.RelDataTypeFactory;
+import org.apache.calcite.rel.type.RelProtoDataType;
+import org.apache.calcite.sql.type.SqlTypeName;
+import org.apache.commons.csv.CSVFormat;
+import org.apache.commons.csv.CSVPrinter;
+import org.junit.AfterClass;
+import org.junit.BeforeClass;
+import org.junit.Rule;
+import org.junit.Test;
+
+/**
+ * Tests for {@code BeamTextCSVTable}.
+ */
+public class BeamTextCSVTableTest {
+
+  @Rule public TestPipeline pipeline = TestPipeline.create();
+  @Rule public TestPipeline pipeline2 = TestPipeline.create();
+
+  /**
+   * testData.
+   *
+   * 
+   * The types of the csv fields are:
+   * integer,bigint,float,double,string
+   * 
+   */
+  private static Object[] data1 = new Object[] { 1, 1L, 1.1F, 1.1, "james" };
+  private static Object[] data2 = new Object[] { 2, 2L, 2.2F, 2.2, "bond" };
+
+  private static List testData = Arrays.asList(data1, data2);
+  private static List testDataRows = new ArrayList() {{
+for (Object[] data : testData) {
+  add(buildRow(data));
+}
+  }};
+
+  private static Path tempFolder;
+  private static File readerSourceFile;
+  private static File writerTargetFile;
+
+  @Test public void testBuildIOReader() {
+PCollection rows = new BeamTextCSVTable(buildBeamSqlRowType(),
+readerSourceFile.getAbsolutePath()).buildIOReader(pipeline);
+PAssert.that(rows).containsInAnyOrder(testDataRows);
+pipeline.run();
+  }
+
+  @Test public void testBuildIOWriter() {
+new BeamTextCSVTable(buildBeamSqlRowType(),
+readerSourceFile.getAbsolutePath()).buildIOReader(pipeline)
+.apply(new BeamTextCSVTable(buildBeamSqlRowType(), 
writerTargetFile.getAbsolutePath())
+.buildIOWriter());
+pipeline.run();
+
+PCollection rows = new BeamTextCSVTable(buildBeamSqlRowType(),
+writerTargetFile.getAbsolutePath()).buildIOReader(pipeline2);
+
+// confirm the two reads match
+PAssert.that(rows).containsInAnyOrder(testDataRows);
+pipeline2.run();
+  }
+
+  @BeforeClass public static void setUp() throws IOException {
+tempFolder = Files.createTempDirectory("BeamTextTableTest");
+readerSourceFile = writeToFile(testData, "readerSourceFile.txt");
+writerTargetFile = writeToFile(testData, "writerTargetFile.txt");
+  }
+
+  @AfterClass public static void teardownClass() throws IOException {
+Files.walkFileTree(tempFolder, new SimpleFileVisitor() {
+
+  @Override public FileVisitResult visitFile(Path file, 
BasicFileAttributes attrs)
+  throws IOException {
+

[jira] [Commented] (BEAM-2671) CreateStreamTest.testFirstElementLate validatesRunner test fails on Spark runner

[JIRA]

[ 
https://issues.apache.org/jira/browse/BEAM-2671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16110272#comment-16110272
 ] 

Jean-Baptiste Onofré commented on BEAM-2671:


I found the issue which cause the issue:

{code}
commit 935c077341de5804b29ffee3926795acf403
Author: Kenneth Knowles 
Date:   Thu Jun 22 18:43:39 2017 -0700

Process timer firings for a window together
{code}

The {{CreateStreamTest}} worked just before this commit and started to fail 
after this commit. I'm digging around that.

> CreateStreamTest.testFirstElementLate validatesRunner test fails on Spark 
> runner
> 
>
> Key: BEAM-2671
> URL: https://issues.apache.org/jira/browse/BEAM-2671
> Project: Beam
>  Issue Type: Bug
>  Components: runner-spark
>Reporter: Etienne Chauchot
>Assignee: Jean-Baptiste Onofré
> Fix For: 2.1.0, 2.2.0
>
>
> Error message:
> Flatten.Iterables/FlattenIterables/FlatMap/ParMultiDo(Anonymous).out0: 
> Expected: iterable over [] in any order
>  but: Not matched: "late"



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Build failed in Jenkins: beam_PostCommit_Python_Verify #2848

[Apache Jenkins Server]

See 


--
[...truncated 561.96 KB...]
test_par_do_with_multiple_outputs_and_using_yield 
(apache_beam.transforms.ptransform_test.PTransformTest) ... ok
:135:
 UserWarning: Using fallback coder for typehint: List[Any].
  warnings.warn('Using fallback coder for typehint: %r.' % typehint)
DEPRECATION: pip install --download has been deprecated and will be removed in 
the future. Pip now has a download command that should be used instead.
Collecting pyhamcrest (from -r postcommit_requirements.txt (line 1))
  File was already downloaded 
/tmp/dataflow-requirements-cache/PyHamcrest-1.9.0.tar.gz
Collecting mock (from -r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/mock-2.0.0.tar.gz
Collecting setuptools (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded 
/tmp/dataflow-requirements-cache/setuptools-36.2.7.zip
Collecting six (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded /tmp/dataflow-requirements-cache/six-1.10.0.tar.gz
Collecting funcsigs>=1 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded 
/tmp/dataflow-requirements-cache/funcsigs-1.0.2.tar.gz
Collecting pbr>=0.11 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/pbr-3.1.1.tar.gz
Successfully downloaded pyhamcrest mock setuptools six funcsigs pbr
test_as_list_and_as_dict_side_inputs 
(apache_beam.transforms.sideinputs_test.SideInputsTest) ... ok
DEPRECATION: pip install --download has been deprecated and will be removed in 
the future. Pip now has a download command that should be used instead.
Collecting pyhamcrest (from -r postcommit_requirements.txt (line 1))
:318:
 SNIMissingWarning: An HTTPS request has been made, but the SNI (Subject Name 
Indication) extension to TLS is not available on this platform. This may cause 
the server to present an incorrect TLS certificate, which can cause validation 
failures. You can upgrade to a newer version of Python to solve this. For more 
information, see 
https://urllib3.readthedocs.io/en/latest/security.html#snimissingwarning.
  SNIMissingWarning
:122:
 InsecurePlatformWarning: A true SSLContext object is not available. This 
prevents urllib3 from configuring SSL appropriately and may cause certain SSL 
connections to fail. You can upgrade to a newer version of Python to solve 
this. For more information, see 
https://urllib3.readthedocs.io/en/latest/security.html#insecureplatformwarning.
  InsecurePlatformWarning
  File was already downloaded 
/tmp/dataflow-requirements-cache/PyHamcrest-1.9.0.tar.gz
Collecting mock (from -r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/mock-2.0.0.tar.gz
Collecting setuptools (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded 
/tmp/dataflow-requirements-cache/setuptools-36.2.7.zip
Collecting six (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded /tmp/dataflow-requirements-cache/six-1.10.0.tar.gz
Collecting funcsigs>=1 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded 
/tmp/dataflow-requirements-cache/funcsigs-1.0.2.tar.gz
Collecting pbr>=0.11 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/pbr-3.1.1.tar.gz
Successfully downloaded pyhamcrest mock setuptools six funcsigs pbr
test_undeclared_outputs (apache_beam.transforms.ptransform_test.PTransformTest) 
... ok
:135:
 UserWarning: Using fallback coder for typehint: List[Any].
  warnings.warn('Using fallback coder for typehint: %r.' % typehint)
test_as_dict_twice (apache_beam.transforms.sideinputs_test.SideInputsTest) ... 
ok
DEPRECATION: pip install --download has been deprecated and will be removed in 
the future. Pip now has a download command that should be used instead.
DEPRECATION: pip install --download has been deprecated and will be removed in 
the future. Pip now has a download command that should be used instead.
Collecting pyhamcrest (from -r postcommit_requirements.txt (line 1))
Collecting pyhamcrest (from -r postcommit_requirements.txt (line 1))
  File was already downloaded 

[jira] [Commented] (BEAM-1323) Add parallelism/splitting in JdbcIO

[JIRA]

[ 
https://issues.apache.org/jira/browse/BEAM-1323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16110238#comment-16110238
 ] 

Jean-Baptiste Onofré commented on BEAM-1323:


Fair enough. I'm closing this Jira.

> Add parallelism/splitting in JdbcIO
> ---
>
> Key: BEAM-1323
> URL: https://issues.apache.org/jira/browse/BEAM-1323
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-extensions
>Reporter: Jean-Baptiste Onofré
>Assignee: Jean-Baptiste Onofré
> Fix For: Not applicable
>
>
> Now, the JDBC IO is basically a {{DoFn}} executed with a {{ParDo}}. So, it 
> means that parallelism is "limited" and executed on one executor.
> We can imagine to create several JDBC {{BoundedSource}}s splitting the SQL 
> query in  subset (for instance using row id paging or any "splitting/limit" 
> we can figure based on the original SQL query) (something similar to what 
> Sqoop is doing).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (BEAM-1323) Add parallelism/splitting in JdbcIO

[JIRA]

 [ 
https://issues.apache.org/jira/browse/BEAM-1323?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jean-Baptiste Onofré resolved BEAM-1323.

   Resolution: Won't Fix
Fix Version/s: Not applicable

> Add parallelism/splitting in JdbcIO
> ---
>
> Key: BEAM-1323
> URL: https://issues.apache.org/jira/browse/BEAM-1323
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-extensions
>Reporter: Jean-Baptiste Onofré
>Assignee: Jean-Baptiste Onofré
> Fix For: Not applicable
>
>
> Now, the JDBC IO is basically a {{DoFn}} executed with a {{ParDo}}. So, it 
> means that parallelism is "limited" and executed on one executor.
> We can imagine to create several JDBC {{BoundedSource}}s splitting the SQL 
> query in  subset (for instance using row id paging or any "splitting/limit" 
> we can figure based on the original SQL query) (something similar to what 
> Sqoop is doing).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BEAM-2706) Create JdbcIO.readAll()

[JIRA]

[ 
https://issues.apache.org/jira/browse/BEAM-2706?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16110237#comment-16110237
 ] 

Jean-Baptiste Onofré commented on BEAM-2706:


I gonna implement this asap. Thanks for the proposal !

> Create JdbcIO.readAll()
> ---
>
> Key: BEAM-2706
> URL: https://issues.apache.org/jira/browse/BEAM-2706
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Reporter: Eugene Kirpichov
>Assignee: Jean-Baptiste Onofré
>
> Similarly to TextIO.readAll(), AvroIO.readAll(), SpannerIO.readAll(), and in 
> the general spirit of making connectors more dynamic: it would be nice to 
> have JdbcIO.readAll() that reads a PCollection of queries, or perhaps better, 
> a parameterized query with a PCollection of parameter values and a user 
> callback for setting the parameters on a query based on the PCollection 
> element.
> JB, as the author of JdbcIO, would you be interested in implementing this at 
> some point?



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Jenkins build is unstable: beam_PostCommit_Java_ValidatesRunner_Dataflow #3698

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-1584) Support clean-up step in integration test

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16110089#comment-16110089
 ] 

ASF GitHub Bot commented on BEAM-1584:
--

GitHub user markflyhigh opened a pull request:

https://github.com/apache/beam/pull/3673

[BEAM-1584] Add clean up for bigquery integration test in Python

Follow this checklist to help us incorporate your contribution quickly and 
easily:

 - [ ] Make sure there is a [JIRA 
issue](https://issues.apache.org/jira/projects/BEAM/issues/) filed for the 
change (usually before you start working on it).  Trivial changes like typos do 
not require a JIRA issue.  Your pull request should address just this issue, 
without pulling in other changes.
 - [ ] Each commit in the pull request should have a meaningful subject 
line and body.
 - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue.
 - [ ] Write a pull request description that is detailed enough to 
understand what the pull request does, how, and why.
 - [ ] Run `mvn clean verify` to make sure basic checks pass. A more 
thorough check will be performed on your pull request automatically.
 - [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).

---

Add clean up util for bigquery integration test and enable it for 
BigqueryTornadoesIT. Clean up will be executed no matter whether pipeline job 
is successful or verification is passed.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/markflyhigh/incubator-beam cleanup

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3673.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3673


commit f0c916fec5641ba576c47838dadf75ffa300221d
Author: Mark Liu 
Date:   2017-08-02T00:46:05Z

[BEAM-1584] Add clean up in bigquery integration test




> Support clean-up step in integration test
> -
>
> Key: BEAM-1584
> URL: https://issues.apache.org/jira/browse/BEAM-1584
> Project: Beam
>  Issue Type: Task
>  Components: testing
>Reporter: Mark Liu
>Assignee: Mark Liu
>
> Idea comes from: 
> https://github.com/apache/beam/pull/2064/files/628fafed098ac5550356a201c6ccdcdcc2e9604e
> Integration tests in all sdks should be able to do clean-up at the end of 
> each run.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] beam pull request #3673: [BEAM-1584] Add clean up for bigquery integration t...

[markflyhigh]

GitHub user markflyhigh opened a pull request:

https://github.com/apache/beam/pull/3673

[BEAM-1584] Add clean up for bigquery integration test in Python

Follow this checklist to help us incorporate your contribution quickly and 
easily:

 - [ ] Make sure there is a [JIRA 
issue](https://issues.apache.org/jira/projects/BEAM/issues/) filed for the 
change (usually before you start working on it).  Trivial changes like typos do 
not require a JIRA issue.  Your pull request should address just this issue, 
without pulling in other changes.
 - [ ] Each commit in the pull request should have a meaningful subject 
line and body.
 - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue.
 - [ ] Write a pull request description that is detailed enough to 
understand what the pull request does, how, and why.
 - [ ] Run `mvn clean verify` to make sure basic checks pass. A more 
thorough check will be performed on your pull request automatically.
 - [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).

---

Add clean up util for bigquery integration test and enable it for 
BigqueryTornadoesIT. Clean up will be executed no matter whether pipeline job 
is successful or verification is passed.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/markflyhigh/incubator-beam cleanup

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3673.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3673


commit f0c916fec5641ba576c47838dadf75ffa300221d
Author: Mark Liu 
Date:   2017-08-02T00:46:05Z

[BEAM-1584] Add clean up in bigquery integration test




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Spark #2756

[Apache Jenkins Server]

See 

[GitHub] beam pull request #3650: Translate ParDo and GroupByKey operations through t...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3650


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/5] beam git commit: Closes #3650

[robertwb]

Repository: beam
Updated Branches:
  refs/heads/master 53c2c8fde -> ba5e31466


Closes #3650


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/ba5e3146
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/ba5e3146
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/ba5e3146

Branch: refs/heads/master
Commit: ba5e314664563f23ac8c5125b89a12ddbbd4e433
Parents: 53c2c8f 2ee7422
Author: Robert Bradshaw 
Authored: Tue Aug 1 17:32:59 2017 -0700
Committer: Robert Bradshaw 
Committed: Tue Aug 1 17:32:59 2017 -0700

--
 sdks/python/apache_beam/pipeline.py | 14 -
 .../apache_beam/runners/direct/direct_runner.py | 27 +++-
 .../apache_beam/runners/pipeline_context.py |  8 +--
 sdks/python/apache_beam/transforms/core.py  | 65 +++-
 sdks/python/apache_beam/utils/urns.py   |  6 ++
 5 files changed, 112 insertions(+), 8 deletions(-)
--

[4/5] beam git commit: More informative references in the proto representation.

[robertwb]

More informative references in the proto representation.


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/2ee7422f
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/2ee7422f
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/2ee7422f

Branch: refs/heads/master
Commit: 2ee7422f1dbd00e3c40ac95e9eaddc745d46fe65
Parents: b179eca
Author: Robert Bradshaw 
Authored: Wed Jul 26 18:21:47 2017 -0700
Committer: Robert Bradshaw 
Committed: Tue Aug 1 17:32:59 2017 -0700

--
 sdks/python/apache_beam/pipeline.py | 3 ++-
 sdks/python/apache_beam/runners/pipeline_context.py | 8 
 2 files changed, 6 insertions(+), 5 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/2ee7422f/sdks/python/apache_beam/pipeline.py
--
diff --git a/sdks/python/apache_beam/pipeline.py 
b/sdks/python/apache_beam/pipeline.py
index 8553f7c..e7c2322 100644
--- a/sdks/python/apache_beam/pipeline.py
+++ b/sdks/python/apache_beam/pipeline.py
@@ -730,7 +730,8 @@ class AppliedPTransform(object):
 return beam_runner_api_pb2.PTransform(
 unique_name=self.full_label,
 spec=transform_to_runner_api(self.transform, context),
-subtransforms=[context.transforms.get_id(part) for part in self.parts],
+subtransforms=[context.transforms.get_id(part, label=part.full_label)
+   for part in self.parts],
 # TODO(BEAM-115): Side inputs.
 inputs={tag: context.pcollections.get_id(pc)
 for tag, pc in self.named_inputs().items()},

http://git-wip-us.apache.org/repos/asf/beam/blob/2ee7422f/sdks/python/apache_beam/runners/pipeline_context.py
--
diff --git a/sdks/python/apache_beam/runners/pipeline_context.py 
b/sdks/python/apache_beam/runners/pipeline_context.py
index a40069b..f4de42a 100644
--- a/sdks/python/apache_beam/runners/pipeline_context.py
+++ b/sdks/python/apache_beam/runners/pipeline_context.py
@@ -43,18 +43,18 @@ class _PipelineContextMap(object):
 self._id_to_proto = proto_map if proto_map else {}
 self._counter = 0
 
-  def _unique_ref(self, obj=None):
+  def _unique_ref(self, obj=None, label=None):
 self._counter += 1
 return "ref_%s_%s_%s" % (
-self._obj_type.__name__, type(obj).__name__, self._counter)
+self._obj_type.__name__, label or type(obj).__name__, self._counter)
 
   def populate_map(self, proto_map):
 for id, proto in self._id_to_proto.items():
   proto_map[id].CopyFrom(proto)
 
-  def get_id(self, obj):
+  def get_id(self, obj, label=None):
 if obj not in self._obj_to_id:
-  id = self._unique_ref(obj)
+  id = self._unique_ref(obj, label)
   self._id_to_obj[id] = obj
   self._obj_to_id[obj] = id
   self._id_to_proto[id] = obj.to_runner_api(self._pipeline_context)

[2/5] beam git commit: Translate GroupByKey[Only] through the Runner API.

[robertwb]

Translate GroupByKey[Only] through the Runner API.


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/b179eca9
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/b179eca9
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/b179eca9

Branch: refs/heads/master
Commit: b179eca90f2af262aed637a2a0c099680a0822d2
Parents: 3c0c337
Author: Robert Bradshaw 
Authored: Wed Jul 26 18:21:20 2017 -0700
Committer: Robert Bradshaw 
Committed: Tue Aug 1 17:32:59 2017 -0700

--
 sdks/python/apache_beam/transforms/core.py | 14 ++
 sdks/python/apache_beam/utils/urns.py  |  2 ++
 2 files changed, 16 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/b179eca9/sdks/python/apache_beam/transforms/core.py
--
diff --git a/sdks/python/apache_beam/transforms/core.py 
b/sdks/python/apache_beam/transforms/core.py
index 671fea4..cff6dbe 100644
--- a/sdks/python/apache_beam/transforms/core.py
+++ b/sdks/python/apache_beam/transforms/core.py
@@ -1184,6 +1184,13 @@ class GroupByKey(PTransform):
   | 'GroupByKey' >> _GroupByKeyOnly()
   | 'GroupByWindow' >> _GroupAlsoByWindow(pcoll.windowing))
 
+  def to_runner_api_parameter(self, unused_context):
+return urns.GROUP_BY_KEY_TRANSFORM, None
+
+  @PTransform.register_urn(urns.GROUP_BY_KEY_TRANSFORM, None)
+  def from_runner_api_parameter(unused_payload, unused_context):
+return GroupByKey()
+
 
 @typehints.with_input_types(typehints.KV[K, V])
 @typehints.with_output_types(typehints.KV[K, typehints.Iterable[V]])
@@ -1197,6 +1204,13 @@ class _GroupByKeyOnly(PTransform):
 self._check_pcollection(pcoll)
 return pvalue.PCollection(pcoll.pipeline)
 
+  def to_runner_api_parameter(self, unused_context):
+return urns.GROUP_BY_KEY_ONLY_TRANSFORM, None
+
+  @PTransform.register_urn(urns.GROUP_BY_KEY_ONLY_TRANSFORM, None)
+  def from_runner_api_parameter(unused_payload, unused_context):
+return _GroupByKeyOnly()
+
 
 @typehints.with_input_types(typehints.KV[K, typehints.Iterable[V]])
 @typehints.with_output_types(typehints.KV[K, typehints.Iterable[V]])

http://git-wip-us.apache.org/repos/asf/beam/blob/b179eca9/sdks/python/apache_beam/utils/urns.py
--
diff --git a/sdks/python/apache_beam/utils/urns.py 
b/sdks/python/apache_beam/utils/urns.py
index e7ef80b..0013cb3 100644
--- a/sdks/python/apache_beam/utils/urns.py
+++ b/sdks/python/apache_beam/utils/urns.py
@@ -39,6 +39,8 @@ PICKLED_CODER = "beam:coder:pickled_python:v0.1"
 
 PICKLED_TRANSFORM = "beam:ptransform:pickled_python:v0.1"
 PARDO_TRANSFORM = "beam:ptransform:pardo:v0.1"
+GROUP_BY_KEY_TRANSFORM = "beam:ptransform:group_by_key:v0.1"
+GROUP_BY_KEY_ONLY_TRANSFORM = "beam:ptransform:group_by_key_only:v0.1"
 GROUP_ALSO_BY_WINDOW_TRANSFORM = "beam:ptransform:group_also_by_window:v0.1"
 COMBINE_PER_KEY_TRANSFORM = "beam:ptransform:combine_per_key:v0.1"
 COMBINE_GROUPED_VALUES_TRANSFORM = 
"beam:ptransform:combine_grouped_values:v0.1"

[3/5] beam git commit: Translate ParDo operations through the Runner API.

[robertwb]

Translate ParDo operations through the Runner API.


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/7d64b769
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/7d64b769
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/7d64b769

Branch: refs/heads/master
Commit: 7d64b769a7099e767a81c693294b5a66791a441b
Parents: 53c2c8f
Author: Robert Bradshaw 
Authored: Wed Jul 26 16:30:56 2017 -0700
Committer: Robert Bradshaw 
Committed: Tue Aug 1 17:32:59 2017 -0700

--
 sdks/python/apache_beam/pipeline.py| 11 +++
 sdks/python/apache_beam/transforms/core.py | 38 -
 sdks/python/apache_beam/utils/urns.py  |  3 ++
 3 files changed, 51 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/7d64b769/sdks/python/apache_beam/pipeline.py
--
diff --git a/sdks/python/apache_beam/pipeline.py 
b/sdks/python/apache_beam/pipeline.py
index fe36d85..8553f7c 100644
--- a/sdks/python/apache_beam/pipeline.py
+++ b/sdks/python/apache_beam/pipeline.py
@@ -66,6 +66,7 @@ from apache_beam.options.pipeline_options import 
StandardOptions
 from apache_beam.options.pipeline_options import TypeOptions
 from apache_beam.options.pipeline_options_validator import 
PipelineOptionsValidator
 from apache_beam.utils.annotations import deprecated
+from apache_beam.utils import urns
 
 
 __all__ = ['Pipeline']
@@ -474,6 +475,12 @@ class Pipeline(object):
 return str, ('Pickled pipeline stub.',)
 
   def _verify_runner_api_compatible(self):
+if self._options.view_as(TypeOptions).runtime_type_check:
+  # This option is incompatible with the runner API as it requires
+  # the runner to inspect non-serialized hints on the transform
+  # itself.
+  return False
+
 class Visitor(PipelineVisitor):  # pylint: disable=used-before-assignment
   ok = True  # Really a nonlocal.
 
@@ -745,6 +752,10 @@ class AppliedPTransform(object):
 result.outputs = {
 None if tag == 'None' else tag: context.pcollections.get_by_id(id)
 for tag, id in proto.outputs.items()}
+# This annotation is expected by some runners.
+if proto.spec.urn == urns.PARDO_TRANSFORM:
+  result.transform.output_tags = set(proto.outputs.keys()).difference(
+  {'None'})
 if not result.parts:
   for tag, pc in result.outputs.items():
 if pc not in result.inputs:

http://git-wip-us.apache.org/repos/asf/beam/blob/7d64b769/sdks/python/apache_beam/transforms/core.py
--
diff --git a/sdks/python/apache_beam/transforms/core.py 
b/sdks/python/apache_beam/transforms/core.py
index 25fe39f..9168a89 100644
--- a/sdks/python/apache_beam/transforms/core.py
+++ b/sdks/python/apache_beam/transforms/core.py
@@ -23,10 +23,13 @@ import copy
 import inspect
 import types
 
+from google.protobuf import wrappers_pb2
+
 from apache_beam import pvalue
 from apache_beam import typehints
 from apache_beam import coders
 from apache_beam.coders import typecoders
+from apache_beam.internal import pickler
 from apache_beam.internal import util
 from apache_beam.portability.api import beam_runner_api_pb2
 from apache_beam.transforms import ptransform
@@ -50,6 +53,7 @@ from apache_beam.typehints.decorators import TypeCheckError
 from apache_beam.typehints.decorators import WithTypeHints
 from apache_beam.typehints.trivial_inference import element_type
 from apache_beam.typehints.typehints import is_consistent_with
+from apache_beam.utils import proto_utils
 from apache_beam.utils import urns
 from apache_beam.options.pipeline_options import TypeOptions
 
@@ -136,7 +140,7 @@ class DoFnProcessContext(DoFnContext):
   self.windows = windowed_value.windows
 
 
-class DoFn(WithTypeHints, HasDisplayData):
+class DoFn(WithTypeHints, HasDisplayData, urns.RunnerApiFn):
   """A function object used by a transform with custom processing.
 
   The ParDo transform is such a transform. The ParDo.apply
@@ -236,6 +240,8 @@ class DoFn(WithTypeHints, HasDisplayData):
   return False # Method is a classmethod
 return True
 
+  urns.RunnerApiFn.register_pickle_urn(urns.PICKLED_DO_FN)
+
 
 def _fn_takes_side_inputs(fn):
   try:
@@ -686,6 +692,36 @@ class ParDo(PTransformWithSideInputs):
   raise ValueError('Unexpected keyword arguments: %s' % main_kw.keys())
 return _MultiParDo(self, tags, main_tag)
 
+  def _pardo_fn_data(self):
+si_tags_and_types = []
+windowing = None
+return self.fn, self.args, self.kwargs, si_tags_and_types, windowing
+
+  def to_runner_api_parameter(self, context):
+return (
+urns.PARDO_TRANSFORM,
+beam_runner_api_pb2.ParDoPayload(
+

[5/5] beam git commit: Streaming fixes.

[robertwb]

Streaming fixes.


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/3c0c3374
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/3c0c3374
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/3c0c3374

Branch: refs/heads/master
Commit: 3c0c337401ca164093c06df5b9021106ba7e5eb9
Parents: 7d64b76
Author: Robert Bradshaw 
Authored: Wed Jul 26 18:01:49 2017 -0700
Committer: Robert Bradshaw 
Committed: Tue Aug 1 17:32:59 2017 -0700

--
 .../apache_beam/runners/direct/direct_runner.py | 27 ++--
 sdks/python/apache_beam/transforms/core.py  | 13 ++
 sdks/python/apache_beam/utils/urns.py   |  1 +
 3 files changed, 39 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/3c0c3374/sdks/python/apache_beam/runners/direct/direct_runner.py
--
diff --git a/sdks/python/apache_beam/runners/direct/direct_runner.py 
b/sdks/python/apache_beam/runners/direct/direct_runner.py
index 1a94b3d..7a88d0e 100644
--- a/sdks/python/apache_beam/runners/direct/direct_runner.py
+++ b/sdks/python/apache_beam/runners/direct/direct_runner.py
@@ -26,6 +26,8 @@ from __future__ import absolute_import
 import collections
 import logging
 
+from google.protobuf import wrappers_pb2
+
 import apache_beam as beam
 from apache_beam import typehints
 from apache_beam.metrics.execution import MetricsEnvironment
@@ -35,6 +37,7 @@ from apache_beam.runners.runner import PipelineResult
 from apache_beam.runners.runner import PipelineRunner
 from apache_beam.runners.runner import PipelineState
 from apache_beam.runners.runner import PValueCache
+from apache_beam.transforms.ptransform import PTransform
 from apache_beam.transforms.core import _GroupAlsoByWindow
 from apache_beam.transforms.core import _GroupByKeyOnly
 from apache_beam.options.pipeline_options import DirectOptions
@@ -54,14 +57,34 @@ V = typehints.TypeVariable('V')
 @typehints.with_output_types(typehints.KV[K, typehints.Iterable[V]])
 class _StreamingGroupByKeyOnly(_GroupByKeyOnly):
   """Streaming GroupByKeyOnly placeholder for overriding in DirectRunner."""
-  pass
+  urn = "direct_runner:streaming_gbko:v0.1"
+
+  # These are needed due to apply overloads.
+  def to_runner_api_parameter(self, unused_context):
+return _StreamingGroupByKeyOnly.urn, None
+
+  @PTransform.register_urn(urn, None)
+  def from_runner_api_parameter(unused_payload, unused_context):
+return _StreamingGroupByKeyOnly()
 
 
 @typehints.with_input_types(typehints.KV[K, typehints.Iterable[V]])
 @typehints.with_output_types(typehints.KV[K, typehints.Iterable[V]])
 class _StreamingGroupAlsoByWindow(_GroupAlsoByWindow):
   """Streaming GroupAlsoByWindow placeholder for overriding in DirectRunner."""
-  pass
+  urn = "direct_runner:streaming_gabw:v0.1"
+
+  # These are needed due to apply overloads.
+  def to_runner_api_parameter(self, context):
+return (
+_StreamingGroupAlsoByWindow.urn,
+wrappers_pb2.BytesValue(value=context.windowing_strategies.get_id(
+self.windowing)))
+
+  @PTransform.register_urn(urn, wrappers_pb2.BytesValue)
+  def from_runner_api_parameter(payload, context):
+return _StreamingGroupAlsoByWindow(
+context.windowing_strategies.get_by_id(payload.value))
 
 
 class DirectRunner(PipelineRunner):

http://git-wip-us.apache.org/repos/asf/beam/blob/3c0c3374/sdks/python/apache_beam/transforms/core.py
--
diff --git a/sdks/python/apache_beam/transforms/core.py 
b/sdks/python/apache_beam/transforms/core.py
index 9168a89..671fea4 100644
--- a/sdks/python/apache_beam/transforms/core.py
+++ b/sdks/python/apache_beam/transforms/core.py
@@ -698,6 +698,7 @@ class ParDo(PTransformWithSideInputs):
 return self.fn, self.args, self.kwargs, si_tags_and_types, windowing
 
   def to_runner_api_parameter(self, context):
+assert self.__class__ is ParDo
 return (
 urns.PARDO_TRANSFORM,
 beam_runner_api_pb2.ParDoPayload(
@@ -1210,6 +1211,18 @@ class _GroupAlsoByWindow(ParDo):
 self._check_pcollection(pcoll)
 return pvalue.PCollection(pcoll.pipeline)
 
+  def to_runner_api_parameter(self, context):
+return (
+urns.GROUP_ALSO_BY_WINDOW_TRANSFORM,
+wrappers_pb2.BytesValue(value=context.windowing_strategies.get_id(
+self.windowing)))
+
+  @PTransform.register_urn(
+  urns.GROUP_ALSO_BY_WINDOW_TRANSFORM, wrappers_pb2.BytesValue)
+  def from_runner_api_parameter(payload, context):
+return _GroupAlsoByWindow(
+context.windowing_strategies.get_by_id(payload.value))
+
 
 class _GroupAlsoByWindowDoFn(DoFn):
   # TODO(robertwb): Support combiner lifting.

[jira] [Updated] (BEAM-2713) Support Python native types in Beam typehints

[Ahmet Altay (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ahmet Altay updated BEAM-2713:
--
Issue Type: Sub-task  (was: New Feature)
Parent: BEAM-1251

> Support Python native types in Beam typehints
> -
>
> Key: BEAM-2713
> URL: https://issues.apache.org/jira/browse/BEAM-2713
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py
>Reporter: Chuan Yu Foo
>Assignee: Ahmet Altay
>
> Add support for Python's native typing type to Beam typehints.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BEAM-2713) Support Python native types in Beam typehints

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16110016#comment-16110016
 ] 

ASF GitHub Bot commented on BEAM-2713:
--

GitHub user chuanyu opened a pull request:

https://github.com/apache/beam/pull/3672

[BEAM-2713] Support Python native types in Beam typehints

Add support for Python native types in Beam typehints.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/chuanyu/beam support_python_types

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3672.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3672


commit 54f12e1b5295e77bd3c1837c14be24accf54b848
Author: Chuan Yu Foo 
Date:   2017-08-01T00:10:45Z

Add support for Python's native type hint types in Beam's type hint 
annotations.




> Support Python native types in Beam typehints
> -
>
> Key: BEAM-2713
> URL: https://issues.apache.org/jira/browse/BEAM-2713
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-py
>Reporter: Chuan Yu Foo
>Assignee: Ahmet Altay
>
> Add support for Python's native typing type to Beam typehints.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] beam pull request #3672: [BEAM-2713] Support Python native types in Beam typ...

[chuanyu]

GitHub user chuanyu opened a pull request:

https://github.com/apache/beam/pull/3672

[BEAM-2713] Support Python native types in Beam typehints

Add support for Python native types in Beam typehints.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/chuanyu/beam support_python_types

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3672.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3672


commit 54f12e1b5295e77bd3c1837c14be24accf54b848
Author: Chuan Yu Foo 
Date:   2017-08-01T00:10:45Z

Add support for Python's native type hint types in Beam's type hint 
annotations.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (BEAM-2713) Support Python native types in Beam typehints

[Chuan Yu Foo (JIRA)]

Chuan Yu Foo created BEAM-2713:
--

 Summary: Support Python native types in Beam typehints
 Key: BEAM-2713
 URL: https://issues.apache.org/jira/browse/BEAM-2713
 Project: Beam
  Issue Type: New Feature
  Components: sdk-py
Reporter: Chuan Yu Foo
Assignee: Ahmet Altay


Add support for Python's native typing type to Beam typehints.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (BEAM-2712) SerializablePipelineOptions should not call FileSystems.setDefaultPipelineOptions.

[Eugene Kirpichov (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2712?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eugene Kirpichov reassigned BEAM-2712:
--

Assignee: (was: Kenneth Knowles)

> SerializablePipelineOptions should not call 
> FileSystems.setDefaultPipelineOptions.
> --
>
> Key: BEAM-2712
> URL: https://issues.apache.org/jira/browse/BEAM-2712
> Project: Beam
>  Issue Type: Bug
>  Components: runner-apex, runner-core, runner-flink, runner-spark
>Reporter: Eugene Kirpichov
>
> https://github.com/apache/beam/pull/3654 introduces 
> SerializablePipelineOptions, which on deserialization calls 
> FileSystems.setDefaultPipelineOptions.
> This is obviously problematic and racy in case the same process uses 
> SerializablePipelineOptions with different filesystem-related options in them.
> The reason the PR does this is, Flink and Apex runners were already doing it 
> in their respective SerializablePipelineOptions-like classes (being removed 
> in the PR); and Spark wasn't but probably should have.
> I believe this is done for the sake of having the proper filesystem options 
> automatically available on workers in all places where any kind of 
> PipelineOptions are used. Instead, all 3 runners should pick a better place 
> to initialize their workers, and explicitly call 
> FileSystems.setDefaultPipelineOptions there.
> It would be even better if FileSystems.setDefaultPipelineOptions didn't exist 
> at all, but that's a topic for a separate JIRA.
> CC'ing runner contributors [~aljoscha] [~aviemzur] [~thw]



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (BEAM-2712) SerializablePipelineOptions should not call FileSystems.setDefaultPipelineOptions.

[Eugene Kirpichov (JIRA)]

Eugene Kirpichov created BEAM-2712:
--

 Summary: SerializablePipelineOptions should not call 
FileSystems.setDefaultPipelineOptions.
 Key: BEAM-2712
 URL: https://issues.apache.org/jira/browse/BEAM-2712
 Project: Beam
  Issue Type: Bug
  Components: runner-apex, runner-core, runner-flink, runner-spark
Reporter: Eugene Kirpichov
Assignee: Kenneth Knowles


https://github.com/apache/beam/pull/3654 introduces 
SerializablePipelineOptions, which on deserialization calls 
FileSystems.setDefaultPipelineOptions.

This is obviously problematic and racy in case the same process uses 
SerializablePipelineOptions with different filesystem-related options in them.

The reason the PR does this is, Flink and Apex runners were already doing it in 
their respective SerializablePipelineOptions-like classes (being removed in the 
PR); and Spark wasn't but probably should have.

I believe this is done for the sake of having the proper filesystem options 
automatically available on workers in all places where any kind of 
PipelineOptions are used. Instead, all 3 runners should pick a better place to 
initialize their workers, and explicitly call 
FileSystems.setDefaultPipelineOptions there.

It would be even better if FileSystems.setDefaultPipelineOptions didn't exist 
at all, but that's a topic for a separate JIRA.

CC'ing runner contributors [~aljoscha] [~aviemzur] [~thw]



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Closed] (BEAM-2710) Outputing "finished" when processing is not called in ParDoTest.TestDoFn leads to unexpected ElementCount

[Rafael Fernandez (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2710?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rafael Fernandez closed BEAM-2710.
--
   Resolution: Not A Problem
Fix Version/s: Not applicable

The issue is broader than test -- it's in worker code.

> Outputing "finished" when processing is not called in ParDoTest.TestDoFn 
> leads to unexpected ElementCount
> -
>
> Key: BEAM-2710
> URL: https://issues.apache.org/jira/browse/BEAM-2710
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Rafael Fernandez
>Assignee: Davor Bonaci
>Priority: Trivial
> Fix For: Not applicable
>
>
> While it is legal to call finishBundle even if processElement was not called 
> (say, when processing an empty input PCollection), it's best to guard against 
> this case if we're examining correctness of outputs. Specifically, I was 
> surprised at seeing ElementCount = 1 instead of 0 -- essentially, "finished" 
> was being counted here.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] beam pull request #3671: Fix typo in dataflow_runner.

[robertwb]

GitHub user robertwb opened a pull request:

https://github.com/apache/beam/pull/3671

Fix typo in dataflow_runner.

Follow this checklist to help us incorporate your contribution quickly and 
easily:

 - [ ] Make sure there is a [JIRA 
issue](https://issues.apache.org/jira/projects/BEAM/issues/) filed for the 
change (usually before you start working on it).  Trivial changes like typos do 
not require a JIRA issue.  Your pull request should address just this issue, 
without pulling in other changes.
 - [ ] Each commit in the pull request should have a meaningful subject 
line and body.
 - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue.
 - [ ] Write a pull request description that is detailed enough to 
understand what the pull request does, how, and why.
 - [ ] Run `mvn clean verify` to make sure basic checks pass. A more 
thorough check will be performed on your pull request automatically.
 - [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/robertwb/incubator-beam patch-10

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3671.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3671


commit 009e6ec204b6ff7cfabe8e480a831a0c5121b5eb
Author: Robert Bradshaw 
Date:   2017-08-01T23:18:15Z

Fix typo in dataflow_runner.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (BEAM-2711) ByteKeyRangeTracker.getFractionConsumed() fails when out of range positions are claimed

[Chamikara Jayalath (JIRA)]

Chamikara Jayalath created BEAM-2711:


 Summary: ByteKeyRangeTracker.getFractionConsumed() fails when out 
of range positions are claimed
 Key: BEAM-2711
 URL: https://issues.apache.org/jira/browse/BEAM-2711
 Project: Beam
  Issue Type: Bug
  Components: sdk-java-core
Reporter: Chamikara Jayalath


ByteKeyRangeTracker.getFractionConsumed() invokes 
range.estimateFractionForKey(position) at following location.
https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/io/range/ByteKeyRangeTracker.java#L127

This invocation fails for out of range positions.
https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/io/range/ByteKeyRange.java#L170

But ByteKeyRangeTracker may accept out of range positions at following location.
https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/io/range/ByteKeyRangeTracker.java#L80

We can fix this by updating ByteKeyRangeTracker.getFractionConsumed() to return 
1.0 for positions that are larger than the stop position, similar to 
OffsetRangeTracker.
https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/io/range/OffsetRangeTracker.java#L176



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Build failed in Jenkins: beam_PostCommit_Python_Verify #2846

[Apache Jenkins Server]

See 


Changes:

[tgroh] Remove References to CloudObject from the Java Harness

--
[...truncated 559.75 KB...]
Collecting pbr>=0.11 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/pbr-3.1.1.tar.gz
Successfully downloaded pyhamcrest mock setuptools six funcsigs pbr
test_par_do_with_multiple_outputs_and_using_return 
(apache_beam.transforms.ptransform_test.PTransformTest) ... ok
:135:
 UserWarning: Using fallback coder for typehint: List[Any].
  warnings.warn('Using fallback coder for typehint: %r.' % typehint)
DEPRECATION: pip install --download has been deprecated and will be removed in 
the future. Pip now has a download command that should be used instead.
Collecting pyhamcrest (from -r postcommit_requirements.txt (line 1))
  File was already downloaded 
/tmp/dataflow-requirements-cache/PyHamcrest-1.9.0.tar.gz
Collecting mock (from -r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/mock-2.0.0.tar.gz
Collecting setuptools (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded 
/tmp/dataflow-requirements-cache/setuptools-36.2.6.zip
Collecting six (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded /tmp/dataflow-requirements-cache/six-1.10.0.tar.gz
Collecting funcsigs>=1 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded 
/tmp/dataflow-requirements-cache/funcsigs-1.0.2.tar.gz
Collecting pbr>=0.11 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/pbr-3.1.1.tar.gz
Successfully downloaded pyhamcrest mock setuptools six funcsigs pbr
test_undeclared_outputs (apache_beam.transforms.ptransform_test.PTransformTest) 
... ok
:135:
 UserWarning: Using fallback coder for typehint: List[Any].
  warnings.warn('Using fallback coder for typehint: %r.' % typehint)
DEPRECATION: pip install --download has been deprecated and will be removed in 
the future. Pip now has a download command that should be used instead.
Collecting pyhamcrest (from -r postcommit_requirements.txt (line 1))
  File was already downloaded 
/tmp/dataflow-requirements-cache/PyHamcrest-1.9.0.tar.gz
Collecting mock (from -r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/mock-2.0.0.tar.gz
Collecting setuptools (from pyhamcrest->-r postcommit_requirements.txt (line 1))
test_as_dict_twice (apache_beam.transforms.sideinputs_test.SideInputsTest) ... 
ok
  File was already downloaded 
/tmp/dataflow-requirements-cache/setuptools-36.2.6.zip
Collecting six (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded /tmp/dataflow-requirements-cache/six-1.10.0.tar.gz
DEPRECATION: pip install --download has been deprecated and will be removed in 
the future. Pip now has a download command that should be used instead.
Collecting pyhamcrest (from -r postcommit_requirements.txt (line 1))
  File was already downloaded 
/tmp/dataflow-requirements-cache/PyHamcrest-1.9.0.tar.gz
Collecting funcsigs>=1 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded 
/tmp/dataflow-requirements-cache/funcsigs-1.0.2.tar.gz
Collecting pbr>=0.11 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/pbr-3.1.1.tar.gz
Collecting mock (from -r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/mock-2.0.0.tar.gz
Successfully downloaded pyhamcrest mock setuptools six funcsigs pbr
Collecting setuptools (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded 
/tmp/dataflow-requirements-cache/setuptools-36.2.6.zip
Collecting six (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded /tmp/dataflow-requirements-cache/six-1.10.0.tar.gz
Collecting funcsigs>=1 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded 
/tmp/dataflow-requirements-cache/funcsigs-1.0.2.tar.gz
Collecting pbr>=0.11 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/pbr-3.1.1.tar.gz
Successfully downloaded pyhamcrest mock setuptools six funcsigs pbr
test_as_list_twice (apache_beam.transforms.sideinputs_test.SideInputsTest) ... 
ok
DEPRECATION: pip install --download has been deprecated and will be removed in 
the future. Pip now has a download command that should be used instead.
Collecting pyhamcrest (from -r 

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Dataflow #3696

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-2707) DSL_SQL branch broken by PR 3334

[Xu Mingmin (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16109896#comment-16109896
 ] 

Xu Mingmin commented on BEAM-2707:
--

#3334 should not impact us as we're in DSL_SQL branch.

I check out the PR and build successfully with steps:
{code}
git fetch origin pull/3666/head:extention_sql

git checkout extention_sql

//the old dsls folder is still there maybe due to my own settings
rm -rf dsls/

mvn clean install
{code}

[~takidau] can you try it again?

> DSL_SQL branch broken by PR 3334
> 
>
> Key: BEAM-2707
> URL: https://issues.apache.org/jira/browse/BEAM-2707
> Project: Beam
>  Issue Type: Bug
>  Components: runner-core
>Reporter: Tyler Akidau
>Assignee: Thomas Groh
>
> The DSL_SQL branch recently started seeing a number of failures in tests when 
> running mvn verify, e.g.:
> {code:java}
> [ERROR] testUdaf(org.apache.beam.dsls.sql.BeamSqlDslUdfUdafTest)  Time 
> elapsed: 0.069 s  <<< ERROR!
> org.apache.beam.runners.direct.repackaged.runners.core.construction.java.repackaged.com.google.common.util.concurrent.UncheckedExecutionException:
>  java.lang.IllegalArgumentException
>   at 
> org.apache.beam.runners.direct.repackaged.runners.core.construction.java.repackaged.com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2214)
>   at 
> org.apache.beam.runners.direct.repackaged.runners.core.construction.java.repackaged.com.google.common.cache.LocalCache.get(LocalCache.java:4053)
>   at 
> org.apache.beam.runners.direct.repackaged.runners.core.construction.java.repackaged.com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:4057)
>   at 
> org.apache.beam.runners.direct.repackaged.runners.core.construction.java.repackaged.com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4986)
>   at 
> org.apache.beam.runners.direct.repackaged.runners.core.construction.RehydratedComponents.getCoder(RehydratedComponents.java:154)
>   at 
> org.apache.beam.runners.direct.repackaged.runners.core.construction.TestStreamTranslation.fromProto(TestStreamTranslation.java:69)
>   at 
> org.apache.beam.runners.direct.repackaged.runners.core.construction.TestStreamTranslation.getTestStream(TestStreamTranslation.java:102)
>   at 
> org.apache.beam.runners.direct.TestStreamEvaluatorFactory$DirectTestStreamFactory.getReplacementTransform(TestStreamEvaluatorFactory.java:179)
>   at org.apache.beam.sdk.Pipeline.applyReplacement(Pipeline.java:540)
>   at org.apache.beam.sdk.Pipeline.replace(Pipeline.java:280)
>   at org.apache.beam.sdk.Pipeline.replaceAll(Pipeline.java:201)
>   at 
> org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:169)
>   at org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:62)
>   at org.apache.beam.sdk.Pipeline.run(Pipeline.java:303)
>   at org.apache.beam.sdk.testing.TestPipeline.run(TestPipeline.java:344)
>   at org.apache.beam.sdk.testing.TestPipeline.run(TestPipeline.java:331)
>   at 
> org.apache.beam.dsls.sql.BeamSqlDslUdfUdafTest.testUdaf(BeamSqlDslUdfUdafTest.java:64)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
>   at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
>   at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
>   at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
>   at 
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
>   at 
> org.junit.rules.ExpectedException$ExpectedExceptionStatement.evaluate(ExpectedException.java:239)
>   at 
> org.apache.beam.sdk.testing.TestPipeline$1.evaluate(TestPipeline.java:320)
>   at org.junit.rules.RunRules.evaluate(RunRules.java:20)
>   at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
>   at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
>   at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
>   at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
>   at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
>   at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
>   at 

Build failed in Jenkins: beam_PostCommit_Python_Verify #2845

[Apache Jenkins Server]

See 


--
[...truncated 559.36 KB...]
  File was already downloaded 
/tmp/dataflow-requirements-cache/funcsigs-1.0.2.tar.gz
Collecting pbr>=0.11 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/pbr-3.1.1.tar.gz
Successfully downloaded pyhamcrest mock setuptools six funcsigs pbr
test_par_do_with_multiple_outputs_and_using_yield 
(apache_beam.transforms.ptransform_test.PTransformTest) ... ok
:135:
 UserWarning: Using fallback coder for typehint: List[Any].
  warnings.warn('Using fallback coder for typehint: %r.' % typehint)
DEPRECATION: pip install --download has been deprecated and will be removed in 
the future. Pip now has a download command that should be used instead.
Collecting pyhamcrest (from -r postcommit_requirements.txt (line 1))
  File was already downloaded 
/tmp/dataflow-requirements-cache/PyHamcrest-1.9.0.tar.gz
Collecting mock (from -r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/mock-2.0.0.tar.gz
Collecting setuptools (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded 
/tmp/dataflow-requirements-cache/setuptools-36.2.6.zip
Collecting six (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded /tmp/dataflow-requirements-cache/six-1.10.0.tar.gz
Collecting funcsigs>=1 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded 
/tmp/dataflow-requirements-cache/funcsigs-1.0.2.tar.gz
Collecting pbr>=0.11 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/pbr-3.1.1.tar.gz
Successfully downloaded pyhamcrest mock setuptools six funcsigs pbr
test_as_dict_twice (apache_beam.transforms.sideinputs_test.SideInputsTest) ... 
ok
DEPRECATION: pip install --download has been deprecated and will be removed in 
the future. Pip now has a download command that should be used instead.
Collecting pyhamcrest (from -r postcommit_requirements.txt (line 1))
  File was already downloaded 
/tmp/dataflow-requirements-cache/PyHamcrest-1.9.0.tar.gz
Collecting mock (from -r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/mock-2.0.0.tar.gz
Collecting setuptools (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded 
/tmp/dataflow-requirements-cache/setuptools-36.2.6.zip
Collecting six (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded /tmp/dataflow-requirements-cache/six-1.10.0.tar.gz
Collecting funcsigs>=1 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded 
/tmp/dataflow-requirements-cache/funcsigs-1.0.2.tar.gz
Collecting pbr>=0.11 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/pbr-3.1.1.tar.gz
Successfully downloaded pyhamcrest mock setuptools six funcsigs pbr
test_undeclared_outputs (apache_beam.transforms.ptransform_test.PTransformTest) 
... ok
:135:
 UserWarning: Using fallback coder for typehint: List[Any].
  warnings.warn('Using fallback coder for typehint: %r.' % typehint)
DEPRECATION: pip install --download has been deprecated and will be removed in 
the future. Pip now has a download command that should be used instead.
Collecting pyhamcrest (from -r postcommit_requirements.txt (line 1))
  File was already downloaded 
/tmp/dataflow-requirements-cache/PyHamcrest-1.9.0.tar.gz
Collecting mock (from -r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/mock-2.0.0.tar.gz
Collecting setuptools (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded 
/tmp/dataflow-requirements-cache/setuptools-36.2.6.zip
Collecting six (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded /tmp/dataflow-requirements-cache/six-1.10.0.tar.gz
Collecting funcsigs>=1 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded 
/tmp/dataflow-requirements-cache/funcsigs-1.0.2.tar.gz
Collecting pbr>=0.11 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/pbr-3.1.1.tar.gz
Successfully downloaded pyhamcrest mock setuptools six funcsigs pbr
test_as_list_and_as_dict_side_inputs 
(apache_beam.transforms.sideinputs_test.SideInputsTest) ... ok
DEPRECATION: pip install --download has been deprecated and will be removed in 
the future. Pip now has a download command that should be used instead.
Collecting pyhamcrest 

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Spark #2755

[Apache Jenkins Server]

See 

[jira] [Created] (BEAM-2710) Outputing "finished" when processing is not called in ParDoTest.TestDoFn leads to unexpected ElementCount

[Rafael Fernandez (JIRA)]

Rafael Fernandez created BEAM-2710:
--

 Summary: Outputing "finished" when processing is not called in 
ParDoTest.TestDoFn leads to unexpected ElementCount
 Key: BEAM-2710
 URL: https://issues.apache.org/jira/browse/BEAM-2710
 Project: Beam
  Issue Type: Bug
  Components: sdk-java-core
Reporter: Rafael Fernandez
Assignee: Davor Bonaci
Priority: Trivial


While it is legal to call finishBundle even if processElement was not called 
(say, when processing an empty input PCollection), it's best to guard against 
this case if we're examining correctness of outputs. Specifically, I was 
surprised at seeing ElementCount = 1 instead of 0 -- essentially, "finished" 
was being counted here.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BEAM-2708) Decompressing bzip2 files with multiple "streams" only reads the first stream

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16109810#comment-16109810
 ] 

ASF GitHub Bot commented on BEAM-2708:
--

GitHub user bjchambers opened a pull request:

https://github.com/apache/beam/pull/3670

[BEAM-2708] Configure BZIP2 to read all "streams"

Without this, CompressionMode.BZIP2 only supports "standard" bz2 files
containing a single stream. With this change, BZIP2 also supports bz2 files
containing multiple streams, such as those produced by pbzip2.

Follow this checklist to help us incorporate your contribution quickly and 
easily:

 - [ ] Make sure there is a [JIRA 
issue](https://issues.apache.org/jira/projects/BEAM/issues/) filed for the 
change (usually before you start working on it).  Trivial changes like typos do 
not require a JIRA issue.  Your pull request should address just this issue, 
without pulling in other changes.
 - [ ] Each commit in the pull request should have a meaningful subject 
line and body.
 - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue.
 - [ ] Write a pull request description that is detailed enough to 
understand what the pull request does, how, and why.
 - [ ] Run `mvn clean verify` to make sure basic checks pass. A more 
thorough check will be performed on your pull request automatically.
 - [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/bjchambers/beam pbzip2-cherrypick

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3670.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3670


commit 371a89ec7bdf4b92ebd7e346b376160253416a32
Author: bchambers 
Date:   2017-08-01T21:11:24Z

[BEAM-2708] Configure BZIP2 to read all "streams"

Without this, CompressionMode.BZIP2 only supports "standard" bz2 files
containing a single stream. With this change, BZIP2 also supports bz2 files
containing multiple streams, such as those produced by pbzip2.




> Decompressing bzip2 files with multiple "streams" only reads the first stream
> -
>
> Key: BEAM-2708
> URL: https://issues.apache.org/jira/browse/BEAM-2708
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions, sdk-py
>Reporter: Pablo Estrada
>Assignee: Ben Chambers
>
> I'm not sure which components to file this against. A user has observed that 
> pbzip2 files are not being properly decompressed:
> https://stackoverflow.com/questions/45439117/google-dataflow-only-partly-uncompressing-files-compressed-with-pbzip2



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] beam pull request #3670: [BEAM-2708] Configure BZIP2 to read all "streams"

[bjchambers]

GitHub user bjchambers opened a pull request:

https://github.com/apache/beam/pull/3670

[BEAM-2708] Configure BZIP2 to read all "streams"

Without this, CompressionMode.BZIP2 only supports "standard" bz2 files
containing a single stream. With this change, BZIP2 also supports bz2 files
containing multiple streams, such as those produced by pbzip2.

Follow this checklist to help us incorporate your contribution quickly and 
easily:

 - [ ] Make sure there is a [JIRA 
issue](https://issues.apache.org/jira/projects/BEAM/issues/) filed for the 
change (usually before you start working on it).  Trivial changes like typos do 
not require a JIRA issue.  Your pull request should address just this issue, 
without pulling in other changes.
 - [ ] Each commit in the pull request should have a meaningful subject 
line and body.
 - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue.
 - [ ] Write a pull request description that is detailed enough to 
understand what the pull request does, how, and why.
 - [ ] Run `mvn clean verify` to make sure basic checks pass. A more 
thorough check will be performed on your pull request automatically.
 - [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/bjchambers/beam pbzip2-cherrypick

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3670.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3670


commit 371a89ec7bdf4b92ebd7e346b376160253416a32
Author: bchambers 
Date:   2017-08-01T21:11:24Z

[BEAM-2708] Configure BZIP2 to read all "streams"

Without this, CompressionMode.BZIP2 only supports "standard" bz2 files
containing a single stream. With this change, BZIP2 also supports bz2 files
containing multiple streams, such as those produced by pbzip2.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-2566) Java SDK harness should not depend on any runner

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16109803#comment-16109803
 ] 

ASF GitHub Bot commented on BEAM-2566:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3647


> Java SDK harness should not depend on any runner
> 
>
> Key: BEAM-2566
> URL: https://issues.apache.org/jira/browse/BEAM-2566
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Kenneth Knowles
>Assignee: Thomas Groh
>
> Right now there is a dependency on the Dataflow runner. I believe this is 
> legacy due to using {{CloudObject}} temporarily but I do not claim to 
> understand the full nature of the dependency.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] beam pull request #3647: [BEAM-2566] Remove References to CloudObject from t...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3647


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/2] beam git commit: Remove References to CloudObject from the Java Harness

[tgroh]

Remove References to CloudObject from the Java Harness

Migrates to using the shared Runner API definitions.


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/64cf18fc
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/64cf18fc
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/64cf18fc

Branch: refs/heads/master
Commit: 64cf18fcdb4237189a5212b6476bdadf73a2ac7f
Parents: 3c81766
Author: Thomas Groh 
Authored: Wed Jul 26 15:34:23 2017 -0700
Committer: Thomas Groh 
Committed: Tue Aug 1 14:22:21 2017 -0700

--
 .../beam/fn/harness/BeamFnDataReadRunner.java   | 27 --
 .../beam/fn/harness/BeamFnDataWriteRunner.java  | 22 ++-
 .../fn/harness/BeamFnDataReadRunnerTest.java| 28 +--
 .../fn/harness/BeamFnDataWriteRunnerTest.java   | 24 ++--
 .../beam/fn/harness/FnApiDoFnRunnerTest.java| 29 
 5 files changed, 41 insertions(+), 89 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/64cf18fc/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/BeamFnDataReadRunner.java
--
diff --git 
a/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/BeamFnDataReadRunner.java
 
b/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/BeamFnDataReadRunner.java
index e2c17b0..1e611db 100644
--- 
a/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/BeamFnDataReadRunner.java
+++ 
b/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/BeamFnDataReadRunner.java
@@ -24,7 +24,6 @@ import com.fasterxml.jackson.databind.ObjectMapper;
 import com.google.auto.service.AutoService;
 import com.google.common.collect.ImmutableMap;
 import com.google.common.collect.Multimap;
-import com.google.protobuf.BytesValue;
 import java.io.IOException;
 import java.util.Collection;
 import java.util.Map;
@@ -35,8 +34,8 @@ import org.apache.beam.fn.harness.data.BeamFnDataClient;
 import org.apache.beam.fn.harness.fn.ThrowingConsumer;
 import org.apache.beam.fn.harness.fn.ThrowingRunnable;
 import org.apache.beam.fn.v1.BeamFnApi;
-import org.apache.beam.runners.dataflow.util.CloudObject;
-import org.apache.beam.runners.dataflow.util.CloudObjects;
+import org.apache.beam.runners.core.construction.CoderTranslation;
+import org.apache.beam.runners.core.construction.RehydratedComponents;
 import org.apache.beam.sdk.coders.Coder;
 import org.apache.beam.sdk.common.runner.v1.RunnerApi;
 import org.apache.beam.sdk.options.PipelineOptions;
@@ -91,8 +90,9 @@ public class BeamFnDataReadRunner {
   .setPrimitiveTransformReference(pTransformId)
   .setName(getOnlyElement(pTransform.getOutputsMap().keySet()))
   .build();
-  RunnerApi.Coder coderSpec = coders.get(pCollections.get(
-  getOnlyElement(pTransform.getOutputsMap().values())).getCoderId());
+  RunnerApi.Coder coderSpec =
+  coders.get(
+  
pCollections.get(getOnlyElement(pTransform.getOutputsMap().values())).getCoderId());
   Collection> consumers =
   (Collection) pCollectionIdsToConsumers.get(
   getOnlyElement(pTransform.getOutputsMap().values()));
@@ -102,6 +102,7 @@ public class BeamFnDataReadRunner {
   processBundleInstructionId,
   target,
   coderSpec,
+  coders,
   beamFnDataClient,
   consumers);
   addStartFunction.accept(runner::registerInputLocation);
@@ -124,6 +125,7 @@ public class BeamFnDataReadRunner {
   Supplier processBundleInstructionIdSupplier,
   BeamFnApi.Target inputTarget,
   RunnerApi.Coder coderSpec,
+  Map coders,
   BeamFnDataClient beamFnDataClientFactory,
   Collection> consumers)
   throws IOException {
@@ -137,17 +139,10 @@ public class BeamFnDataReadRunner {
 @SuppressWarnings("unchecked")
 Coder coder =
 (Coder)
-CloudObjects.coderFromCloudObject(
-CloudObject.fromSpec(
-OBJECT_MAPPER.readValue(
-coderSpec
-.getSpec()
-.getSpec()
-.getParameter()
-.unpack(BytesValue.class)
-.getValue()
-.newInput(),
-Map.class)));
+CoderTranslation.fromProto(
+coderSpec,
+RehydratedComponents.forComponents(
+
RunnerApi.Components.newBuilder().putAllCoders(coders).build()));
 this.coder = coder;
   }
 

[1/2] beam git commit: This closes #3647

[tgroh]

Repository: beam
Updated Branches:
  refs/heads/master 3c81766e2 -> 53c2c8fde


This closes #3647


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/53c2c8fd
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/53c2c8fd
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/53c2c8fd

Branch: refs/heads/master
Commit: 53c2c8fdecd82c77db7a7a87ac10d812e93d924b
Parents: 3c81766 64cf18f
Author: Thomas Groh 
Authored: Tue Aug 1 14:22:21 2017 -0700
Committer: Thomas Groh 
Committed: Tue Aug 1 14:22:21 2017 -0700

--
 .../beam/fn/harness/BeamFnDataReadRunner.java   | 27 --
 .../beam/fn/harness/BeamFnDataWriteRunner.java  | 22 ++-
 .../fn/harness/BeamFnDataReadRunnerTest.java| 28 +--
 .../fn/harness/BeamFnDataWriteRunnerTest.java   | 24 ++--
 .../beam/fn/harness/FnApiDoFnRunnerTest.java| 29 
 5 files changed, 41 insertions(+), 89 deletions(-)
--

[GitHub] beam pull request #3669: [BEAM-2708] Configure BZIP2 to read all "streams"

[bjchambers]

GitHub user bjchambers opened a pull request:

https://github.com/apache/beam/pull/3669

[BEAM-2708] Configure BZIP2 to read all "streams"

Without this, CompressionMode.BZIP2 only supports "standard" bz2 files
containing a single stream. With this change, BZIP2 also supports bz2 files
containing multiple streams, such as those produced by pbzip2.

Follow this checklist to help us incorporate your contribution quickly and 
easily:

 - [ ] Make sure there is a [JIRA 
issue](https://issues.apache.org/jira/projects/BEAM/issues/) filed for the 
change (usually before you start working on it).  Trivial changes like typos do 
not require a JIRA issue.  Your pull request should address just this issue, 
without pulling in other changes.
 - [ ] Each commit in the pull request should have a meaningful subject 
line and body.
 - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue.
 - [ ] Write a pull request description that is detailed enough to 
understand what the pull request does, how, and why.
 - [ ] Run `mvn clean verify` to make sure basic checks pass. A more 
thorough check will be performed on your pull request automatically.
 - [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/bjchambers/beam pbzip2

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3669.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3669


commit 5f82ea2569db3c2d287344700b212bdf3d9b7ea6
Author: bchambers 
Date:   2017-08-01T21:11:24Z

[BEAM-2708] Configure BZIP2 to read all "streams"

Without this, CompressionMode.BZIP2 only supports "standard" bz2 files
containing a single stream. With this change, BZIP2 also supports bz2 files
containing multiple streams, such as those produced by pbzip2.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-2708) Decompressing bzip2 files with multiple "streams" only reads the first stream

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16109787#comment-16109787
 ] 

ASF GitHub Bot commented on BEAM-2708:
--

GitHub user bjchambers opened a pull request:

https://github.com/apache/beam/pull/3669

[BEAM-2708] Configure BZIP2 to read all "streams"

Without this, CompressionMode.BZIP2 only supports "standard" bz2 files
containing a single stream. With this change, BZIP2 also supports bz2 files
containing multiple streams, such as those produced by pbzip2.

Follow this checklist to help us incorporate your contribution quickly and 
easily:

 - [ ] Make sure there is a [JIRA 
issue](https://issues.apache.org/jira/projects/BEAM/issues/) filed for the 
change (usually before you start working on it).  Trivial changes like typos do 
not require a JIRA issue.  Your pull request should address just this issue, 
without pulling in other changes.
 - [ ] Each commit in the pull request should have a meaningful subject 
line and body.
 - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue.
 - [ ] Write a pull request description that is detailed enough to 
understand what the pull request does, how, and why.
 - [ ] Run `mvn clean verify` to make sure basic checks pass. A more 
thorough check will be performed on your pull request automatically.
 - [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/bjchambers/beam pbzip2

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3669.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3669


commit 5f82ea2569db3c2d287344700b212bdf3d9b7ea6
Author: bchambers 
Date:   2017-08-01T21:11:24Z

[BEAM-2708] Configure BZIP2 to read all "streams"

Without this, CompressionMode.BZIP2 only supports "standard" bz2 files
containing a single stream. With this change, BZIP2 also supports bz2 files
containing multiple streams, such as those produced by pbzip2.




> Decompressing bzip2 files with multiple "streams" only reads the first stream
> -
>
> Key: BEAM-2708
> URL: https://issues.apache.org/jira/browse/BEAM-2708
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions, sdk-py
>Reporter: Pablo Estrada
>Assignee: Ben Chambers
>
> I'm not sure which components to file this against. A user has observed that 
> pbzip2 files are not being properly decompressed:
> https://stackoverflow.com/questions/45439117/google-dataflow-only-partly-uncompressing-files-compressed-with-pbzip2



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BEAM-2709) Add TezRunner

[Brandon Scheller (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16109754#comment-16109754
 ] 

Brandon Scheller commented on BEAM-2709:


Can this be assigned to me?

> Add TezRunner
> -
>
> Key: BEAM-2709
> URL: https://issues.apache.org/jira/browse/BEAM-2709
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-ideas
>Reporter: Brandon Scheller
>Assignee: Davor Bonaci
>
> Add a TezRunner to Beam



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)