[jira] [Work logged] (BEAM-4167) Implement UNNEST
[ https://issues.apache.org/jira/browse/BEAM-4167?focusedWorklogId=105999=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-105999 ] ASF GitHub Bot logged work on BEAM-4167: Author: ASF GitHub Bot Created on: 25/May/18 20:16 Start Date: 25/May/18 20:16 Worklog Time Spent: 10m Work Description: kennknowles commented on issue #5428: [BEAM-4167] Implement UNNEST URL: https://github.com/apache/beam/pull/5428#issuecomment-392168402 Thanks! Rewriting and merging. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 105999) Time Spent: 4h 10m (was: 4h) > Implement UNNEST > > > Key: BEAM-4167 > URL: https://issues.apache.org/jira/browse/BEAM-4167 > Project: Beam > Issue Type: New Feature > Components: dsl-sql >Reporter: Anton Kedin >Assignee: Kenneth Knowles >Priority: Major > Time Spent: 4h 10m > Remaining Estimate: 0h > > We need to be able to convert collections to relations in the query to > perform any meaningful operations on them. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-4167) Implement UNNEST
[ https://issues.apache.org/jira/browse/BEAM-4167?focusedWorklogId=105427=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-105427 ] ASF GitHub Bot logged work on BEAM-4167: Author: ASF GitHub Bot Created on: 24/May/18 02:16 Start Date: 24/May/18 02:16 Worklog Time Spent: 10m Work Description: kennknowles commented on issue #5428: [BEAM-4167] Implement UNNEST URL: https://github.com/apache/beam/pull/5428#issuecomment-391565591 Autoformat and permanent enforcement are now submitted prior to this PR and I have rebased it. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 105427) Time Spent: 4h (was: 3h 50m) > Implement UNNEST > > > Key: BEAM-4167 > URL: https://issues.apache.org/jira/browse/BEAM-4167 > Project: Beam > Issue Type: New Feature > Components: dsl-sql >Reporter: Anton Kedin >Assignee: Kenneth Knowles >Priority: Major > Time Spent: 4h > Remaining Estimate: 0h > > We need to be able to convert collections to relations in the query to > perform any meaningful operations on them. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-4167) Implement UNNEST
[ https://issues.apache.org/jira/browse/BEAM-4167?focusedWorklogId=105320=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-105320 ] ASF GitHub Bot logged work on BEAM-4167: Author: ASF GitHub Bot Created on: 23/May/18 20:59 Start Date: 23/May/18 20:59 Worklog Time Spent: 10m Work Description: kennknowles commented on issue #5428: [BEAM-4167] Implement UNNEST URL: https://github.com/apache/beam/pull/5428#issuecomment-391494400 R: @reuvenlax looking for a committer - can you take the second pass? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 105320) Time Spent: 3h 50m (was: 3h 40m) > Implement UNNEST > > > Key: BEAM-4167 > URL: https://issues.apache.org/jira/browse/BEAM-4167 > Project: Beam > Issue Type: New Feature > Components: dsl-sql >Reporter: Anton Kedin >Assignee: Kenneth Knowles >Priority: Major > Time Spent: 3h 50m > Remaining Estimate: 0h > > We need to be able to convert collections to relations in the query to > perform any meaningful operations on them. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-4167) Implement UNNEST
[ https://issues.apache.org/jira/browse/BEAM-4167?focusedWorklogId=105298=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-105298 ] ASF GitHub Bot logged work on BEAM-4167: Author: ASF GitHub Bot Created on: 23/May/18 19:54 Start Date: 23/May/18 19:54 Worklog Time Spent: 10m Work Description: kennknowles commented on issue #5428: [BEAM-4167] Implement UNNEST URL: https://github.com/apache/beam/pull/5428#issuecomment-391475475 Agreed. FWIW I've undone the formatting and just left checkstyle broken for now. To get a clean history I could put a format commit underneath this PR. I'm inclined to do it across the SQL module and then lock down the format. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 105298) Time Spent: 3h 40m (was: 3.5h) > Implement UNNEST > > > Key: BEAM-4167 > URL: https://issues.apache.org/jira/browse/BEAM-4167 > Project: Beam > Issue Type: New Feature > Components: dsl-sql >Reporter: Anton Kedin >Assignee: Kenneth Knowles >Priority: Major > Time Spent: 3h 40m > Remaining Estimate: 0h > > We need to be able to convert collections to relations in the query to > perform any meaningful operations on them. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-4167) Implement UNNEST
[ https://issues.apache.org/jira/browse/BEAM-4167?focusedWorklogId=105292=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-105292 ] ASF GitHub Bot logged work on BEAM-4167: Author: ASF GitHub Bot Created on: 23/May/18 19:48 Start Date: 23/May/18 19:48 Worklog Time Spent: 10m Work Description: apilloud commented on issue #5428: [BEAM-4167] Implement UNNEST URL: https://github.com/apache/beam/pull/5428#issuecomment-391474027 I don't disagree with the format. It would just be nice to have it in a separate commit. We should probably just run format on everything and then add a precommit to verify. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 105292) Time Spent: 3.5h (was: 3h 20m) > Implement UNNEST > > > Key: BEAM-4167 > URL: https://issues.apache.org/jira/browse/BEAM-4167 > Project: Beam > Issue Type: New Feature > Components: dsl-sql >Reporter: Anton Kedin >Assignee: Kenneth Knowles >Priority: Major > Time Spent: 3.5h > Remaining Estimate: 0h > > We need to be able to convert collections to relations in the query to > perform any meaningful operations on them. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-4167) Implement UNNEST
[ https://issues.apache.org/jira/browse/BEAM-4167?focusedWorklogId=105217=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-105217 ] ASF GitHub Bot logged work on BEAM-4167: Author: ASF GitHub Bot Created on: 23/May/18 18:14 Start Date: 23/May/18 18:14 Worklog Time Spent: 10m Work Description: kennknowles commented on issue #5428: [BEAM-4167] Implement UNNEST URL: https://github.com/apache/beam/pull/5428#issuecomment-391446537 (there's just a ton of little changes and it seems a waste of time to try to scope the formatting) This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 105217) Time Spent: 3h 20m (was: 3h 10m) > Implement UNNEST > > > Key: BEAM-4167 > URL: https://issues.apache.org/jira/browse/BEAM-4167 > Project: Beam > Issue Type: New Feature > Components: dsl-sql >Reporter: Anton Kedin >Assignee: Kenneth Knowles >Priority: Major > Time Spent: 3h 20m > Remaining Estimate: 0h > > We need to be able to convert collections to relations in the query to > perform any meaningful operations on them. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-4167) Implement UNNEST
[ https://issues.apache.org/jira/browse/BEAM-4167?focusedWorklogId=105216=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-105216 ] ASF GitHub Bot logged work on BEAM-4167: Author: ASF GitHub Bot Created on: 23/May/18 18:13 Start Date: 23/May/18 18:13 Worklog Time Spent: 10m Work Description: kennknowles commented on issue #5428: [BEAM-4167] Implement UNNEST URL: https://github.com/apache/beam/pull/5428#issuecomment-391446452 How about this - since Gradle makes it clear that the only failing task is the checkStyle, we finish LGTM with a committer, and I just autoformat prior to merging. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 105216) Time Spent: 3h 10m (was: 3h) > Implement UNNEST > > > Key: BEAM-4167 > URL: https://issues.apache.org/jira/browse/BEAM-4167 > Project: Beam > Issue Type: New Feature > Components: dsl-sql >Reporter: Anton Kedin >Assignee: Kenneth Knowles >Priority: Major > Time Spent: 3h 10m > Remaining Estimate: 0h > > We need to be able to convert collections to relations in the query to > perform any meaningful operations on them. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-4167) Implement UNNEST
[ https://issues.apache.org/jira/browse/BEAM-4167?focusedWorklogId=105215=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-105215 ] ASF GitHub Bot logged work on BEAM-4167: Author: ASF GitHub Bot Created on: 23/May/18 18:11 Start Date: 23/May/18 18:11 Worklog Time Spent: 10m Work Description: kennknowles commented on issue #5428: [BEAM-4167] Implement UNNEST URL: https://github.com/apache/beam/pull/5428#issuecomment-391445686 I can drop that last document commit and redo it with more tight scope. Some formatting is required because the new parameter makes lines too long. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 105215) Time Spent: 3h (was: 2h 50m) > Implement UNNEST > > > Key: BEAM-4167 > URL: https://issues.apache.org/jira/browse/BEAM-4167 > Project: Beam > Issue Type: New Feature > Components: dsl-sql >Reporter: Anton Kedin >Assignee: Kenneth Knowles >Priority: Major > Time Spent: 3h > Remaining Estimate: 0h > > We need to be able to convert collections to relations in the query to > perform any meaningful operations on them. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-4167) Implement UNNEST
[ https://issues.apache.org/jira/browse/BEAM-4167?focusedWorklogId=105178=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-105178 ] ASF GitHub Bot logged work on BEAM-4167: Author: ASF GitHub Bot Created on: 23/May/18 17:08 Start Date: 23/May/18 17:08 Worklog Time Spent: 10m Work Description: apilloud commented on issue #5428: [BEAM-4167] Implement UNNEST URL: https://github.com/apache/beam/pull/5428#issuecomment-391426096 Ugh on mixing the autoformat with functional changes. But LGTM. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 105178) Time Spent: 2h 50m (was: 2h 40m) > Implement UNNEST > > > Key: BEAM-4167 > URL: https://issues.apache.org/jira/browse/BEAM-4167 > Project: Beam > Issue Type: New Feature > Components: dsl-sql >Reporter: Anton Kedin >Assignee: Kenneth Knowles >Priority: Major > Time Spent: 2h 50m > Remaining Estimate: 0h > > We need to be able to convert collections to relations in the query to > perform any meaningful operations on them. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-4167) Implement UNNEST
[ https://issues.apache.org/jira/browse/BEAM-4167?focusedWorklogId=105107=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-105107 ] ASF GitHub Bot logged work on BEAM-4167: Author: ASF GitHub Bot Created on: 23/May/18 13:57 Start Date: 23/May/18 13:57 Worklog Time Spent: 10m Work Description: kennknowles commented on issue #5428: [BEAM-4167] Implement UNNEST URL: https://github.com/apache/beam/pull/5428#issuecomment-391356767 Fix checkstyle with autoformat. They were all tests where I didn't change the tests at all, just changed signatures. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 105107) Time Spent: 2h 40m (was: 2.5h) > Implement UNNEST > > > Key: BEAM-4167 > URL: https://issues.apache.org/jira/browse/BEAM-4167 > Project: Beam > Issue Type: New Feature > Components: dsl-sql >Reporter: Anton Kedin >Assignee: Kenneth Knowles >Priority: Major > Time Spent: 2h 40m > Remaining Estimate: 0h > > We need to be able to convert collections to relations in the query to > perform any meaningful operations on them. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-4167) Implement UNNEST
[
https://issues.apache.org/jira/browse/BEAM-4167?focusedWorklogId=104910=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-104910
]
ASF GitHub Bot logged work on BEAM-4167:
Author: ASF GitHub Bot
Created on: 23/May/18 00:46
Start Date: 23/May/18 00:46
Worklog Time Spent: 10m
Work Description: kennknowles commented on a change in pull request
#5428: [BEAM-4167] Implement UNNEST
URL: https://github.com/apache/beam/pull/5428#discussion_r190095184
##
File path:
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/BeamSqlPrimitive.java
##
@@ -111,54 +112,57 @@ public boolean accept() {
}
switch (outputType) {
-case BIGINT:
- return value instanceof Long;
-case DECIMAL:
- return value instanceof BigDecimal;
-case DOUBLE:
- return value instanceof Double;
-case FLOAT:
- return value instanceof Float;
-case INTEGER:
- return value instanceof Integer;
-case SMALLINT:
- return value instanceof Short;
-case TINYINT:
- return value instanceof Byte;
-case BOOLEAN:
- return value instanceof Boolean;
-case CHAR:
-case VARCHAR:
- return value instanceof String || value instanceof NlsString;
-case TIME:
- return value instanceof ReadableInstant;
-case TIMESTAMP:
-case DATE:
- return value instanceof ReadableInstant;
-case INTERVAL_SECOND:
-case INTERVAL_MINUTE:
-case INTERVAL_HOUR:
-case INTERVAL_DAY:
-case INTERVAL_MONTH:
-case INTERVAL_YEAR:
- return value instanceof BigDecimal;
-case SYMBOL:
- // for SYMBOL, it supports anything...
- return true;
-case ARRAY:
- return value instanceof List;
-case MAP:
- return value instanceof Map;
-case ROW:
- return value instanceof Row;
-default:
- throw new UnsupportedOperationException(
- "Unsupported Beam SQL type in expression: " + outputType.name());
+ case BIGINT:
+return value instanceof Long;
+ case DECIMAL:
+return value instanceof BigDecimal;
+ case DOUBLE:
+return value instanceof Double;
+ case FLOAT:
+return value instanceof Float;
+ case INTEGER:
+return value instanceof Integer;
+ case SMALLINT:
+return value instanceof Short;
+ case TINYINT:
+return value instanceof Byte;
+ case BOOLEAN:
+return value instanceof Boolean;
+ case CHAR:
+ case VARCHAR:
+return value instanceof String || value instanceof NlsString;
+ case TIME:
+return value instanceof ReadableInstant;
+ case TIMESTAMP:
+ case DATE:
+return value instanceof ReadableInstant;
+ case INTERVAL_SECOND:
+ case INTERVAL_MINUTE:
+ case INTERVAL_HOUR:
+ case INTERVAL_DAY:
+ case INTERVAL_MONTH:
+ case INTERVAL_YEAR:
+return value instanceof BigDecimal;
+ case SYMBOL:
+// for SYMBOL, it supports anything...
+return true;
+ case ARRAY:
+return value instanceof List;
+ case MAP:
+return value instanceof Map;
+ case ROW:
+return value instanceof Row;
+ case MULTISET:
+return value instanceof Iterable;
Review comment:
Yes, I believe that is correct. It seems to make semantic sense - anything
where you can get the values out of it, basically. And MULTISET is actually the
least constrained of any collection (am I right?) so I think anything works.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 104910)
Time Spent: 2h 20m (was: 2h 10m)
> Implement UNNEST
>
>
> Key: BEAM-4167
> URL: https://issues.apache.org/jira/browse/BEAM-4167
> Project: Beam
> Issue Type: New Feature
> Components: dsl-sql
>Reporter: Anton Kedin
>Assignee: Kenneth Knowles
>Priority: Major
> Time Spent: 2h 20m
> Remaining Estimate: 0h
>
> We need to be able to convert collections to relations in the query to
> perform any meaningful operations on them.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-4167) Implement UNNEST
[
https://issues.apache.org/jira/browse/BEAM-4167?focusedWorklogId=104911=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-104911
]
ASF GitHub Bot logged work on BEAM-4167:
Author: ASF GitHub Bot
Created on: 23/May/18 00:46
Start Date: 23/May/18 00:46
Worklog Time Spent: 10m
Work Description: kennknowles commented on a change in pull request
#5428: [BEAM-4167] Implement UNNEST
URL: https://github.com/apache/beam/pull/5428#discussion_r190095082
##
File path:
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/BeamSqlCorrelVariableExpression.java
##
@@ -0,0 +1,56 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.impl.interpreter.operator;
+
+import static com.google.common.base.Preconditions.checkState;
+
+import com.google.common.collect.ImmutableMap;
+import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
+import org.apache.beam.sdk.values.Row;
+import org.apache.calcite.sql.type.SqlTypeName;
+
+/** A primitive operation for deferencing a correlation variable. */
+public class BeamSqlCorrelVariableExpression extends BeamSqlExpression {
+
+ private final int correlationId;
+
+ public BeamSqlCorrelVariableExpression(SqlTypeName sqlTypeName, int
correlationId) {
+super(null, sqlTypeName);
+this.correlationId = correlationId;
+ }
+
+ @Override
+ public boolean accept() {
+return true;
+ }
+
+ @Override
+ public BeamSqlPrimitive evaluate(
+ Row inputRow, BoundedWindow window, ImmutableMap
correlateEnv) {
Review comment:
Oh, I think that the correlation expression - this class - will work in
general. It can be nested arbitrarily AFAIK. The place where that support is
missing is at the `Rel` level. If we had a general `BeamCorrelateRel` then it
would have to add its var to the environment just like the `BeamUnnestRel`
does. And to be general, all of the `Rel` bits would also need to also know
about the environment and do something clever. I'm not sure that would be the
overall solution - likely we want to successfully use Calcite's decorrelate. It
did not seem to actually decorrelate this particular query, though I might have
just been using it wrong.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 104911)
Time Spent: 2h 20m (was: 2h 10m)
> Implement UNNEST
>
>
> Key: BEAM-4167
> URL: https://issues.apache.org/jira/browse/BEAM-4167
> Project: Beam
> Issue Type: New Feature
> Components: dsl-sql
>Reporter: Anton Kedin
>Assignee: Kenneth Knowles
>Priority: Major
> Time Spent: 2h 20m
> Remaining Estimate: 0h
>
> We need to be able to convert collections to relations in the query to
> perform any meaningful operations on them.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-4167) Implement UNNEST
[
https://issues.apache.org/jira/browse/BEAM-4167?focusedWorklogId=104912=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-104912
]
ASF GitHub Bot logged work on BEAM-4167:
Author: ASF GitHub Bot
Created on: 23/May/18 00:46
Start Date: 23/May/18 00:46
Worklog Time Spent: 10m
Work Description: kennknowles commented on a change in pull request
#5428: [BEAM-4167] Implement UNNEST
URL: https://github.com/apache/beam/pull/5428#discussion_r190095340
##
File path:
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamUnnestRel.java
##
@@ -0,0 +1,157 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.impl.rel;
+
+import static com.google.common.base.Preconditions.checkArgument;
+import static com.google.common.base.Preconditions.checkState;
+
+import com.google.common.collect.ImmutableMap;
+import java.util.List;
+import
org.apache.beam.sdk.extensions.sql.impl.interpreter.BeamSqlExpressionExecutor;
+import org.apache.beam.sdk.extensions.sql.impl.interpreter.BeamSqlFnExecutor;
+import org.apache.beam.sdk.extensions.sql.impl.schema.BeamTableUtils;
+import org.apache.beam.sdk.extensions.sql.impl.utils.CalciteUtils;
+import org.apache.beam.sdk.schemas.Schema;
+import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.transforms.PTransform;
+import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.PCollectionTuple;
+import org.apache.beam.sdk.values.Row;
+import org.apache.calcite.plan.RelOptCluster;
+import org.apache.calcite.plan.RelTraitSet;
+import org.apache.calcite.rel.RelNode;
+import org.apache.calcite.rel.core.Correlate;
+import org.apache.calcite.rel.core.CorrelationId;
+import org.apache.calcite.rel.core.Uncollect;
+import org.apache.calcite.sql.SemiJoinType;
+import org.apache.calcite.util.ImmutableBitSet;
+
+/**
+ * {@link BeamRelNode} to implement UNNEST, supporting specifically only
{@link Correlate} with
+ * {@link Uncollect}.
+ */
+public class BeamUnnestRel extends Correlate implements BeamRelNode {
+
+ public BeamUnnestRel(
+ RelOptCluster cluster,
+ RelTraitSet traits,
+ RelNode left,
+ RelNode right,
+ CorrelationId correlationId,
+ ImmutableBitSet requiredColumns,
+ SemiJoinType joinType) {
+super(cluster, traits, left, right, correlationId, requiredColumns,
joinType);
+ }
+
+ @Override
+ public PTransform toPTransform() {
+return new Transform();
+ }
+
+ @Override
+ public Correlate copy(
+ RelTraitSet relTraitSet,
+ RelNode left,
+ RelNode right,
+ CorrelationId correlationId,
+ ImmutableBitSet requireColumns,
+ SemiJoinType joinType) {
+return new BeamUnnestRel(
+getCluster(), relTraitSet, left, right, correlationId,
requiredColumns, joinType);
+ }
+
+ private class Transform extends PTransform {
+@Override
+public PCollection expand(PCollectionTuple inputPCollections) {
+ String stageName = BeamSqlRelUtils.getStageName(BeamUnnestRel.this);
+
+ // The set of rows where we run the correlated unnest for each row
+ PCollection outer =
+
inputPCollections.apply(BeamSqlRelUtils.getBeamRelInput(left).toPTransform());
+
+ // The correlated subquery
+ BeamUncollectRel uncollect = (BeamUncollectRel)
BeamSqlRelUtils.getBeamRelInput(right);
+ Schema innerSchema = CalciteUtils.toBeamSchema(uncollect.getRowType());
+ checkArgument(
+ innerSchema.getFieldCount() == 1, "Can only UNNEST a single column",
getClass());
+
+ BeamSqlExpressionExecutor expr =
+ new
BeamSqlFnExecutor(BeamSqlRelUtils.getBeamRelInput(uncollect.getInput()));
+
+ Schema joinedSchema = CalciteUtils.toBeamSchema(rowType);
+
+ return outer
+ .apply(
+ stageName,
+ ParDo.of(
+ new UnnestFn(correlationId.getId(),
[jira] [Work logged] (BEAM-4167) Implement UNNEST
[
https://issues.apache.org/jira/browse/BEAM-4167?focusedWorklogId=104815=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-104815
]
ASF GitHub Bot logged work on BEAM-4167:
Author: ASF GitHub Bot
Created on: 22/May/18 21:44
Start Date: 22/May/18 21:44
Worklog Time Spent: 10m
Work Description: akedin commented on a change in pull request #5428:
[BEAM-4167] Implement UNNEST
URL: https://github.com/apache/beam/pull/5428#discussion_r190063251
##
File path:
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/BeamSqlCorrelVariableExpression.java
##
@@ -0,0 +1,56 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.impl.interpreter.operator;
+
+import static com.google.common.base.Preconditions.checkState;
+
+import com.google.common.collect.ImmutableMap;
+import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
+import org.apache.beam.sdk.values.Row;
+import org.apache.calcite.sql.type.SqlTypeName;
+
+/** A primitive operation for deferencing a correlation variable. */
+public class BeamSqlCorrelVariableExpression extends BeamSqlExpression {
+
+ private final int correlationId;
+
+ public BeamSqlCorrelVariableExpression(SqlTypeName sqlTypeName, int
correlationId) {
+super(null, sqlTypeName);
+this.correlationId = correlationId;
+ }
+
+ @Override
+ public boolean accept() {
+return true;
+ }
+
+ @Override
+ public BeamSqlPrimitive evaluate(
+ Row inputRow, BoundedWindow window, ImmutableMap
correlateEnv) {
Review comment:
I see. Would it work for complex cases, like multiple nested correlated
queries? What the plan would look like? I don't see where this is handled,
maybe I missed it.
If not, then it probably makes sense to comment on correlation logic that
it's only there to support unnest and is limited. And is there a need for a map
in this case? Wouldn't a single id work?
And should we wrap the map into some kind of execution context class?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 104815)
Time Spent: 2h 10m (was: 2h)
> Implement UNNEST
>
>
> Key: BEAM-4167
> URL: https://issues.apache.org/jira/browse/BEAM-4167
> Project: Beam
> Issue Type: New Feature
> Components: dsl-sql
>Reporter: Anton Kedin
>Assignee: Kenneth Knowles
>Priority: Major
> Time Spent: 2h 10m
> Remaining Estimate: 0h
>
> We need to be able to convert collections to relations in the query to
> perform any meaningful operations on them.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-4167) Implement UNNEST
[ https://issues.apache.org/jira/browse/BEAM-4167?focusedWorklogId=104746=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-104746 ] ASF GitHub Bot logged work on BEAM-4167: Author: ASF GitHub Bot Created on: 22/May/18 18:35 Start Date: 22/May/18 18:35 Worklog Time Spent: 10m Work Description: kennknowles commented on issue #5428: [BEAM-4167] Implement UNNEST URL: https://github.com/apache/beam/pull/5428#issuecomment-391096205 run java precommit This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 104746) Time Spent: 2h (was: 1h 50m) > Implement UNNEST > > > Key: BEAM-4167 > URL: https://issues.apache.org/jira/browse/BEAM-4167 > Project: Beam > Issue Type: New Feature > Components: dsl-sql >Reporter: Anton Kedin >Assignee: Kenneth Knowles >Priority: Major > Time Spent: 2h > Remaining Estimate: 0h > > We need to be able to convert collections to relations in the query to > perform any meaningful operations on them. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-4167) Implement UNNEST
[ https://issues.apache.org/jira/browse/BEAM-4167?focusedWorklogId=104210=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-104210 ] ASF GitHub Bot logged work on BEAM-4167: Author: ASF GitHub Bot Created on: 21/May/18 19:57 Start Date: 21/May/18 19:57 Worklog Time Spent: 10m Work Description: kennknowles commented on a change in pull request #5428: [BEAM-4167] Implement UNNEST URL: https://github.com/apache/beam/pull/5428#discussion_r189695916 ## File path: sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/BeamSqlCorrelVariableExpression.java ## @@ -0,0 +1,56 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.beam.sdk.extensions.sql.impl.interpreter.operator; + +import static com.google.common.base.Preconditions.checkState; + +import com.google.common.collect.ImmutableMap; +import org.apache.beam.sdk.transforms.windowing.BoundedWindow; +import org.apache.beam.sdk.values.Row; +import org.apache.calcite.sql.type.SqlTypeName; + +/** A primitive operation for deferencing a correlation variable. */ Review comment: Done This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 104210) Time Spent: 1h 50m (was: 1h 40m) > Implement UNNEST > > > Key: BEAM-4167 > URL: https://issues.apache.org/jira/browse/BEAM-4167 > Project: Beam > Issue Type: New Feature > Components: dsl-sql >Reporter: Anton Kedin >Assignee: Kenneth Knowles >Priority: Major > Time Spent: 1h 50m > Remaining Estimate: 0h > > We need to be able to convert collections to relations in the query to > perform any meaningful operations on them. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-4167) Implement UNNEST
[
https://issues.apache.org/jira/browse/BEAM-4167?focusedWorklogId=104209=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-104209
]
ASF GitHub Bot logged work on BEAM-4167:
Author: ASF GitHub Bot
Created on: 21/May/18 19:57
Start Date: 21/May/18 19:57
Worklog Time Spent: 10m
Work Description: kennknowles commented on a change in pull request
#5428: [BEAM-4167] Implement UNNEST
URL: https://github.com/apache/beam/pull/5428#discussion_r189696204
##
File path:
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/BeamSqlCorrelVariableExpression.java
##
@@ -0,0 +1,56 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.impl.interpreter.operator;
+
+import static com.google.common.base.Preconditions.checkState;
+
+import com.google.common.collect.ImmutableMap;
+import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
+import org.apache.beam.sdk.values.Row;
+import org.apache.calcite.sql.type.SqlTypeName;
+
+/** A primitive operation for deferencing a correlation variable. */
+public class BeamSqlCorrelVariableExpression extends BeamSqlExpression {
+
+ private final int correlationId;
+
+ public BeamSqlCorrelVariableExpression(SqlTypeName sqlTypeName, int
correlationId) {
+super(null, sqlTypeName);
+this.correlationId = correlationId;
+ }
+
+ @Override
+ public boolean accept() {
+return true;
Review comment:
Good to know. In this case, it cannot even be checked properly. The `accept`
method is trying to do type checking but with no type environment. I could
extend them all to plumb a type `correlateEnv` but if it was broken it means
Calcite is broken since it adds the variables.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 104209)
Time Spent: 1h 40m (was: 1.5h)
> Implement UNNEST
>
>
> Key: BEAM-4167
> URL: https://issues.apache.org/jira/browse/BEAM-4167
> Project: Beam
> Issue Type: New Feature
> Components: dsl-sql
>Reporter: Anton Kedin
>Assignee: Kenneth Knowles
>Priority: Major
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> We need to be able to convert collections to relations in the query to
> perform any meaningful operations on them.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-4167) Implement UNNEST
[
https://issues.apache.org/jira/browse/BEAM-4167?focusedWorklogId=104197=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-104197
]
ASF GitHub Bot logged work on BEAM-4167:
Author: ASF GitHub Bot
Created on: 21/May/18 19:05
Start Date: 21/May/18 19:05
Worklog Time Spent: 10m
Work Description: kennknowles commented on a change in pull request
#5428: [BEAM-4167] Implement UNNEST
URL: https://github.com/apache/beam/pull/5428#discussion_r189684154
##
File path:
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/BeamSqlCorrelVariableExpression.java
##
@@ -0,0 +1,56 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.impl.interpreter.operator;
+
+import static com.google.common.base.Preconditions.checkState;
+
+import com.google.common.collect.ImmutableMap;
+import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
+import org.apache.beam.sdk.values.Row;
+import org.apache.calcite.sql.type.SqlTypeName;
+
+/** A primitive operation for deferencing a correlation variable. */
+public class BeamSqlCorrelVariableExpression extends BeamSqlExpression {
+
+ private final int correlationId;
+
+ public BeamSqlCorrelVariableExpression(SqlTypeName sqlTypeName, int
correlationId) {
+super(null, sqlTypeName);
+this.correlationId = correlationId;
+ }
+
+ @Override
+ public boolean accept() {
+return true;
+ }
+
+ @Override
+ public BeamSqlPrimitive evaluate(
+ Row inputRow, BoundedWindow window, ImmutableMap
correlateEnv) {
Review comment:
Only in BeamUnnestRel. Other than that, it is just passed around. The
meaning is that the main PCollection input is an outer loop, and each time
through the current row (or whatever) gets set to the $cor0 in the
correlateEnv. Then the inner loop runs the project expression and pulls out the
fields from the current row.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 104197)
Time Spent: 1.5h (was: 1h 20m)
> Implement UNNEST
>
>
> Key: BEAM-4167
> URL: https://issues.apache.org/jira/browse/BEAM-4167
> Project: Beam
> Issue Type: New Feature
> Components: dsl-sql
>Reporter: Anton Kedin
>Assignee: Kenneth Knowles
>Priority: Major
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> We need to be able to convert collections to relations in the query to
> perform any meaningful operations on them.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-4167) Implement UNNEST
[
https://issues.apache.org/jira/browse/BEAM-4167?focusedWorklogId=104161=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-104161
]
ASF GitHub Bot logged work on BEAM-4167:
Author: ASF GitHub Bot
Created on: 21/May/18 17:51
Start Date: 21/May/18 17:51
Worklog Time Spent: 10m
Work Description: akedin commented on a change in pull request #5428:
[BEAM-4167] Implement UNNEST
URL: https://github.com/apache/beam/pull/5428#discussion_r189663755
##
File path:
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/BeamSqlCorrelVariableExpression.java
##
@@ -0,0 +1,56 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.impl.interpreter.operator;
+
+import static com.google.common.base.Preconditions.checkState;
+
+import com.google.common.collect.ImmutableMap;
+import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
+import org.apache.beam.sdk.values.Row;
+import org.apache.calcite.sql.type.SqlTypeName;
+
+/** A primitive operation for deferencing a correlation variable. */
+public class BeamSqlCorrelVariableExpression extends BeamSqlExpression {
+
+ private final int correlationId;
+
+ public BeamSqlCorrelVariableExpression(SqlTypeName sqlTypeName, int
correlationId) {
+super(null, sqlTypeName);
+this.correlationId = correlationId;
+ }
+
+ @Override
+ public boolean accept() {
+return true;
+ }
+
+ @Override
+ public BeamSqlPrimitive evaluate(
+ Row inputRow, BoundedWindow window, ImmutableMap
correlateEnv) {
Review comment:
where does correlateEnv get populated?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 104161)
Time Spent: 1h 20m (was: 1h 10m)
> Implement UNNEST
>
>
> Key: BEAM-4167
> URL: https://issues.apache.org/jira/browse/BEAM-4167
> Project: Beam
> Issue Type: New Feature
> Components: dsl-sql
>Reporter: Anton Kedin
>Assignee: Kenneth Knowles
>Priority: Major
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> We need to be able to convert collections to relations in the query to
> perform any meaningful operations on them.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-4167) Implement UNNEST
[
https://issues.apache.org/jira/browse/BEAM-4167?focusedWorklogId=104158=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-104158
]
ASF GitHub Bot logged work on BEAM-4167:
Author: ASF GitHub Bot
Created on: 21/May/18 17:51
Start Date: 21/May/18 17:51
Worklog Time Spent: 10m
Work Description: akedin commented on a change in pull request #5428:
[BEAM-4167] Implement UNNEST
URL: https://github.com/apache/beam/pull/5428#discussion_r189654159
##
File path:
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/BeamSqlPrimitive.java
##
@@ -111,54 +112,57 @@ public boolean accept() {
}
switch (outputType) {
-case BIGINT:
- return value instanceof Long;
-case DECIMAL:
- return value instanceof BigDecimal;
-case DOUBLE:
- return value instanceof Double;
-case FLOAT:
- return value instanceof Float;
-case INTEGER:
- return value instanceof Integer;
-case SMALLINT:
- return value instanceof Short;
-case TINYINT:
- return value instanceof Byte;
-case BOOLEAN:
- return value instanceof Boolean;
-case CHAR:
-case VARCHAR:
- return value instanceof String || value instanceof NlsString;
-case TIME:
- return value instanceof ReadableInstant;
-case TIMESTAMP:
-case DATE:
- return value instanceof ReadableInstant;
-case INTERVAL_SECOND:
-case INTERVAL_MINUTE:
-case INTERVAL_HOUR:
-case INTERVAL_DAY:
-case INTERVAL_MONTH:
-case INTERVAL_YEAR:
- return value instanceof BigDecimal;
-case SYMBOL:
- // for SYMBOL, it supports anything...
- return true;
-case ARRAY:
- return value instanceof List;
-case MAP:
- return value instanceof Map;
-case ROW:
- return value instanceof Row;
-default:
- throw new UnsupportedOperationException(
- "Unsupported Beam SQL type in expression: " + outputType.name());
+ case BIGINT:
+return value instanceof Long;
+ case DECIMAL:
+return value instanceof BigDecimal;
+ case DOUBLE:
+return value instanceof Double;
+ case FLOAT:
+return value instanceof Float;
+ case INTEGER:
+return value instanceof Integer;
+ case SMALLINT:
+return value instanceof Short;
+ case TINYINT:
+return value instanceof Byte;
+ case BOOLEAN:
+return value instanceof Boolean;
+ case CHAR:
+ case VARCHAR:
+return value instanceof String || value instanceof NlsString;
+ case TIME:
+return value instanceof ReadableInstant;
+ case TIMESTAMP:
+ case DATE:
+return value instanceof ReadableInstant;
+ case INTERVAL_SECOND:
+ case INTERVAL_MINUTE:
+ case INTERVAL_HOUR:
+ case INTERVAL_DAY:
+ case INTERVAL_MONTH:
+ case INTERVAL_YEAR:
+return value instanceof BigDecimal;
+ case SYMBOL:
+// for SYMBOL, it supports anything...
+return true;
+ case ARRAY:
+return value instanceof List;
+ case MAP:
+return value instanceof Map;
+ case ROW:
+return value instanceof Row;
+ case MULTISET:
+return value instanceof Iterable;
Review comment:
this says that Multiset can be represented by any iterable, is this correct?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 104158)
Time Spent: 1h (was: 50m)
> Implement UNNEST
>
>
> Key: BEAM-4167
> URL: https://issues.apache.org/jira/browse/BEAM-4167
> Project: Beam
> Issue Type: New Feature
> Components: dsl-sql
>Reporter: Anton Kedin
>Assignee: Kenneth Knowles
>Priority: Major
> Time Spent: 1h
> Remaining Estimate: 0h
>
> We need to be able to convert collections to relations in the query to
> perform any meaningful operations on them.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-4167) Implement UNNEST
[ https://issues.apache.org/jira/browse/BEAM-4167?focusedWorklogId=104159=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-104159 ] ASF GitHub Bot logged work on BEAM-4167: Author: ASF GitHub Bot Created on: 21/May/18 17:51 Start Date: 21/May/18 17:51 Worklog Time Spent: 10m Work Description: akedin commented on a change in pull request #5428: [BEAM-4167] Implement UNNEST URL: https://github.com/apache/beam/pull/5428#discussion_r189662258 ## File path: sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/BeamSqlCorrelVariableExpression.java ## @@ -0,0 +1,56 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.beam.sdk.extensions.sql.impl.interpreter.operator; + +import static com.google.common.base.Preconditions.checkState; + +import com.google.common.collect.ImmutableMap; +import org.apache.beam.sdk.transforms.windowing.BoundedWindow; +import org.apache.beam.sdk.values.Row; +import org.apache.calcite.sql.type.SqlTypeName; + +/** A primitive operation for deferencing a correlation variable. */ Review comment: type: dereferencing? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 104159) Time Spent: 1h 10m (was: 1h) > Implement UNNEST > > > Key: BEAM-4167 > URL: https://issues.apache.org/jira/browse/BEAM-4167 > Project: Beam > Issue Type: New Feature > Components: dsl-sql >Reporter: Anton Kedin >Assignee: Kenneth Knowles >Priority: Major > Time Spent: 1h 10m > Remaining Estimate: 0h > > We need to be able to convert collections to relations in the query to > perform any meaningful operations on them. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-4167) Implement UNNEST
[
https://issues.apache.org/jira/browse/BEAM-4167?focusedWorklogId=104160=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-104160
]
ASF GitHub Bot logged work on BEAM-4167:
Author: ASF GitHub Bot
Created on: 21/May/18 17:51
Start Date: 21/May/18 17:51
Worklog Time Spent: 10m
Work Description: akedin commented on a change in pull request #5428:
[BEAM-4167] Implement UNNEST
URL: https://github.com/apache/beam/pull/5428#discussion_r189662435
##
File path:
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/BeamSqlCorrelVariableExpression.java
##
@@ -0,0 +1,56 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.impl.interpreter.operator;
+
+import static com.google.common.base.Preconditions.checkState;
+
+import com.google.common.collect.ImmutableMap;
+import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
+import org.apache.beam.sdk.values.Row;
+import org.apache.calcite.sql.type.SqlTypeName;
+
+/** A primitive operation for deferencing a correlation variable. */
+public class BeamSqlCorrelVariableExpression extends BeamSqlExpression {
+
+ private final int correlationId;
+
+ public BeamSqlCorrelVariableExpression(SqlTypeName sqlTypeName, int
correlationId) {
+super(null, sqlTypeName);
+this.correlationId = correlationId;
+ }
+
+ @Override
+ public boolean accept() {
+return true;
Review comment:
fyi, there's a bug in fn executor where we don't even call this method for a
bunch of expressions
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 104160)
> Implement UNNEST
>
>
> Key: BEAM-4167
> URL: https://issues.apache.org/jira/browse/BEAM-4167
> Project: Beam
> Issue Type: New Feature
> Components: dsl-sql
>Reporter: Anton Kedin
>Assignee: Kenneth Knowles
>Priority: Major
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> We need to be able to convert collections to relations in the query to
> perform any meaningful operations on them.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-4167) Implement UNNEST
[ https://issues.apache.org/jira/browse/BEAM-4167?focusedWorklogId=103838=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-103838 ] ASF GitHub Bot logged work on BEAM-4167: Author: ASF GitHub Bot Created on: 21/May/18 03:58 Start Date: 21/May/18 03:58 Worklog Time Spent: 10m Work Description: kennknowles commented on issue #5428: [BEAM-4167] Implement UNNEST URL: https://github.com/apache/beam/pull/5428#issuecomment-390546863 R: @XuMingmin WDYT? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 103838) Time Spent: 50m (was: 40m) > Implement UNNEST > > > Key: BEAM-4167 > URL: https://issues.apache.org/jira/browse/BEAM-4167 > Project: Beam > Issue Type: New Feature > Components: dsl-sql >Reporter: Anton Kedin >Assignee: Kenneth Knowles >Priority: Major > Time Spent: 50m > Remaining Estimate: 0h > > We need to be able to convert collections to relations in the query to > perform any meaningful operations on them. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-4167) Implement UNNEST
[ https://issues.apache.org/jira/browse/BEAM-4167?focusedWorklogId=103836=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-103836 ] ASF GitHub Bot logged work on BEAM-4167: Author: ASF GitHub Bot Created on: 21/May/18 03:57 Start Date: 21/May/18 03:57 Worklog Time Spent: 10m Work Description: kennknowles commented on issue #5428: [BEAM-4167] Implement UNNEST URL: https://github.com/apache/beam/pull/5428#issuecomment-390546800 run java precommit This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 103836) Time Spent: 40m (was: 0.5h) > Implement UNNEST > > > Key: BEAM-4167 > URL: https://issues.apache.org/jira/browse/BEAM-4167 > Project: Beam > Issue Type: New Feature > Components: dsl-sql >Reporter: Anton Kedin >Assignee: Kenneth Knowles >Priority: Major > Time Spent: 40m > Remaining Estimate: 0h > > We need to be able to convert collections to relations in the query to > perform any meaningful operations on them. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-4167) Implement UNNEST
[ https://issues.apache.org/jira/browse/BEAM-4167?focusedWorklogId=103593=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-103593 ] ASF GitHub Bot logged work on BEAM-4167: Author: ASF GitHub Bot Created on: 18/May/18 23:31 Start Date: 18/May/18 23:31 Worklog Time Spent: 10m Work Description: kennknowles commented on issue #5428: [BEAM-4167] Implement UNNEST URL: https://github.com/apache/beam/pull/5428#issuecomment-390357844 run java precommit This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 103593) Time Spent: 0.5h (was: 20m) > Implement UNNEST > > > Key: BEAM-4167 > URL: https://issues.apache.org/jira/browse/BEAM-4167 > Project: Beam > Issue Type: New Feature > Components: dsl-sql >Reporter: Anton Kedin >Assignee: Kenneth Knowles >Priority: Major > Time Spent: 0.5h > Remaining Estimate: 0h > > We need to be able to convert collections to relations in the query to > perform any meaningful operations on them. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-4167) Implement UNNEST
[ https://issues.apache.org/jira/browse/BEAM-4167?focusedWorklogId=103579=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-103579 ] ASF GitHub Bot logged work on BEAM-4167: Author: ASF GitHub Bot Created on: 18/May/18 22:53 Start Date: 18/May/18 22:53 Worklog Time Spent: 10m Work Description: kennknowles commented on issue #5428: [BEAM-4167] Implement UNNEST URL: https://github.com/apache/beam/pull/5428#issuecomment-390352654 @apilloud @akedin finally This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 103579) Time Spent: 20m (was: 10m) > Implement UNNEST > > > Key: BEAM-4167 > URL: https://issues.apache.org/jira/browse/BEAM-4167 > Project: Beam > Issue Type: New Feature > Components: dsl-sql >Reporter: Anton Kedin >Assignee: Kenneth Knowles >Priority: Major > Time Spent: 20m > Remaining Estimate: 0h > > We need to be able to convert collections to relations in the query to > perform any meaningful operations on them. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-4167) Implement UNNEST
[ https://issues.apache.org/jira/browse/BEAM-4167?focusedWorklogId=103578=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-103578 ] ASF GitHub Bot logged work on BEAM-4167: Author: ASF GitHub Bot Created on: 18/May/18 22:43 Start Date: 18/May/18 22:43 Worklog Time Spent: 10m Work Description: kennknowles opened a new pull request #5428: [BEAM-4167] WIP Implement UNNEST URL: https://github.com/apache/beam/pull/5428 DO NOT REVIEW - opening for self-review Follow this checklist to help us incorporate your contribution quickly and easily: - [x] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [x] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). It will help us expedite review of your Pull Request if you tag someone (e.g. `@username`) to look at it. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 103578) Time Spent: 10m Remaining Estimate: 0h > Implement UNNEST > > > Key: BEAM-4167 > URL: https://issues.apache.org/jira/browse/BEAM-4167 > Project: Beam > Issue Type: New Feature > Components: dsl-sql >Reporter: Anton Kedin >Assignee: Kenneth Knowles >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > We need to be able to convert collections to relations in the query to > perform any meaningful operations on them. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
