[jira] [Updated] (BEAM-5202) register UDF/UDAF with ServiceLoader

[Xu Mingmin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5202?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xu Mingmin updated BEAM-5202:
-
Fix Version/s: (was: Not applicable)
   2.8.0

> register UDF/UDAF with ServiceLoader
> 
>
> Key: BEAM-5202
> URL: https://issues.apache.org/jira/browse/BEAM-5202
> Project: Beam
>  Issue Type: Improvement
>  Components: dsl-sql
>Reporter: Xu Mingmin
>Assignee: Xu Mingmin
>Priority: Major
> Fix For: 2.8.0
>
>  Time Spent: 3h 10m
>  Remaining Estimate: 0h
>
> create a ServiceLoader for UDF/UDAF register, if the autoLoad option is set 
> to TRUE.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-5690) Issue with GroupByKey in BeamSql using SparkRunner

[Xu Mingmin (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-5690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16643769#comment-16643769
 ] 

Xu Mingmin commented on BEAM-5690:
--

Is this the error specifically? Seems duplicated {{0}} counts here,
{code:java}
{"count_col1":1,"Col3":"string6","WST":"2018-10-09  09-55-00   
+","WET":"2018-10-09  09-56-00   +"}
{"count_col1":0,"Col3":"string6","WST":"2018-10-09  09-55-00   
+","WET":"2018-10-09  09-56-00   +"}
{"count_col1":0,"Col3":"string6","WST":"2018-10-09  09-55-00   
+","WET":"2018-10-09  09-56-00   +"}
{"count_col1":0,"Col3":"string6","WST":"2018-10-09  09-55-00   
+","WET":"2018-10-09  09-56-00   +"}
{"count_col1":0,"Col3":"string6","WST":"2018-10-09  09-55-00   
+","WET":"2018-10-09  09-56-00   +"}
{"count_col1":0,"Col3":"string6","WST":"2018-10-09  09-55-00   
+","WET":"2018-10-09  09-56-00   +"}
{"count_col1":0,"Col3":"string6","WST":"2018-10-09  09-55-00   
+","WET":"2018-10-09  09-56-00 0}{code}

> Issue with GroupByKey in BeamSql using SparkRunner
> --
>
> Key: BEAM-5690
> URL: https://issues.apache.org/jira/browse/BEAM-5690
> Project: Beam
>  Issue Type: Task
>  Components: runner-spark
>Reporter: Kenneth Knowles
>Priority: Major
>
> Reported on user@
> {quote}We are trying to setup a pipeline with using BeamSql and the trigger 
> used is default (AfterWatermark crosses the window). 
> Below is the pipeline:
>   
>KafkaSource (KafkaIO) 
>---> Windowing (FixedWindow 1min)
>---> BeamSql
>---> KafkaSink (KafkaIO)
>  
> We are using Spark Runner for this. 
> The BeamSql query is:
> {code}select Col3, count(*) as count_col1 from PCOLLECTION GROUP BY Col3{code}
> We are grouping by Col3 which is a string. It can hold values string[0-9]. 
>  
> The records are getting emitted out at 1 min to kafka sink, but the output 
> record in kafka is not as expected.
> Below is the output observed: (WST and WET are indicators for window start 
> time and window end time)
> {code}
> {"count_col1":1,"Col3":"string5","WST":"2018-10-09  09-55-00   
> +","WET":"2018-10-09  09-56-00   +"}
> {"count_col1":3,"Col3":"string7","WST":"2018-10-09  09-55-00   
> +","WET":"2018-10-09  09-56-00   +"}
> {"count_col1":2,"Col3":"string8","WST":"2018-10-09  09-55-00   
> +","WET":"2018-10-09  09-56-00   +"}
> {"count_col1":1,"Col3":"string2","WST":"2018-10-09  09-55-00   
> +","WET":"2018-10-09  09-56-00   +"}
> {"count_col1":1,"Col3":"string6","WST":"2018-10-09  09-55-00   
> +","WET":"2018-10-09  09-56-00   +"}
> {"count_col1":0,"Col3":"string6","WST":"2018-10-09  09-55-00   
> +","WET":"2018-10-09  09-56-00   +"}
> {"count_col1":0,"Col3":"string6","WST":"2018-10-09  09-55-00   
> +","WET":"2018-10-09  09-56-00   +"}
> {"count_col1":0,"Col3":"string6","WST":"2018-10-09  09-55-00   
> +","WET":"2018-10-09  09-56-00   +"}
> {"count_col1":0,"Col3":"string6","WST":"2018-10-09  09-55-00   
> +","WET":"2018-10-09  09-56-00   +"}
> {"count_col1":0,"Col3":"string6","WST":"2018-10-09  09-55-00   
> +","WET":"2018-10-09  09-56-00   +"}
> {"count_col1":0,"Col3":"string6","WST":"2018-10-09  09-55-00   
> +","WET":"2018-10-09  09-56-00 0}
> {code}
> {quote}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Closed] (BEAM-1775) fix issue of start_from_previous_offset in KafkaIO

[Xu Mingmin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1775?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xu Mingmin closed BEAM-1775.

   Resolution: Won't Fix
Fix Version/s: Not applicable

> fix issue of start_from_previous_offset in KafkaIO
> --
>
> Key: BEAM-1775
> URL: https://issues.apache.org/jira/browse/BEAM-1775
> Project: Beam
>  Issue Type: Improvement
>  Components: io-java-kafka
>Reporter: Xu Mingmin
>Assignee: Xu Mingmin
>Priority: Minor
> Fix For: Not applicable
>
>
> Jins George jins.geo...@aeris.net via aermail.onmicrosoft.com 
>   
> 5:50 PM (15 hours ago)
>   
> to user
> Hello,
> I am writing a Beam pipeline(streaming) with Flink runner to consume data 
> from Kafka and apply some transformations and persist to Hbase.
> If I restart the application ( due to failure/manual restart), consumer does 
> not resume from the offset where it was prior to restart. It always resume 
> from the latest offset.
> If I enable Flink checkpionting with hdfs state back-end, system appears to 
> be resuming from the earliest offset
> Is there a recommended way to resume from the offset where it was stopped ?
> Thanks,
> Jins George



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-5210) VARBINARY is not supported in BeamSqlPrimitive

[Xu Mingmin (JIRA)]

Xu Mingmin created BEAM-5210:


 Summary: VARBINARY is not supported in BeamSqlPrimitive
 Key: BEAM-5210
 URL: https://issues.apache.org/jira/browse/BEAM-5210
 Project: Beam
  Issue Type: Bug
  Components: dsl-sql
Reporter: Xu Mingmin
Assignee: Xu Mingmin


See error as below

{code}
Caused by: java.lang.UnsupportedOperationException: Unsupported Beam SQL type 
in expression: VARBINARY
at 
org.apache.beam.sdk.extensions.sql.impl.interpreter.operator.BeamSqlPrimitive.accept(BeamSqlPrimitive.java:158)
at 
org.apache.beam.sdk.extensions.sql.impl.interpreter.operator.BeamSqlPrimitive.(BeamSqlPrimitive.java:44)
at 
org.apache.beam.sdk.extensions.sql.impl.interpreter.operator.BeamSqlPrimitive.of(BeamSqlPrimitive.java:56)
at 
org.apache.beam.sdk.extensions.sql.impl.interpreter.operator.BeamSqlInputRefExpression.evaluate(BeamSqlInputRefExpression.java:42)
at 
org.apache.beam.sdk.extensions.sql.impl.interpreter.BeamSqlExpressionEnvironments$ListEnvironment.getLocalRef(BeamSqlExpressionEnvironments.java:92)
at 
org.apache.beam.sdk.extensions.sql.impl.interpreter.operator.BeamSqlLocalRefExpression.evaluate(BeamSqlLocalRefExpression.java:44)
at 
org.apache.beam.sdk.extensions.sql.impl.interpreter.BeamSqlFnExecutor.lambda$execute$0(BeamSqlFnExecutor.java:557)
at 
java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
at 
java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1374)
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
at 
java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
at 
java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at 
java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)
at 
org.apache.beam.sdk.extensions.sql.impl.interpreter.BeamSqlFnExecutor.execute(BeamSqlFnExecutor.java:558)
at 
org.apache.beam.sdk.extensions.sql.impl.rel.BeamCalcRel$CalcFn.processElement(BeamCalcRel.java:116)
{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-5204) reduce restriction of JOIN

[Xu Mingmin (JIRA)]

Xu Mingmin created BEAM-5204:


 Summary: reduce restriction of JOIN
 Key: BEAM-5204
 URL: https://issues.apache.org/jira/browse/BEAM-5204
 Project: Beam
  Issue Type: Improvement
  Components: dsl-sql
Reporter: Xu Mingmin
Assignee: Xu Mingmin


ATM streaming-streaming JOIN is acceptable only when 
\{{triggersOncePerWindow(windowingStrategy)}}. With PaneInfo exposed by UDF, we 
should allow JOIN with multiple triggers as developers can filter out what they 
want now.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-5203) expose PaneInfo and BoundedWindow as UDF

[Xu Mingmin (JIRA)]

Xu Mingmin created BEAM-5203:


 Summary: expose PaneInfo and BoundedWindow as UDF
 Key: BEAM-5203
 URL: https://issues.apache.org/jira/browse/BEAM-5203
 Project: Beam
  Issue Type: Improvement
  Components: dsl-sql
Reporter: Xu Mingmin
Assignee: Xu Mingmin


besides adding new keywords in Calcite, there's an alternative way to expose 
PaneInfo and BoundedWindow of Row by UDF.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-5202) register UDF/UDAF with ServiceLoader

[Xu Mingmin (JIRA)]

Xu Mingmin created BEAM-5202:


 Summary: register UDF/UDAF with ServiceLoader
 Key: BEAM-5202
 URL: https://issues.apache.org/jira/browse/BEAM-5202
 Project: Beam
  Issue Type: Improvement
  Components: dsl-sql
Reporter: Xu Mingmin
Assignee: Xu Mingmin


create a ServiceLoader for UDF/UDAF register, if the autoLoad option is set to 
TRUE.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-5198) add functions to add/remove/list tables in BeamSqlEnv

[Xu Mingmin (JIRA)]

Xu Mingmin created BEAM-5198:


 Summary: add functions to add/remove/list tables in BeamSqlEnv
 Key: BEAM-5198
 URL: https://issues.apache.org/jira/browse/BEAM-5198
 Project: Beam
  Issue Type: Task
  Components: dsl-sql
Reporter: Xu Mingmin
Assignee: Xu Mingmin


We use a customized CLI implementation to submit streaming queries with 
BeamSql, the capability to add/remove/list tables in BeamSqlEnv is helpful for 
user-interactive.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (BEAM-2990) support data type MAP

[Xu Mingmin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xu Mingmin resolved BEAM-2990.
--
   Resolution: Fixed
Fix Version/s: 2.5.0

> support data type MAP
> -
>
> Key: BEAM-2990
> URL: https://issues.apache.org/jira/browse/BEAM-2990
> Project: Beam
>  Issue Type: New Feature
>  Components: dsl-sql
>Reporter: Xu Mingmin
>Assignee: Xu Mingmin
>Priority: Major
> Fix For: 2.5.0
>
>  Time Spent: 6h 10m
>  Remaining Estimate: 0h
>
> support Non-scalar types:
> MAP   Collection of keys mapped to values
> ARRAY Ordered, contiguous collection that may contain duplicates



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (BEAM-2990) support data type MAP

[Xu Mingmin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xu Mingmin reassigned BEAM-2990:


Assignee: Xu Mingmin  (was: Shayang Zang)

> support data type MAP
> -
>
> Key: BEAM-2990
> URL: https://issues.apache.org/jira/browse/BEAM-2990
> Project: Beam
>  Issue Type: New Feature
>  Components: dsl-sql
>Reporter: Xu Mingmin
>Assignee: Xu Mingmin
>Priority: Major
>
> support Non-scalar types:
> MAP   Collection of keys mapped to values
> ARRAY Ordered, contiguous collection that may contain duplicates



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-2990) support data type MAP

[Xu Mingmin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xu Mingmin updated BEAM-2990:
-
Summary: support data type MAP  (was: support data type MAP/ARRAY )

> support data type MAP
> -
>
> Key: BEAM-2990
> URL: https://issues.apache.org/jira/browse/BEAM-2990
> Project: Beam
>  Issue Type: New Feature
>  Components: dsl-sql
>Reporter: Xu Mingmin
>Assignee: Shayang Zang
>Priority: Major
>
> support Non-scalar types:
> MAP   Collection of keys mapped to values
> ARRAY Ordered, contiguous collection that may contain duplicates



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-3918) test failure BeamSqlDslAggregationTest.testTriggeredTumble

[Xu Mingmin (JIRA)]

Xu Mingmin created BEAM-3918:


 Summary: test failure BeamSqlDslAggregationTest.testTriggeredTumble
 Key: BEAM-3918
 URL: https://issues.apache.org/jira/browse/BEAM-3918
 Project: Beam
  Issue Type: Bug
  Components: dsl-sql, testing
Reporter: Xu Mingmin
Assignee: Kenneth Knowles


I cannot pass the test case added in 
[https://github.com/apache/beam/pull/4826,] here's the error message: (run with 
{{mvn clean install -pl sdks/java/extensions/sql/}})
{code}
[ERROR] Failures:
[ERROR]   BeamSqlDslAggregationTest.testTriggeredTumble:384 Windowed 
Query/BeamProjectRel.Transform/BEAMPROJECTREL_1566_149/ParMultiDo(BeamSqlProject).output:
Expected: iterable over [,
 ,
 ]
 in any order
 but: No item matches: ,
 
 in []
{code}

What it confuses me is, the failure is not found in precommit jobs.
 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Closed] (BEAM-3749) support customized trigger/accumulationMode in BeamSql

[Xu Mingmin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3749?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xu Mingmin closed BEAM-3749.

Resolution: Won't Fix

> support customized trigger/accumulationMode in BeamSql
> --
>
> Key: BEAM-3749
> URL: https://issues.apache.org/jira/browse/BEAM-3749
> Project: Beam
>  Issue Type: Improvement
>  Components: dsl-sql
>Reporter: Xu Mingmin
>Assignee: Xu Mingmin
>Priority: Major
> Fix For: 2.5.0
>
>  Time Spent: 3h 20m
>  Remaining Estimate: 0h
>
> Currently BeamSql use {{DefaultTrigger}} for aggregation operations. 
> By adding two options {{withTrigger(Trigger)}} and 
> {{withAccumulationMode(AccumulationMode)}}, developers can specify their own 
> aggregation strategies with BeamSql.
> [~xumingming] [~kedin] [~kenn] for any comments.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-3749) support customized trigger/accumulationMode in BeamSql

[Xu Mingmin (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16390324#comment-16390324
 ] 

Xu Mingmin commented on BEAM-3749:
--

Do you run a test to verify that?

> support customized trigger/accumulationMode in BeamSql
> --
>
> Key: BEAM-3749
> URL: https://issues.apache.org/jira/browse/BEAM-3749
> Project: Beam
>  Issue Type: Improvement
>  Components: dsl-sql
>Reporter: Xu Mingmin
>Assignee: Xu Mingmin
>Priority: Major
> Fix For: 2.5.0
>
>  Time Spent: 2h 10m
>  Remaining Estimate: 0h
>
> Currently BeamSql use {{DefaultTrigger}} for aggregation operations. 
> By adding two options {{withTrigger(Trigger)}} and 
> {{withAccumulationMode(AccumulationMode)}}, developers can specify their own 
> aggregation strategies with BeamSql.
> [~xumingming] [~kedin] [~kenn] for any comments.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-3749) support customized trigger/accumulationMode in BeamSql

[Xu Mingmin (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16390271#comment-16390271
 ] 

Xu Mingmin commented on BEAM-3749:
--

I get your point, below code doesn't use the trigger set in 
{{Window.configure().triggering}}, it use a {{DefaultTrigger}}:
{code}
input.apply(Window.configure().triggering( ... )
.accumulatingFiredPanes()
.withAllowedLateness(Duration.ZERO)
)
.apply( BeamSql.query("SELECT count(*) from PCOLLECTION GROUP BY 
TUMBLE(CURRENT_TIMESTAMP, INTERVAL '1' MINUTE)"));
{code}

So I suppose {{Window.configure().triggering}} should always use with 
{{Window.into}} together.

> support customized trigger/accumulationMode in BeamSql
> --
>
> Key: BEAM-3749
> URL: https://issues.apache.org/jira/browse/BEAM-3749
> Project: Beam
>  Issue Type: Improvement
>  Components: dsl-sql
>Reporter: Xu Mingmin
>Assignee: Xu Mingmin
>Priority: Major
> Fix For: 2.5.0
>
>  Time Spent: 2h 10m
>  Remaining Estimate: 0h
>
> Currently BeamSql use {{DefaultTrigger}} for aggregation operations. 
> By adding two options {{withTrigger(Trigger)}} and 
> {{withAccumulationMode(AccumulationMode)}}, developers can specify their own 
> aggregation strategies with BeamSql.
> [~xumingming] [~kedin] [~kenn] for any comments.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-3749) support customized trigger/accumulationMode in BeamSql

[Xu Mingmin (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16390124#comment-16390124
 ] 

Xu Mingmin commented on BEAM-3749:
--

It doesn't, with {{Window.configure()}} the code is
{code}
input
  .apply(Window.into(FixedWindows.of(Duration.standardMinutes(2
  .apply(Window.configure().triggering( ... )
.accumulatingFiredPanes()

.withAllowedLateness(Duration.ZERO)
)
  .apply( BeamSql.query("SELECT count(*)  from PCOLLECTION " ))
{code}
 

> support customized trigger/accumulationMode in BeamSql
> --
>
> Key: BEAM-3749
> URL: https://issues.apache.org/jira/browse/BEAM-3749
> Project: Beam
>  Issue Type: Improvement
>  Components: dsl-sql
>Reporter: Xu Mingmin
>Assignee: Xu Mingmin
>Priority: Major
> Fix For: 2.5.0
>
>  Time Spent: 2h 10m
>  Remaining Estimate: 0h
>
> Currently BeamSql use {{DefaultTrigger}} for aggregation operations. 
> By adding two options {{withTrigger(Trigger)}} and 
> {{withAccumulationMode(AccumulationMode)}}, developers can specify their own 
> aggregation strategies with BeamSql.
> [~xumingming] [~kedin] [~kenn] for any comments.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-3749) support customized trigger/accumulationMode in BeamSql

[Xu Mingmin (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16381653#comment-16381653
 ] 

Xu Mingmin commented on BEAM-3749:
--

Sink-based trigger has a limitation that it cannot handle the case when 
there're multiple aggregations in the whole pipeline, I suppose we need to have 
a way to set in each query.

> support customized trigger/accumulationMode in BeamSql
> --
>
> Key: BEAM-3749
> URL: https://issues.apache.org/jira/browse/BEAM-3749
> Project: Beam
>  Issue Type: Improvement
>  Components: dsl-sql
>Reporter: Xu Mingmin
>Assignee: Xu Mingmin
>Priority: Major
> Fix For: 2.5.0
>
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> Currently BeamSql use {{DefaultTrigger}} for aggregation operations. 
> By adding two options {{withTrigger(Trigger)}} and 
> {{withAccumulationMode(AccumulationMode)}}, developers can specify their own 
> aggregation strategies with BeamSql.
> [~xumingming] [~kedin] [~kenn] for any comments.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-3749) support customized trigger/accumulationMode in BeamSql

[Xu Mingmin (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16380953#comment-16380953
 ] 

Xu Mingmin commented on BEAM-3749:
--

{{Window.triggering(trigger).discardingFiredPanes().withAllowedLateness()}} is 
a good idea, based on my applications it's not enough, for example multiple 
queries sharing the same source table.

I would submit a PR and we should make it comparable when 
{{Window.triggering(trigger)...}} works in future.

 

> support customized trigger/accumulationMode in BeamSql
> --
>
> Key: BEAM-3749
> URL: https://issues.apache.org/jira/browse/BEAM-3749
> Project: Beam
>  Issue Type: Improvement
>  Components: dsl-sql
>Reporter: Xu Mingmin
>Assignee: Xu Mingmin
>Priority: Major
> Fix For: 2.4.0
>
>
> Currently BeamSql use {{DefaultTrigger}} for aggregation operations. 
> By adding two options {{withTrigger(Trigger)}} and 
> {{withAccumulationMode(AccumulationMode)}}, developers can specify their own 
> aggregation strategies with BeamSql.
> [~xumingming] [~kedin] [~kenn] for any comments.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Closed] (BEAM-3739) @Parameter annotation does not work for UDFs in Beam SQL

[Xu Mingmin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3739?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xu Mingmin closed BEAM-3739.

   Resolution: Fixed
Fix Version/s: 2.4.0

> @Parameter annotation does not work for UDFs in Beam SQL
> 
>
> Key: BEAM-3739
> URL: https://issues.apache.org/jira/browse/BEAM-3739
> Project: Beam
>  Issue Type: Bug
>  Components: dsl-sql
>Affects Versions: 2.3.0
>Reporter: Samuel Waggoner
>Assignee: Xu Mingmin
>Priority: Major
> Fix For: 2.4.0
>
>  Time Spent: 50m
>  Remaining Estimate: 0h
>
> BeamSqlUdf javadoc indicates you can have optional parameters, but this 
> functionality is not working. I implemented the following copy/paste from the 
> doc 
> https://beam.apache.org/documentation/sdks/javadoc/2.3.0/org/apache/beam/sdk/extensions/sql/BeamSqlUdf.html:
> {code:java}
> public static class MyLeftFunction implements BeamSqlUdf {
>  public String eval(
>  @Parameter(name = "s") String s,
>  @Parameter(name = "n", optional = true) Integer n) {
>  return s.substring(0, n == null ? 1 : n);
>  }
> }{code}
> I modify a query in BeamSqlExample.java to use it. With all parameters 
> supplied, it completes successfully:
> {code:java}
> //Case 1. run a simple SQL query over input PCollection with 
> BeamSql.simpleQuery;
> PCollection outputStream = inputTable.apply(
> BeamSql.query("select c1, leftfn('string1', 1) as c2, c3 from PCOLLECTION 
> where c1 > 1")
> .registerUdf("leftfn", MyLeftFunction.class));{code}
> With the optional parameter left off, I get an exception:
> {code:java}
> //Case 1. run a simple SQL query over input PCollection with 
> BeamSql.simpleQuery;
> PCollection outputStream = inputTable.apply(
>  BeamSql.query("select c1, leftfn('string1') as c2, c3 from PCOLLECTION where 
> c1 > 1")
>  .registerUdf("leftfn", MyLeftFunction.class));{code}
> {code:java}
> Exception in thread "main" java.lang.IllegalStateException: 
> java.lang.UnsupportedOperationException: Operator: DEFAULT is not supported 
> yet!
>  at 
> org.apache.beam.sdk.extensions.sql.QueryTransform.expand(QueryTransform.java:75)
>  at 
> org.apache.beam.sdk.extensions.sql.QueryTransform.expand(QueryTransform.java:47)
>  at org.apache.beam.sdk.Pipeline.applyInternal(Pipeline.java:537)
>  at org.apache.beam.sdk.Pipeline.applyTransform(Pipeline.java:472)
>  at org.apache.beam.sdk.values.PCollection.apply(PCollection.java:286)
>  at 
> org.apache.beam.sdk.extensions.sql.example.BeamSqlExample.main(BeamSqlExample.java:76)
> Caused by: java.lang.UnsupportedOperationException: Operator: DEFAULT is not 
> supported yet!
>  at 
> org.apache.beam.sdk.extensions.sql.impl.interpreter.BeamSqlFnExecutor.buildExpression(BeamSqlFnExecutor.java:424)
>  at 
> org.apache.beam.sdk.extensions.sql.impl.interpreter.BeamSqlFnExecutor.buildExpression(BeamSqlFnExecutor.java:201)
>  at 
> org.apache.beam.sdk.extensions.sql.impl.interpreter.BeamSqlFnExecutor.(BeamSqlFnExecutor.java:125)
>  at 
> org.apache.beam.sdk.extensions.sql.impl.rel.BeamProjectRel.buildBeamPipeline(BeamProjectRel.java:70)
>  at 
> org.apache.beam.sdk.extensions.sql.QueryTransform.expand(QueryTransform.java:73)
>  ... 5 more{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-3752) PAssert doesn't use the correct Coder

[Xu Mingmin (JIRA)]

Xu Mingmin created BEAM-3752:


 Summary: PAssert doesn't use the correct Coder
 Key: BEAM-3752
 URL: https://issues.apache.org/jira/browse/BEAM-3752
 Project: Beam
  Issue Type: Bug
  Components: testing
Affects Versions: 2.3.0
Reporter: Xu Mingmin


Run into issue with below test, that it can pass although I'm sure it should 
fail. It seems {{PAssert}} doesn't use the right Coder to compare 
expected/actual values.

See test case here 
[https://github.com/XuMingmin/beam/blob/5d2bc2aa7c344b8b9adf4ceeac2d7450d6ddec1e/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/BeamSqlDslUdfUdafTest.java#L93-L104]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-3751) enforce data validation in org.apache.beam.sdk.values.Row

[Xu Mingmin (JIRA)]

Xu Mingmin created BEAM-3751:


 Summary: enforce data validation in org.apache.beam.sdk.values.Row
 Key: BEAM-3751
 URL: https://issues.apache.org/jira/browse/BEAM-3751
 Project: Beam
  Issue Type: Bug
  Components: dsl-sql, sdk-java-core
Affects Versions: 2.3.0
Reporter: Xu Mingmin


It doesn't verify data when creating a {{Row}} record, below lines have 
type-mismatch but it can pass without error.
{code:java}
RowType subStrRowType = RowSqlType.builder()
.withIntegerField("f_int")
.withBigIntField("f_bigint")
.build();
Row subStrRow = Row.withRowType(subStrRowType).addValues(2, "s").build();
{code}

CC: [~kedin] [~kenn] [~xumingming]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-3739) @Parameter annotation does not work for UDFs in Beam SQL

[Xu Mingmin (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16377748#comment-16377748
 ] 

Xu Mingmin commented on BEAM-3739:
--

[~samwagg], thanks to report the issue, will look into it. With a quick glance, 
it's because that {{DEFAULT}} keyword is not supported.

> @Parameter annotation does not work for UDFs in Beam SQL
> 
>
> Key: BEAM-3739
> URL: https://issues.apache.org/jira/browse/BEAM-3739
> Project: Beam
>  Issue Type: Bug
>  Components: dsl-sql
>Affects Versions: 2.3.0
>Reporter: Samuel Waggoner
>Assignee: Xu Mingmin
>Priority: Major
>
> BeamSqlUdf javadoc indicates you can have optional parameters, but this 
> functionality is not working. I implemented the following copy/paste from the 
> doc 
> https://beam.apache.org/documentation/sdks/javadoc/2.3.0/org/apache/beam/sdk/extensions/sql/BeamSqlUdf.html:
> {code:java}
> public static class MyLeftFunction implements BeamSqlUdf {
>  public String eval(
>  @Parameter(name = "s") String s,
>  @Parameter(name = "n", optional = true) Integer n) {
>  return s.substring(0, n == null ? 1 : n);
>  }
> }{code}
> I modify a query in BeamSqlExample.java to use it. With all parameters 
> supplied, it completes successfully:
> {code:java}
> //Case 1. run a simple SQL query over input PCollection with 
> BeamSql.simpleQuery;
> PCollection outputStream = inputTable.apply(
> BeamSql.query("select c1, leftfn('string1', 1) as c2, c3 from PCOLLECTION 
> where c1 > 1")
> .registerUdf("leftfn", MyLeftFunction.class));{code}
> With the optional parameter left off, I get an exception:
> {code:java}
> //Case 1. run a simple SQL query over input PCollection with 
> BeamSql.simpleQuery;
> PCollection outputStream = inputTable.apply(
>  BeamSql.query("select c1, leftfn('string1') as c2, c3 from PCOLLECTION where 
> c1 > 1")
>  .registerUdf("leftfn", MyLeftFunction.class));{code}
> {code:java}
> Exception in thread "main" java.lang.IllegalStateException: 
> java.lang.UnsupportedOperationException: Operator: DEFAULT is not supported 
> yet!
>  at 
> org.apache.beam.sdk.extensions.sql.QueryTransform.expand(QueryTransform.java:75)
>  at 
> org.apache.beam.sdk.extensions.sql.QueryTransform.expand(QueryTransform.java:47)
>  at org.apache.beam.sdk.Pipeline.applyInternal(Pipeline.java:537)
>  at org.apache.beam.sdk.Pipeline.applyTransform(Pipeline.java:472)
>  at org.apache.beam.sdk.values.PCollection.apply(PCollection.java:286)
>  at 
> org.apache.beam.sdk.extensions.sql.example.BeamSqlExample.main(BeamSqlExample.java:76)
> Caused by: java.lang.UnsupportedOperationException: Operator: DEFAULT is not 
> supported yet!
>  at 
> org.apache.beam.sdk.extensions.sql.impl.interpreter.BeamSqlFnExecutor.buildExpression(BeamSqlFnExecutor.java:424)
>  at 
> org.apache.beam.sdk.extensions.sql.impl.interpreter.BeamSqlFnExecutor.buildExpression(BeamSqlFnExecutor.java:201)
>  at 
> org.apache.beam.sdk.extensions.sql.impl.interpreter.BeamSqlFnExecutor.(BeamSqlFnExecutor.java:125)
>  at 
> org.apache.beam.sdk.extensions.sql.impl.rel.BeamProjectRel.buildBeamPipeline(BeamProjectRel.java:70)
>  at 
> org.apache.beam.sdk.extensions.sql.QueryTransform.expand(QueryTransform.java:73)
>  ... 5 more{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Closed] (BEAM-3688) add setup/teardown for BeamSqlSeekableTable

[Xu Mingmin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3688?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xu Mingmin closed BEAM-3688.

   Resolution: Fixed
Fix Version/s: 2.4.0

> add setup/teardown for BeamSqlSeekableTable
> ---
>
> Key: BEAM-3688
> URL: https://issues.apache.org/jira/browse/BEAM-3688
> Project: Beam
>  Issue Type: Improvement
>  Components: dsl-sql
>Reporter: Xu Mingmin
>Assignee: Xu Mingmin
>Priority: Major
> Fix For: 2.4.0
>
>  Time Spent: 2h
>  Remaining Estimate: 0h
>
> Add interface \{{setup}} and \{{teardown}} for initial/cleanup work of 
> \{{BeamSqlSeekableTable}}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Closed] (BEAM-3093) add an option 'FirstPollOffsetStrategy' to KafkaIO

[Xu Mingmin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3093?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xu Mingmin closed BEAM-3093.

   Resolution: Won't Fix
Fix Version/s: Not applicable

> add an option 'FirstPollOffsetStrategy' to KafkaIO
> --
>
> Key: BEAM-3093
> URL: https://issues.apache.org/jira/browse/BEAM-3093
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Xu Mingmin
>Assignee: Xu Mingmin
>Priority: Major
> Fix For: Not applicable
>
>
> This is a feature borrowed from Storm KafkaSpout.
> *What's the issue?*
> In KafkaIO, when offset is stored either in checkpoint or auto_committed, it 
> cannot be changed in application, to force to read from earliest/latest. 
> --This feature is important to reset the start offset when relaunching a job.
> *Proposed solution:*
> By borrowing the FirstPollOffsetStrategy concept, users can have more options:
> 1). *{{EARLIEST}}*: always start_from_beginning no matter of what's in 
> checkpoint/auto_commit;
> 2). *{{LATEST}}*: always start_from_latest no matter of what's in 
> checkpoint/auto_commit;
> 3). *{{UNCOMMITTED_EARLIEST}}*: if no offset in checkpoint/auto_commit then 
> start_from_beginning if, otherwise start_from_previous_offset;
> 4). *{{UNCOMMITTED_LATEST}}*: if no offset in checkpoint/auto_commit then 
> start_from_latest, otherwise start_from_previous_offset;
> [~rangadi], any comments?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Closed] (BEAM-3699) RecordTimestamp should be the default Watermark in KafkaIO

[Xu Mingmin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3699?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xu Mingmin closed BEAM-3699.

   Resolution: Duplicate
Fix Version/s: (was: 2.4.0)
   Not applicable

> RecordTimestamp should be the default Watermark in KafkaIO
> --
>
> Key: BEAM-3699
> URL: https://issues.apache.org/jira/browse/BEAM-3699
> Project: Beam
>  Issue Type: Bug
>  Components: io-java-kafka
>Reporter: Xu Mingmin
>Assignee: Xu Mingmin
>Priority: Major
> Fix For: Not applicable
>
>  Time Spent: 1h
>  Remaining Estimate: 0h
>
> Currently, the priority to get Watermark Instance in KafkaIO is:
> {code}
> getWatermarkFn().apply(curRecord)
>   getTimestampFn().apply(record)
>     Instant.now()
> {code}
> I would propose to change it as below to leverage {{KafkaRecord.timestamp}} 
> if no {{WatermarkFn()}} or {{TimestampFn()}} is available:
> {code}
> getWatermarkFn().apply(curRecord)
>   getTimestampFn().apply(record)
> KafkaRecord(Beam.KafkaIO).timestamp
> {code}
>  It equals to
> {code}
> getWatermarkFn().apply(curRecord)
>   getTimestampFn().apply(record)
> KafkaRawRecord(Kafka_client).timestamp
>   Instant.now()
> {code}
> [~rangadi] any comments?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-3699) RecordTimestamp should be the default Watermark in KafkaIO

[Xu Mingmin (JIRA)]

Xu Mingmin created BEAM-3699:


 Summary: RecordTimestamp should be the default Watermark in KafkaIO
 Key: BEAM-3699
 URL: https://issues.apache.org/jira/browse/BEAM-3699
 Project: Beam
  Issue Type: Bug
  Components: sdk-java-extensions
Reporter: Xu Mingmin
Assignee: Xu Mingmin
 Fix For: 2.4.0


Currently, the priority to get Watermark Instance in KafkaIO is:

{code}
getWatermarkFn().apply(curRecord)
  getTimestampFn().apply(record)
    Instant.now()
{code}

I would propose to change it as below to leverage {{KafkaRecord.timestamp}} if 
no {{WatermarkFn()}} or {{TimestampFn()}} is available:
{code}
getWatermarkFn().apply(curRecord)
  getTimestampFn().apply(record)
KafkaRecord(Beam.KafkaIO).timestamp
{code}

 It equals to
{code}
getWatermarkFn().apply(curRecord)
  getTimestampFn().apply(record)
KafkaRawRecord(Kafka_client).timestamp
  Instant.now()
{code}

[~rangadi] any comments?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-3679) Upgrade calcite to release 1.16

[Xu Mingmin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xu Mingmin updated BEAM-3679:
-
Fix Version/s: 2.4.0

> Upgrade calcite to release 1.16
> ---
>
> Key: BEAM-3679
> URL: https://issues.apache.org/jira/browse/BEAM-3679
> Project: Beam
>  Issue Type: Improvement
>  Components: dsl-sql
>Reporter: Ted Yu
>Assignee: Xu Mingmin
>Priority: Minor
> Fix For: 2.4.0
>
>
> Currently Beam uses Calcite 1.13.0
> This issue is to upgrade to Calcite 1.16.0



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-3688) add setup/teardown for BeamSqlSeekableTable

[Xu Mingmin (JIRA)]

Xu Mingmin created BEAM-3688:


 Summary: add setup/teardown for BeamSqlSeekableTable
 Key: BEAM-3688
 URL: https://issues.apache.org/jira/browse/BEAM-3688
 Project: Beam
  Issue Type: Improvement
  Components: dsl-sql
Reporter: Xu Mingmin
Assignee: Xu Mingmin


Add interface \{{setup}} and \{{teardown}} for initial/cleanup work of 
\{{BeamSqlSeekableTable}}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-2445) DSL SQL to use service locator pattern to automatically register UDFs

[Xu Mingmin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xu Mingmin updated BEAM-2445:
-
Priority: Major  (was: Minor)

> DSL SQL to use service locator pattern to automatically register UDFs
> -
>
> Key: BEAM-2445
> URL: https://issues.apache.org/jira/browse/BEAM-2445
> Project: Beam
>  Issue Type: Improvement
>  Components: dsl-sql
>Reporter: Luke Cwik
>Priority: Major
> Fix For: 2.4.0
>
>
> Use a service locator pattern to find UDFs that can be registered. The 
> service loader can be used to register UDFs for standard functions via DSL 
> SQL, additional UDFs registered by third party libraries, and end user 
> created UDFs.
> Example ServiceLoader usage within Apache Beam to find coder providers:
> https://github.com/apache/beam/blob/7126fdc6ee5671e99a2dede3f25ba616aa0e8fa4/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/CoderRegistry.java#L147



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (BEAM-2445) DSL SQL to use service locator pattern to automatically register UDFs

[Xu Mingmin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xu Mingmin reassigned BEAM-2445:


Assignee: Xu Mingmin

> DSL SQL to use service locator pattern to automatically register UDFs
> -
>
> Key: BEAM-2445
> URL: https://issues.apache.org/jira/browse/BEAM-2445
> Project: Beam
>  Issue Type: Improvement
>  Components: dsl-sql
>Reporter: Luke Cwik
>Assignee: Xu Mingmin
>Priority: Major
> Fix For: 2.4.0
>
>
> Use a service locator pattern to find UDFs that can be registered. The 
> service loader can be used to register UDFs for standard functions via DSL 
> SQL, additional UDFs registered by third party libraries, and end user 
> created UDFs.
> Example ServiceLoader usage within Apache Beam to find coder providers:
> https://github.com/apache/beam/blob/7126fdc6ee5671e99a2dede3f25ba616aa0e8fa4/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/CoderRegistry.java#L147



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-2445) DSL SQL to use service locator pattern to automatically register UDFs

[Xu Mingmin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xu Mingmin updated BEAM-2445:
-
Fix Version/s: 2.4.0

> DSL SQL to use service locator pattern to automatically register UDFs
> -
>
> Key: BEAM-2445
> URL: https://issues.apache.org/jira/browse/BEAM-2445
> Project: Beam
>  Issue Type: Improvement
>  Components: dsl-sql
>Reporter: Luke Cwik
>Priority: Major
> Fix For: 2.4.0
>
>
> Use a service locator pattern to find UDFs that can be registered. The 
> service loader can be used to register UDFs for standard functions via DSL 
> SQL, additional UDFs registered by third party libraries, and end user 
> created UDFs.
> Example ServiceLoader usage within Apache Beam to find coder providers:
> https://github.com/apache/beam/blob/7126fdc6ee5671e99a2dede3f25ba616aa0e8fa4/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/CoderRegistry.java#L147



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Closed] (BEAM-3171) convert a join into lookup

[Xu Mingmin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xu Mingmin closed BEAM-3171.

   Resolution: Fixed
Fix Version/s: 2.3.0

> convert a join into lookup
> --
>
> Key: BEAM-3171
> URL: https://issues.apache.org/jira/browse/BEAM-3171
> Project: Beam
>  Issue Type: New Feature
>  Components: dsl-sql
>Reporter: Xu Mingmin
>Assignee: Xu Mingmin
>Priority: Major
>  Labels: experimental
> Fix For: 2.3.0
>
>
> We use BeamSQL to run streaming jobs mostly, and  add a join_as_lookup 
> improvement(internal branch) to cover the streaming-to-batch case(similar as 
> [1]). I could submit a PR as experimental if people are interested. 
> The rough solution is, if one source of join node implements 
> {{BeamSeekableTable}} and the other is not, then the join node is converted 
> to a fact-lookup operation.
> Ref:
> [1] 
> https://docs.google.com/document/d/1B-XnUwXh64lbswRieckU0BxtygSV58hysqZbpZmk03A/edit?usp=sharing
>  
> [~xumingming] [~takidau] for any comments



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (BEAM-3558) aggregation expression can't apply to math or arithmetic expressions

[Xu Mingmin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3558?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xu Mingmin reassigned BEAM-3558:


Assignee: (was: Xu Mingmin)

> aggregation expression can't apply to math or arithmetic expressions
> 
>
> Key: BEAM-3558
> URL: https://issues.apache.org/jira/browse/BEAM-3558
> Project: Beam
>  Issue Type: Sub-task
>  Components: dsl-sql
>Reporter: Kai Jiang
>Priority: Major
>
> fails when executing sql
> 'select sum(c1)+2 from PCOLLECTION group by c2'



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (BEAM-3481) Query with subquery and aggregates cannot be implemented.

[Xu Mingmin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3481?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xu Mingmin reassigned BEAM-3481:


Assignee: (was: Xu Mingmin)

> Query with subquery and aggregates cannot be implemented.
> -
>
> Key: BEAM-3481
> URL: https://issues.apache.org/jira/browse/BEAM-3481
> Project: Beam
>  Issue Type: Bug
>  Components: dsl-sql
>Affects Versions: 2.2.0
>Reporter: Austin Haas
>Priority: Major
>
> This query results in the error below:
> {noformat}
> "SELECT (COUNT(`p`))
>  FROM (SELECT `p`
>FROM `contains`
>GROUP BY `p`) AS `t1`"{noformat}
> This works correctly:
> {noformat}
> "SELECT (COUNT(`p`))
>  FROM (SELECT `p`, CURRENT_TIME
>FROM `contains`
>GROUP BY `p`) AS `t1`"{noformat}
> Error:
>  
> {noformat}
> [nREPL-worker-5] INFO 
> org.apache.beam.sdk.extensions.sql.impl.planner.BeamQueryPlanner - SQL:
> SELECT COUNT(`t1`.`p`)
> FROM (SELECT `contains`.`p`
> FROM `contains` AS `contains`
> GROUP BY `contains`.`p`) AS `t1`
> [nREPL-worker-5] INFO 
> org.apache.beam.sdk.extensions.sql.impl.planner.BeamQueryPlanner - SQLPlan>
> LogicalAggregate(group=[{}], EXPR$0=[COUNT()])
>  LogicalAggregate(group=[{0}])
>  LogicalProject(p=[$0])
>  LogicalTableScan(table=[[contains]])
> CannotPlanException Node [rel#157:Subset#3.BEAM_LOGICAL.[]] could not be 
> implemented; planner state:
> Root: rel#157:Subset#3.BEAM_LOGICAL.[]
> Original rel:
> LogicalAggregate(subset=[rel#157:Subset#3.BEAM_LOGICAL.[]], group=[{}], 
> EXPR$0=[COUNT()]): rowcount = 1.0, cumulative cost = {1.125 rows, 0.0 cpu, 
> 0.0 io}, id = 155
>  LogicalAggregate(subset=[rel#154:Subset#2.NONE.[]], group=[{0}]): rowcount = 
> 10.0, cumulative cost = {10.0 rows, 0.0 cpu, 0.0 io}, id = 153
>  LogicalProject(subset=[rel#152:Subset#1.NONE.[]], p=[$0]): rowcount = 100.0, 
> cumulative cost = {100.0 rows, 100.0 cpu, 0.0 io}, id = 151
>  LogicalTableScan(subset=[rel#150:Subset#0.NONE.[]], table=[[contains]]): 
> rowcount = 100.0, cumulative cost = {100.0 rows, 101.0 cpu, 0.0 io}, id = 146
> Sets:
> Set#0, type: RecordType(VARCHAR p, VARCHAR s, BIGINT c)
>  rel#150:Subset#0.NONE.[], best=null, importance=0.6561
>  rel#146:LogicalTableScan.NONE.[](table=[contains]), rowcount=100.0, 
> cumulative cost={inf}
>  rel#162:Subset#0.BEAM_LOGICAL.[], best=rel#164, importance=0.32805
>  rel#164:BeamIOSourceRel.BEAM_LOGICAL.[](table=[contains]), rowcount=100.0, 
> cumulative cost={100.0 rows, 101.0 cpu, 0.0 io}
> Set#1, type: RecordType(VARCHAR p)
>  rel#152:Subset#1.NONE.[], best=null, importance=0.7291
>  rel#151:LogicalProject.NONE.[](input=rel#150:Subset#0.NONE.[],p=$0), 
> rowcount=100.0, cumulative cost={inf}
>  rel#159:Subset#1.BEAM_LOGICAL.[], best=rel#163, 
> importance=0.36455
>  
> rel#163:BeamProjectRel.BEAM_LOGICAL.[](input=rel#162:Subset#0.BEAM_LOGICAL.[],p=$0),
>  rowcount=100.0, cumulative cost={200.0 rows, 201.0 cpu, 0.0 io}
> Set#2, type: RecordType(VARCHAR p)
>  rel#154:Subset#2.NONE.[], best=null, importance=0.81
>  rel#153:LogicalAggregate.NONE.[](input=rel#152:Subset#1.NONE.[],group={0}), 
> rowcount=10.0, cumulative cost={inf}
>  rel#161:Subset#2.BEAM_LOGICAL.[], best=rel#160, importance=0.405
>  
> rel#160:BeamAggregationRel.BEAM_LOGICAL.[](group={0},window=org.apache.beam.sdk.transforms.windowing.GlobalWindows,trigger=Repeatedly.forever(AfterWatermark.pastEndOfWindow())),
>  rowcount=10.0, cumulative cost={210.0 rows, 201.0 cpu, 0.0 io}
> Set#3, type: RecordType(BIGINT EXPR$0)
>  rel#156:Subset#3.NONE.[], best=null, importance=0.9
>  
> rel#155:LogicalAggregate.NONE.[](input=rel#154:Subset#2.NONE.[],group={},EXPR$0=COUNT()),
>  rowcount=1.0, cumulative cost={inf}
>  rel#157:Subset#3.BEAM_LOGICAL.[], best=null,
>  importance=1.0
>  
> rel#158:AbstractConverter.BEAM_LOGICAL.[](input=rel#156:Subset#3.NONE.[],convention=BEAM_LOGICAL,sort=[]),
>  rowcount=1.0, cumulative cost={inf}
> org.apache.beam.sdks.java.extensions.sql.repackaged.org.apache.calcite.plan.volcano.RelSubset$CheapestPlanReplacer.visit
>  (RelSubset.java:441)
> {noformat}
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (BEAM-3386) Dependency conflict when Calcite is included in a project.

[Xu Mingmin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xu Mingmin reassigned BEAM-3386:


Assignee: (was: Xu Mingmin)

> Dependency conflict when Calcite is included in a project.
> --
>
> Key: BEAM-3386
> URL: https://issues.apache.org/jira/browse/BEAM-3386
> Project: Beam
>  Issue Type: Bug
>  Components: dsl-sql
>Affects Versions: 2.2.0
>Reporter: Austin Haas
>Priority: Major
>
> When Calcite (v. 1.13.0) is included in a project that also includes Beam and 
> the Beam SQL extension, then the following error is thrown when trying to run 
> Beam code.
> ClassCastException 
> org.apache.beam.sdk.extensions.sql.impl.planner.BeamRelDataTypeSystem cannot 
> be cast to org.apache.calcite.rel.type.RelDataTypeSystem
> org.apache.calcite.jdbc.CalciteConnectionImpl. 
> (CalciteConnectionImpl.java:120)
> 
> org.apache.calcite.jdbc.CalciteJdbc41Factory$CalciteJdbc41Connection. 
> (CalciteJdbc41Factory.java:114)
> org.apache.calcite.jdbc.CalciteJdbc41Factory.newConnection 
> (CalciteJdbc41Factory.java:59)
> org.apache.calcite.jdbc.CalciteJdbc41Factory.newConnection 
> (CalciteJdbc41Factory.java:44)
> org.apache.calcite.jdbc.CalciteFactory.newConnection 
> (CalciteFactory.java:53)
> org.apache.calcite.avatica.UnregisteredDriver.connect 
> (UnregisteredDriver.java:138)
> java.sql.DriverManager.getConnection (DriverManager.java:664)
> java.sql.DriverManager.getConnection (DriverManager.java:208)
> 
> org.apache.beam.sdks.java.extensions.sql.repackaged.org.apache.calcite.tools.Frameworks.withPrepare
>  (Frameworks.java:145)
> 
> org.apache.beam.sdks.java.extensions.sql.repackaged.org.apache.calcite.tools.Frameworks.withPlanner
>  (Frameworks.java:106)
> 
> org.apache.beam.sdks.java.extensions.sql.repackaged.org.apache.calcite.prepare.PlannerImpl.ready
>  (PlannerImpl.java:140)
> 
> org.apache.beam.sdks.java.extensions.sql.repackaged.org.apache.calcite.prepare.PlannerImpl.parse
>  (PlannerImpl.java:170)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (BEAM-3509) PARTITION BY in Beam SQL In Select Command

[Xu Mingmin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3509?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xu Mingmin reassigned BEAM-3509:


Assignee: (was: Xu Mingmin)

> PARTITION BY in Beam SQL In Select Command
> --
>
> Key: BEAM-3509
> URL: https://issues.apache.org/jira/browse/BEAM-3509
> Project: Beam
>  Issue Type: New Feature
>  Components: dsl-sql
>Affects Versions: 2.2.0
>Reporter: Kishan Kumar
>Priority: Major
>  Labels: performance
>
> Partition By Option Will Be Very Help Full for DataFlow Developer To Migrate 
> Query and Do Transformation on That because of Many *Netezza Query and Oracle 
> Query* Consists Of Partition By Which Makes SQL Query More Efficient. *The 
> alternative is Making Joins And Filtering It Can Be Done But It Makes Code 
> Unreadable And Performance Become bad for DataFlow Job.*
> Examples: SELECT MIN(COLUMN) OVER (PARTITION BY COLUMN NAME) FROM TABLENAME



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Closed] (BEAM-3348) Beam SQL DSL not support non-ascii characters in sql

[Xu Mingmin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3348?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xu Mingmin closed BEAM-3348.

   Resolution: Fixed
Fix Version/s: 2.3.0

> Beam SQL DSL not support non-ascii characters in sql
> 
>
> Key: BEAM-3348
> URL: https://issues.apache.org/jira/browse/BEAM-3348
> Project: Beam
>  Issue Type: Bug
>  Components: dsl-sql
>Affects Versions: 2.2.0
>Reporter: Yiyan Lu
>Assignee: Xu Mingmin
>Priority: Major
> Fix For: 2.3.0
>
>
> When I have non-ascii in Beam SQL:
> ```
> SELECT * FROM TABLE_A WHERE f_string = '第四行'
> ```
>  calcite planner will throw exception:
> ```
> Caused by: org.apache.calcite.runtime.CalciteException: Failed to encode 
> '第四行' in character set 'ISO-8859-1'
> ```
> cause by calcite using ISO-8859-1 for default charset



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Closed] (BEAM-3110) The transform Read(UnboundedKafkaSource) is currently not supported

[Xu Mingmin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3110?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xu Mingmin closed BEAM-3110.

   Resolution: Fixed
Fix Version/s: 2.3.0

> The transform Read(UnboundedKafkaSource) is currently not supported
> ---
>
> Key: BEAM-3110
> URL: https://issues.apache.org/jira/browse/BEAM-3110
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Affects Versions: 2.2.0, 2.3.0
>Reporter: Xu Mingmin
>Assignee: Xu Mingmin
>Priority: Major
> Fix For: 2.3.0
>
>
> I see this issue when submitting a job to Flink cluster. It appears after 
> build {{2.2.0-20170912.083349-51}}.
> {code}
> org.apache.flink.client.program.ProgramInvocationException: The main method 
> caused an error.
>   at 
> org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:545)
>   at 
> org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:419)
>   at 
> org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:381)
>   at 
> org.apache.flink.client.CliFrontend.executeProgram(CliFrontend.java:838)
>   at org.apache.flink.client.CliFrontend.run(CliFrontend.java:259)
>   at 
> org.apache.flink.client.CliFrontend.parseParameters(CliFrontend.java:1086)
>   at org.apache.flink.client.CliFrontend$2.call(CliFrontend.java:1133)
>   at org.apache.flink.client.CliFrontend$2.call(CliFrontend.java:1130)
>   at 
> org.apache.flink.runtime.security.HadoopSecurityContext$1.run(HadoopSecurityContext.java:43)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at javax.security.auth.Subject.doAs(Subject.java:415)
>   at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
>   at 
> org.apache.flink.runtime.security.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:40)
>   at org.apache.flink.client.CliFrontend.main(CliFrontend.java:1129)
> Caused by: java.lang.UnsupportedOperationException: The transform 
> Read(UnboundedKafkaSource) is currently not supported.
>   at 
> org.apache.beam.runners.flink.FlinkStreamingPipelineTranslator.visitPrimitiveTransform(FlinkStreamingPipelineTranslator.java:113)
>   at 
> org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:666)
>   at 
> org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:658)
>   at 
> org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:658)
>   at 
> org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:658)
>   at 
> org.apache.beam.sdk.runners.TransformHierarchy$Node.access$600(TransformHierarchy.java:311)
>   at 
> org.apache.beam.sdk.runners.TransformHierarchy.visit(TransformHierarchy.java:245)
>   at org.apache.beam.sdk.Pipeline.traverseTopologically(Pipeline.java:451)
>   at 
> org.apache.beam.runners.flink.FlinkPipelineTranslator.translate(FlinkPipelineTranslator.java:38)
>   at 
> org.apache.beam.runners.flink.FlinkStreamingPipelineTranslator.translate(FlinkStreamingPipelineTranslator.java:69)
>   at 
> org.apache.beam.runners.flink.FlinkPipelineExecutionEnvironment.translate(FlinkPipelineExecutionEnvironment.java:104)
>   at org.apache.beam.runners.flink.FlinkRunner.run(FlinkRunner.java:113)
>   at org.apache.beam.sdk.Pipeline.run(Pipeline.java:304)
>   at org.apache.beam.sdk.Pipeline.run(Pipeline.java:290)
> {code} 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-3325) optional parameter is not supported in BeamSQL

[Xu Mingmin (JIRA)]

Xu Mingmin created BEAM-3325:


 Summary: optional parameter is not supported in BeamSQL
 Key: BEAM-3325
 URL: https://issues.apache.org/jira/browse/BEAM-3325
 Project: Beam
  Issue Type: Bug
  Components: dsl-sql
Reporter: Xu Mingmin


I'm wring a UDF with optional parameters, the method code is:
{code}
  public static String eval(String kvString, String keyName
  , @Parameter(name = "pd", optional = true) String pairSeperator
  , @Parameter(name = "kvd", optional = true) String kvSeperator){
//...
  }
{code}

And I see this error when assembling a Beam pipeline:
{code}
Execution Plan: 
BeamProjectRel(ITEM_ID=[CAST(KV_EXT($0, 'itemId', DEFAULT(), 
DEFAULT())):BIGINT], TRANSACTION_ID=[CAST(KV_EXT($0, 'transactionId', 
DEFAULT(), DEFAULT())):BIGINT], TRANSACTION_TIME=[KV_EXT($0, 'createdDT', 
DEFAULT(), DEFAULT())])
  BeamIOSourceRel(table=[[]])

Exception in thread "main" java.lang.RuntimeException: 
java.lang.UnsupportedOperationException: Operator: DEFAULT is not supported yet!
at .
Caused by: java.lang.UnsupportedOperationException: Operator: DEFAULT is not 
supported yet!
at 
org.apache.beam.sdk.extensions.sql.impl.interpreter.BeamSqlFnExecutor.buildExpression(BeamSqlFnExecutor.java:425)
at 
org.apache.beam.sdk.extensions.sql.impl.interpreter.BeamSqlFnExecutor.buildExpression(BeamSqlFnExecutor.java:202)
at 
org.apache.beam.sdk.extensions.sql.impl.interpreter.BeamSqlFnExecutor.buildExpression(BeamSqlFnExecutor.java:202)
at 
org.apache.beam.sdk.extensions.sql.impl.interpreter.BeamSqlFnExecutor.(BeamSqlFnExecutor.java:126)
at 
org.apache.beam.sdk.extensions.sql.impl.rel.BeamProjectRel.buildBeamPipeline(BeamProjectRel.java:70)
at 
org.apache.beam.sdk.extensions.sql.impl.planner.BeamQueryPlanner.compileBeamPipeline(BeamQueryPlanner.java:116)
at 
org.apache.beam.sdk.extensions.sql.BeamSqlCli.compilePipeline(BeamSqlCli.java:112)
{code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (BEAM-3110) The transform Read(UnboundedKafkaSource) is currently not supported

[Xu Mingmin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3110?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xu Mingmin reassigned BEAM-3110:


Assignee: Xu Mingmin  (was: Aljoscha Krettek)

> The transform Read(UnboundedKafkaSource) is currently not supported
> ---
>
> Key: BEAM-3110
> URL: https://issues.apache.org/jira/browse/BEAM-3110
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Affects Versions: 2.2.0, 2.3.0
>Reporter: Xu Mingmin
>Assignee: Xu Mingmin
>
> I see this issue when submitting a job to Flink cluster. It appears after 
> build {{2.2.0-20170912.083349-51}}.
> {code}
> org.apache.flink.client.program.ProgramInvocationException: The main method 
> caused an error.
>   at 
> org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:545)
>   at 
> org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:419)
>   at 
> org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:381)
>   at 
> org.apache.flink.client.CliFrontend.executeProgram(CliFrontend.java:838)
>   at org.apache.flink.client.CliFrontend.run(CliFrontend.java:259)
>   at 
> org.apache.flink.client.CliFrontend.parseParameters(CliFrontend.java:1086)
>   at org.apache.flink.client.CliFrontend$2.call(CliFrontend.java:1133)
>   at org.apache.flink.client.CliFrontend$2.call(CliFrontend.java:1130)
>   at 
> org.apache.flink.runtime.security.HadoopSecurityContext$1.run(HadoopSecurityContext.java:43)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at javax.security.auth.Subject.doAs(Subject.java:415)
>   at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
>   at 
> org.apache.flink.runtime.security.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:40)
>   at org.apache.flink.client.CliFrontend.main(CliFrontend.java:1129)
> Caused by: java.lang.UnsupportedOperationException: The transform 
> Read(UnboundedKafkaSource) is currently not supported.
>   at 
> org.apache.beam.runners.flink.FlinkStreamingPipelineTranslator.visitPrimitiveTransform(FlinkStreamingPipelineTranslator.java:113)
>   at 
> org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:666)
>   at 
> org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:658)
>   at 
> org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:658)
>   at 
> org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:658)
>   at 
> org.apache.beam.sdk.runners.TransformHierarchy$Node.access$600(TransformHierarchy.java:311)
>   at 
> org.apache.beam.sdk.runners.TransformHierarchy.visit(TransformHierarchy.java:245)
>   at org.apache.beam.sdk.Pipeline.traverseTopologically(Pipeline.java:451)
>   at 
> org.apache.beam.runners.flink.FlinkPipelineTranslator.translate(FlinkPipelineTranslator.java:38)
>   at 
> org.apache.beam.runners.flink.FlinkStreamingPipelineTranslator.translate(FlinkStreamingPipelineTranslator.java:69)
>   at 
> org.apache.beam.runners.flink.FlinkPipelineExecutionEnvironment.translate(FlinkPipelineExecutionEnvironment.java:104)
>   at org.apache.beam.runners.flink.FlinkRunner.run(FlinkRunner.java:113)
>   at org.apache.beam.sdk.Pipeline.run(Pipeline.java:304)
>   at org.apache.beam.sdk.Pipeline.run(Pipeline.java:290)
> {code} 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BEAM-3110) The transform Read(UnboundedKafkaSource) is currently not supported

[Xu Mingmin (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3110?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16267580#comment-16267580
 ] 

Xu Mingmin commented on BEAM-3110:
--

Add more details on what's happening in the backend, as I see it again in 
another build.

The exact root cause is file 
{{META-INF/services/org.apache.beam.runners.core.construction.TransformPayloadTranslatorRegistrar}},
 it exists in several libraries(some are relocated/renamed). By default 
maven-shade use the first one and ignore others, but we need to merge them. So 
the steps I take are:
1). make sure {{beam-runners-direct-java}} is excluded when package as it 
relocates 
{{org.apache.beam.runners.core.construction.TransformPayloadTranslatorRegistrar}}
 which introduces additional chaos;
2). in you pom.xml, add these configuration:
{code}

  org.apache.maven.plugins
  maven-shade-plugin
  3.1.0
  

servicesmerge
  package
  
shade
  
  

  

  
  
  
  
  
  

{code}

> The transform Read(UnboundedKafkaSource) is currently not supported
> ---
>
> Key: BEAM-3110
> URL: https://issues.apache.org/jira/browse/BEAM-3110
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Affects Versions: 2.2.0, 2.3.0
>Reporter: Xu Mingmin
>Assignee: Aljoscha Krettek
>
> I see this issue when submitting a job to Flink cluster. It appears after 
> build {{2.2.0-20170912.083349-51}}.
> {code}
> org.apache.flink.client.program.ProgramInvocationException: The main method 
> caused an error.
>   at 
> org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:545)
>   at 
> org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:419)
>   at 
> org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:381)
>   at 
> org.apache.flink.client.CliFrontend.executeProgram(CliFrontend.java:838)
>   at org.apache.flink.client.CliFrontend.run(CliFrontend.java:259)
>   at 
> org.apache.flink.client.CliFrontend.parseParameters(CliFrontend.java:1086)
>   at org.apache.flink.client.CliFrontend$2.call(CliFrontend.java:1133)
>   at org.apache.flink.client.CliFrontend$2.call(CliFrontend.java:1130)
>   at 
> org.apache.flink.runtime.security.HadoopSecurityContext$1.run(HadoopSecurityContext.java:43)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at javax.security.auth.Subject.doAs(Subject.java:415)
>   at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
>   at 
> org.apache.flink.runtime.security.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:40)
>   at org.apache.flink.client.CliFrontend.main(CliFrontend.java:1129)
> Caused by: java.lang.UnsupportedOperationException: The transform 
> Read(UnboundedKafkaSource) is currently not supported.
>   at 
> org.apache.beam.runners.flink.FlinkStreamingPipelineTranslator.visitPrimitiveTransform(FlinkStreamingPipelineTranslator.java:113)
>   at 
> org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:666)
>   at 
> org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:658)
>   at 
> org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:658)
>   at 
> org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:658)
>   at 
> org.apache.beam.sdk.runners.TransformHierarchy$Node.access$600(TransformHierarchy.java:311)
>   at 
> org.apache.beam.sdk.runners.TransformHierarchy.visit(TransformHierarchy.java:245)
>   at org.apache.beam.sdk.Pipeline.traverseTopologically(Pipeline.java:451)
>   at 
> org.apache.beam.runners.flink.FlinkPipelineTranslator.translate(FlinkPipelineTranslator.java:38)
>   at 
> org.apache.beam.runners.flink.FlinkStreamingPipelineTranslator.translate(FlinkStreamingPipelineTranslator.java:69)
>   at 
> org.apache.beam.runners.flink.FlinkPipelineExecutionEnvironment.translate(FlinkPipelineExecutionEnvironment.java:104)
>   at org.apache.beam.runners.flink.FlinkRunner.run(FlinkRunner.java:113)
>   at org.apache.beam.sdk.Pipeline.run(Pipeline.java:304)
>   at org.apache.beam.sdk.Pipeline.run(Pipeline.java:290)
> {code} 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BEAM-3171) convert a join into lookup

[Xu Mingmin (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3171?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16248314#comment-16248314
 ] 

Xu Mingmin commented on BEAM-3171:
--

[~takidau] Yes, and here's the definition
{code}
public interface BeamSeekableTable extends Serializable{
  List seekRecord(BeamRecord lookupSubRecord);
}
{code}

I'm ok with either SeekableIO or refresh SideInput in the long term. To me this 
feature is necessary to run streaming jobs, as a short term solution. 


> convert a join into lookup
> --
>
> Key: BEAM-3171
> URL: https://issues.apache.org/jira/browse/BEAM-3171
> Project: Beam
>  Issue Type: New Feature
>  Components: dsl-sql
>Reporter: Xu Mingmin
>Assignee: Xu Mingmin
>  Labels: experimental
>
> We use BeamSQL to run streaming jobs mostly, and  add a join_as_lookup 
> improvement(internal branch) to cover the streaming-to-batch case(similar as 
> [1]). I could submit a PR as experimental if people are interested. 
> The rough solution is, if one source of join node implements 
> {{BeamSeekableTable}} and the other is not, then the join node is converted 
> to a fact-lookup operation.
> Ref:
> [1] 
> https://docs.google.com/document/d/1B-XnUwXh64lbswRieckU0BxtygSV58hysqZbpZmk03A/edit?usp=sharing
>  
> [~xumingming] [~takidau] for any comments



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (BEAM-3171) convert a join into lookup

[Xu Mingmin (JIRA)]

Xu Mingmin created BEAM-3171:


 Summary: convert a join into lookup
 Key: BEAM-3171
 URL: https://issues.apache.org/jira/browse/BEAM-3171
 Project: Beam
  Issue Type: New Feature
  Components: dsl-sql
Reporter: Xu Mingmin
Assignee: Xu Mingmin


We use BeamSQL to run streaming jobs mostly, and  add a join_as_lookup 
improvement(internal branch) to cover the streaming-to-batch case(similar as 
[1]). I could submit a PR as experimental if people are interested. 

The rough solution is, if one source of join node implements 
{{BeamSeekableTable}} and the other is not, then the join node is converted to 
a fact-lookup operation.

Ref:
[1] 
https://docs.google.com/document/d/1B-XnUwXh64lbswRieckU0BxtygSV58hysqZbpZmk03A/edit?usp=sharing
 

[~xumingming] [~takidau] for any comments



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (BEAM-3170) support topicPartition in BeamKafkaTable

[Xu Mingmin (JIRA)]

Xu Mingmin created BEAM-3170:


 Summary: support topicPartition in BeamKafkaTable
 Key: BEAM-3170
 URL: https://issues.apache.org/jira/browse/BEAM-3170
 Project: Beam
  Issue Type: Improvement
  Components: dsl-sql
Reporter: Xu Mingmin
Assignee: Xu Mingmin


In {{KafkaIO}} there're two options to set the source topics/partitions, and we 
only support the 1st one now:
{code}
1 .withTopics(topics)
2. withTopicPartitions(topicPartitions)
{code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BEAM-3110) The transform Read(UnboundedKafkaSource) is currently not supported

[Xu Mingmin (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3110?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16244960#comment-16244960
 ] 

Xu Mingmin commented on BEAM-3110:
--

seems the bug is not there in latest master branch, will do more test to verify.

> The transform Read(UnboundedKafkaSource) is currently not supported
> ---
>
> Key: BEAM-3110
> URL: https://issues.apache.org/jira/browse/BEAM-3110
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Affects Versions: 2.2.0, 2.3.0
>Reporter: Xu Mingmin
>Assignee: Aljoscha Krettek
>
> I see this issue when submitting a job to Flink cluster. It appears after 
> build {{2.2.0-20170912.083349-51}}.
> {code}
> org.apache.flink.client.program.ProgramInvocationException: The main method 
> caused an error.
>   at 
> org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:545)
>   at 
> org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:419)
>   at 
> org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:381)
>   at 
> org.apache.flink.client.CliFrontend.executeProgram(CliFrontend.java:838)
>   at org.apache.flink.client.CliFrontend.run(CliFrontend.java:259)
>   at 
> org.apache.flink.client.CliFrontend.parseParameters(CliFrontend.java:1086)
>   at org.apache.flink.client.CliFrontend$2.call(CliFrontend.java:1133)
>   at org.apache.flink.client.CliFrontend$2.call(CliFrontend.java:1130)
>   at 
> org.apache.flink.runtime.security.HadoopSecurityContext$1.run(HadoopSecurityContext.java:43)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at javax.security.auth.Subject.doAs(Subject.java:415)
>   at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
>   at 
> org.apache.flink.runtime.security.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:40)
>   at org.apache.flink.client.CliFrontend.main(CliFrontend.java:1129)
> Caused by: java.lang.UnsupportedOperationException: The transform 
> Read(UnboundedKafkaSource) is currently not supported.
>   at 
> org.apache.beam.runners.flink.FlinkStreamingPipelineTranslator.visitPrimitiveTransform(FlinkStreamingPipelineTranslator.java:113)
>   at 
> org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:666)
>   at 
> org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:658)
>   at 
> org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:658)
>   at 
> org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:658)
>   at 
> org.apache.beam.sdk.runners.TransformHierarchy$Node.access$600(TransformHierarchy.java:311)
>   at 
> org.apache.beam.sdk.runners.TransformHierarchy.visit(TransformHierarchy.java:245)
>   at org.apache.beam.sdk.Pipeline.traverseTopologically(Pipeline.java:451)
>   at 
> org.apache.beam.runners.flink.FlinkPipelineTranslator.translate(FlinkPipelineTranslator.java:38)
>   at 
> org.apache.beam.runners.flink.FlinkStreamingPipelineTranslator.translate(FlinkStreamingPipelineTranslator.java:69)
>   at 
> org.apache.beam.runners.flink.FlinkPipelineExecutionEnvironment.translate(FlinkPipelineExecutionEnvironment.java:104)
>   at org.apache.beam.runners.flink.FlinkRunner.run(FlinkRunner.java:113)
>   at org.apache.beam.sdk.Pipeline.run(Pipeline.java:304)
>   at org.apache.beam.sdk.Pipeline.run(Pipeline.java:290)
> {code} 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (BEAM-3084) Support for window analytic functions

[Xu Mingmin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3084?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xu Mingmin reassigned BEAM-3084:


Assignee: (was: Xu Mingmin)

> Support for window analytic functions
> -
>
> Key: BEAM-3084
> URL: https://issues.apache.org/jira/browse/BEAM-3084
> Project: Beam
>  Issue Type: New Feature
>  Components: dsl-sql
>Reporter: Xu Mingmin
>
> Calcite streaming documentation includes examples for using SQL window
> analytic functions
> https://calcite.apache.org/docs/stream.html#sliding-windows
> From: Kobi Salant 
> d...@beam.apache.org



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (BEAM-3161) Cannot output with timestamp XXXX

[Xu Mingmin (JIRA)]

Xu Mingmin created BEAM-3161:


 Summary: Cannot output with timestamp 
 Key: BEAM-3161
 URL: https://issues.apache.org/jira/browse/BEAM-3161
 Project: Beam
  Issue Type: Bug
  Components: dsl-sql
Reporter: Xu Mingmin
Assignee: Xu Mingmin
 Fix For: 2.3.0


BeamSQL throws exception when running query
{code}
select siteId, count(*), TUMBLE_START(unix_timestamp_to_date(eventTimestamp), 
INTERVAL '1' HOUR) from RHEOS_SOJEVENT_TOTAL GROUP BY siteId, 
TUMBLE(unix_timestamp_to_date(eventTimestamp), INTERVAL '1' HOUR)
{code}

Exception details:
{code}
Caused by: java.lang.IllegalArgumentException: Cannot output with timestamp 
2017-11-08T21:37:46.000Z. Output timestamps must be no earlier than the 
timestamp of the current input (2017-11-08T21:37:49.322Z) minus the allowed 
skew (0 milliseconds). See the DoFn#getAllowedTimestampSkew() Javadoc for 
details on changing the allowed skew.
at 
org.apache.beam.runners.core.SimpleDoFnRunner$DoFnProcessContext.checkTimestamp(SimpleDoFnRunner.java:463)
at 
org.apache.beam.runners.core.SimpleDoFnRunner$DoFnProcessContext.outputWithTimestamp(SimpleDoFnRunner.java:429)
at org.apache.Caused by: java.lang.IllegalArgumentException: Cannot 
output with timestamp 2017-11-08T21:37:46.000Z. Output timestamps must be no 
earlier than the timestamp of the current input (2017-11-08T21:37:49.322Z) 
minus the allowed skew (0 milliseconds). See the DoFn#getAllowedTimestampSkew() 
Javadoc for details on changing the allowed skew.
at 
org.apache.beam.runners.core.SimpleDoFnRunner$DoFnProcessContext.checkTimestamp(SimpleDoFnRunner.java:463)
at 
org.apache.beam.runners.core.SimpleDoFnRunner$DoFnProcessContext.outputWithTimestamp(SimpleDoFnRunner.java:429)
at 
org.apache.beam.sdk.transforms.WithTimestamps$AddTimestampsDoFn.processElement(WithTimestamps.java:138)beam.sdk.transforms.WithTimestamps$AddTimestampsDoFn.processElement(WithTimestamps.java:138)
{code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Closed] (BEAM-307) Upgrade/Test to Kafka 0.10

[Xu Mingmin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-307?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xu Mingmin closed BEAM-307.
---
   Resolution: Fixed
Fix Version/s: Not applicable

> Upgrade/Test to Kafka 0.10
> --
>
> Key: BEAM-307
> URL: https://issues.apache.org/jira/browse/BEAM-307
> Project: Beam
>  Issue Type: Task
>  Components: sdk-java-extensions
>Reporter: Jean-Baptiste Onofré
>Assignee: Xu Mingmin
>  Labels: backward-incompatible
> Fix For: Not applicable
>
>
> I gonna test at least that the KafkaIO works fine with Kafka 0.10 (I'm 
> preparing new complete samples around that).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (BEAM-3110) The transform Read(UnboundedKafkaSource) is currently not supported

[Xu Mingmin (JIRA)]

Xu Mingmin created BEAM-3110:


 Summary: The transform Read(UnboundedKafkaSource) is currently not 
supported
 Key: BEAM-3110
 URL: https://issues.apache.org/jira/browse/BEAM-3110
 Project: Beam
  Issue Type: Bug
  Components: runner-flink
Affects Versions: 2.2.0, 2.3.0
Reporter: Xu Mingmin
Assignee: Aljoscha Krettek


I see this issue when submitting a job to Flink cluster. It appears after build 
{{2.2.0-20170912.083349-51}}.

{code}
org.apache.flink.client.program.ProgramInvocationException: The main method 
caused an error.
at 
org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:545)
at 
org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:419)
at 
org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:381)
at 
org.apache.flink.client.CliFrontend.executeProgram(CliFrontend.java:838)
at org.apache.flink.client.CliFrontend.run(CliFrontend.java:259)
at 
org.apache.flink.client.CliFrontend.parseParameters(CliFrontend.java:1086)
at org.apache.flink.client.CliFrontend$2.call(CliFrontend.java:1133)
at org.apache.flink.client.CliFrontend$2.call(CliFrontend.java:1130)
at 
org.apache.flink.runtime.security.HadoopSecurityContext$1.run(HadoopSecurityContext.java:43)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
at 
org.apache.flink.runtime.security.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:40)
at org.apache.flink.client.CliFrontend.main(CliFrontend.java:1129)
Caused by: java.lang.UnsupportedOperationException: The transform 
Read(UnboundedKafkaSource) is currently not supported.
at 
org.apache.beam.runners.flink.FlinkStreamingPipelineTranslator.visitPrimitiveTransform(FlinkStreamingPipelineTranslator.java:113)
at 
org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:666)
at 
org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:658)
at 
org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:658)
at 
org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:658)
at 
org.apache.beam.sdk.runners.TransformHierarchy$Node.access$600(TransformHierarchy.java:311)
at 
org.apache.beam.sdk.runners.TransformHierarchy.visit(TransformHierarchy.java:245)
at org.apache.beam.sdk.Pipeline.traverseTopologically(Pipeline.java:451)
at 
org.apache.beam.runners.flink.FlinkPipelineTranslator.translate(FlinkPipelineTranslator.java:38)
at 
org.apache.beam.runners.flink.FlinkStreamingPipelineTranslator.translate(FlinkStreamingPipelineTranslator.java:69)
at 
org.apache.beam.runners.flink.FlinkPipelineExecutionEnvironment.translate(FlinkPipelineExecutionEnvironment.java:104)
at org.apache.beam.runners.flink.FlinkRunner.run(FlinkRunner.java:113)
at org.apache.beam.sdk.Pipeline.run(Pipeline.java:304)
at org.apache.beam.sdk.Pipeline.run(Pipeline.java:290)
{code} 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (BEAM-2770) Mark all SQL APIs @Experimental

[Xu Mingmin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2770?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xu Mingmin resolved BEAM-2770.
--
   Resolution: Fixed
Fix Version/s: Not applicable

> Mark all SQL APIs @Experimental
> ---
>
> Key: BEAM-2770
> URL: https://issues.apache.org/jira/browse/BEAM-2770
> Project: Beam
>  Issue Type: Bug
>  Components: dsl-sql
>Reporter: Tyler Akidau
>Assignee: Tyler Akidau
> Fix For: Not applicable
>
>
> Some, but not all, of the public SQL API classes are marked \@Experiemental. 
> This should be consistent across the APIs for the initial release.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Closed] (BEAM-2209) NoSuchFiled exception when use beam-dsls-sql jar

[Xu Mingmin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2209?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xu Mingmin closed BEAM-2209.

   Resolution: Fixed
Fix Version/s: Not applicable

close task, PR 3140 is merged.

> NoSuchFiled exception when use beam-dsls-sql jar
> 
>
> Key: BEAM-2209
> URL: https://issues.apache.org/jira/browse/BEAM-2209
> Project: Beam
>  Issue Type: Bug
>  Components: dsl-sql
> Environment: jdk1.8 idea 
>Reporter: Fei Feng
>Assignee: Fei Feng
> Fix For: Not applicable
>
>
> when use beam-dsls-sql jar to filter BeamSqlRecord will cause an NoSuchField 
> Exception。
> change
> from 
> for (RexNode subNode : node.operands) {
> subExps.add(buildExpression(subNode));
>  }
> to 
> for (RexNode subNode : node.getOperands()) {
> subExps.add(buildExpression(subNode));
>   }
> will fix this error.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BEAM-3093) add an option 'FirstPollOffsetStrategy' to KafkaIO

[Xu Mingmin (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16216282#comment-16216282
 ] 

Xu Mingmin commented on BEAM-3093:
--

Mostly it's about case (1) and (2), but I think it's more clear if we list the 
4 scenarios together in one method.

Reset offset with admin commands may not be easy for developers, especially in 
production environment. (I don't have the permission.)

> add an option 'FirstPollOffsetStrategy' to KafkaIO
> --
>
> Key: BEAM-3093
> URL: https://issues.apache.org/jira/browse/BEAM-3093
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Xu Mingmin
>Assignee: Kenneth Knowles
>
> This is a feature borrowed from Storm KafkaSpout.
> *What's the issue?*
> In KafkaIO, when offset is stored either in checkpoint or auto_committed, it 
> cannot be changed in application, to force to read from earliest/latest. 
> --This feature is important to reset the start offset when relaunching a job.
> *Proposed solution:*
> By borrowing the FirstPollOffsetStrategy concept, users can have more options:
> 1). *{{EARLIEST}}*: always start_from_beginning no matter of what's in 
> checkpoint/auto_commit;
> 2). *{{LATEST}}*: always start_from_latest no matter of what's in 
> checkpoint/auto_commit;
> 3). *{{UNCOMMITTED_EARLIEST}}*: if no offset in checkpoint/auto_commit then 
> start_from_beginning if, otherwise start_from_previous_offset;
> 4). *{{UNCOMMITTED_LATEST}}*: if no offset in checkpoint/auto_commit then 
> start_from_latest, otherwise start_from_previous_offset;
> [~rangadi], any comments?



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (BEAM-3093) add an option 'FirstPollOffsetStrategy' to KafkaIO

[Xu Mingmin (JIRA)]

Xu Mingmin created BEAM-3093:


 Summary: add an option 'FirstPollOffsetStrategy' to KafkaIO
 Key: BEAM-3093
 URL: https://issues.apache.org/jira/browse/BEAM-3093
 Project: Beam
  Issue Type: Improvement
  Components: sdk-java-core
Reporter: Xu Mingmin
Assignee: Kenneth Knowles


This is a feature borrowed from Storm KafkaSpout.

*What's the issue?*
In KafkaIO, when offset is stored either in checkpoint or auto_committed, it 
cannot be changed in application, to force to read from earliest/latest. --This 
feature is important to reset the start offset when relaunching a job.

*Proposed solution:*
By borrowing the FirstPollOffsetStrategy concept, users can have more options:
1). *{{EARLIEST}}*: always start_from_beginning no matter of what's in 
checkpoint/auto_commit;
2). *{{LATEST}}*: always start_from_latest no matter of what's in 
checkpoint/auto_commit;
3). *{{UNCOMMITTED_EARLIEST}}*: if no offset in checkpoint/auto_commit then 
start_from_beginning if, otherwise start_from_previous_offset;
4). *{{UNCOMMITTED_LATEST}}*: if no offset in checkpoint/auto_commit then 
start_from_latest, otherwise start_from_previous_offset;

[~rangadi], any comments?



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Closed] (BEAM-2227) add a post to explain how Beam SQL works

[Xu Mingmin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2227?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xu Mingmin closed BEAM-2227.

   Resolution: Fixed
Fix Version/s: Not applicable

> add a post to explain how Beam SQL works
> 
>
> Key: BEAM-2227
> URL: https://issues.apache.org/jira/browse/BEAM-2227
> Project: Beam
>  Issue Type: Task
>  Components: dsl-sql, website
>Reporter: Xu Mingmin
>Assignee: Xu Mingmin
>  Labels: dsl_sql_merge
> Fix For: Not applicable
>
>
> As mentioned in maillist, it's important to clarify what Beam SQL does, and 
> how it works.
> {quote}
> There's some confusion where people think we're just doing a pass through
> to the framework's SQL engine. We'll have to make sure we're clear on how
> Beam's SQL works in the docs.
> {quote}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Closed] (BEAM-1705) visualize the DAG of Pipeline

[Xu Mingmin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1705?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xu Mingmin closed BEAM-1705.

   Resolution: Won't Fix
Fix Version/s: Not applicable

> visualize the DAG of Pipeline
> -
>
> Key: BEAM-1705
> URL: https://issues.apache.org/jira/browse/BEAM-1705
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-core
>Reporter: Xu Mingmin
>Assignee: Xu Mingmin
>Priority: Minor
> Fix For: Not applicable
>
>
> After pipeline is compiled and run in different runners, such as Spark, 
> Flink, it shows the data flow as a graph. There's limitation here:
> 1. the job must be run first;
> 2. The pipeline in a runner is not exact same as original designed, some 
> steps are merged for example.
> What to do:
> a method or tool to show the DAG of pipeline in a graph, to display the 
> native logic components, flows.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Closed] (BEAM-2125) update JavaDoc of BoundedWindow

[Xu Mingmin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2125?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xu Mingmin closed BEAM-2125.

   Resolution: Fixed
Fix Version/s: Not applicable

> update JavaDoc of BoundedWindow
> ---
>
> Key: BEAM-2125
> URL: https://issues.apache.org/jira/browse/BEAM-2125
> Project: Beam
>  Issue Type: Task
>  Components: sdk-java-core
>Reporter: Xu Mingmin
>Assignee: Xu Mingmin
>Priority: Minor
> Fix For: Not applicable
>
>
> {{BoundedWindow}} represents the window information of element, not 
> represents a finite grouping of elements. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Closed] (BEAM-2672) test merge in BeamSqlUdaf

[Xu Mingmin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2672?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xu Mingmin closed BEAM-2672.

   Resolution: Fixed
Fix Version/s: Not applicable

> test merge in BeamSqlUdaf
> -
>
> Key: BEAM-2672
> URL: https://issues.apache.org/jira/browse/BEAM-2672
> Project: Beam
>  Issue Type: Test
>  Components: dsl-sql
>Reporter: Xu Mingmin
>Assignee: Xu Mingmin
>Priority: Minor
> Fix For: Not applicable
>
>
> The integration test with {{DirectRunner}} cannot cover method {{merge}} in 
> {{BeamSqlUdaf}}.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (BEAM-3084) Support for window analytic functions

[Xu Mingmin (JIRA)]

Xu Mingmin created BEAM-3084:


 Summary: Support for window analytic functions
 Key: BEAM-3084
 URL: https://issues.apache.org/jira/browse/BEAM-3084
 Project: Beam
  Issue Type: New Feature
  Components: dsl-sql
Reporter: Xu Mingmin
Assignee: Xu Mingmin


Calcite streaming documentation includes examples for using SQL window
analytic functions
https://calcite.apache.org/docs/stream.html#sliding-windows

From: Kobi Salant 
d...@beam.apache.org



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BEAM-2998) add IT test for SQL

[Xu Mingmin (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2998?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16185018#comment-16185018
 ] 

Xu Mingmin commented on BEAM-2998:
--

seems CI job cannot run IT tests on shaded libraries, true?

> add IT test for SQL
> ---
>
> Key: BEAM-2998
> URL: https://issues.apache.org/jira/browse/BEAM-2998
> Project: Beam
>  Issue Type: Test
>  Components: dsl-sql, testing
>Reporter: Xu Mingmin
>
> Add IT test for SQL module
> https://github.com/apache/beam/blob/master/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/example/BeamSqlExample.java
>  is the base example.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (BEAM-2998) add IT test for SQL

[Xu Mingmin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2998?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xu Mingmin updated BEAM-2998:
-
Component/s: testing

> add IT test for SQL
> ---
>
> Key: BEAM-2998
> URL: https://issues.apache.org/jira/browse/BEAM-2998
> Project: Beam
>  Issue Type: Test
>  Components: dsl-sql, testing
>Reporter: Xu Mingmin
>
> Add IT test for SQL module
> https://github.com/apache/beam/blob/master/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/example/BeamSqlExample.java
>  is the base example.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (BEAM-2998) add IT test for SQL

[Xu Mingmin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2998?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xu Mingmin updated BEAM-2998:
-
Fix Version/s: (was: 2.2.0)

> add IT test for SQL
> ---
>
> Key: BEAM-2998
> URL: https://issues.apache.org/jira/browse/BEAM-2998
> Project: Beam
>  Issue Type: Test
>  Components: dsl-sql
>Reporter: Xu Mingmin
>
> Add IT test for SQL module
> https://github.com/apache/beam/blob/master/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/example/BeamSqlExample.java
>  is the base example.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (BEAM-2998) add IT test for SQL

[Xu Mingmin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2998?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xu Mingmin updated BEAM-2998:
-
Description: 
Add IT test for SQL module

https://github.com/apache/beam/blob/master/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/example/BeamSqlExample.java
 is the base example.

  was:Add IT test for SQL module


> add IT test for SQL
> ---
>
> Key: BEAM-2998
> URL: https://issues.apache.org/jira/browse/BEAM-2998
> Project: Beam
>  Issue Type: Test
>  Components: dsl-sql
>Reporter: Xu Mingmin
> Fix For: 2.2.0
>
>
> Add IT test for SQL module
> https://github.com/apache/beam/blob/master/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/example/BeamSqlExample.java
>  is the base example.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (BEAM-2998) add IT test for SQL

[Xu Mingmin (JIRA)]

Xu Mingmin created BEAM-2998:


 Summary: add IT test for SQL
 Key: BEAM-2998
 URL: https://issues.apache.org/jira/browse/BEAM-2998
 Project: Beam
  Issue Type: Test
  Components: dsl-sql
Reporter: Xu Mingmin
 Fix For: 2.2.0


Add IT test for SQL module



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (BEAM-2990) support data type MAP/ARRAY

[Xu Mingmin (JIRA)]

Xu Mingmin created BEAM-2990:


 Summary: support data type MAP/ARRAY 
 Key: BEAM-2990
 URL: https://issues.apache.org/jira/browse/BEAM-2990
 Project: Beam
  Issue Type: New Feature
  Components: dsl-sql
Reporter: Xu Mingmin
Assignee: Shayang Zang


support Non-scalar types:
MAP Collection of keys mapped to values
ARRAY   Ordered, contiguous collection that may contain duplicates



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BEAM-2968) Master build broken due to error when running autocomplete_test.py

[Xu Mingmin (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16173499#comment-16173499
 ] 

Xu Mingmin commented on BEAM-2968:
--

seems the error is gone, [~aviemzur].

> Master build broken due to error when running autocomplete_test.py
> --
>
> Key: BEAM-2968
> URL: https://issues.apache.org/jira/browse/BEAM-2968
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core
>Reporter: Aviem Zur
>Assignee: Ahmet Altay
>
> Master build fails with:
> {code}
> org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute 
> goal org.codehaus.mojo:exec-maven-plugin:1.5.0:exec (setuptools-test) on 
> project beam-sdks-python: Command execution failed.
> {code}
> Last nightly build: 
> https://builds.apache.org/job/beam_Release_NightlySnapshot/536/
> All recent pre-commit builds: 
> https://builds.apache.org/view/A-D/view/Beam/job/beam_PreCommit_Java_MavenInstall/
> {code}
> py27gcp runtests: commands[3] | python 
> apache_beam/examples/complete/autocomplete_test.py
> Traceback (most recent call last):
>   File "apache_beam/examples/complete/autocomplete_test.py", line 22, in 
> 
> import apache_beam as beam
>   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Java_MavenInstall/sdks/python/apache_beam/__init__.py",
>  line 87, in 
> from apache_beam import io
>   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Java_MavenInstall/sdks/python/apache_beam/io/__init__.py",
>  line 21, in 
> from apache_beam.io.avroio import *
>   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Java_MavenInstall/sdks/python/apache_beam/io/avroio.py",
>  line 56, in 
> from apache_beam.io import filebasedsink
>   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Java_MavenInstall/sdks/python/apache_beam/io/filebasedsink.py",
>  line 32, in 
> from apache_beam.io.filesystems import FileSystems
>   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Java_MavenInstall/sdks/python/apache_beam/io/filesystems.py",
>  line 30, in 
> from apache_beam.io.gcp.gcsfilesystem import GCSFileSystem
>   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Java_MavenInstall/sdks/python/apache_beam/io/gcp/gcsfilesystem.py",
>  line 27, in 
> from apache_beam.io.gcp import gcsio
>   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Java_MavenInstall/sdks/python/apache_beam/io/gcp/gcsio.py",
>  line 37, in 
> from apache_beam.utils import retry
>   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Java_MavenInstall/sdks/python/apache_beam/utils/retry.py",
>  line 40, in 
> from apitools.base.py.exceptions import HttpError
>   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Java_MavenInstall/sdks/python/target/.tox/py27gcp/local/lib/python2.7/site-packages/apitools/base/py/__init__.py",
>  line 21, in 
> from apitools.base.py.base_api import *
>   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Java_MavenInstall/sdks/python/target/.tox/py27gcp/local/lib/python2.7/site-packages/apitools/base/py/base_api.py",
>  line 31, in 
> from apitools.base.protorpclite import message_types
>   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Java_MavenInstall/sdks/python/target/.tox/py27gcp/local/lib/python2.7/site-packages/apitools/base/protorpclite/message_types.py",
>  line 25, in 
> from apitools.base.protorpclite import messages
>   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Java_MavenInstall/sdks/python/target/.tox/py27gcp/local/lib/python2.7/site-packages/apitools/base/protorpclite/messages.py",
>  line 1165, in 
> class Field(six.with_metaclass(_FieldMeta, object)):
> TypeError: Error when calling the metaclass bases
> metaclass conflict: the metaclass of a derived class must be a 
> (non-strict) subclass of the metaclasses of all its bases
> ERROR: InvocationError: 
> '/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Java_MavenInstall/sdks/python/target/.tox/py27gcp/bin/python
>  apache_beam/examples/complete/autocomplete_test.py'
> ___ summary 
> 
> ERROR:   docs: commands failed
>   lint: commands succeeded
>   py27: commands succeeded
>   py27cython: commands succeeded
> ERROR:   py27gcp: commands failed
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (BEAM-2954) update shade configurations in extension/sql

[Xu Mingmin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xu Mingmin updated BEAM-2954:
-
Summary: update shade configurations in extension/sql  (was: opt-out shade 
in extension/sql)

> update shade configurations in extension/sql
> 
>
> Key: BEAM-2954
> URL: https://issues.apache.org/jira/browse/BEAM-2954
> Project: Beam
>  Issue Type: Bug
>  Components: dsl-sql
>Reporter: Xu Mingmin
>Assignee: Xu Mingmin
>  Labels: 2.2.0, shade, sql
> Fix For: 2.2.0
>
>
> {{guava}} is shaded by default in beam modules, while {{calcite}}(a 
> dependency of SQL) includes {{guava}} which is not shaded. Below error is 
> thrown when calling {{calcite}} methods with guava classed.
> {code}
> Exception in thread "main" java.lang.NoSuchMethodError: 
> org.apache.calcite.rel.core.Aggregate.getGroupSets()Lorg/apache/beam/sdks/java/extensions/sql/repackaged/com/google/common/collect/ImmutableList;
>   at 
> org.apache.beam.sdk.extensions.sql.impl.rule.BeamAggregationRule.updateWindowTrigger(BeamAggregationRule.java:139)
>   at 
> org.apache.beam.sdk.extensions.sql.impl.rule.BeamAggregationRule.onMatch(BeamAggregationRule.java:73)
>   at 
> org.apache.calcite.plan.volcano.VolcanoRuleCall.onMatch(VolcanoRuleCall.java:212)
>   at 
> org.apache.calcite.plan.volcano.VolcanoPlanner.findBestExp(VolcanoPlanner.java:650)
>   at 
> org.apache.calcite.tools.Programs$RuleSetProgram.run(Programs.java:368)
>   at 
> org.apache.calcite.prepare.PlannerImpl.transform(PlannerImpl.java:313)
>   at 
> org.apache.beam.sdk.extensions.sql.impl.planner.BeamQueryPlanner.convertToBeamRel(BeamQueryPlanner.java:149)
>   at 
> org.apache.beam.sdk.extensions.sql.impl.planner.BeamQueryPlanner.validateAndConvert(BeamQueryPlanner.java:140)
>   at 
> org.apache.beam.sdk.extensions.sql.impl.planner.BeamQueryPlanner.convertToBeamRel(BeamQueryPlanner.java:128)
>   at 
> org.apache.beam.sdk.extensions.sql.impl.planner.BeamQueryPlanner.compileBeamPipeline(BeamQueryPlanner.java:113)
>   at 
> org.apache.beam.sdk.extensions.sql.impl.BeamSqlCli.compilePipeline(BeamSqlCli.java:62)
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BEAM-2954) opt-out shade in extension/sql

[Xu Mingmin (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16165338#comment-16165338
 ] 

Xu Mingmin commented on BEAM-2954:
--

make sense to me, I'll turn to shade Calcite dependencies.

> opt-out shade in extension/sql
> --
>
> Key: BEAM-2954
> URL: https://issues.apache.org/jira/browse/BEAM-2954
> Project: Beam
>  Issue Type: Bug
>  Components: dsl-sql
>Reporter: Xu Mingmin
>Assignee: Xu Mingmin
>  Labels: 2.2.0, shade, sql
> Fix For: 2.2.0
>
>
> {{guava}} is shaded by default in beam modules, while {{calcite}}(a 
> dependency of SQL) includes {{guava}} which is not shaded. Below error is 
> thrown when calling {{calcite}} methods with guava classed.
> {code}
> Exception in thread "main" java.lang.NoSuchMethodError: 
> org.apache.calcite.rel.core.Aggregate.getGroupSets()Lorg/apache/beam/sdks/java/extensions/sql/repackaged/com/google/common/collect/ImmutableList;
>   at 
> org.apache.beam.sdk.extensions.sql.impl.rule.BeamAggregationRule.updateWindowTrigger(BeamAggregationRule.java:139)
>   at 
> org.apache.beam.sdk.extensions.sql.impl.rule.BeamAggregationRule.onMatch(BeamAggregationRule.java:73)
>   at 
> org.apache.calcite.plan.volcano.VolcanoRuleCall.onMatch(VolcanoRuleCall.java:212)
>   at 
> org.apache.calcite.plan.volcano.VolcanoPlanner.findBestExp(VolcanoPlanner.java:650)
>   at 
> org.apache.calcite.tools.Programs$RuleSetProgram.run(Programs.java:368)
>   at 
> org.apache.calcite.prepare.PlannerImpl.transform(PlannerImpl.java:313)
>   at 
> org.apache.beam.sdk.extensions.sql.impl.planner.BeamQueryPlanner.convertToBeamRel(BeamQueryPlanner.java:149)
>   at 
> org.apache.beam.sdk.extensions.sql.impl.planner.BeamQueryPlanner.validateAndConvert(BeamQueryPlanner.java:140)
>   at 
> org.apache.beam.sdk.extensions.sql.impl.planner.BeamQueryPlanner.convertToBeamRel(BeamQueryPlanner.java:128)
>   at 
> org.apache.beam.sdk.extensions.sql.impl.planner.BeamQueryPlanner.compileBeamPipeline(BeamQueryPlanner.java:113)
>   at 
> org.apache.beam.sdk.extensions.sql.impl.BeamSqlCli.compilePipeline(BeamSqlCli.java:62)
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (BEAM-2954) opt-out shade in extension/sql

[Xu Mingmin (JIRA)]

Xu Mingmin created BEAM-2954:


 Summary: opt-out shade in extension/sql
 Key: BEAM-2954
 URL: https://issues.apache.org/jira/browse/BEAM-2954
 Project: Beam
  Issue Type: Bug
  Components: dsl-sql
Reporter: Xu Mingmin
Assignee: Xu Mingmin
 Fix For: 2.2.0


{{guava}} is shaded by default in beam modules, while {{calcite}}(a dependency 
of SQL) includes {{guava}} which is not shaded. Below error is thrown when 
calling {{calcite}} methods with guava classed.

{code}
Exception in thread "main" java.lang.NoSuchMethodError: 
org.apache.calcite.rel.core.Aggregate.getGroupSets()Lorg/apache/beam/sdks/java/extensions/sql/repackaged/com/google/common/collect/ImmutableList;
at 
org.apache.beam.sdk.extensions.sql.impl.rule.BeamAggregationRule.updateWindowTrigger(BeamAggregationRule.java:139)
at 
org.apache.beam.sdk.extensions.sql.impl.rule.BeamAggregationRule.onMatch(BeamAggregationRule.java:73)
at 
org.apache.calcite.plan.volcano.VolcanoRuleCall.onMatch(VolcanoRuleCall.java:212)
at 
org.apache.calcite.plan.volcano.VolcanoPlanner.findBestExp(VolcanoPlanner.java:650)
at 
org.apache.calcite.tools.Programs$RuleSetProgram.run(Programs.java:368)
at 
org.apache.calcite.prepare.PlannerImpl.transform(PlannerImpl.java:313)
at 
org.apache.beam.sdk.extensions.sql.impl.planner.BeamQueryPlanner.convertToBeamRel(BeamQueryPlanner.java:149)
at 
org.apache.beam.sdk.extensions.sql.impl.planner.BeamQueryPlanner.validateAndConvert(BeamQueryPlanner.java:140)
at 
org.apache.beam.sdk.extensions.sql.impl.planner.BeamQueryPlanner.convertToBeamRel(BeamQueryPlanner.java:128)
at 
org.apache.beam.sdk.extensions.sql.impl.planner.BeamQueryPlanner.compileBeamPipeline(BeamQueryPlanner.java:113)
at 
org.apache.beam.sdk.extensions.sql.impl.BeamSqlCli.compilePipeline(BeamSqlCli.java:62)
at 
com.ebay.dss.beam_sql_demo.ToraSqlCli.executeQuery(ToraSqlCli.java:72)
at com.ebay.dss.beam_sql_demo.ToraSqlCli.exec(ToraSqlCli.java:111)
at com.ebay.dss.beam_sql_demo.ToraSqlCli.main(ToraSqlCli.java:130)
{code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Closed] (BEAM-301) Add a Beam SQL DSL

[Xu Mingmin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-301?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xu Mingmin closed BEAM-301.
---
Resolution: Fixed

> Add a Beam SQL DSL
> --
>
> Key: BEAM-301
> URL: https://issues.apache.org/jira/browse/BEAM-301
> Project: Beam
>  Issue Type: New Feature
>  Components: dsl-sql, sdk-ideas
>Reporter: Jean-Baptiste Onofré
>Assignee: Xu Mingmin
> Fix For: 2.2.0
>
>
> The SQL DSL helps developers to build a Beam pipeline from SQL statement in 
> String directly. 
> In Phase I, it starts to support INSERT/SELECT queries with FILTERs, one 
> example SQL as below:
> {code}
> INSERT INTO `SUB_USEREVENT` (`SITEID`, `PAGEID`, `PAGENAME`, `EVENTTIMESTAMP`)
> (SELECT STREAM `USEREVENT`.`SITEID`, `USEREVENT`.`PAGEID`, 
> `USEREVENT`.`PAGENAME`, `USEREVENT`.`EVENTTIMESTAMP`
> FROM `USEREVENT` AS `USEREVENT`
> WHERE `USEREVENT`.`SITEID` > 10)
> {code}
> A design doc is available at 
> https://docs.google.com/document/d/1Uc5xYTpO9qsLXtT38OfuoqSLimH_0a1Bz5BsCROMzCU/edit?usp=sharing.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BEAM-2804) support TIMESTAMP in Sort

[Xu Mingmin (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2804?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16159429#comment-16159429
 ] 

Xu Mingmin commented on BEAM-2804:
--

[~vectorijk], I intend to keep the scope of this task as a small fix in 
BeamSortRel. 

+1 that we need to revisit the type mapping in {{BeamRecordSqlType}}, for 
DATE/TIME/TIMESTAMP AFAIK, feel free to create a new task.

> support TIMESTAMP in Sort
> -
>
> Key: BEAM-2804
> URL: https://issues.apache.org/jira/browse/BEAM-2804
> Project: Beam
>  Issue Type: Improvement
>  Components: dsl-sql
>Reporter: Xu Mingmin
>Assignee: Shayang Zang
>Priority: Minor
>  Labels: beginner
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BEAM-2842) python test failure in local maven build

[Xu Mingmin (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2842?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16157745#comment-16157745
 ] 

Xu Mingmin commented on BEAM-2842:
--

duplicated with https://issues.apache.org/jira/browse/BEAM-2861

> python test failure in local maven build
> 
>
> Key: BEAM-2842
> URL: https://issues.apache.org/jira/browse/BEAM-2842
> Project: Beam
>  Issue Type: Test
>  Components: sdk-ideas, website
>Reporter: Xu Mingmin
>Assignee: Reuven Lax
>Priority: Minor
> Fix For: Not applicable
>
>
> See below error when running {{mvn clean install}} locally (Mac OS), with a 
> fresh environment. 
> I resolve it following the guideline mentioned in error log, and wanna what's 
> the proper way to improve it:
> 1). add a task in prepare step of 'Contribution Guide';
> 2). option for shared credential?
> {code}
> File 
> “/Users//Desktop/beam/sdks/python/target/.tox/py27gcp/lib/python2.7/site-packages/google/auth/_default.py”,
>  line 282, in default
> raise exceptions.DefaultCredentialsError(_HELP_MESSAGE)
> DefaultCredentialsError: Could not automatically determine credentials. 
> Please set GOOGLE_APPLICATION_CREDENTIALS or
> explicitly create credential and re-run the application. For more
> information, please see
> https://developers.google.com/accounts/docs/application-default-credentials.
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Closed] (BEAM-2842) python test failure in local maven build

[Xu Mingmin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2842?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xu Mingmin closed BEAM-2842.

   Resolution: Duplicate
Fix Version/s: Not applicable

> python test failure in local maven build
> 
>
> Key: BEAM-2842
> URL: https://issues.apache.org/jira/browse/BEAM-2842
> Project: Beam
>  Issue Type: Test
>  Components: sdk-ideas, website
>Reporter: Xu Mingmin
>Assignee: Reuven Lax
>Priority: Minor
> Fix For: Not applicable
>
>
> See below error when running {{mvn clean install}} locally (Mac OS), with a 
> fresh environment. 
> I resolve it following the guideline mentioned in error log, and wanna what's 
> the proper way to improve it:
> 1). add a task in prepare step of 'Contribution Guide';
> 2). option for shared credential?
> {code}
> File 
> “/Users//Desktop/beam/sdks/python/target/.tox/py27gcp/lib/python2.7/site-packages/google/auth/_default.py”,
>  line 282, in default
> raise exceptions.DefaultCredentialsError(_HELP_MESSAGE)
> DefaultCredentialsError: Could not automatically determine credentials. 
> Please set GOOGLE_APPLICATION_CREDENTIALS or
> explicitly create credential and re-run the application. For more
> information, please see
> https://developers.google.com/accounts/docs/application-default-credentials.
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (BEAM-2149) BeamKafkaCSVTable: support column type other than string

[Xu Mingmin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xu Mingmin updated BEAM-2149:
-
Fix Version/s: (was: 2.2.0)
   Not applicable

> BeamKafkaCSVTable: support column type other than string
> 
>
> Key: BEAM-2149
> URL: https://issues.apache.org/jira/browse/BEAM-2149
> Project: Beam
>  Issue Type: Improvement
>  Components: dsl-sql
>Reporter: James Xu
>Assignee: James Xu
>Priority: Minor
> Fix For: Not applicable
>
>
> Currently, BeamKafkaCSVTable only supports String column, need to support 
> other types.
> Also, use a more robust library to parse CSV.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (BEAM-301) Add a Beam SQL DSL

[Xu Mingmin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-301?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xu Mingmin updated BEAM-301:

Fix Version/s: 2.2.0

> Add a Beam SQL DSL
> --
>
> Key: BEAM-301
> URL: https://issues.apache.org/jira/browse/BEAM-301
> Project: Beam
>  Issue Type: New Feature
>  Components: dsl-sql, sdk-ideas
>Reporter: Jean-Baptiste Onofré
>Assignee: Xu Mingmin
> Fix For: 2.2.0
>
>
> The SQL DSL helps developers to build a Beam pipeline from SQL statement in 
> String directly. 
> In Phase I, it starts to support INSERT/SELECT queries with FILTERs, one 
> example SQL as below:
> {code}
> INSERT INTO `SUB_USEREVENT` (`SITEID`, `PAGEID`, `PAGENAME`, `EVENTTIMESTAMP`)
> (SELECT STREAM `USEREVENT`.`SITEID`, `USEREVENT`.`PAGEID`, 
> `USEREVENT`.`PAGENAME`, `USEREVENT`.`EVENTTIMESTAMP`
> FROM `USEREVENT` AS `USEREVENT`
> WHERE `USEREVENT`.`SITEID` > 10)
> {code}
> A design doc is available at 
> https://docs.google.com/document/d/1Uc5xYTpO9qsLXtT38OfuoqSLimH_0a1Bz5BsCROMzCU/edit?usp=sharing.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Reopened] (BEAM-2159) CAST operator support

[Xu Mingmin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xu Mingmin reopened BEAM-2159:
--

> CAST operator support
> -
>
> Key: BEAM-2159
> URL: https://issues.apache.org/jira/browse/BEAM-2159
> Project: Beam
>  Issue Type: Task
>  Components: dsl-sql
>Reporter: Xu Mingmin
>Assignee: Tarush Grover
>  Labels: dsl_sql_merge
> Fix For: Not applicable
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Reopened] (BEAM-2204) support TUMBLE/HOP/SESSION_START

[Xu Mingmin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xu Mingmin reopened BEAM-2204:
--

> support TUMBLE/HOP/SESSION_START
> 
>
> Key: BEAM-2204
> URL: https://issues.apache.org/jira/browse/BEAM-2204
> Project: Beam
>  Issue Type: Bug
>  Components: dsl-sql
>Reporter: Xu Mingmin
>Assignee: Xu Mingmin
>  Labels: dsl_sql_merge
> Fix For: Not applicable
>
>
> As described in {{CALCITE-1761}}, TUMBLE/HOP/SESSION_START/END do not resolve 
> time field correctly.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (BEAM-2255) support ORDER BY clause

[Xu Mingmin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2255?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xu Mingmin updated BEAM-2255:
-
Fix Version/s: (was: 2.2.0)
   Not applicable

> support ORDER BY clause
> ---
>
> Key: BEAM-2255
> URL: https://issues.apache.org/jira/browse/BEAM-2255
> Project: Beam
>  Issue Type: New Feature
>  Components: dsl-sql
>Reporter: James Xu
>Assignee: James Xu
> Fix For: Not applicable
>
>
> Currently Beam does not have `global sort` support[1]. So we will implement a 
> limited version of `ORDER BY` using `org.apache.beam.sdk.transforms.Top` for 
> now. So it will support sql like:
> {code:sql}
> select * from t order by id desc limit 10;
> {code}
> but not support `ORDER BY` without a `limit`:
> {code:sql}
> select * from t order by id desc
> {code}
> and also not support `ORDER BY` with a `offset`:
> {code:sql}
> select * from t order by id desc limit 10, 10;
> {code}
> [1]. 
> https://lists.apache.org/thread.html/bc0e65a3bb653b8fd0db96bcd4c9da5af71a71af5a5639a472167808@1464278191@%3Cdev.beam.apache.org%3E



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Closed] (BEAM-2289) add unit tests for BeamSQLFilterFn and BeamSQLProjectFn

[Xu Mingmin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xu Mingmin closed BEAM-2289.

   Resolution: Fixed
Fix Version/s: (was: 2.2.0)
   Not applicable

> add unit tests for BeamSQLFilterFn and BeamSQLProjectFn
> ---
>
> Key: BEAM-2289
> URL: https://issues.apache.org/jira/browse/BEAM-2289
> Project: Beam
>  Issue Type: Improvement
>  Components: dsl-sql
>Reporter: Xu Mingmin
>Assignee: Xu Mingmin
>Priority: Minor
>  Labels: dsl_sql_merge
> Fix For: Not applicable
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Closed] (BEAM-2204) support TUMBLE/HOP/SESSION_START

[Xu Mingmin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xu Mingmin closed BEAM-2204.

   Resolution: Fixed
Fix Version/s: (was: 2.2.0)
   Not applicable

> support TUMBLE/HOP/SESSION_START
> 
>
> Key: BEAM-2204
> URL: https://issues.apache.org/jira/browse/BEAM-2204
> Project: Beam
>  Issue Type: Bug
>  Components: dsl-sql
>Reporter: Xu Mingmin
>Assignee: Xu Mingmin
>  Labels: dsl_sql_merge
> Fix For: Not applicable
>
>
> As described in {{CALCITE-1761}}, TUMBLE/HOP/SESSION_START/END do not resolve 
> time field correctly.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (BEAM-2195) support conditional functions & operators

[Xu Mingmin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xu Mingmin updated BEAM-2195:
-
Fix Version/s: (was: 2.2.0)
   Not applicable

> support conditional functions & operators
> -
>
> Key: BEAM-2195
> URL: https://issues.apache.org/jira/browse/BEAM-2195
> Project: Beam
>  Issue Type: New Feature
>  Components: dsl-sql
>Reporter: James Xu
>Assignee: James Xu
> Fix For: Not applicable
>
>
> https://calcite.apache.org/docs/reference.html#conditional-functions-and-operators



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Closed] (BEAM-2159) CAST operator support

[Xu Mingmin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xu Mingmin closed BEAM-2159.

   Resolution: Fixed
Fix Version/s: (was: 2.2.0)
   Not applicable

> CAST operator support
> -
>
> Key: BEAM-2159
> URL: https://issues.apache.org/jira/browse/BEAM-2159
> Project: Beam
>  Issue Type: Task
>  Components: dsl-sql
>Reporter: Xu Mingmin
>Assignee: Tarush Grover
>  Labels: dsl_sql_merge
> Fix For: Not applicable
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (BEAM-2310) support data type: DECIMAL

[Xu Mingmin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2310?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xu Mingmin updated BEAM-2310:
-
Fix Version/s: (was: 2.2.0)
   Not applicable

> support data type: DECIMAL
> --
>
> Key: BEAM-2310
> URL: https://issues.apache.org/jira/browse/BEAM-2310
> Project: Beam
>  Issue Type: Bug
>  Components: dsl-sql
>Reporter: James Xu
>Assignee: James Xu
> Fix For: Not applicable
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Reopened] (BEAM-2435) test SQL DSL methods

[Xu Mingmin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xu Mingmin reopened BEAM-2435:
--

> test SQL DSL methods
> 
>
> Key: BEAM-2435
> URL: https://issues.apache.org/jira/browse/BEAM-2435
> Project: Beam
>  Issue Type: Test
>  Components: dsl-sql
>Reporter: Xu Mingmin
>Assignee: Xu Mingmin
>  Labels: dsl_sql_merge
> Fix For: Not applicable
>
>
> Test supported SQL operators in BeamRelNode, using DSL methods. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (BEAM-2309) support relation operator: Values

[Xu Mingmin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2309?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xu Mingmin updated BEAM-2309:
-
Fix Version/s: (was: 2.2.0)
   Not applicable

> support relation operator: Values
> -
>
> Key: BEAM-2309
> URL: https://issues.apache.org/jira/browse/BEAM-2309
> Project: Beam
>  Issue Type: Bug
>  Components: dsl-sql
>Reporter: James Xu
>Assignee: James Xu
> Fix For: Not applicable
>
>
> Values operator corresponds to SQL like:
> {code:sql}
> select LOCALTIME
> {code}
> which does not query from any table.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (BEAM-2384) CEIL, FLOOR, TRUNCATE, PI, ATAN2 math function

[Xu Mingmin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2384?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xu Mingmin updated BEAM-2384:
-
Fix Version/s: (was: 2.2.0)
   Not applicable

> CEIL, FLOOR, TRUNCATE, PI, ATAN2 math function
> --
>
> Key: BEAM-2384
> URL: https://issues.apache.org/jira/browse/BEAM-2384
> Project: Beam
>  Issue Type: Sub-task
>  Components: dsl-sql
>Reporter: Tarush Grover
>Assignee: Tarush Grover
>  Labels: dsl_sql_merge
> Fix For: Not applicable
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (BEAM-2355) support comparison operator

[Xu Mingmin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xu Mingmin updated BEAM-2355:
-
Fix Version/s: (was: 2.2.0)
   Not applicable

> support comparison operator
> ---
>
> Key: BEAM-2355
> URL: https://issues.apache.org/jira/browse/BEAM-2355
> Project: Beam
>  Issue Type: New Feature
>  Components: dsl-sql
>Reporter: James Xu
>Assignee: James Xu
> Fix For: Not applicable
>
>
> https://calcite.apache.org/docs/reference.html#comparison-operators



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Reopened] (BEAM-2289) add unit tests for BeamSQLFilterFn and BeamSQLProjectFn

[Xu Mingmin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xu Mingmin reopened BEAM-2289:
--

> add unit tests for BeamSQLFilterFn and BeamSQLProjectFn
> ---
>
> Key: BEAM-2289
> URL: https://issues.apache.org/jira/browse/BEAM-2289
> Project: Beam
>  Issue Type: Improvement
>  Components: dsl-sql
>Reporter: Xu Mingmin
>Assignee: Xu Mingmin
>Priority: Minor
>  Labels: dsl_sql_merge
> Fix For: 2.2.0
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Closed] (BEAM-2435) test SQL DSL methods

[Xu Mingmin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xu Mingmin closed BEAM-2435.

   Resolution: Fixed
Fix Version/s: (was: 2.2.0)
   Not applicable

> test SQL DSL methods
> 
>
> Key: BEAM-2435
> URL: https://issues.apache.org/jira/browse/BEAM-2435
> Project: Beam
>  Issue Type: Test
>  Components: dsl-sql
>Reporter: Xu Mingmin
>Assignee: Xu Mingmin
>  Labels: dsl_sql_merge
> Fix For: Not applicable
>
>
> Test supported SQL operators in BeamRelNode, using DSL methods. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (BEAM-2442) DSL SQL: Public classes/methods should not expose/use calcite types

[Xu Mingmin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2442?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xu Mingmin updated BEAM-2442:
-
Fix Version/s: (was: 2.2.0)
   Not applicable

> DSL SQL: Public classes/methods should not expose/use calcite types
> ---
>
> Key: BEAM-2442
> URL: https://issues.apache.org/jira/browse/BEAM-2442
> Project: Beam
>  Issue Type: Improvement
>  Components: dsl-sql
>Reporter: Luke Cwik
>Assignee: James Xu
>  Labels: dsl_sql_merge
> Fix For: Not applicable
>
>
> Calcite is an internal implementation detail of how Beam SQL is operating. To 
> prevent a hard dependence on Calcite, public methods and classes should not 
> rely on consuming/producing Calcite types.
> For example, BeamSqlRecordType uses org.apache.calcite.sql.type.SqlTypeName 
> instead of using the Java SQL types 
> (https://docs.oracle.com/javase/8/docs/api/java/sql/Types.html).
> This task is to create an ApiSurfaceTest to help find, fix, and prevent 
> org.apache.calcite.* from being exposed. Example ApiSurfaceTest: 
> https://github.com/apache/beam/blob/367fcb28d544934797d25cb34d54136b2d7d6e99/sdks/java/core/src/test/java/org/apache/beam/SdkCoreApiSurfaceTest.java



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Closed] (BEAM-2520) add UDF/UDAF in BeamSql.query/simpleQuery

[Xu Mingmin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2520?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xu Mingmin closed BEAM-2520.

   Resolution: Fixed
Fix Version/s: (was: 2.2.0)
   Not applicable

> add UDF/UDAF in BeamSql.query/simpleQuery
> -
>
> Key: BEAM-2520
> URL: https://issues.apache.org/jira/browse/BEAM-2520
> Project: Beam
>  Issue Type: Improvement
>  Components: dsl-sql
>Reporter: Xu Mingmin
>Assignee: Xu Mingmin
>  Labels: dsl_sql_merge
> Fix For: Not applicable
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (BEAM-2462) LOG10 Function

[Xu Mingmin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2462?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xu Mingmin updated BEAM-2462:
-
Fix Version/s: (was: 2.2.0)
   Not applicable

> LOG10 Function
> --
>
> Key: BEAM-2462
> URL: https://issues.apache.org/jira/browse/BEAM-2462
> Project: Beam
>  Issue Type: Sub-task
>  Components: dsl-sql
>Reporter: Tarush Grover
>Assignee: Tarush Grover
>  Labels: dsl_sql_merge
> Fix For: Not applicable
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (BEAM-2440) DSL SQL: Reduce visibility to simplify backwards compatibility

[Xu Mingmin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xu Mingmin updated BEAM-2440:
-
Fix Version/s: (was: 2.2.0)
   Not applicable

> DSL SQL: Reduce visibility to simplify backwards compatibility
> --
>
> Key: BEAM-2440
> URL: https://issues.apache.org/jira/browse/BEAM-2440
> Project: Beam
>  Issue Type: Improvement
>  Components: dsl-sql
>Reporter: Luke Cwik
>Assignee: James Xu
>  Labels: dsl_sql_merge
> Fix For: Not applicable
>
>
> The package namespace should be flattened into one java package and 
> everything made package private except for the public classes like BeamSql, 
> BeamSqlCli, BeamSqlRow, BeamSqlRowCoder, ...
> This will simplify the backwards compatibility story after merging since it 
> reduces the visible surface that users can interact with.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Reopened] (BEAM-2520) add UDF/UDAF in BeamSql.query/simpleQuery

[Xu Mingmin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2520?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xu Mingmin reopened BEAM-2520:
--

> add UDF/UDAF in BeamSql.query/simpleQuery
> -
>
> Key: BEAM-2520
> URL: https://issues.apache.org/jira/browse/BEAM-2520
> Project: Beam
>  Issue Type: Improvement
>  Components: dsl-sql
>Reporter: Xu Mingmin
>Assignee: Xu Mingmin
>  Labels: dsl_sql_merge
> Fix For: Not applicable
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (BEAM-2515) BeamSql: refactor the MockedBeamSqlTable and related tests

[Xu Mingmin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2515?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xu Mingmin updated BEAM-2515:
-
Fix Version/s: (was: 2.2.0)
   Not applicable

> BeamSql: refactor the MockedBeamSqlTable and related tests
> --
>
> Key: BEAM-2515
> URL: https://issues.apache.org/jira/browse/BEAM-2515
> Project: Beam
>  Issue Type: Bug
>  Components: dsl-sql
>Reporter: James Xu
>Assignee: James Xu
> Fix For: Not applicable
>
>
> MockedBeamSqlTable is only for Bounded data sources, after another Unbounded 
> mock added, some refactor will be needed.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (BEAM-2463) EXP Function

[Xu Mingmin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2463?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xu Mingmin updated BEAM-2463:
-
Fix Version/s: (was: 2.2.0)
   Not applicable

> EXP Function
> 
>
> Key: BEAM-2463
> URL: https://issues.apache.org/jira/browse/BEAM-2463
> Project: Beam
>  Issue Type: Sub-task
>  Components: dsl-sql
>Reporter: Tarush Grover
>Assignee: Tarush Grover
>  Labels: dsl_sql_merge
> Fix For: Not applicable
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Reopened] (BEAM-2550) test JOINs with DSL methods

[Xu Mingmin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xu Mingmin reopened BEAM-2550:
--

> test JOINs with DSL methods
> ---
>
> Key: BEAM-2550
> URL: https://issues.apache.org/jira/browse/BEAM-2550
> Project: Beam
>  Issue Type: Task
>  Components: dsl-sql
>Reporter: Xu Mingmin
>Assignee: James Xu
>  Labels: dsl_sql_merge
> Fix For: Not applicable
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Reopened] (BEAM-2527) Test queries on unbounded PCollections with BeamSql dsl API

[Xu Mingmin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2527?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xu Mingmin reopened BEAM-2527:
--

> Test queries on unbounded PCollections with BeamSql dsl API
> ---
>
> Key: BEAM-2527
> URL: https://issues.apache.org/jira/browse/BEAM-2527
> Project: Beam
>  Issue Type: Improvement
>  Components: dsl-sql
>Reporter: Xu Mingmin
>Assignee: Xu Mingmin
>  Labels: dsl_sql_merge
> Fix For: Not applicable
>
>
> Beam SQL supports both BOUNDED and UNBOUNDED PCollections. 
> This task will add unit tests for UNBOUNDED PCollection. For BOUNDED 
> PCollection, see BEAM-2452.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Closed] (BEAM-2550) test JOINs with DSL methods

[Xu Mingmin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xu Mingmin closed BEAM-2550.

   Resolution: Fixed
Fix Version/s: (was: 2.2.0)
   Not applicable

> test JOINs with DSL methods
> ---
>
> Key: BEAM-2550
> URL: https://issues.apache.org/jira/browse/BEAM-2550
> Project: Beam
>  Issue Type: Task
>  Components: dsl-sql
>Reporter: Xu Mingmin
>Assignee: James Xu
>  Labels: dsl_sql_merge
> Fix For: Not applicable
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (BEAM-2559) Add integration test for the sql functions

[Xu Mingmin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2559?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xu Mingmin updated BEAM-2559:
-
Fix Version/s: (was: 2.2.0)
   Not applicable

> Add integration test for the sql functions
> --
>
> Key: BEAM-2559
> URL: https://issues.apache.org/jira/browse/BEAM-2559
> Project: Beam
>  Issue Type: Task
>  Components: dsl-sql
>Reporter: James Xu
>  Labels: dsl_sql_merge
> Fix For: Not applicable
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Closed] (BEAM-2559) Add integration test for the sql functions

[Xu Mingmin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2559?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xu Mingmin closed BEAM-2559.


> Add integration test for the sql functions
> --
>
> Key: BEAM-2559
> URL: https://issues.apache.org/jira/browse/BEAM-2559
> Project: Beam
>  Issue Type: Task
>  Components: dsl-sql
>Reporter: James Xu
>  Labels: dsl_sql_merge
> Fix For: Not applicable
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)