[jira] [Created] (FLINK-5795) Improve “UDTF" to support with parameter constructor
sunjincheng created FLINK-5795: -- Summary: Improve “UDTF" to support with parameter constructor Key: FLINK-5795 URL: https://issues.apache.org/jira/browse/FLINK-5795 Project: Flink Issue Type: Sub-task Components: Table API & SQL Reporter: sunjincheng Assignee: sunjincheng -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (FLINK-5803) Add [partitioned] processing time OVER RANGE BETWEEN UNBOUNDED PRECEDING aggregation to SQL
sunjincheng created FLINK-5803: -- Summary: Add [partitioned] processing time OVER RANGE BETWEEN UNBOUNDED PRECEDING aggregation to SQL Key: FLINK-5803 URL: https://issues.apache.org/jira/browse/FLINK-5803 Project: Flink Issue Type: Sub-task Components: Table API & SQL Reporter: sunjincheng Assignee: sunjincheng -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (FLINK-5804) Add [non-partitioned] processing time OVER RANGE BETWEEN UNBOUNDED PRECEDING aggregation to SQL
sunjincheng created FLINK-5804: -- Summary: Add [non-partitioned] processing time OVER RANGE BETWEEN UNBOUNDED PRECEDING aggregation to SQL Key: FLINK-5804 URL: https://issues.apache.org/jira/browse/FLINK-5804 Project: Flink Issue Type: Sub-task Components: Table API & SQL Reporter: sunjincheng Assignee: sunjincheng -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (FLINK-5776) Improve XXMapRunner support create instance by carrying constructor parameters
sunjincheng created FLINK-5776: -- Summary: Improve XXMapRunner support create instance by carrying constructor parameters Key: FLINK-5776 URL: https://issues.apache.org/jira/browse/FLINK-5776 Project: Flink Issue Type: Improvement Components: Table API & SQL Reporter: sunjincheng Assignee: sunjincheng At present, MapRunner FlatMapRunner only supports create non-parameter instance, but sometimes we need to carry constructor parameters to instantiate, so I would like to improve XXMapRunner support create instance by carrying constructor parameters. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (FLINK-5105) Improve ReduceState: value put into ReducingState should always be a copy
sunjincheng created FLINK-5105: -- Summary: Improve ReduceState: value put into ReducingState should always be a copy Key: FLINK-5105 URL: https://issues.apache.org/jira/browse/FLINK-5105 Project: Flink Issue Type: Bug Components: State Backends, Checkpointing Reporter: sunjincheng In case of overlapping sliding windows, multiple references are hold on the same object. If we modify value1 or value2 the results are incorrect. The value that is put into a ReducingState is always copied. That would allow to modify and emit one of the two input values (the one which is comes from the state). The FoldingState has the same problem. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (FLINK-5106) improving IncrementalAggregateReduceFunction
sunjincheng created FLINK-5106: -- Summary: improving IncrementalAggregateReduceFunction Key: FLINK-5106 URL: https://issues.apache.org/jira/browse/FLINK-5106 Project: Flink Issue Type: Improvement Components: Table API & SQL Reporter: sunjincheng Assignee: sunjincheng Please refer to FLINK-4937. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (FLINK-5386) Refacturing Window Clause
sunjincheng created FLINK-5386: -- Summary: Refacturing Window Clause Key: FLINK-5386 URL: https://issues.apache.org/jira/browse/FLINK-5386 Project: Flink Issue Type: Improvement Components: Table API & SQL Reporter: sunjincheng Assignee: sunjincheng -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (FLINK-6204) Improve Event-Time OVER ROWS BETWEEN UNBOUNDED PRECEDING aggregation to SQL
sunjincheng created FLINK-6204: -- Summary: Improve Event-Time OVER ROWS BETWEEN UNBOUNDED PRECEDING aggregation to SQL Key: FLINK-6204 URL: https://issues.apache.org/jira/browse/FLINK-6204 Project: Flink Issue Type: Bug Components: Table API & SQL Reporter: sunjincheng Assignee: sunjincheng Currently `event time OVER ROWS BETWEEN UNBOUNDED PRECEDING aggregation to SQL` implementation class: ` UnboundedEventTimeOverProcessFunction` use data size uncontrollable memory data structures`sortedTimestamps: util.LinkedList [Long] cache data timestamps and sort timestamps. IMO,It's not a good way, because in the production environment there are millions of window data pre millisecond in our application scenario.So, I want to improve it. Welcome anyone to give me feedback. What do you think? [~fhueske] and [~Yuhong_kyo] -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (FLINK-6219) Add a state backend which supports sorting
sunjincheng created FLINK-6219:
--
Summary: Add a state backend which supports sorting
Key: FLINK-6219
URL: https://issues.apache.org/jira/browse/FLINK-6219
Project: Flink
Issue Type: New Feature
Components: State Backends, Checkpointing, Table API & SQL
Reporter: sunjincheng
When we implement the OVER window of
[FLIP11|https://cwiki.apache.org/confluence/display/FLINK/FLIP-11%3A+Table+API+Stream+Aggregations]
We notice that we need a state backend which supports sorting, allows for
efficient insertion, traversal in order, and removal from the head.
For example: In event-time OVER window, we need to sort by time，If the datas as
follow:
{code}
(1L, 1, Hello)
(2L, 2, Hello)
(5L, 5, Hello)
(4L, 4, Hello)
{code}
We randomly insert the datas, just like:
{code}
put((2L, 2, Hello)),put((1L, 1, Hello)),put((5L, 5, Hello)),put((4L, 4, Hello)),
{code}
We deal with elements in time order:
{code}
process((1L, 1, Hello)),process((2L, 2, Hello)),process((4L, 4,
Hello)),process((5L, 5, Hello))
{code}
Welcome anyone to give feedback，And what do you think? [~xiaogang.shi]
[~aljoscha] [~fhueske]
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Created] (FLINK-6200) Add event time OVER RANGE BETWEEN UNBOUNDED PRECEDING aggregation to SQL
sunjincheng created FLINK-6200: -- Summary: Add event time OVER RANGE BETWEEN UNBOUNDED PRECEDING aggregation to SQL Key: FLINK-6200 URL: https://issues.apache.org/jira/browse/FLINK-6200 Project: Flink Issue Type: New Feature Components: Table API & SQL Reporter: sunjincheng The goal of this issue is to add support for OVER RANGE aggregations on event time streams to the SQL interface. Queries similar to the following should be supported: SELECT a, SUM(b) OVER (PARTITION BY c ORDER BY rowTime() RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW) AS sumB, MIN(b) OVER (PARTITION BY c ORDER BY rowTime() RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW) AS minB FROM myStream The following restrictions should initially apply: All OVER clauses in the same SELECT clause must be exactly the same. The PARTITION BY clause is optional (no partitioning results in single threaded execution). The ORDER BY clause may only have rowTime() as parameter. rowTime() is a parameterless scalar function that just indicates processing time mode. bounded PRECEDING is not supported (see FLINK-5655) FOLLOWING is not supported. The restrictions will be resolved in follow up issues. If we find that some of the restrictions are trivial to address, we can add the functionality in this issue as well. This issue includes: Design of the DataStream operator to compute OVER ROW aggregates Translation from Calcite's RelNode representation (LogicalProject with RexOver expression). -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (FLINK-6228) Integrating the OVER windows in the Table API
sunjincheng created FLINK-6228:
--
Summary: Integrating the OVER windows in the Table API
Key: FLINK-6228
URL: https://issues.apache.org/jira/browse/FLINK-6228
Project: Flink
Issue Type: Sub-task
Components: Table API & SQL
Reporter: sunjincheng
Assignee: sunjincheng
Syntax:
{code}
table
.overWindows(
(Rows|Range [ partitionBy value_expression , ... [ n ]]
(preceding
UNBOUNDED|value_specification.(rows|milli|second|minute|hour|day|month|year)|CURRENTROW)
[following
UNBOUNDED|value_specification.(rows|milli|second|minute|hour|day|month|year)|CURRENTROW]
as alias,...[n])
)
.select( [col1,...[n]], (agg(col1) OVER overWindowAlias, … [n])
{code}
Implement restrictions:
** All OVER clauses in the same SELECT clause must be exactly the same.
** The PARTITION BY clause is optional (no partitioning results in single
threaded execution).
** The ORDER BY clause is hidden in tableApi, According to time characteristic
automatic identification.
** FOLLOWING is not supported.
I will soon add a user interface design document.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Created] (FLINK-6074) Fix
sunjincheng created FLINK-6074: -- Summary: Fix Key: FLINK-6074 URL: https://issues.apache.org/jira/browse/FLINK-6074 Project: Flink Issue Type: Bug Reporter: sunjincheng -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (FLINK-6138) Improve UnboundedNonPartitionedProcessingOverProcessFunction
sunjincheng created FLINK-6138: -- Summary: Improve UnboundedNonPartitionedProcessingOverProcessFunction Key: FLINK-6138 URL: https://issues.apache.org/jira/browse/FLINK-6138 Project: Flink Issue Type: Improvement Components: Table API & SQL Reporter: sunjincheng Assignee: sunjincheng Hi,[~fhueske] Just like you suggested in FLINK-5804, I completed the FLINK-5995 bug fix, now I want Create the ListStateDescriptor with the aggregationStateType instead of a serializer. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (FLINK-6097) Guaranteed the order of the extracted field references
sunjincheng created FLINK-6097: -- Summary: Guaranteed the order of the extracted field references Key: FLINK-6097 URL: https://issues.apache.org/jira/browse/FLINK-6097 Project: Flink Issue Type: Improvement Components: Table API & SQL Reporter: sunjincheng Assignee: sunjincheng The current `ProjectionTranslator # identifyFieldReferences` method uses the` Set` temporary save field, causing the field to be out of order, although the out of order is also working, but I think the order is better than out of order. So I want to improve it,Orderly extraction field. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (FLINK-6055) Supported setting timers on a Non-Keyed Stream
sunjincheng created FLINK-6055: -- Summary: Supported setting timers on a Non-Keyed Stream Key: FLINK-6055 URL: https://issues.apache.org/jira/browse/FLINK-6055 Project: Flink Issue Type: New Feature Reporter: sunjincheng Assignee: sunjincheng After [FLINK-4460] Allow ProcessFunction on non-keyed streams, I want supported setting timers on a Non-Keyed Stream. What do you think? [~aljoscha] -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (FLINK-6259) Fix a small spelling error
sunjincheng created FLINK-6259:
--
Summary: Fix a small spelling error
Key: FLINK-6259
URL: https://issues.apache.org/jira/browse/FLINK-6259
Project: Flink
Issue Type: Bug
Components: Gelly
Reporter: sunjincheng
Assignee: sunjincheng
flink-gelly-scala/pom.xml {{har-with-dependencies}} -> {{
jar-with-dependencies}}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Created] (FLINK-6266) Remove the useless import in FlinkRelBuilder
sunjincheng created FLINK-6266: -- Summary: Remove the useless import in FlinkRelBuilder Key: FLINK-6266 URL: https://issues.apache.org/jira/browse/FLINK-6266 Project: Flink Issue Type: Improvement Components: Table API & SQL Reporter: sunjincheng Assignee: sunjincheng Remove FLINK-6037 legacy useless import. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (FLINK-6267) Remove the useless import in FlinkRelBuilder
sunjincheng created FLINK-6267: -- Summary: Remove the useless import in FlinkRelBuilder Key: FLINK-6267 URL: https://issues.apache.org/jira/browse/FLINK-6267 Project: Flink Issue Type: Improvement Components: Table API & SQL Reporter: sunjincheng Assignee: sunjincheng Remove FLINK-6037 legacy useless import. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (FLINK-6304) Clear a lot of useless import
sunjincheng created FLINK-6304:
--
Summary: Clear a lot of useless import
Key: FLINK-6304
URL: https://issues.apache.org/jira/browse/FLINK-6304
Project: Flink
Issue Type: Improvement
Components: Table API & SQL
Reporter: sunjincheng
Assignee: sunjincheng
There are some classes(as follow) have useless import, I want clear them before
release 1.3.
{code}
DataSetSlideTimeWindowAggFlatMapFunction
CommonScan
FlinkRel
StreamTableSourceScanRule
DataStreamOverAggregateRule
DataStreamAggregateRule
{code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Created] (FLINK-6018) Minor improvements about `AbstractKeyedStateBackend#getPartitionedState` method
sunjincheng created FLINK-6018:
--
Summary: Minor improvements about
`AbstractKeyedStateBackend#getPartitionedState` method
Key: FLINK-6018
URL: https://issues.apache.org/jira/browse/FLINK-6018
Project: Flink
Issue Type: Improvement
Components: DataStream API, State Backends, Checkpointing
Reporter: sunjincheng
Assignee: sunjincheng
The code snippet currently in the `AbstractKeyedStateBackend #
getPartitionedState` method, as follows:
{code}
// TODO: This is wrong, it should throw an exception that the initialization
has not properly happened
if (!stateDescriptor.isSerializerInitialized()) {
stateDescriptor.initializeSerializerUnlessSet(new
ExecutionConfig());
}
{code}
I hope this code has a little improvement to the following:
approach 1:
According to the `TODO` information we throw an exception
{code}
if (!stateDescriptor.isSerializerInitialized()) {
throw new IllegalStateException("The serializer of the
descriptor has not been initialized!");
}
{code}
approach 2:
Try to initialize and remove `if (!stateDescriptor.isSerializerInitialized())
{` logic.
{code}
stateDescriptor.initializeSerializerUnlessSet(new ExecutionConfig());
{code}
Meanwhile, If we use the approach 2, I suggest that `AbstractKeyedStateBackend`
add a `private final ExecutionConfig executionConfig` property. then we can
change the code like this:
{code}
stateDescriptor.initializeSerializerUnlessSet(executionConfig);
{code}
Are the above suggestions reasonable for you?
Welcome anybody's feedback and corrections.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Created] (FLINK-5957) Remove `getAccumulatorType` method from `AggregateFunction`
sunjincheng created FLINK-5957:
--
Summary: Remove `getAccumulatorType` method from
`AggregateFunction`
Key: FLINK-5957
URL: https://issues.apache.org/jira/browse/FLINK-5957
Project: Flink
Issue Type: Sub-task
Reporter: sunjincheng
Assignee: sunjincheng
Build-in aggregateFunction need not implement the `getAccumulatorType` method.
We can get TypeInformation by `TypeInformation.of() ` or
`TypeInformation.of(new TypeHint[AGG.type](){})`.
What do you think? [~fhueske] [~shaoxuan]
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Created] (FLINK-5995) Get a Exception when creating the ListStateDescriptor with a TypeInformation
sunjincheng created FLINK-5995:
--
Summary: Get a Exception when creating the ListStateDescriptor
with a TypeInformation
Key: FLINK-5995
URL: https://issues.apache.org/jira/browse/FLINK-5995
Project: Flink
Issue Type: Bug
Components: Core
Reporter: sunjincheng
When use OperatorState and creating the ListStateDescriptor with a
TypeInformation，I got a exception. The Exception info is：
{code}
org.apache.flink.runtime.client.JobExecutionException: Job execution failed.
at
org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1$$anonfun$applyOrElse$7.apply$mcV$sp(JobManager.scala:915)
at
org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1$$anonfun$applyOrElse$7.apply(JobManager.scala:858)
at
org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1$$anonfun$applyOrElse$7.apply(JobManager.scala:858)
at
scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24)
at
scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24)
at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:40)
at
akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:397)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at
scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at
scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at
scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
Caused by: java.lang.IllegalStateException: Serializer not yet initialized.
at
org.apache.flink.api.common.state.StateDescriptor.getSerializer(StateDescriptor.java:169)
at
org.apache.flink.api.common.state.ListStateDescriptor.getElementSerializer(ListStateDescriptor.java:93)
at
org.apache.flink.runtime.state.DefaultOperatorStateBackend.getOperatorState(DefaultOperatorStateBackend.java:110)
at
org.apache.flink.runtime.state.DefaultOperatorStateBackend.getOperatorState(DefaultOperatorStateBackend.java:91)
at
org.apache.flink.table.runtime.aggregate.UnboundedNonPartitionedProcessingOverProcessFunction.initializeState(UnboundedNonPartitionedProcessingOverProcessFunction.scala:104)
at
org.apache.flink.streaming.api.functions.util.StreamingFunctionUtils.tryRestoreFunction(StreamingFunctionUtils.java:178)
at
org.apache.flink.streaming.api.functions.util.StreamingFunctionUtils.restoreFunctionState(StreamingFunctionUtils.java:160)
at
org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.initializeState(AbstractUdfStreamOperator.java:106)
at
org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:242)
at
org.apache.flink.streaming.runtime.tasks.StreamTask.initializeOperators(StreamTask.java:681)
at
org.apache.flink.streaming.runtime.tasks.StreamTask.initializeState(StreamTask.java:669)
at
org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:251)
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:670)
at java.lang.Thread.run(Thread.java:745)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Created] (FLINK-6145) Add [non-partitioned] event time OVER ROWS BETWEEN x PRECEDING aggregation to SQL
sunjincheng created FLINK-6145:
--
Summary: Add [non-partitioned] event time OVER ROWS BETWEEN x
PRECEDING aggregation to SQL
Key: FLINK-6145
URL: https://issues.apache.org/jira/browse/FLINK-6145
Project: Flink
Issue Type: Sub-task
Components: Table API & SQL
Reporter: sunjincheng
Assignee: sunjincheng
The goal of this issue is to add support for OVER ROWS aggregations on event
time streams to the SQL interface.
Queries similar to the following should be supported:
{code}
SELECT
a,
SUM(b) OVER (ORDER BY rowTime() ROWS BETWEEN 2 PRECEDING AND CURRENT ROW) AS
sumB,
MIN(b) OVER (ORDER BY rowTime() ROWS BETWEEN 2 PRECEDING AND CURRENT ROW) AS
minB
FROM myStream
{code}
The following restrictions should initially apply:
- All OVER clauses in the same SELECT clause must be exactly the same.
- The ORDER BY clause may only have rowTime() as parameter. rowTime() is a
parameterless scalar function that just indicates event time mode.
- UNBOUNDED PRECEDING is not supported (see FLINK-5804)
- FOLLOWING is not supported.
The restrictions will be resolved in follow up issues. If we find that some of
the restrictions are trivial to address, we can add the functionality in this
issue as well.
This issue includes:
- Design of the DataStream operator to compute OVER ROW aggregates
- Translation from Calcite's RelNode representation (LogicalProject with
RexOver expression).
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Created] (FLINK-7342) Add support for Regular Expression Operators (RLIKE)
sunjincheng created FLINK-7342:
--
Summary: Add support for Regular Expression Operators (RLIKE)
Key: FLINK-7342
URL: https://issues.apache.org/jira/browse/FLINK-7342
Project: Flink
Issue Type: Bug
Components: Table API & SQL
Reporter: sunjincheng
Assignee: sunjincheng
Currently Regular Expression Operators (RLIKE) are not supported. e.g.:
{code}
select * from emps name rlike '^(K|G|W)'
{code}
we'll get Error:
{code}
Error: Error while executing SQL "select * from emps name rlike '^(K|G|W)'":
parse failed: Encountered "rlike" at line 1, column 25.
{code}
So, after https://issues.apache.org/jira/browse/CALCITE-1918 fixed. I want add
RLIKE supported in flink (TableAPI and SQL).
Reference:
MySql: https://dev.mysql.com/doc/refman/5.7/en/regexp.html#operator_regexp
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (FLINK-7358) Add implicitly converts support for User-defined function
sunjincheng created FLINK-7358:
--
Summary: Add implicitly converts support for User-defined function
Key: FLINK-7358
URL: https://issues.apache.org/jira/browse/FLINK-7358
Project: Flink
Issue Type: Improvement
Components: Table API & SQL
Reporter: sunjincheng
Assignee: sunjincheng
Currently if user defined a UDF as follows:
{code}
object Func extends ScalarFunction {
def eval(a: Int, b: Long): String = {
...
}
}
{code}
And if the table schema is (a: Int, b: int, c: String), then we can not call
the UDF `Func('a, 'b)`. So
I want add implicitly converts when we call UDF.
*Note:
In this JIRA. only for TableAPI, And SQL will be fixed in
https://issues.apache.org/jira/browse/CALCITE-1908.*
What do you think? [~fhueske]
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (FLINK-7464) Add useful build-in Aggregate function into TabalAPI
sunjincheng created FLINK-7464: -- Summary: Add useful build-in Aggregate function into TabalAPI Key: FLINK-7464 URL: https://issues.apache.org/jira/browse/FLINK-7464 Project: Flink Issue Type: New Feature Components: Table API & SQL Reporter: sunjincheng In this JIRA, will create some sub-task for add specific build-in aggregate function, such as FIRST_VALUE, LAST_VALUE, BloomFilterCount etc. Welcome anybody to add the sub-task. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (FLINK-7465) Add build-in BloomFilterCount on TableAPI
sunjincheng created FLINK-7465:
--
Summary: Add build-in BloomFilterCount on TableAPI
Key: FLINK-7465
URL: https://issues.apache.org/jira/browse/FLINK-7465
Project: Flink
Issue Type: Sub-task
Components: Table API & SQL
Reporter: sunjincheng
Assignee: sunjincheng
In this JIRA. use BloomFilter to implement counting functions.
BloomFilter Algorithm description:
An empty Bloom filter is a bit array of m bits, all set to 0. There must also
be k different hash functions defined, each of which maps or hashes some set
element to one of the m array positions, generating a uniform random
distribution. Typically, k is a constant, much smaller than m, which is
proportional to the number of elements to be added; the precise choice of k and
the constant of proportionality of m are determined by the intended false
positive rate of the filter.
To add an element, feed it to each of the k hash functions to get k array
positions. Set the bits at all these positions to 1.
To query for an element (test whether it is in the set), feed it to each of the
k hash functions to get k array positions. If any of the bits at these
positions is 0, the element is definitely not in the set – if it were, then all
the bits would have been set to 1 when it was inserted. If all are 1, then
either the element is in the set, or the bits have by chance been set to 1
during the insertion of other elements, resulting in a false positive.
An example of a Bloom filter, representing the set {x, y, z}. The colored
arrows show the positions in the bit array that each set element is mapped to.
The element w is not in the set {x, y, z}, because it hashes to one bit-array
position containing 0. For this figure, m = 18 and k = 3. The sketch as follows:
!https://en.wikipedia.org/wiki/Bloom_filter#/media/File:Bloom_filter.svg!
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (FLINK-7205) Add UUID supported in TableAPI/SQL
sunjincheng created FLINK-7205: -- Summary: Add UUID supported in TableAPI/SQL Key: FLINK-7205 URL: https://issues.apache.org/jira/browse/FLINK-7205 Project: Flink Issue Type: Sub-task Affects Versions: 1.4.0 Reporter: sunjincheng UUID() returns a value that conforms to UUID version 1 as described in RFC 4122. The value is a 128-bit number represented as a utf8 string of five hexadecimal numbers in ---- format: The first three numbers are generated from the low, middle, and high parts of a timestamp. The high part also includes the UUID version number. The fourth number preserves temporal uniqueness in case the timestamp value loses monotonicity (for example, due to daylight saving time). The fifth number is an IEEE 802 node number that provides spatial uniqueness. A random number is substituted if the latter is not available (for example, because the host device has no Ethernet card, or it is unknown how to find the hardware address of an interface on the host operating system). In this case, spatial uniqueness cannot be guaranteed. Nevertheless, a collision should have very low probability. See: [RFC 4122: http://www.ietf.org/rfc/rfc4122.txt|http://www.ietf.org/rfc/rfc4122.txt] See detailed semantics: MySql: [https://dev.mysql.com/doc/refman/5.7/en/miscellaneous-functions.html#function_uuid|https://dev.mysql.com/doc/refman/5.7/en/miscellaneous-functions.html#function_uuid] Welcome anybody feedback -:). -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (FLINK-7101) Fix Non-windowed group-aggregate error when using `minIdleStateRetentionTime` config and retract agg
sunjincheng created FLINK-7101:
--
Summary: Fix Non-windowed group-aggregate error when using
`minIdleStateRetentionTime` config and retract agg
Key: FLINK-7101
URL: https://issues.apache.org/jira/browse/FLINK-7101
Project: Flink
Issue Type: Bug
Components: Table API & SQL
Affects Versions: 1.3.1, 1.3.0
Reporter: sunjincheng
Assignee: sunjincheng
Fix For: 1.4.0
When Non-windowed group-aggregate using {{minIdleStateRetentionTime}} config
and retract AGG, Will emit "NULL" agg value which we do not expect.
For example: ({{IntSumWithRetractAggFunction}})
1. Receive: CRow(Row.of(6L: JLong, 5: JInt, "aaa"), true)
2. Cleanup state
3. Receive: CRow(Row.of(6L: JLong, 5: JInt, "aaa"), false) // acc.f1 = -1,
getValue= null
So, we must change the logic of {{GroupAggProcessFunction}} as follows:
{code}
if (inputCnt != 0) {
...
} else {
...
}
{code}
TO
{code}
if (inputCnt > 0) {
...
} else {
if( null != prevRow.row){
...
}
}
{code}
What do you think? [~fhueske] [~hequn8128]
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (FLINK-6392) Change the alias of Window from optional to essential.
sunjincheng created FLINK-6392:
--
Summary: Change the alias of Window from optional to essential.
Key: FLINK-6392
URL: https://issues.apache.org/jira/browse/FLINK-6392
Project: Flink
Issue Type: Improvement
Components: Table API & SQL
Affects Versions: 1.3.0
Reporter: sunjincheng
Assignee: sunjincheng
Fix For: 1.3.0
Currently, The window clause use case looks like:
{code}
tab //Table('a,'b,'c)
.window( Slide over 10.milli every 5.milli as 'w)
.groupBy('w,'a,'b) // WindowGroupedTable
.select('a, 'b, 'c.sum, 'w.start, 'w.end)
{code}
As we see the alias of window is essential. But the current implementation of
the TableAPI does not have the constraint for the alias,So we must refactoring
the API definition using TYPE SYSTEM lead to constraint for the alias.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Created] (FLINK-6406) Cleanup useless import
sunjincheng created FLINK-6406: -- Summary: Cleanup useless import Key: FLINK-6406 URL: https://issues.apache.org/jira/browse/FLINK-6406 Project: Flink Issue Type: Improvement Components: Table API & SQL Affects Versions: 1.3.0 Reporter: sunjincheng Assignee: sunjincheng When browsing the code, it is found that there are some useless reference in the following file which need cleanup. *packages.scala *ExternalCatalogTable *arithmetic.scala *array.scala *ColumnStats -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (FLINK-7471) Improve bounded OVER support non-retract method AGG
sunjincheng created FLINK-7471:
--
Summary: Improve bounded OVER support non-retract method AGG
Key: FLINK-7471
URL: https://issues.apache.org/jira/browse/FLINK-7471
Project: Flink
Issue Type: Improvement
Components: Table API & SQL
Reporter: sunjincheng
Assignee: sunjincheng
Currently BOUNDED OVER WINDOW only support have {{retract}} method AGG. In this
JIRA. will add non-retract method support.
What do you think? [~fhueske]
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (FLINK-6583) Enable QueryConfig in count base GroupWindow
sunjincheng created FLINK-6583: -- Summary: Enable QueryConfig in count base GroupWindow Key: FLINK-6583 URL: https://issues.apache.org/jira/browse/FLINK-6583 Project: Flink Issue Type: New Feature Components: Table API & SQL Affects Versions: 1.3.0, 1.4.0 Reporter: sunjincheng Assignee: sunjincheng Enable QueryConfig in count base GroupWindow by Add a custom Trigger `CountTriggerWithCleanupState`. See more in FLINK-6491. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (FLINK-6618) Fix `GroupWindow` JAVA logical plans not consistent with SCALA logical plans.
sunjincheng created FLINK-6618:
--
Summary: Fix `GroupWindow` JAVA logical plans not consistent with
SCALA logical plans.
Key: FLINK-6618
URL: https://issues.apache.org/jira/browse/FLINK-6618
Project: Flink
Issue Type: Bug
Components: Table API & SQL
Affects Versions: 1.3.0
Reporter: sunjincheng
Assignee: sunjincheng
I find 2 bugs as follows:
1. `GroupWindowStringExpressionTest` testcase bug,
`Assert.assertEquals("Logical Plans do not match", resJava.logicalPlan,
resJava.logicalPlan)` -> `Assert.assertEquals("Logical Plans do not match",
resJava.logicalPlan, resScala.logicalPlan)`
2. When i fix the bug above, we got anther bug:
{code}
java.lang.AssertionError: Logical Plans do not match
Expected :Project(ListBuffer('string, 'TMP_4, 'TMP_5, 'TMP_6, ('TMP_7 * 2) as
'_c4),WindowAggregate(List('string),SlidingGroupWindow('w, 'rowtime,
1440.millis, 720.millis),List(),List(CountAggFunction(List('string)) as
'TMP_4, sum('int) as 'TMP_5, WeightedAvg(List('long, 'int)) as 'TMP_6,
WeightedAvg(List('int, 'int)) as 'TMP_7),Project(ArrayBuffer('string, 'int,
'long,
'rowtime),CatalogNode(WrappedArray(_DataStreamTable_0),RecordType(INTEGER int,
BIGINT long, VARCHAR(2147483647) string, TIMESTAMP(3) rowtime)
Actual :Project(ListBuffer('string, 'TMP_0, 'TMP_1, 'TMP_2, ('TMP_3 * 2) as
'_c4),WindowAggregate(ArrayBuffer('string),SlidingGroupWindow('w, 'rowtime,
1440.millis,
720.millis),List(),List(CountAggFunction(ArrayBuffer('string)) as 'TMP_0,
sum('int) as 'TMP_1, WeightedAvg(ArrayBuffer('long, 'int)) as 'TMP_2,
WeightedAvg(ArrayBuffer('int, 'int)) as 'TMP_3),Project(ArrayBuffer('string,
'int, 'long,
'rowtime),CatalogNode(WrappedArray(_DataStreamTable_0),RecordType(INTEGER int,
BIGINT long, VARCHAR(2147483647) string, TIMESTAMP(3) rowtime)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Created] (FLINK-6617) Improve JAVA and SCALA logical plans consistent test
sunjincheng created FLINK-6617: -- Summary: Improve JAVA and SCALA logical plans consistent test Key: FLINK-6617 URL: https://issues.apache.org/jira/browse/FLINK-6617 Project: Flink Issue Type: Test Components: Table API & SQL Affects Versions: 1.3.0 Reporter: sunjincheng Assignee: sunjincheng Currently,we need some `StringExpression` test,for all JAVA and SCALA API. Such as:`GroupAggregations`,`GroupWindowAggregaton`(Session,Tumble),`Calc` etc. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (FLINK-6619) Check Table API & SQL support for 1.3.0 RC01 Release
sunjincheng created FLINK-6619: -- Summary: Check Table API & SQL support for 1.3.0 RC01 Release Key: FLINK-6619 URL: https://issues.apache.org/jira/browse/FLINK-6619 Project: Flink Issue Type: Test Components: Table API & SQL Affects Versions: 1.3.0 Reporter: sunjincheng Assignee: sunjincheng In this JIRA. I will do the following tasks for Flink 1.3.0 RC01 Release. * Check that the JAVA and SCALA logical plans are consistent. * Check that the SQL and Table API logical plans are consistent. * Check that UDF, UDTF, and UDAF are working properly in group-windows and over-windows. * Check that all built-in Agg on Batch and Stream are working properly. When I do the task above, I'll created some sub-task. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (FLINK-6598) Fix `DataStreamGroupAggregateRule` matches error.
sunjincheng created FLINK-6598: -- Summary: Fix `DataStreamGroupAggregateRule` matches error. Key: FLINK-6598 URL: https://issues.apache.org/jira/browse/FLINK-6598 Project: Flink Issue Type: Bug Components: Table API & SQL Reporter: sunjincheng Assignee: sunjincheng Fix For: 1.3.0 Change retrun from `!distinctAggs && !groupSets && !agg.indicator` to `!distinctAggs && !groupSets && !agg.indicator && !agg.getAggCallList.isEmpty`. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (FLINK-6597) Cleanup WindowOperatorTest useless imports
sunjincheng created FLINK-6597: -- Summary: Cleanup WindowOperatorTest useless imports Key: FLINK-6597 URL: https://issues.apache.org/jira/browse/FLINK-6597 Project: Flink Issue Type: Test Reporter: sunjincheng Assignee: sunjincheng Fix For: 1.3.0 Don't miss doing any good thing no matter how insignificant it looks. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (FLINK-6959) Add documentation for SQL OVER windows
sunjincheng created FLINK-6959: -- Summary: Add documentation for SQL OVER windows Key: FLINK-6959 URL: https://issues.apache.org/jira/browse/FLINK-6959 Project: Flink Issue Type: Improvement Components: Documentation, Table API & SQL Affects Versions: 1.4.0 Reporter: sunjincheng Assignee: sunjincheng Add documentation for SQL OVER windows. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (FLINK-6960) Add E(2.7182818284590452354),PI(3.14159265358979323846) supported in SQL
sunjincheng created FLINK-6960: -- Summary: Add E(2.7182818284590452354),PI(3.14159265358979323846) supported in SQL Key: FLINK-6960 URL: https://issues.apache.org/jira/browse/FLINK-6960 Project: Flink Issue Type: Sub-task Components: Table API & SQL Reporter: sunjincheng Assignee: sunjincheng E=Math.E PI=Math.PI -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (FLINK-6973) Add L/RPAD supported in TableAPI
sunjincheng created FLINK-6973: -- Summary: Add L/RPAD supported in TableAPI Key: FLINK-6973 URL: https://issues.apache.org/jira/browse/FLINK-6973 Project: Flink Issue Type: Sub-task Components: Table API & SQL Affects Versions: 1.4.0 Reporter: sunjincheng See FLINK-6892 for detail. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (FLINK-6974) Add BIN supported in TableAPI
sunjincheng created FLINK-6974: -- Summary: Add BIN supported in TableAPI Key: FLINK-6974 URL: https://issues.apache.org/jira/browse/FLINK-6974 Project: Flink Issue Type: Sub-task Components: Table API & SQL Affects Versions: 1.4.0 Reporter: sunjincheng See FLINK-6893 for detail. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (FLINK-6975) Add CONCAT/CONCAT_WS supported in TableAPI
sunjincheng created FLINK-6975: -- Summary: Add CONCAT/CONCAT_WS supported in TableAPI Key: FLINK-6975 URL: https://issues.apache.org/jira/browse/FLINK-6975 Project: Flink Issue Type: Sub-task Components: Table API & SQL Affects Versions: 1.4.0 Reporter: sunjincheng See FLINK-6925 for detail. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (FLINK-6978) Add documentation for Register User-Defined Functions.
sunjincheng created FLINK-6978: -- Summary: Add documentation for Register User-Defined Functions. Key: FLINK-6978 URL: https://issues.apache.org/jira/browse/FLINK-6978 Project: Flink Issue Type: Improvement Components: Documentation, Table API & SQL Affects Versions: 1.4.0 Reporter: sunjincheng The User-defined Functions documentation is currently lacking a description of Register User-Defined Functions. The page has a placeholder section with a TODO: ./docs/dev/table/udfs.md. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (FLINK-6977) Add MD5/SHA1/SHA2 supported in TableAPI
sunjincheng created FLINK-6977: -- Summary: Add MD5/SHA1/SHA2 supported in TableAPI Key: FLINK-6977 URL: https://issues.apache.org/jira/browse/FLINK-6977 Project: Flink Issue Type: Sub-task Components: Table API & SQL Affects Versions: 1.4.0 Reporter: sunjincheng See FLINK-6895 for detail. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (FLINK-7020) Upgrade calcite to calcite 1.13
sunjincheng created FLINK-7020: -- Summary: Upgrade calcite to calcite 1.13 Key: FLINK-7020 URL: https://issues.apache.org/jira/browse/FLINK-7020 Project: Flink Issue Type: Improvement Components: Table API & SQL Affects Versions: 1.4.0 Reporter: sunjincheng Assignee: sunjincheng Calcite 1.13 has released, So I'll try to upgrade the dependence. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (FLINK-6650) Fix Non-windowed group-aggregate error when using append-table mode.
sunjincheng created FLINK-6650:
--
Summary: Fix Non-windowed group-aggregate error when using
append-table mode.
Key: FLINK-6650
URL: https://issues.apache.org/jira/browse/FLINK-6650
Project: Flink
Issue Type: Sub-task
Reporter: sunjincheng
Assignee: sunjincheng
When I test Non-windowed group-aggregate with {{stream.toTable(tEnv, 'a, 'b,
'c).select('a.sum, weightAvgFun('a, 'b)).toAppendStream[Row].addSink(new
StreamITCase.StringSink)}}, I got the error as follows:
{code}
org.apache.flink.table.api.TableException: Table is not an append-only table.
Output needs to handle update and delete changes.
at
org.apache.flink.table.api.StreamTableEnvironment.translate(StreamTableEnvironment.scala:631)
at
org.apache.flink.table.api.StreamTableEnvironment.translate(StreamTableEnvironment.scala:607)
at
org.apache.flink.table.api.scala.StreamTableEnvironment.toAppendStream(StreamTableEnvironment.scala:219)
at
org.apache.flink.table.api.scala.StreamTableEnvironment.toAppendStream(StreamTableEnvironment.scala:195)
at
org.apache.flink.table.api.scala.TableConversions.toAppendStream(TableConversions.scala:121)
{code}
The reason is {{DataStreamGroupAggregate#producesUpdates}} as follows:
{code}
override def producesUpdates = true
{code}
I think in the view of the user, what user want are(for example):
Data:
{code}
val data = List(
(1L, 1, "Hello"),
(2L, 2, "Hello"),
(3L, 3, "Hello"),
(4L, 4, "Hello"),
(5L, 5, "Hello"),
(6L, 6, "Hello"),
(7L, 7, "Hello World"),
(8L, 8, "Hello World"),
(20L, 20, "Hello World"))
{code}
*Case1:
TableAPI
{code}
stream.toTable(tEnv, 'a, 'b, 'c).select('a.sum).toRetractStream[Row]
.addSink(new StreamITCase.RetractingSink)
{code}
Result
{code}
1
3
6
10
15
21
28
36
56
{code}
* Case 2:
TableAPI
{code}
stream.toTable(tEnv, 'a, 'b, 'c).select('a.sum).toRetractStream[Row]
.addSink(new StreamITCase.RetractingSink)
{code}
Result:
{code}
56
{code}
In fact about #Case 1,we can using unbounded OVER windows, as follows:
TableAPI
{code}
stream.toTable(tEnv, 'a, 'b, 'c, 'proctime.proctime)
.window(Over orderBy 'proctime preceding UNBOUNDED_ROW as 'w)
.select('a.sum over 'w)
.toAppendStream[Row].addSink(new StreamITCase.StringSink)
{code}
Result
{code}
Same as #Case1
{code}
But after the [FLINK-6649 | https://issues.apache.org/jira/browse/FLINK-6649]
OVER can not express the #Case1 with earlyFiring.
So I still think Non-windowed group-aggregate not always update-table, user can
decide which mode to use.
Is there any drawback to this improvement? Welcome anyone feedback?
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Created] (FLINK-6649) Improve Non-window group aggregate with configurable `earlyFire`.
sunjincheng created FLINK-6649: -- Summary: Improve Non-window group aggregate with configurable `earlyFire`. Key: FLINK-6649 URL: https://issues.apache.org/jira/browse/FLINK-6649 Project: Flink Issue Type: Improvement Components: Table API & SQL Affects Versions: 1.4.0 Reporter: sunjincheng Assignee: sunjincheng Currently, Non-windowed group aggregate is earlyFiring at count(1), that is every row will emit a aggregate result. But some times user want config count number (`early firing with count[N]`) , to reduce the downstream pressure. This JIRA. will enable the config of e`arlyFiring` for Non-windowed group aggregate. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (FLINK-6686) Improve UDXF(UDF,UDTF,UDAF) test case
sunjincheng created FLINK-6686: -- Summary: Improve UDXF(UDF,UDTF,UDAF) test case Key: FLINK-6686 URL: https://issues.apache.org/jira/browse/FLINK-6686 Project: Flink Issue Type: Sub-task Components: Table API & SQL Affects Versions: 1.3.0 Reporter: sunjincheng Assignee: sunjincheng 1. Add Check that UDF, UDTF, and UDAF are working properly in group-windows and over-windows. 2. Add Check that all built-in Agg on Batch and Stream are working properly. Let types such as Timestamp, BigDecimal or Pojo flow through UDF. UDTF, UDAF (input and output types) -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (FLINK-6736) Fix UDTF codegen bug when window follow by join( UDTF)
sunjincheng created FLINK-6736:
--
Summary: Fix UDTF codegen bug when window follow by join( UDTF)
Key: FLINK-6736
URL: https://issues.apache.org/jira/browse/FLINK-6736
Project: Flink
Issue Type: Sub-task
Components: Table API & SQL
Affects Versions: 1.3.0
Reporter: sunjincheng
Assignee: sunjincheng
When we run the tableAPI as follows:
{code}
val table = stream.toTable(tEnv, 'long.rowtime, 'int, 'double, 'float, 'bigdec,
'date,'pojo, 'string)
val windowedTable = table
.join(udtf2('string) as ('a, 'b))
.window(Slide over 5.milli every 2.milli on 'long as 'w)
.groupBy('w)
.select('int.count, agg1('pojo, 'bigdec, 'date, 'int), 'w.start, 'w.end)
{code}
We will get the error message:
{code}
org.apache.flink.runtime.client.JobExecutionException: Job execution failed.
at
org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1$$anonfun$applyOrElse$7.apply$mcV$sp(JobManager.scala:933)
at
org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1$$anonfun$applyOrElse$7.apply(JobManager.scala:876)
at
org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1$$anonfun$applyOrElse$7.apply(JobManager.scala:876)
at
scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24)
at
scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24)
at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:40)
at
akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:397)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at
scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at
scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at
scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
Caused by: org.apache.flink.api.common.InvalidProgramException: Table program
cannot be compiled. This is a bug. Please file an issue.
at
org.apache.flink.table.codegen.Compiler$class.compile(Compiler.scala:36)
at
org.apache.flink.table.runtime.CRowCorrelateProcessRunner.compile(CRowCorrelateProcessRunner.scala:35)
at
org.apache.flink.table.runtime.CRowCorrelateProcessRunner.open(CRowCorrelateProcessRunner.scala:59)
at
org.apache.flink.api.common.functions.util.FunctionUtils.openFunction(FunctionUtils.java:36)
at
org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.open(AbstractUdfStreamOperator.java:111)
at
org.apache.flink.streaming.api.operators.ProcessOperator.open(ProcessOperator.java:56)
at
org.apache.flink.streaming.runtime.tasks.StreamTask.openAllOperators(StreamTask.java:377)
at
org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:254)
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:702)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.codehaus.commons.compiler.CompileException: Line 77, Column 62:
Unknown variable or type "in2"
at
org.codehaus.janino.UnitCompiler.compileError(UnitCompiler.java:11523)
at org.codehaus.janino.UnitCompiler.getType2(UnitCompiler.java:6292)
at org.codehaus.janino.UnitCompiler.access$12900(UnitCompiler.java:209)
at
org.codehaus.janino.UnitCompiler$18.visitPackage(UnitCompiler.java:5904)
at
org.codehaus.janino.UnitCompiler$18.visitPackage(UnitCompiler.java:5901)
at org.codehaus.janino.Java$Package.accept(Java.java:4074)
at org.codehaus.janino.UnitCompiler.getType(UnitCompiler.java:5901)
at org.codehaus.janino.UnitCompiler.getType2(UnitCompiler.java:6287)
at org.codehaus.janino.UnitCompiler.access$13500(UnitCompiler.java:209)
{code}
The reason is {{val generator = new CodeGenerator(config, false,
inputSchema.physicalTypeInfo)}} `physicalTypeInfo` will remove the
TimeIndicator.
I think we should fix this. What do you think [~fhueske] [~twalthr] , And hope
your suggestions. :)
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Created] (FLINK-6740) Fix "parameterTypeEquals" method error.
sunjincheng created FLINK-6740:
--
Summary: Fix "parameterTypeEquals" method error.
Key: FLINK-6740
URL: https://issues.apache.org/jira/browse/FLINK-6740
Project: Flink
Issue Type: Sub-task
Components: Table API & SQL
Affects Versions: 1.3.0
Reporter: sunjincheng
Assignee: sunjincheng
When we define UDTF as follows:
{code}
class TableFuncPojo extends TableFunction[TPojo] {
def eval(age: Int, name:String): Unit = {
collect(new TPojo(age.toLong,name))
}
def eval(age: Date, name:String): Unit = {
collect(new TPojo(age.getTime,name))
}
}
{code}
TableAPI:
{code}
val table = stream.toTable(tEnv,
'long2, 'int, 'double, 'float, 'bigdec, 'ts, 'date,'pojo, 'string,
'long.rowtime)
val windowedTable = table
.join(udtf('date, 'string) as 'pojo2).select('pojo2)
{code}
We will get the error as following:
{code}
org.apache.flink.table.api.ValidationException: Found multiple 'eval' methods
which match the signature.
at
org.apache.flink.table.functions.utils.UserDefinedFunctionUtils$.getUserDefinedMethod(UserDefinedFunctionUtils.scala:180)
at
org.apache.flink.table.plan.logical.LogicalTableFunctionCall.validate(operators.scala:700)
at org.apache.flink.table.api.Table.join(table.scala:539)
at org.apache.flink.table.api.Table.join(table.scala:328)
at
org.apache.flink.table.runtime.datastream.DataStreamAggregateITCase.test1(DataStreamAggregateITCase.scala:84)
{code}
The reason is in {{ parameterTypeEquals }} method, logical as follows:
{code}
candidate == classOf[Date] && (expected == classOf[Int] || expected ==
classOf[JInt])
{code}
But when we only define one `eval` method, we got different result, as follows:
{code}
// for def eval(age: Int, name:String)
Pojo{id=0, name='Hello'}
Pojo{id=1, name='Hallo'}
Pojo{id=1, name='Hello world'}
Pojo{id=1, name='Hello world'}
Pojo{id=1, name='Hello'}
Pojo{id=1, name='Hi'}
Pojo{id=8, name='Hello'}
// for def eval(age: Date, name:String)
Pojo{id=-2880, name='Hello'}
Pojo{id=5760, name='Hallo'}
Pojo{id=5760, name='Hello world'}
Pojo{id=5760, name='Hello world'}
Pojo{id=5760, name='Hello'}
Pojo{id=5760, name='Hi'}
Pojo{id=66240, name='Hello'}
{code}
So, We should modify the logical of {{ parameterTypeEquals }} method.
What do you think? Welcome anybody feedback...
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Created] (FLINK-6632) Fix parameter case sensitive error for test passing/rejecting filter API
sunjincheng created FLINK-6632:
--
Summary: Fix parameter case sensitive error for test
passing/rejecting filter API
Key: FLINK-6632
URL: https://issues.apache.org/jira/browse/FLINK-6632
Project: Flink
Issue Type: Sub-task
Reporter: sunjincheng
Assignee: sunjincheng
TableAPI {{testAllPassingFilter}}:
{code}
val t = util.addTable[(Int, Long, String)]('int, 'long, 'string)
val resScala = t.filter(Literal(true)).select('int as 'myInt, 'string)
val resJava = t.filter("TrUe").select("int as myInt, string")
{code}
We got error:
{code}
org.apache.flink.table.api.ValidationException: Cannot resolve [TrUe] given
input [int, long, string].
{code}
The error is caused by :
{code}
lazy val boolLiteral: PackratParser[Expression] = ("true" | "false") ^^ {
str => Literal(str.toBoolean)
}
{code}
I want import the method as follow:
{code}
lazy val boolLiteral: PackratParser[Expression] =
("(t|T)(r|R)(u|U)(e|E)".r | "(f|F)(a|A)(l|L)(s|S)(e|E)".r) ^^ { str =>
Literal(str.toBoolean)}
{code}
Is there any drawback to this improvement? Welcome anyone feedback ?
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Created] (FLINK-6760) Fix OverWindowTest alias test error
sunjincheng created FLINK-6760:
--
Summary: Fix OverWindowTest alias test error
Key: FLINK-6760
URL: https://issues.apache.org/jira/browse/FLINK-6760
Project: Flink
Issue Type: Sub-task
Components: Table API & SQL
Affects Versions: 1.3.0
Reporter: sunjincheng
Assignee: sunjincheng
For Sql:
{code}
val sql = "SELECT c, count(a) OVER (ORDER BY proctime ROWS BETWEEN 2 preceding
AND CURRENT ROW) as cnt1 CURRENT ROW from MyTable"
{code}
The alias `cnt1` The alias did not take effect when we generated the plan
string. But we can using the alias in outer layer query, for example:
{code}
val sql = "SELECT cnt1 from (SELECT c, count(a) OVER (ORDER BY proctime ROWS
BETWEEN 2 preceding AND CURRENT ROW) as cnt1 CURRENT ROW from MyTable)"
{code}
So in this JIRA. we just fix the test case for 1.3 release. In another JIRA.
will improve the alias.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Created] (FLINK-6813) Add DATEDIFF as build-in scalar function
sunjincheng created FLINK-6813: -- Summary: Add DATEDIFF as build-in scalar function Key: FLINK-6813 URL: https://issues.apache.org/jira/browse/FLINK-6813 Project: Flink Issue Type: Sub-task Components: Table API & SQL Affects Versions: 1.4.0 Reporter: sunjincheng Assignee: sunjincheng * Syntax DATEDIFF ( datepart , startdate , enddate ) -datepart Is the part of startdate and enddate that specifies the type of boundary crossed. -startdate Is an expression that can be resolved to a time, date. -enddate Same with startdate. * Example SELECT DATEDIFF(year, '2015-12-31 23:59:59.999', '2017-01-01 00:00:00.000') from tab; --> 2 -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (FLINK-6811) Add DATEADD/DATESUB/DATEDIFF as build-in scalar function
sunjincheng created FLINK-6811: -- Summary: Add DATEADD/DATESUB/DATEDIFF as build-in scalar function Key: FLINK-6811 URL: https://issues.apache.org/jira/browse/FLINK-6811 Project: Flink Issue Type: Sub-task Reporter: sunjincheng -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (FLINK-6839) Improve SQL OVER alias When only one OVER window agg in selection.
sunjincheng created FLINK-6839:
--
Summary: Improve SQL OVER alias When only one OVER window agg in
selection.
Key: FLINK-6839
URL: https://issues.apache.org/jira/browse/FLINK-6839
Project: Flink
Issue Type: Improvement
Reporter: sunjincheng
For OVER SQL:
{code}
SELECT a COUNT(c) OVER (ORDER BY proctime RANGE BETWEEN INTERVAL '10' SECOND
PRECEDING AND CURRENT ROW) as cnt1 FROM MyTable
{code}
We expect plan {{DataStreamCalc(select=[a, w0$o0 AS cnt1]) But we get
{{DataStreamCalc(select=[a, w0$o0 AS $1]) }}. this improve only for plan check.
the functional is work well in nested queries，e.g.:
{code}
SELECT cnt1 from (SELECT a COUNT(c) OVER (ORDER BY proctime RANGE BETWEEN
INTERVAL '10' SECOND PRECEDING AND CURRENT ROW) as cnt1 FROM MyTable)
{code}
The SQL above is work well. which mentioned in
[FLINK-6760|https://issues.apache.org/jira/browse/FLINK-6760].
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Created] (FLINK-6837) Fix a small error message bug, And improve some message info.
sunjincheng created FLINK-6837: -- Summary: Fix a small error message bug, And improve some message info. Key: FLINK-6837 URL: https://issues.apache.org/jira/browse/FLINK-6837 Project: Flink Issue Type: Bug Components: Table API & SQL Affects Versions: 1.4.0 Reporter: sunjincheng Assignee: sunjincheng Fix a variable reference error, and improve some error message info. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (FLINK-6841) using TableSourceTable for both Stream and Batch OR remove useless import
sunjincheng created FLINK-6841:
--
Summary: using TableSourceTable for both Stream and Batch OR
remove useless import
Key: FLINK-6841
URL: https://issues.apache.org/jira/browse/FLINK-6841
Project: Flink
Issue Type: Improvement
Components: Table API & SQL
Affects Versions: 1.4.0
Reporter: sunjincheng
Assignee: sunjincheng
1. {{StreamTableSourceTable}} exist useless import of {{TableException}}
2. {{StreamTableSourceTable}} only override {{getRowType}} of {{FlinkTable}},
I think we can override the method in {{TableSourceTable}}, If so we can using
{{TableSourceTable}} for both {{Stream}} and {{Batch}}.
What do you think? [~fhueske] [~twalthr]
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Created] (FLINK-6847) Add TIMESTAMPDIFF supported in TableAPI
sunjincheng created FLINK-6847: -- Summary: Add TIMESTAMPDIFF supported in TableAPI Key: FLINK-6847 URL: https://issues.apache.org/jira/browse/FLINK-6847 Project: Flink Issue Type: Sub-task Components: Table API & SQL Affects Versions: 1.4.0 Reporter: sunjincheng Assignee: sunjincheng see FLINK-6813 -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (FLINK-6845) Cleanup "println(StreamITCase.testResults)" call in test case
sunjincheng created FLINK-6845: -- Summary: Cleanup "println(StreamITCase.testResults)" call in test case Key: FLINK-6845 URL: https://issues.apache.org/jira/browse/FLINK-6845 Project: Flink Issue Type: Test Components: Table API & SQL Affects Versions: 1.4.0 Reporter: sunjincheng Assignee: sunjincheng Cleanup "println(StreamITCase.testResults)" call in test case. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (FLINK-6924) ADD LOG/LPAD/RPAD/BIN supported in TableAPI
sunjincheng created FLINK-6924: -- Summary: ADD LOG/LPAD/RPAD/BIN supported in TableAPI Key: FLINK-6924 URL: https://issues.apache.org/jira/browse/FLINK-6924 Project: Flink Issue Type: Sub-task Components: Table API & SQL Affects Versions: 1.4.0 Reporter: sunjincheng See FLINK-6891/ FLINK-6892/ FLINK-6893 for detail. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (FLINK-6925) Add CONCAT/CONCAT_WS supported in SQL
sunjincheng created FLINK-6925:
--
Summary: Add CONCAT/CONCAT_WS supported in SQL
Key: FLINK-6925
URL: https://issues.apache.org/jira/browse/FLINK-6925
Project: Flink
Issue Type: Sub-task
Components: Table API & SQL
Affects Versions: 1.4.0
Reporter: sunjincheng
CONCAT(str1,str2,...)Returns the string that results from concatenating the
arguments. May have one or more arguments. If all arguments are nonbinary
strings, the result is a nonbinary string. If the arguments include any binary
strings, the result is a binary string. A numeric argument is converted to its
equivalent nonbinary string form.
CONCAT() returns NULL if any argument is NULL.
* Syntax:
CONCAT(str1,str2,...)
* Arguments
** str1,str2,... -
* Return Types
string
* Example:
CONCAT('F', 'lin', 'k') -> 'Flink'
CONCAT('M', NULL, 'L') -> NULL
CONCAT(14.3) -> '14.3'
* See more:
** [MySQL|
https://dev.mysql.com/doc/refman/5.7/en/string-functions.html#function_concat]
CONCAT_WS() stands for Concatenate With Separator and is a special form of
CONCAT(). The first argument is the separator for the rest of the arguments.
The separator is added between the strings to be concatenated. The separator
can be a string, as can the rest of the arguments. If the separator is NULL,
the result is NULL.
* Syntax:
CONCAT_WS(separator,str1,str2,...)
* Arguments
** separator -
** str1,str2,... -
* Return Types
string
* Example:
CONCAT_WS(',','First name','Second name','Last Name') -> 'First name,Second
name,Last Name'
* See more:
** [MySQL|
https://dev.mysql.com/doc/refman/5.7/en/string-functions.html#function_concat-ws]
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (FLINK-6891) Add LOG supported in SQL
sunjincheng created FLINK-6891: -- Summary: Add LOG supported in SQL Key: FLINK-6891 URL: https://issues.apache.org/jira/browse/FLINK-6891 Project: Flink Issue Type: Sub-task Components: Table API & SQL Affects Versions: 1.4.0 Reporter: sunjincheng LONG (N) A single parameter version of the function returns the natural logarithm of N, and if two arguments are called, it returns any radix of the logarithm of N. * Syntax: LOG ( float_expression [, base ] ) * Arguments **float_expression: Is an expression of type float or of a type that can be implicitly converted to float. **base: Optional integer argument that sets the base for the logarithm. * Return Types float * Example: LOG(10) -> 2.30 *See more: **[MSQL|https://docs.microsoft.com/en-us/sql/t-sql/functions/log-transact-sql] **[MySQL| https://dev.mysql.com/doc/refman/5.7/en/mathematical-functions.html#function_log] -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (FLINK-6892) Add LPAD supported in SQL
sunjincheng created FLINK-6892:
--
Summary: Add LPAD supported in SQL
Key: FLINK-6892
URL: https://issues.apache.org/jira/browse/FLINK-6892
Project: Flink
Issue Type: Sub-task
Components: Table API & SQL
Affects Versions: 1.4.0
Reporter: sunjincheng
LPAD(str,len,padstr) Returns the string str, left-padded with the string padstr
to a length of len characters. If str is longer than len, the return value is
shortened to len characters.
* Syntax:
LPAD(str,len,padstr)
* Arguments
**str: -
**len: -
**padstr: -
* Return Types
String
* Example:
LPAD('hi',4,'??') -> '??hi'
LPAD('hi',1,'??') -> 'h'
* See more:
** [MySQL|
https://dev.mysql.com/doc/refman/5.7/en/mathematical-functions.html#function_lpad]
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (FLINK-6893) Add BIN supported in SQL
sunjincheng created FLINK-6893: -- Summary: Add BIN supported in SQL Key: FLINK-6893 URL: https://issues.apache.org/jira/browse/FLINK-6893 Project: Flink Issue Type: Sub-task Components: Table API & SQL Affects Versions: 1.4.0 Reporter: sunjincheng BIN(N) Returns a string representation of the binary value of N, where N is a longlong (BIGINT) number. This is equivalent to CONV(N,10,2). Returns NULL if N is NULL. * Syntax: BIN(num) * Arguments **num: a long/bigint value * Return Types String * Example: BIN(12) -> '1100' * See more: ** [MySQL| https://dev.mysql.com/doc/refman/5.7/en/string-functions.html#function_bin] -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (FLINK-6895) Add STR_TO_DATE supported in SQL
sunjincheng created FLINK-6895:
--
Summary: Add STR_TO_DATE supported in SQL
Key: FLINK-6895
URL: https://issues.apache.org/jira/browse/FLINK-6895
Project: Flink
Issue Type: Sub-task
Components: Table API & SQL
Affects Versions: 1.4.0
Reporter: sunjincheng
STR_TO_DATE(str,format) This is the inverse of the DATE_FORMAT() function. It
takes a string str and a format string format. STR_TO_DATE() returns a DATETIME
value if the format string contains both date and time parts, or a DATE or TIME
value if the string contains only date or time parts. If the date, time, or
datetime value extracted from str is illegal, STR_TO_DATE() returns NULL and
produces a warning.
* Syntax:
STR_TO_DATE(str,format)
* Arguments
**str: -
**format: -
* Return Types
DATAETIME/DATE/TIME
* Example:
STR_TO_DATE('01,5,2013','%d,%m,%Y') -> '2013-05-01'
SELECT STR_TO_DATE('a09:30:17','a%h:%i:%s') -> '09:30:17'
* See more:
** [MySQL|
https://dev.mysql.com/doc/refman/5.7/en/date-and-time-functions.html#function_str-to-date]
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (FLINK-6894) Add DATE_FORMAT supported in SQL
sunjincheng created FLINK-6894:
--
Summary: Add DATE_FORMAT supported in SQL
Key: FLINK-6894
URL: https://issues.apache.org/jira/browse/FLINK-6894
Project: Flink
Issue Type: Sub-task
Components: Table API & SQL
Affects Versions: 1.4.0
Reporter: sunjincheng
DATE_FORMAT(date,format) Formats the date value according to the format string.
* Syntax:
DATE_FORMAT(date,format)
* Arguments
**date: -
**format: -
* Return Types
String
* Example:
DATE_FORMAT('2009-10-04 22:23:00', '%W %M %Y') -> 'Sunday October 2009'
DATE_FORMAT('1900-10-04 22:23:00','%D %y %a %d %m %b %j') -> '4th 00 Thu 04
10 Oct 277'
* See more:
** [MySQL|
https://dev.mysql.com/doc/refman/5.7/en/date-and-time-functions.html#function_date-format]
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (FLINK-6941) Selecting window start / end on over window causes field not resolve exception
sunjincheng created FLINK-6941:
--
Summary: Selecting window start / end on over window causes field
not resolve exception
Key: FLINK-6941
URL: https://issues.apache.org/jira/browse/FLINK-6941
Project: Flink
Issue Type: Bug
Components: Table API & SQL
Affects Versions: 1.4.0
Reporter: sunjincheng
Assignee: sunjincheng
Selecting window start / end on over window causes field not resolve exception.
The following program:
{code}
table
.window(
Over partitionBy 'c orderBy 'proctime preceding UNBOUNDED_ROW as 'w)
.select('c, countFun('b) over 'w, 'w.start, 'w.end)
{code}
causes
{code}
org.apache.flink.table.api.ValidationException: Cannot resolve [w] given input
[a, b, c, proctime].
at
org.apache.flink.table.plan.logical.LogicalNode.failValidation(LogicalNode.scala:143)
at
org.apache.flink.table.plan.logical.LogicalNode$$anonfun$validate$1.applyOrElse(LogicalNode.scala:86)
at
org.apache.flink.table.plan.logical.LogicalNode$$anonfun$validate$1.applyOrElse(LogicalNode.scala:83)
at
org.apache.flink.table.plan.TreeNode.postOrderTransform(TreeNode.scala:72)
at
org.apache.flink.table.plan.TreeNode$$anonfun$1.apply(TreeNode.scala:46)
at scala.collection.Iterator$$anon$11.next(Iterator.scala:328)
at scala.collection.Iterator$class.foreach(Iterator.scala:727)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
at
scala.collection.generic.Growable$class.$plus$plus$eq(Growable.scala:48)
{code}
We should validate that the start and end window properties are not accessed on
over windows.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (FLINK-6942) Add a document for the FLINK-6810 related scalar functions
sunjincheng created FLINK-6942: -- Summary: Add a document for the FLINK-6810 related scalar functions Key: FLINK-6942 URL: https://issues.apache.org/jira/browse/FLINK-6942 Project: Flink Issue Type: Sub-task Components: Table API & SQL Affects Versions: 1.4.0 Reporter: sunjincheng Add a document for the FLINK-6810 related scalar functions -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (FLINK-6886) Fix Timestamp field can not be selected in event time case when toDataStream[T], `T` not a `Row` Type.
sunjincheng created FLINK-6886: -- Summary: Fix Timestamp field can not be selected in event time case when toDataStream[T], `T` not a `Row` Type. Key: FLINK-6886 URL: https://issues.apache.org/jira/browse/FLINK-6886 Project: Flink Issue Type: Bug Components: Table API & SQL Affects Versions: 1.4.0 Reporter: sunjincheng Assignee: sunjincheng Currently for event-time window(group/over), When contain `Timestamp` type field in `SELECT Clause`, And toDataStream[T], `T` not a `Row` Type, Such `PojoType`, will throw a exception. In this JIRA. will fix this bug. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (FLINK-6479) Hot fix FLINK-5906 bug
sunjincheng created FLINK-6479:
--
Summary: Hot fix FLINK-5906 bug
Key: FLINK-6479
URL: https://issues.apache.org/jira/browse/FLINK-6479
Project: Flink
Issue Type: Bug
Components: Table API & SQL
Affects Versions: 1.3.0
Reporter: sunjincheng
Assignee: sunjincheng
{code}
val windowedTable = table
.window(Slide over 2.rows every 1.rows on 'proctime as 'w)
.groupBy('w, 'string)
.select(weightAvgFun('long, 'int))
{code}
will got exception:
{code}
java.lang.IndexOutOfBoundsException: index (2) must be less than size (2)
at
com.google.common.base.Preconditions.checkElementIndex(Preconditions.java:310)
at
com.google.common.base.Preconditions.checkElementIndex(Preconditions.java:292)
at
com.google.common.collect.RegularImmutableList.get(RegularImmutableList.java:65)
at org.apache.calcite.util.Pair$6.get(Pair.java:347)
at
scala.collection.convert.Wrappers$JListWrapper.apply(Wrappers.scala:88)
at
org.apache.flink.table.plan.nodes.CommonAggregate$$anonfun$groupingToString$1.apply(CommonAggregate.scala:34)
at
org.apache.flink.table.plan.nodes.CommonAggregate$$anonfun$groupingToString$1.apply(CommonAggregate.scala:34)
at
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
at
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
at
scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
at scala.collection.mutable.ArrayOps$ofInt.foreach(ArrayOps.scala:156)
at scala.collection.TraversableLike$class.map(TraversableLike.scala:244)
at scala.collection.mutable.ArrayOps$ofInt.map(ArrayOps.scala:156)
at
org.apache.flink.table.plan.nodes.CommonAggregate$class.groupingToString(CommonAggregate.scala:34)
at
org.apache.flink.table.plan.nodes.datastream.DataStreamGroupWindowAggregate.groupingToString(DataStreamGroupWindowAggregate.scala:44)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Created] (FLINK-6426) Correct the method parameter type error in the group-window table API document
sunjincheng created FLINK-6426: -- Summary: Correct the method parameter type error in the group-window table API document Key: FLINK-6426 URL: https://issues.apache.org/jira/browse/FLINK-6426 Project: Flink Issue Type: Sub-task Components: Documentation, Table API & SQL Affects Versions: 1.3.0 Reporter: sunjincheng Assignee: sunjincheng Correct the document from ` .window([w: Window] as 'w)` to ` .window([w: WindowWithoutAlias] as 'w)` -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (FLINK-6462) Add requires over indicators for UDAF
sunjincheng created FLINK-6462: -- Summary: Add requires over indicators for UDAF Key: FLINK-6462 URL: https://issues.apache.org/jira/browse/FLINK-6462 Project: Flink Issue Type: Improvement Components: Table API & SQL Reporter: sunjincheng Assignee: sunjincheng The window function in the standard database only supports the "over" window, such as `LAG`,` LEAD`, `FIRST_VALUE`,` LAST_VALUE`. These window functions do not apply to `Slide`,` Tumble`, `Session` window. So when the user-defined AGG need to be clearly defined. In calcite `SqlAggFunction` will using `requiresOver` to distinguish. In this JIRA. will deal with this feature. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (FLINK-6851) Improve built-in scalar function when upgrade calcite version to 1.13
sunjincheng created FLINK-6851:
--
Summary: Improve built-in scalar function when upgrade calcite
version to 1.13
Key: FLINK-6851
URL: https://issues.apache.org/jira/browse/FLINK-6851
Project: Flink
Issue Type: Improvement
Components: Table API & SQL
Reporter: sunjincheng
Assignee: sunjincheng
I found that the difference between calcite 1.12 and calcite master affects the
semantics of the build-in scalar function, for example:
The SQL:
{code}
select
timestampadd(MONTH, 1, date '2016-05-31'),
timestampadd(MONTH, 5, date '2016-01-31'),
timestampadd(MONTH, -1, date '2016-03-31') from depts;
{code}
Calcite 1.12 Result:
{code}
++++
| EXPR$0 | EXPR$1 | EXPR$2 |
++++
| 2016-07-01 | 2016-07-01 | 2016-03-01 |
++++
3 rows selected (1.06 seconds)
{code}
Calcite Master result:
{code}
++++
| EXPR$0 | EXPR$1 | EXPR$2 |
++++
| 2016-06-30 | 2016-06-30 | 2016-02-29 |
++++
3 rows selected (1.047 seconds)
{code}
The MSSQL result:
{code}
select
dateadd(MONTH, 1, '2016-05-31'),
dateadd(MONTH, 5, '2016-01-31'),
dateadd(MONTH, -1,'2016-03-31') from pvt;
2016-06-30 00:00:00 2016-06-30 00:00:00 2016-02-29 00:00:00
{code}
So, when we flink upgrade the calcite version to 1.13. we can open some useful
feature. e.g: {{TIMESTAMPADD(SqlTypeFamily.ANY, SqlTypeFamily.INTEGER,
SqlTypeFamily.DATE)}}.
So, In this JIRA. will create some subtask to improve flink according to
calcite 1.13.
I appreciated if you can give me some feedback?
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Created] (FLINK-7025) Using NullByteKeySelector for Unbounded ProcTime NonPartitioned Over
sunjincheng created FLINK-7025: -- Summary: Using NullByteKeySelector for Unbounded ProcTime NonPartitioned Over Key: FLINK-7025 URL: https://issues.apache.org/jira/browse/FLINK-7025 Project: Flink Issue Type: Bug Reporter: sunjincheng Assignee: sunjincheng Currently we added `Cleanup State` feature. But It not work well if we enabled the stateCleaning on Unbounded ProcTime NonPartitioned Over window, Because in `ProcessFunctionWithCleanupState` we has using the keyed state. So, In this JIRA. I'll change the `Unbounded ProcTime NonPartitioned Over` to `partitioned Over` by using NullByteKeySelector. OR created a `NonKeyedProcessFunctionWithCleanupState`. But I think the first way is simpler. What do you think? [~fhueske] -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (FLINK-7024) Add supported for selecting window proctime/rowtime on row-based Tumble/Slide window
sunjincheng created FLINK-7024:
--
Summary: Add supported for selecting window proctime/rowtime on
row-based Tumble/Slide window
Key: FLINK-7024
URL: https://issues.apache.org/jira/browse/FLINK-7024
Project: Flink
Issue Type: Bug
Components: Table API & SQL
Affects Versions: 1.4.0
Reporter: sunjincheng
Assignee: sunjincheng
We get validate exception,when selecting window.proctime/rowtime on row-based
group window.
{code}
table
.window(Tumble over 2.rows on 'proctime as 'w)
.groupBy('w, 'string)
.select('string, countFun('string) as 'cnt, 'w.rowtime as 'proctime)
.window(Over partitionBy 'string orderBy 'proctime preceding
UNBOUNDED_RANGE following CURRENT_RANGE as 'w2)
.select('string, 'cnt.sum over 'w2 as 'cnt)
{code}
Exception:
{code}
org.apache.flink.table.api.ValidationException: Window start and Window end
cannot be selected for a row-count Tumbling window.
at
org.apache.flink.table.plan.logical.LogicalNode.failValidation(LogicalNode.scala:143)
at
org.apache.flink.table.plan.logical.WindowAggregate.validate(operators.scala:660)
{code}
We should add window.proctime/rowtime check in `validate ` method.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (FLINK-7618) Add BINARY supported in FlinkTypeFactory
sunjincheng created FLINK-7618:
--
Summary: Add BINARY supported in FlinkTypeFactory
Key: FLINK-7618
URL: https://issues.apache.org/jira/browse/FLINK-7618
Project: Flink
Issue Type: Bug
Components: Table API & SQL
Reporter: sunjincheng
Assignee: sunjincheng
We will get the following exception when we deal with the BINARY type.
{code}
org.apache.flink.table.api.TableException: Type is not supported: BINARY
at org.apache.flink.table.api.TableException$.apply(exceptions.scala:53)
at
org.apache.flink.table.calcite.FlinkTypeFactory$.toTypeInfo(FlinkTypeFactory.scala:377)
at
org.apache.flink.table.codegen.CodeGenerator.visitCall(CodeGenerator.scala:741)
at
org.apache.flink.table.codegen.CodeGenerator.visitCall(CodeGenerator.scala:67)
at org.apache.calcite.rex.RexCall.accept(RexCall.java:104)
at
org.apache.flink.table.codegen.CodeGenerator$$anonfun$16.apply(CodeGenerator.scala:754)
at
org.apache.flink.table.codegen.CodeGenerator$$anonfun$16.apply(CodeGenerator.scala:744)
{code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (FLINK-7558) Improve SQL ValidationException message.
sunjincheng created FLINK-7558: -- Summary: Improve SQL ValidationException message. Key: FLINK-7558 URL: https://issues.apache.org/jira/browse/FLINK-7558 Project: Flink Issue Type: Improvement Components: Table API & SQL Reporter: sunjincheng Assignee: sunjincheng org.apache.flink.table.api.ValidationException: SQL validation failed. Operand types of could not be inferred. at org.apache.flink.table.calcite.FlinkPlannerImpl.validate(FlinkPlannerImpl.scala:91) at org.apache.flink.table.api.TableEnvironment.sql(TableEnvironment.scala:513) at com.alibaba.blink.scala.tool.util.SqlJobAdapter.dealInserts(SqlJobAdapter.java:292) at com.alibaba.blink.scala.tool.util.JobBuildHelper.buildSqlJob(JobBuildHelper.java:80) at com.alibaba.blink.scala.tool.JobLauncher.main(JobLauncher.java:138) Caused by: org.apache.flink.table.api.ValidationException: Operand types of could not be inferred. at org.apache.flink.table.functions.utils.ScalarSqlFunction$$anon$2$$anonfun$2.apply(ScalarSqlFunction.scala:110) at org.apache.flink.table.functions.utils.ScalarSqlFunction$$anon$2$$anonfun$2.apply(ScalarSqlFunction.scala:110) at scala.Option.getOrElse(Option.scala:121) at org.apache.flink.table.functions.utils.ScalarSqlFunction$$anon$2.inferOperandTypes(ScalarSqlFunction.scala:110) at org.apache.calcite.sql.validate.SqlValidatorImpl.inferUnknownTypes(SqlValidatorImpl.java:1769) at -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (FLINK-7485) Using DataView interface to improve (MIN/MAX)WithRetractAggFunction.
sunjincheng created FLINK-7485:
--
Summary: Using DataView interface to improve
(MIN/MAX)WithRetractAggFunction.
Key: FLINK-7485
URL: https://issues.apache.org/jira/browse/FLINK-7485
Project: Flink
Issue Type: Improvement
Reporter: sunjincheng
Currently MIN/MAX using memory structure {{HashMap}} to store all values,
after FLINK-7206 we can improve them by using {{DataView}}.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (FLINK-7572) Improve TableSchema ValidationException message.
sunjincheng created FLINK-7572:
--
Summary: Improve TableSchema ValidationException message.
Key: FLINK-7572
URL: https://issues.apache.org/jira/browse/FLINK-7572
Project: Flink
Issue Type: Improvement
Reporter: sunjincheng
Assignee: sunjincheng
When sql is very complicated(e.g.: more than 100+ tables), the following
exception information is very confusing:
{code}
org.apache.flink.table.api.TableException: Table column names must be unique.
at org.apache.flink.table.api.TableSchema.(TableSchema.scala:36)
{code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (FLINK-8323) Fix Mod scala function bug
sunjincheng created FLINK-8323: -- Summary: Fix Mod scala function bug Key: FLINK-8323 URL: https://issues.apache.org/jira/browse/FLINK-8323 Project: Flink Issue Type: Sub-task Components: Table API & SQL Affects Versions: 1.5.0 Reporter: sunjincheng Assignee: sunjincheng As we know mod(151435632,6)=0, but currently we get `-15488` when call `MOD(151435632,6)`. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (FLINK-8355) DataSet Should not union a NULL row for AGG without GROUP BY clause.
sunjincheng created FLINK-8355:
--
Summary: DataSet Should not union a NULL row for AGG without GROUP
BY clause.
Key: FLINK-8355
URL: https://issues.apache.org/jira/browse/FLINK-8355
Project: Flink
Issue Type: Bug
Components: Table API & SQL
Affects Versions: 1.5.0
Reporter: sunjincheng
Currently {{DataSetAggregateWithNullValuesRule}} will UINON a NULL row for non
grouped aggregate query. when {{CountAggFunction}} support
{{COUNT(*)}}(FLINK-8325). the result will incorrect.
for example, if Tabble {{T1}} has 3 records. when we run the follow SQL in
DataSet:
{code}
SELECT COUNT(*) as cnt from Tab // cnt = 4(incorrect).
{code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (FLINK-8325) Add support for constant parameters in GROUP WINDOW aggregate
sunjincheng created FLINK-8325:
--
Summary: Add support for constant parameters in GROUP WINDOW
aggregate
Key: FLINK-8325
URL: https://issues.apache.org/jira/browse/FLINK-8325
Project: Flink
Issue Type: Bug
Components: Table API & SQL
Affects Versions: 1.5.0
Reporter: sunjincheng
Assignee: sunjincheng
COUNT(1) with Group Window, always output 0.
e.g.
DATA:
{code}
val data = List(
(1L, 1, "Hi"),
(2L, 2, "Hello"),
(4L, 2, "Hello"),
(8L, 3, "Hello world"),
(16L, 3, "Hello world"))
{code}
SQL:
{code}
SELECT b, COUNT(1) FROM MyTable GROUP BY Hop(proctime, interval '0.001' SECOND,
interval '0.002' SECOND),b
{code}
OUTPUT:
{code}
1,0,1,
1,0,1,
2,0,1,
2,0,1,
2,0,2,
3,0,1,
3,0,1
{code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (FLINK-10639) Fix java syntax error in document
sunjincheng created FLINK-10639: --- Summary: Fix java syntax error in document Key: FLINK-10639 URL: https://issues.apache.org/jira/browse/FLINK-10639 Project: Flink Issue Type: Bug Components: Documentation, Table API SQL Affects Versions: 1.6.1, 1.7.0 Reporter: sunjincheng Assignee: Hequn Cheng Fix For: 1.7.0, 1.6.1 Attachments: image-2018-10-22-16-54-10-305.png Due to the [StreamTableSourceFactory|https://github.com/apache/flink/blob/master/flink-libraries/flink-table/src/main/scala/org/apache/flink/table/factories/StreamTableSourceFactory.scala] is a trait. So the java example in the document should using "implements" keyword. !image-2018-10-22-16-54-10-305.png! -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (FLINK-10676) Add 'as' method for OverWindowWithOrderBy in Java API
sunjincheng created FLINK-10676:
---
Summary: Add 'as' method for OverWindowWithOrderBy in Java API
Key: FLINK-10676
URL: https://issues.apache.org/jira/browse/FLINK-10676
Project: Flink
Issue Type: Improvement
Components: Table API SQL
Affects Versions: 1.7.0
Reporter: sunjincheng
Fix For: 1.7.0
The preceding clause of OVER Window in the traditional database is optional.
The default is UNBOUNDED. So we can add the "as" method to
OverWindowWithOrderBy. This way OVERWindow is written more easily. e.g.:
{code:java}
.window(Over partitionBy 'c orderBy 'proctime preceding UNBOUNDED_ROW as
'w){code}
Can be simplified as follows:
{code:java}
.window(Over partitionBy 'c orderBy 'proctime as 'w){code}
What do you think?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (FLINK-10819) The instability problem of CI, JobManagerHAProcessFailureRecoveryITCase.testDispatcherProcessFailure test fail.
sunjincheng created FLINK-10819: --- Summary: The instability problem of CI, JobManagerHAProcessFailureRecoveryITCase.testDispatcherProcessFailure test fail. Key: FLINK-10819 URL: https://issues.apache.org/jira/browse/FLINK-10819 Project: Flink Issue Type: Test Components: Tests Reporter: sunjincheng Fix For: 1.7.1 Found the following error in the process of CI： Results : Tests in error: JobManagerHAProcessFailureRecoveryITCase.testDispatcherProcessFailure:331 » IllegalArgument Tests run: 1463, Failures: 0, Errors: 1, Skipped: 29 18:40:55.828 [INFO] 18:40:55.829 [INFO] BUILD FAILURE 18:40:55.829 [INFO] 18:40:55.830 [INFO] Total time: 30:19 min 18:40:55.830 [INFO] Finished at: 2018-11-07T18:40:55+00:00 18:40:56.294 [INFO] Final Memory: 92M/678M 18:40:56.294 [INFO] 18:40:56.294 [WARNING] The requested profile "include-kinesis" could not be activated because it does not exist. 18:40:56.295 [ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.18.1:test (integration-tests) on project flink-tests_2.11: There are test failures. 18:40:56.295 [ERROR] 18:40:56.295 [ERROR] Please refer to /home/travis/build/sunjincheng121/flink/flink-tests/target/surefire-reports for the individual test results. 18:40:56.295 [ERROR] -> [Help 1] 18:40:56.295 [ERROR] 18:40:56.295 [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. 18:40:56.295 [ERROR] Re-run Maven using the -X switch to enable full debug logging. 18:40:56.295 [ERROR] 18:40:56.295 [ERROR] For more information about the errors and possible solutions, please read the following articles: 18:40:56.295 [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException MVN exited with EXIT CODE: 1. Trying to KILL watchdog (11329). ./tools/travis_mvn_watchdog.sh: line 269: 11329 Terminated watchdog PRODUCED build artifacts. But after the rerun, the error disappeared. Currently，no specific reasons are found, and will continue to pay attention. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (FLINK-10834) TableAPI flatten calculated value error
sunjincheng created FLINK-10834:
---
Summary: TableAPI flatten calculated value error
Key: FLINK-10834
URL: https://issues.apache.org/jira/browse/FLINK-10834
Project: Flink
Issue Type: Bug
Components: Table API SQL
Reporter: sunjincheng
Fix For: 1.7.1
We have a UDF as follows:
object FuncRow extends ScalarFunction {
def eval(v: Int): Row = {
val version = "" + new Random().nextInt()
val row = new Row(3)
row.setField(0, version)
row.setField(1, version)
row.setField(2, version)
row
}
override def isDeterministic: Boolean = false
override def getResultType(signature: Array[Class[_]]): TypeInformation[_] =
Types.ROW(Types.STRING, Types.STRING, Types.STRING)
}
...
val data = new mutable.MutableList[(Int, Long, String)]
data.+=((1, 1L, "Hi"))
val ds = env.fromCollection(data).toTable(tEnv, 'a, 'b,'c)
.select(FuncRow('a).flatten()).as('v1, 'v2, 'v3)
...
The result is : -1189206469,-151367792,1988676906
The result expected by the user should be: v1==v2==v3 .
It looks the real reason is that there is no result of the reuse in codegen.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (FLINK-10875) Add `toTableWithTimestamp` method in `DataStreamConversions`
sunjincheng created FLINK-10875:
---
Summary: Add `toTableWithTimestamp` method in
`DataStreamConversions`
Key: FLINK-10875
URL: https://issues.apache.org/jira/browse/FLINK-10875
Project: Flink
Issue Type: Improvement
Components: Table API SQL
Reporter: sunjincheng
Assignee: sunjincheng
Fix For: 1.7.1
Currently we convert a `DataStream` to a `Table` by
`DataStreamConversions#toTable`, e.g.:
{code:java}
// Without TimeAttribute
...
val stream = env.fromCollection(...)
val tab = stream.toTable(tEnv, 'a, 'b, 'c)
val result = tab.select('a, 'b)
// With TimeAttribute
...
val stream = env.fromCollection(...).assignTimestampsAndWatermarks(...)
val tab = stream.toTable(tEnv, 'a, 'b, 'c, 'ts.rowtime)
val result = tab.window(Session withGap 5.milli on 'ts as 'w)
...{code}
I think the fieldNames parameter in the `toTable` method is reasonable in the
conversion without the time attribute, because the fieldNames will actually
correspond to the fields of the physical table, but when applied to the
conversion with the time attribute, the time attribute column is silently added
to the table. This feeling is very Magical, so I recommend adding a method that
allows the user to display the time attribute added to the physical table:
`toTableWithTimestamp`, which is automatically named to the time attribute
column named by user input and TimeCharacteristic, eg:
{code:java}
env.setStreamTimeCharacteristic(TimeCharacteristic.EventTime)
...
val table = stream.toTableWithTimestamp(tEnv, 'count, 'size, 'name, 'ts)
.window(Tumble over 2.rows on 'ts as 'w)
...
{code}
In the example above the flink will mark `ts` ad a `RowtimeAttribute`.
What do you think ?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (FLINK-11242) The predicate `'o_c_id === Null(Types.STRING)` not work in TableAPI
sunjincheng created FLINK-11242:
---
Summary: The predicate `'o_c_id === Null(Types.STRING)` not work
in TableAPI
Key: FLINK-11242
URL: https://issues.apache.org/jira/browse/FLINK-11242
Project: Flink
Issue Type: Bug
Components: Table API SQL
Affects Versions: 1.8.0
Reporter: sunjincheng
Schema:
{code:java}
Table(c_id:String, o_c_id:String){code}
Data:
{code:java}
c_001,c_001
c_002,c_002
c_003,null{code}
TableAPI：
{code:java}
val result = tab
.select('c_id, 'o_c_id)
.where('o_c_id === Null(Types.STRING)){code}
Result:
We expect `c_003, null`, but got empty.
BTW, we can got the correct result by follows TableAPI:
{code:java}
val result = tab
.select('c_id, 'o_c_id)
.where('o_c_id isNull){code}
I think is better for we let `.where('o_c_id === Null(Types.STRING))` work
well. What do you think?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (FLINK-11243) GroupBy udf() can not be select in TableAPI
sunjincheng created FLINK-11243:
---
Summary: GroupBy udf() can not be select in TableAPI
Key: FLINK-11243
URL: https://issues.apache.org/jira/browse/FLINK-11243
Project: Flink
Issue Type: Bug
Components: Table API SQL
Affects Versions: 1.8.0
Reporter: sunjincheng
The group key can not be select in the follows TableAPI:
{code:java}
val result = order
.groupBy('o_time.substring(1, 16))
.select('o_time.substring(1, 16), 'o_id.count){code}
Exception:
{code:java}
org.apache.flink.table.api.ValidationException: Cannot resolve field [o_time]
given input [('o_time).substring(1, 16), TMP_0].
at
org.apache.flink.table.plan.logical.LogicalNode.failValidation(LogicalNode.scala:156){code}
BTW, We can only use the following description to avoid the above problem.
{code:java}
val result = order
.select('o_id, 'c_id, 'o_time.substring(1, 16) as 'key)
.groupBy('key)
.select('key, 'o_id.count)
{code}
But I think it is better to make `groupby udf()` can be select in TableAPI.
What do you think?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (FLINK-11246) Fix distinct AGG visibility issues
sunjincheng created FLINK-11246: --- Summary: Fix distinct AGG visibility issues Key: FLINK-11246 URL: https://issues.apache.org/jira/browse/FLINK-11246 Project: Flink Issue Type: Bug Components: Table API SQL Reporter: sunjincheng 1. DistinctAggregateFunction# distinct -> remove private[flink] 2. case class DistinctAgg#distinct -> remove private[flink] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (FLINK-11220) Can not Select row time field in JOIN query
sunjincheng created FLINK-11220:
---
Summary: Can not Select row time field in JOIN query
Key: FLINK-11220
URL: https://issues.apache.org/jira/browse/FLINK-11220
Project: Flink
Issue Type: Bug
Components: Table API SQL
Affects Versions: 1.8.0
Reporter: sunjincheng
SQL:
{code:java}
Orders...toTable(tEnv, 'orderId, 'orderTime.rowtime)
Payment...toTable(tEnv, 'orderId, 'payTime.rowtime)
SELECT orderTime, o.orderId, payTime
FROM Orders AS o JOIN Payment AS p
ON o.orderId = p.orderId AND
p.payTime BETWEEN orderTime AND orderTime + INTERVAL '1' HOUR
{code}
Execption:
{code:java}
org.apache.flink.table.api.TableException: Found more than one rowtime field:
[orderTime, payTime] in the table that should be converted to a DataStream.
Please select the rowtime field that should be used as event-time timestamp for
the DataStream by casting all other fields to TIMESTAMP.
at
org.apache.flink.table.api.StreamTableEnvironment.translate(StreamTableEnvironment.scala:906)
{code}
The reason for the error is that we have 2 time fields `orderTime` and
`payTime`. I think we do not need throw the exception, and we can remove the
logic of `plan.process(new OutputRowtimeProcessFunction[A](conversion,
rowtimeFields.head.getIndex))`, if we want using the timestamp after
toDataSteram, we should using `assignTimestampsAndWatermarks()`.
What do you think ? [~twalthr] [~fhueske]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (FLINK-11223) Time zone problem of row time field
sunjincheng created FLINK-11223:
---
Summary: Time zone problem of row time field
Key: FLINK-11223
URL: https://issues.apache.org/jira/browse/FLINK-11223
Project: Flink
Issue Type: Bug
Components: Table API SQL
Affects Versions: 1.7.2, 1.8.0
Reporter: sunjincheng
SQL:
{code:java}
Data.+=(("001", "flink", new Timestamp(1545803501000L)))
tab...toTable(tEnv, 'a, 'b, 'rowtime2, 'rowtime.rowtime)
val sqlQuery = "SELECT rowtime2, rowtime FROM tab"{code}
Result:
{code:java}
2018-12-26 13:51:41.0, 2018-12-26 05:51:41.0
{code}
My local time zone is UTC/GMT+08.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (FLINK-11123) Improve ml quick start doc
sunjincheng created FLINK-11123: --- Summary: Improve ml quick start doc Key: FLINK-11123 URL: https://issues.apache.org/jira/browse/FLINK-11123 Project: Flink Issue Type: Improvement Components: Documentation, Machine Learning Library Affects Versions: 1.7.0 Reporter: sunjincheng Assignee: sunjincheng Fix For: 1.7.1, 1.7.0 The user cannot run the sample through the ml quick launch document because the import description of the class is missing. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (FLINK-11178) Check flink-test module for 1.7.1-rc2
sunjincheng created FLINK-11178: --- Summary: Check flink-test module for 1.7.1-rc2 Key: FLINK-11178 URL: https://issues.apache.org/jira/browse/FLINK-11178 Project: Flink Issue Type: Test Components: Tests Reporter: sunjincheng Will create some sub JIRAs for flink-test test error. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (FLINK-11181) SimpleRecoveryITCaseBase test error
sunjincheng created FLINK-11181: --- Summary: SimpleRecoveryITCaseBase test error Key: FLINK-11181 URL: https://issues.apache.org/jira/browse/FLINK-11181 Project: Flink Issue Type: Sub-task Reporter: sunjincheng Run many times always fail. at org.apache.flink.test.recovery.SimpleRecoveryITCaseBase.executeAndRunAssertions(SimpleRecoveryITCaseBase.java:124) at org.apache.flink.test.recovery.SimpleRecoveryITCaseBase.testRestart(SimpleRecoveryITCaseBase.java:150) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) at org.junit.rules.RunRules.evaluate(RunRules.java:20) at org.junit.runners.ParentRunner.run(ParentRunner.java:363) at org.junit.runners.Suite.runChild(Suite.java:128) at org.junit.runners.Suite.runChild(Suite.java:27) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.runners.ParentRunner.run(ParentRunner.java:363) at org.junit.runner.JUnitCore.run(JUnitCore.java:137) at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68) at com.intellij.rt.execution.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:47) at com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:242) at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:70) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (FLINK-11179) JoinCancelingITCase#testCancelSortMatchWhileDoingHeavySorting test error
sunjincheng created FLINK-11179: --- Summary: JoinCancelingITCase#testCancelSortMatchWhileDoingHeavySorting test error Key: FLINK-11179 URL: https://issues.apache.org/jira/browse/FLINK-11179 Project: Flink Issue Type: Sub-task Components: Tests Affects Versions: 1.7.0 Reporter: sunjincheng tag: release-1.7.1-rc2 java.util.concurrent.ExecutionException: org.apache.flink.runtime.messages.FlinkJobNotFoundException: Could not find Flink job (f8abcfa2bf2f9bf13024075e51891d2e) at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1895) at org.apache.flink.client.program.MiniClusterClient.cancel(MiniClusterClient.java:118) at org.apache.flink.test.cancelling.CancelingTestBase.runAndCancelJob(CancelingTestBase.java:109) at org.apache.flink.test.cancelling.JoinCancelingITCase.executeTaskWithGenerator(JoinCancelingITCase.java:94) at org.apache.flink.test.cancelling.JoinCancelingITCase.testCancelSortMatchWhileDoingHeavySorting(JoinCancelingITCase.java:99) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55) at org.junit.rules.RunRules.evaluate(RunRules.java:20) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) at org.junit.rules.RunRules.evaluate(RunRules.java:20) at org.junit.runners.ParentRunner.run(ParentRunner.java:363) at org.junit.runner.JUnitCore.run(JUnitCore.java:137) at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68) at com.intellij.rt.execution.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:47) at com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:242) at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:70) Caused by: org.apache.flink.runtime.messages.FlinkJobNotFoundException: Could not find Flink job (f8abcfa2bf2f9bf13024075e51891d2e) at org.apache.flink.runtime.dispatcher.Dispatcher.getJobMasterGatewayFuture(Dispatcher.java:766) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (FLINK-11182) JobManagerHACheckpointRecoveryITCase need be improved
sunjincheng created FLINK-11182: --- Summary: JobManagerHACheckpointRecoveryITCase need be improved Key: FLINK-11182 URL: https://issues.apache.org/jira/browse/FLINK-11182 Project: Flink Issue Type: Sub-task Components: Tests Affects Versions: 1.7.0 Reporter: sunjincheng 警告: An exception was thrown by an exception handler. java.util.concurrent.RejectedExecutionException: Worker has already been shutdown at org.jboss.netty.channel.socket.nio.AbstractNioSelector.registerTask(AbstractNioSelector.java:120) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.executeInIoThread(AbstractNioWorker.java:72) at org.jboss.netty.channel.socket.nio.NioWorker.executeInIoThread(NioWorker.java:36) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.executeInIoThread(AbstractNioWorker.java:56) at org.jboss.netty.channel.socket.nio.NioWorker.executeInIoThread(NioWorker.java:36) at org.jboss.netty.channel.socket.nio.AbstractNioChannelSink.execute(AbstractNioChannelSink.java:34) at org.jboss.netty.channel.DefaultChannelPipeline.execute(DefaultChannelPipeline.java:636) at org.jboss.netty.channel.Channels.fireExceptionCaughtLater(Channels.java:496) at org.jboss.netty.channel.AbstractChannelSink.exceptionCaught(AbstractChannelSink.java:46) at org.jboss.netty.channel.DefaultChannelPipeline.notifyHandlerException(DefaultChannelPipeline.java:658) at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:781) at org.jboss.netty.handler.codec.oneone.OneToOneEncoder.handleDownstream(OneToOneEncoder.java:54) at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:591) at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:784) at org.jboss.netty.channel.SimpleChannelHandler.disconnectRequested(SimpleChannelHandler.java:320) at org.jboss.netty.channel.SimpleChannelHandler.handleDownstream(SimpleChannelHandler.java:274) at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:591) at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:582) at org.jboss.netty.channel.Channels.disconnect(Channels.java:781) at org.jboss.netty.channel.AbstractChannel.disconnect(AbstractChannel.java:219) at akka.remote.transport.netty.NettyTransport$$anonfun$gracefulClose$1.apply(NettyTransport.scala:241) at akka.remote.transport.netty.NettyTransport$$anonfun$gracefulClose$1.apply(NettyTransport.scala:240) at scala.util.Success.foreach(Try.scala:236) at scala.concurrent.Future$$anonfun$foreach$1.apply(Future.scala:206) at scala.concurrent.Future$$anonfun$foreach$1.apply(Future.scala:206) at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:36) at akka.dispatch.BatchingExecutor$AbstractBatch.processBatch(BatchingExecutor.scala:55) at akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply$mcV$sp(BatchingExecutor.scala:91) at akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply(BatchingExecutor.scala:91) at akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply(BatchingExecutor.scala:91) at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:72) at akka.dispatch.BatchingExecutor$BlockableBatch.run(BatchingExecutor.scala:90) at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:39) at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:415) at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (FLINK-11180) ProcessFailureCancelingITCase#testCancelingOnProcessFailure
sunjincheng created FLINK-11180: --- Summary: ProcessFailureCancelingITCase#testCancelingOnProcessFailure Key: FLINK-11180 URL: https://issues.apache.org/jira/browse/FLINK-11180 Project: Flink Issue Type: Sub-task Components: Tests Affects Versions: 1.7.0 Reporter: sunjincheng tag: release-1.7.1-rc2 org.apache.flink.util.FlinkException: Could not create the DispatcherResourceManagerComponent. at org.apache.flink.runtime.entrypoint.component.AbstractDispatcherResourceManagerComponentFactory.create(AbstractDispatcherResourceManagerComponentFactory.java:242) at org.apache.flink.test.recovery.ProcessFailureCancelingITCase.testCancelingOnProcessFailure(ProcessFailureCancelingITCase.java:148) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55) at org.junit.rules.RunRules.evaluate(RunRules.java:20) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.runners.ParentRunner.run(ParentRunner.java:363) at org.junit.runner.JUnitCore.run(JUnitCore.java:137) at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68) at com.intellij.rt.execution.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:47) at com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:242) at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:70) Caused by: java.net.BindException: Address already in use at sun.nio.ch.Net.bind0(Native Method) at sun.nio.ch.Net.bind(Net.java:433) at sun.nio.ch.Net.bind(Net.java:425) at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223) at org.apache.flink.shaded.netty4.io.netty.channel.socket.nio.NioServerSocketChannel.doBind(NioServerSocketChannel.java:128) at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannel$AbstractUnsafe.bind(AbstractChannel.java:558) at org.apache.flink.shaded.netty4.io.netty.channel.DefaultChannelPipeline$HeadContext.bind(DefaultChannelPipeline.java:1358) at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeBind(AbstractChannelHandlerContext.java:501) at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.bind(AbstractChannelHandlerContext.java:486) at org.apache.flink.shaded.netty4.io.netty.channel.DefaultChannelPipeline.bind(DefaultChannelPipeline.java:1019) at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannel.bind(AbstractChannel.java:254) at org.apache.flink.shaded.netty4.io.netty.bootstrap.AbstractBootstrap$2.run(AbstractBootstrap.java:366) at org.apache.flink.shaded.netty4.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163) at org.apache.flink.shaded.netty4.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:404) at org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:463) at org.apache.flink.shaded.netty4.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884) at java.lang.Thread.run(Thread.java:748) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (FLINK-11091) Clear the use of deprecated methods of KeyedStream in table operators
sunjincheng created FLINK-11091: --- Summary: Clear the use of deprecated methods of KeyedStream in table operators Key: FLINK-11091 URL: https://issues.apache.org/jira/browse/FLINK-11091 Project: Flink Issue Type: Improvement Components: Table API SQL Affects Versions: 1.7.0, 1.6.2, 1.5.5 Reporter: sunjincheng The method of `KeyedStream#process(ProcessFunction)` has bend deprecated after FLINK-8560. So It's good to using `KeyedStream#process(KeyProcessFunction)` to implement `DataStreamSort`, `DataStreamGroupAggregate` and `DataStreamOverAggregate` operators. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
