Re: can flink sql handle udf-generated timestamp field

2019-06-06 Thread Yu Yang
Thank you Fabian! We will try the approach that you suggest. On Thu, Jun 6, 2019 at 1:03 AM Fabian Hueske wrote: > Hi Yu, > > When you register a DataStream as a Table, you can create a new attribute > that contains the event timestamp of the DataStream records. > For that, you would need to

Re: can flink sql handle udf-generated timestamp field

2019-06-06 Thread Fabian Hueske
Hi Yu, When you register a DataStream as a Table, you can create a new attribute that contains the event timestamp of the DataStream records. For that, you would need to assign timestamps and generate watermarks before registering the stream: FlinkKafkaConsumer kafkaConsumer = new

Re: can flink sql handle udf-generated timestamp field

2019-06-06 Thread Yu Yang
Hi Jingsong, Thanks for the reply! The following is our code snippet for creating the log stream. Our messages are in thrift format. We use a customized serializer for serializing/deserializing messages ( see https://github.com/apache/flink/pull/8067 for the implementation) . Given that, how

Re: can flink sql handle udf-generated timestamp field

2019-06-06 Thread Yu Yang
+flink-user On Wed, Jun 5, 2019 at 9:58 AM Yu Yang wrote: > Thanks for the reply! In flink-table-planner, TimeIndicatorTypeInfo is an > internal class that cannot be referenced from application. I got "cannot > find symbol" error when I tried to use it. I have also tried to use " >

Re: can flink sql handle udf-generated timestamp field

2019-06-06 Thread JingsongLee
Hi @Yu Yang: Time-based operations such as windows in both the Table API and SQL require information about the notion of time and its origin. Therefore, tables can offer logical time attributes for indicating time and accessing corresponding timestamps in table programs.[1] This mean Window

can flink sql handle udf-generated timestamp field

2019-06-05 Thread Yu Yang
Hi, I am trying to use Flink SQL to do aggregation on a hopping window. In the data stream, we store the timestamp in long type. So I wrote a UDF 'FROM_UNIXTIME' to convert long to Timestamp type. public static class TimestampModifier extends ScalarFunction { public Timestamp eval(long t)