Apply for being a contributor

[highfei2011]

Hi PMCs,
  Good afternoon !
  This is Jeff Yang ,Now it is Beijing time 13:36:20PM, I am from China , and I 
want to contribute to Apache Flink.
Would you please give me the permission as a contributor?
My JIRA ID is Jeff Yang and username is highfei2...@126.com .
  Thanks.


Best,
Jeff Yang

Re: [DISCUSS] Create a Flink ecosystem website

[Congxian Qiu]

Robert and Daryl, thanks for the great work, I tried the website and filed
some issues on Github.
Best,
Congxian


Robert Metzger  于2019年7月17日周三 下午11:28写道：

> Hey all,
>
> Daryl and I have great news to share. We are about to finish adding the
> basic features to the ecosystem page.
> We are at a stage where it is ready to be reviewed and made public.
>
> You can either check out a development instance of the ecosystem page
> here: https://flink-ecosystem-demo.flink-resources.org/
> Or you run it locally, with the instructions from the README.md:
> https://github.com/sorahn/flink-ecosystem
>
> Please report all issues you find here:
> https://github.com/sorahn/flink-ecosystem/issues or in this thread.
>
> The next steps in this project are the following:
> - We fix all issues reported through this testing
> - We set up the site on the INFRA resources Becket has secured [1], do
> some further testing (including email notifications) and pre-fill the page
> with some packages.
> - We set up a packages.flink.apache.org or flink.apache.org/packages
> domain
> - We announce the packages through a short blog post
>
> Happy testing!
>
> Best,
> Robert
>
> [1] https://issues.apache.org/jira/browse/INFRA-18010
>
>
> On Thu, Apr 25, 2019 at 6:23 AM Becket Qin  wrote:
>
>> Thanks for the update, Robert. Looking forward to the website. If there
>> is already a list of software we need to run the website, we can ask Apache
>> infra team to prepare the VM for us, as that may also take some time.
>>
>> On Wed, Apr 24, 2019 at 11:57 PM Robert Metzger 
>> wrote:
>>
>>> Hey all,
>>>
>>> quick update on this project: The frontend and backend code have been
>>> put together into this repository:
>>> https://github.com/sorahn/flink-ecosystem
>>> We also just agreed on an API specification, and will now work on
>>> finishing the backend.
>>>
>>> It will probably take a few more weeks for this to finish, but we are
>>> making progress :)
>>>
>>> Best,
>>> Robert
>>>
>>>
>>> On Mon, Apr 15, 2019 at 11:18 AM Robert Metzger 
>>> wrote:
>>>
 Hey Daryl,

 thanks a lot for posting a link to this first prototype on the mailing
 list! I really like it!

 Becket: Our plan forward is that Congxian is implementing the backend
 for the website. He has already started with the work, but needs at least
 one more week.


 [Re-sending this email because the first one was blocked on dev@f.a.o]


 On Mon, Apr 15, 2019 at 7:59 AM Becket Qin 
 wrote:

> Hi Daryl,
>
> Thanks a lot for the update. The site looks awesome! This is a great
> progress. I really like the conciseness of GUI.
>
> One minor suggestion is that for the same library, there might be
> multiple versions compatible with different Flink versions. It would be
> good to show that somewhere in the project page as it seems important to
> the users.
>
> BTW, will you share the plan to move forward? Would additional hands
> help?
>
> Thanks,
>
> Jiangjie (Becket) Qin
>
> On Sat, Apr 13, 2019 at 7:10 PM Daryl Roberts 
> wrote:
>
>> > Shall we add a guide page to show people how to publish their
>> projects to the website? The exact rules can be discussed and drafted in 
>> a
>> separate email thread IMO
>>
>> This is a good idea. (Both the guise, and separate thread), I think
>> once there is an actual packed in place we’ll be in a lot better position
>> to discuss this.
>>
>> > The "Log in with Github" link doesn't seem to work yet. Will it
>> only allow login for admins and publishers, or for everyone?
>>
>> Correct, all the oauth stuff requires a real server. We are currently
>> just faking everything.
>>
>> I will add a mock-login page (username/password that just accepts
>> anything and displays whatever username you type in) so we can see the
>> add-comment field and add-packages page once they exist.
>>
>>
>>
>>

Re: [DISCUSS] Flink project bylaws

[jincheng sun]

Hi Becket,

Thanks for the proposal.

Regarding the vote of FLIP, preferably at least includes a PMC vote.
Because FLIP is usually a big change or affects the user's interface
changes. What do you think? (I leave the comment in the wiki.)

Best,
Jincheng

Dawid Wysakowicz  于2019年7月17日周三 下午9:12写道：

> Hi all,
>
> Sorry for joining late. I just wanted to say that I really like the
> proposed bylaws!
>
> One comment, I also have the same concerns as expressed by few people
> before that the "committer +1" on code change might be hard to achieve
> currently. On the other hand I would say this would be beneficial for
> the quality/uniformity of our codebase and knowledge sharing.
>
> I was just wondering what should be the next step for this? I think it
> would make sense to already use those bylaws and put them to PMC vote.
>
> Best,
>
> Dawid
>
> On 12/07/2019 13:35, Piotr Nowojski wrote:
> > Hi Aljoscha and Becket
> >
> > Right, 3 days for FLIP voting is fine I think.
> >
> >>> I’m missing this stated somewhere clearly. If we are stating that a
> single
> >>> committers +1 is good enough for code review, with 0 hours delay (de
> facto
> >>> the current state), we should also write down that this is subject to
> the
> >>> best judgement of the committer to respect the components expertise and
> >>> ongoing development plans (also the de facto current state).
> >> Adding the statement would help clarify the intention, but it may be a
> >> little difficult to enforce and follow..
> > I would be fine with that, it’s a soft/vague rule anyway, intended to be
> used with your “best judgemenet". I would like to just avoid a situation
> when someone violates current uncodified state and refers to the bylaws
> which are saying merging with any committer +1 is always fine (like mine +1
> for flink-python or flink-ml).
> >
> > Piotrek
> >
> >> On 12 Jul 2019, at 11:29, Aljoscha Krettek  wrote:
> >>
> >> @Piotr regarding the 3 days voting on the FLIP. This is just about the
> voting, before that there needs to be the discussion thread. If three days
> have passed on a vote and there is consensus (i.e. 3 committers/PMCs have
> voted +1) that seems a high enough bar for me. So far, we have rarely see
> any FLIPs pass that formal bar.
> >>
> >> According to the recent META-FLIP thread, we want to use "lazy
> majority" for the FLIP voting process. I think that should be changed from
> “consensus” in the proposed bylaws.
> >>
> >> Regarding the current state: do we have a current state that we all
> agree on? I have the feeling that if we try to come up with something that
> reflects the common state, according to PMCs/commiters, that might take a
> very long time. In that case I think it’s better to adopt something that we
> all like, rather than trying to capture how we do it now.
> >>
> >> Aljoscha
> >>
> >>> On 12. Jul 2019, at 11:07, Piotr Nowojski  wrote:
> >>>
> >>> Hi,
> >>>
> >>> Thanks for the proposal. Generally speaking +1 from my side to the
> general idea and most of the content. I also see merit to the Chesney's
> proposal to start from the current state. I think either would be fine for
> me.
> >>>
> >>> Couple of comments:
> >>>
> >>> 1.
> >>>
> >>> I also think that requiring +1 from another committer would slow down
> and put even more strain on the current reviewing bottleneck that we are
> having. Even if the change clear and simple, context switch cost is quite
> high, and that’s just one less PR that the second “cross” committer could
> have reviewed somewhere else in that time. Besides, current setup that we
> have (with no cross +1 from another committer required) works quite well
> and I do not feel that’s causing troubles. On the other hand reviewing
> bottleneck is.
> >>>
> >>> 2.
> >>>
>  I think a committer should know when to ask another committer for
> feedback or not.
> >>> I’m missing this stated somewhere clearly. If we are stating that a
> single committers +1 is good enough for code review, with 0 hours delay (de
> facto the current state), we should also write down that this is subject to
> the best judgement of the committer to respect the components expertise and
> ongoing development plans (also the de facto current state).
> >>>
> >>> 3.
> >>>
> >>> Minimum length of 3 days for FLIP I think currently might be
> problematic/too quick and can lead to problems if respected to the letter.
> Again I think it depends highly on whether the committers with highest
> expertise in the affected components managed to respond or not.
> >>>
> >>> Piotrek
> >>>
>  On 12 Jul 2019, at 09:42, Chesnay Schepler 
> wrote:
> 
>  I'm wondering whether we shouldn't first write down Bylaws that
> reflect the current state, and then have separate discussions for
> individual amendments. My gut feeling is that this discussion will quickly
> become a chaotic mess with plenty points being discussed at once.
> 
>  On 11/07/2019 20:03, Bowen Li wrote:
> > On Thu, Jul 11, 2019 at 10:38 AM 

[jira] [Created] (FLINK-13315) Port wmstrategies to api-java

[Jingsong Lee (JIRA)]

Jingsong Lee created FLINK-13315:


 Summary: Port wmstrategies to api-java
 Key: FLINK-13315
 URL: https://issues.apache.org/jira/browse/FLINK-13315
 Project: Flink
  Issue Type: Sub-task
  Components: Table SQL / API
Reporter: Jingsong Lee
 Fix For: 1.9.0, 1.10.0






--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

[jira] [Created] (FLINK-13314) Correct resultType of some PlannerExpression when operands contains DecimalTypeInfo or BigDecimalTypeInfo in Blink planner

[Jing Zhang (JIRA)]

Jing Zhang created FLINK-13314:
--

 Summary: Correct resultType of some PlannerExpression when 
operands contains DecimalTypeInfo or BigDecimalTypeInfo in Blink planner
 Key: FLINK-13314
 URL: https://issues.apache.org/jira/browse/FLINK-13314
 Project: Flink
  Issue Type: Task
  Components: Table SQL / Planner
Reporter: Jing Zhang


Correct resultType of the following PlannerExpression when operands contains 
DecimalTypeInfo or BigDecimalTypeInfo in Blink planner:

Minus/plus/Div/Mul/Ceil/Floor/Round

 



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

Re: flink-python failed on Travis

[Zili Chen]

Hi Dian

Nice fix!

Best,
tison.


Dian Fu  于2019年7月18日周四 上午9:27写道：

> Hi all,
>
> This issue has been fixed in
> https://github.com/apache/flink/commit/200a5bf9dca9d398cf07879d4d1e407a2f41d839
> <
> https://github.com/apache/flink/commit/200a5bf9dca9d398cf07879d4d1e407a2f41d839>.
> Thanks for @WeiZhong's fix.
>
> Regards,
> Dian
>
> > 在 2019年7月18日，上午2:41，Bowen Li  写道：
> >
> > Hi Dian,
> >
> > Is there any update on this? It seems have been failing for a day.
> >
> >
> >
> > On Tue, Jul 16, 2019 at 9:35 PM Dian Fu  wrote:
> >
> >> Thanks for reporting this issue. I will take a look at it.
> >>
> >>> 在 2019年7月17日，上午11:50，Danny Chan  写道：
> >>>
> >>> I have the same issue ~~
> >>>
> >>> Best,
> >>> Danny Chan
> >>> 在 2019年7月17日 +0800 AM11:21，Haibo Sun ，写道：
>  Hi, folks
> 
> 
>  I noticed that all of the Travis tests reported the following failure.
> >> Is anyone working on this issue?
> 
> 
>  ___ summary
> >> 
>  ERROR: py27: InvocationError for command
> >> /home/travis/build/flink-ci/flink/flink-python/dev/.conda/bin/python3.7
> -m
> >> virtualenv --no-download --python
> >>
> /home/travis/build/flink-ci/flink/flink-python/dev/.conda/envs/2.7/bin/python2.7
> >> py27 (exited with code 1)
>  py33: commands succeeded
>  ERROR: py34: InvocationError for command
> >> /home/travis/build/flink-ci/flink/flink-python/dev/.conda/bin/python3.7
> -m
> >> virtualenv --no-download --python
> >>
> /home/travis/build/flink-ci/flink/flink-python/dev/.conda/envs/3.4/bin/python3.4
> >> py34 (exited with code 100)
>  py35: commands succeeded
>  py36: commands succeeded
>  py37: commands succeeded
>  tox checks... [FAILED]
>  PYTHON exited with EXIT CODE: 1.
>  Trying to KILL watchdog (12990).
> 
> 
>  Best,
>  Haibo
> >>
> >>
>
>

Re: flink-python failed on Travis

[Dian Fu]

Hi all,

This issue has been fixed in 
https://github.com/apache/flink/commit/200a5bf9dca9d398cf07879d4d1e407a2f41d839 
.
  Thanks for @WeiZhong's fix.

Regards,
Dian

> 在 2019年7月18日，上午2:41，Bowen Li  写道：
> 
> Hi Dian,
> 
> Is there any update on this? It seems have been failing for a day.
> 
> 
> 
> On Tue, Jul 16, 2019 at 9:35 PM Dian Fu  wrote:
> 
>> Thanks for reporting this issue. I will take a look at it.
>> 
>>> 在 2019年7月17日，上午11:50，Danny Chan  写道：
>>> 
>>> I have the same issue ~~
>>> 
>>> Best,
>>> Danny Chan
>>> 在 2019年7月17日 +0800 AM11:21，Haibo Sun ，写道：
 Hi, folks
 
 
 I noticed that all of the Travis tests reported the following failure.
>> Is anyone working on this issue?
 
 
 ___ summary
>> 
 ERROR: py27: InvocationError for command
>> /home/travis/build/flink-ci/flink/flink-python/dev/.conda/bin/python3.7 -m
>> virtualenv --no-download --python
>> /home/travis/build/flink-ci/flink/flink-python/dev/.conda/envs/2.7/bin/python2.7
>> py27 (exited with code 1)
 py33: commands succeeded
 ERROR: py34: InvocationError for command
>> /home/travis/build/flink-ci/flink/flink-python/dev/.conda/bin/python3.7 -m
>> virtualenv --no-download --python
>> /home/travis/build/flink-ci/flink/flink-python/dev/.conda/envs/3.4/bin/python3.4
>> py34 (exited with code 100)
 py35: commands succeeded
 py36: commands succeeded
 py37: commands succeeded
 tox checks... [FAILED]
 PYTHON exited with EXIT CODE: 1.
 Trying to KILL watchdog (12990).
 
 
 Best,
 Haibo
>> 
>> 

[jira] [Created] (FLINK-13313) create CatalogTableBuilder to support building CatalogTable from descriptors

[Bowen Li (JIRA)]

Bowen Li created FLINK-13313:


 Summary: create CatalogTableBuilder to support building 
CatalogTable from descriptors
 Key: FLINK-13313
 URL: https://issues.apache.org/jira/browse/FLINK-13313
 Project: Flink
  Issue Type: Sub-task
  Components: Table SQL / API
Affects Versions: 1.9.0, 1.10.0
Reporter: Bowen Li
Assignee: Bowen Li
 Fix For: 1.9.0, 1.10.0


Previously, users can create an ExternalCatalogTable (deprecated) from 
descriptors via ExternalCatalogTableBuilder, and this helps smooth user 
experience of Flink Table API. E.g.

{code:java}
  *   ExternalCatalogTableBuilder(
  * new ExternalSystemXYZ()
  *   .version("0.11"))
  *   .withFormat(
  * new Json()
  *   .jsonSchema("{...}")
  *   .failOnMissingField(false))
  *   .withSchema(
  * new Schema()
  *   .field("user-name", "VARCHAR").from("u_name")
  *   .field("count", "DECIMAL")
  *   .supportsStreaming()
  *   .asTableSource()
{code}

We need a similar new class {{CatalogTableBuilder}} for new Catalog APIs

cc [~tzulitai] [~ykt836]



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

[jira] [Created] (FLINK-13312) move tests for data type mappings between Flink and Hive into its own test class

[Bowen Li (JIRA)]

Bowen Li created FLINK-13312:


 Summary: move tests for data type mappings between Flink and Hive 
into its own test class
 Key: FLINK-13312
 URL: https://issues.apache.org/jira/browse/FLINK-13312
 Project: Flink
  Issue Type: Sub-task
  Components: Connectors / Hive
Reporter: Bowen Li
Assignee: Bowen Li
 Fix For: 1.9.0, 1.10.0






--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

Re: flink-python failed on Travis

[Bowen Li]

Hi Dian,

Is there any update on this? It seems have been failing for a day.



On Tue, Jul 16, 2019 at 9:35 PM Dian Fu  wrote:

> Thanks for reporting this issue. I will take a look at it.
>
> > 在 2019年7月17日，上午11:50，Danny Chan  写道：
> >
> > I have the same issue ~~
> >
> > Best,
> > Danny Chan
> > 在 2019年7月17日 +0800 AM11:21，Haibo Sun ，写道：
> >> Hi, folks
> >>
> >>
> >> I noticed that all of the Travis tests reported the following failure.
> Is anyone working on this issue?
> >>
> >>
> >> ___ summary
> 
> >> ERROR: py27: InvocationError for command
> /home/travis/build/flink-ci/flink/flink-python/dev/.conda/bin/python3.7 -m
> virtualenv --no-download --python
> /home/travis/build/flink-ci/flink/flink-python/dev/.conda/envs/2.7/bin/python2.7
> py27 (exited with code 1)
> >> py33: commands succeeded
> >> ERROR: py34: InvocationError for command
> /home/travis/build/flink-ci/flink/flink-python/dev/.conda/bin/python3.7 -m
> virtualenv --no-download --python
> /home/travis/build/flink-ci/flink/flink-python/dev/.conda/envs/3.4/bin/python3.4
> py34 (exited with code 100)
> >> py35: commands succeeded
> >> py36: commands succeeded
> >> py37: commands succeeded
> >> tox checks... [FAILED]
> >> PYTHON exited with EXIT CODE: 1.
> >> Trying to KILL watchdog (12990).
> >>
> >>
> >> Best,
> >> Haibo
>
>

Re: instable checkpointing after migration to flink 1.8

[Bekir Oguz]

Hi Congxian,Yes we have incremental checkpointing enabled on RocksDBBackend.For further investigation, I have logged into one task manager node which had 15 min long snapshotting and found the logs under some /tmp directory.Attaching 2 logs files, one for a long/problematic snapshotting and one log file for a good/fast snapshot.I know the problematic snapshot from the trigger time (14:22 today) on the task manager, that’s only a guess and I extracted the log events around that time in the bad_snapshot.log file.And I also extracted events from 14:11 today from the same task manager in the good_snapshot.log file to be able to compare these two.The only difference I can see is the compaction kicking in during the checkpointing in the bad_snaphot.log file.Do these logs give more insight to explain what is going on?

bad_snapshot.log
Description: Binary data


good_snapshot.log
Description: Binary data
Kind regards,Bekir OguzOn 17 Jul 2019, at 15:29, Congxian Qiu  wrote:Hi BekirSorry for the previous message, I didn't see the second image of your first message :(From the second image of your first message, seems the sync part consumes too much time.57    15:40:24(acknowledgement Time)   15m53s (End to End Duration)  464m(State size)  15M48s(Checkpoint Duration(Sync))  4s(Checkpoint Duration (Async)Do you enable incremental checkpoint or not?If you enable incremental checkpoint, then In the sync part of a checkpoint for a RocksDBStateBackend, we'll 1) flush all data from memory to sst files, 2) snapshot meta, 3) checkpoint the RocksDB, maybe we should check the disk info during the long checkpoint. If you disable incremental checkpoint, then in the sync part of a checkpoint for RocksDBStateBackend, we'll 1) snapshot meta; 2) get a snapshot from RocksDBAnd another question for this is, do you ever change the user jar's logic when migrating from 1.6 to 1.8?Best,CongxianBekir Oguz  于2019年7月17日周三 下午5:15写道：Sending again with reduced image sizes due to Apache mail server error.Begin forwarded message:From: Bekir Oguz Subject: Re: instable checkpointing after migration to flink 1.8Date: 17 July 2019 at 11:10:41 CESTTo: Congxian Qiu Cc: dev@flink.apache.orgHi Congxian,Thanks for your response. Here are the memory/cpu/network usage of the task manager and the job manager pods around that time.The vertical line is the moment the checkpoint is triggered (15:24) and acknowledgement received on 15:40. What we see is the memory usage is jumping around +1GB each time a checkpoint is triggered. We can also see the network bandwidth usage correlates with the checkpointing interval of 5 mins. After the checkpoint is triggered on 15:24 we see a normal network bandwidth usage for 5 mins and then nothing for about 15 mins which is the checkpoint ack time for this task slot. Regards,BekirOn 17 Jul 2019, at 09:16, Congxian Qiu  wrote:Hi BekirFirst of all, I think there is something wrong.  the state size is almost the same,  but the duration is different so much.The checkpoint for RocksDBStatebackend is dump sst files, then copy the needed sst files(if you enable incremental checkpoint, the sst files already on remote will not upload), then complete checkpoint. Can you check the network bandwidth usage during checkpoint?Best,CongxianBekir Oguz  于2019年7月16日周二 下午10:45写道：Hi all,We have a flink job with user state, checkpointing to RocksDBBackend which is externally stored in AWS S3.After we have migrated our cluster from 1.6 to 1.8, we see occasionally that some slots do to acknowledge the checkpoints quick enough. As an example: All slots acknowledge between 30-50 seconds except only one slot acknowledges in 15 mins. Checkpoint sizes are similar to each other, like 200-400 MB.We did not experience this weird behaviour in Flink 1.6. We have 5 min checkpoint interval and this happens sometimes once in an hour sometimes more but not in all the checkpoint requests. Please see the screenshot below.Also another point: For the faulty slots, the duration is consistently 15 mins and some seconds, we couldn’t find out where this 15 mins response time comes from. And each time it is a different task manager, not always the same one.Do you guys aware of any other users having similar issues with the new version and also a suggested bug fix or solution?Thanks in advance,Bekir Oguz

Re: [DISCUSS] Create a Flink ecosystem website

[Robert Metzger]

Hey all,

Daryl and I have great news to share. We are about to finish adding the
basic features to the ecosystem page.
We are at a stage where it is ready to be reviewed and made public.

You can either check out a development instance of the ecosystem page here:
https://flink-ecosystem-demo.flink-resources.org/
Or you run it locally, with the instructions from the README.md:
https://github.com/sorahn/flink-ecosystem

Please report all issues you find here:
https://github.com/sorahn/flink-ecosystem/issues or in this thread.

The next steps in this project are the following:
- We fix all issues reported through this testing
- We set up the site on the INFRA resources Becket has secured [1], do some
further testing (including email notifications) and pre-fill the page with
some packages.
- We set up a packages.flink.apache.org or flink.apache.org/packages domain
- We announce the packages through a short blog post

Happy testing!

Best,
Robert

[1] https://issues.apache.org/jira/browse/INFRA-18010


On Thu, Apr 25, 2019 at 6:23 AM Becket Qin  wrote:

> Thanks for the update, Robert. Looking forward to the website. If there is
> already a list of software we need to run the website, we can ask Apache
> infra team to prepare the VM for us, as that may also take some time.
>
> On Wed, Apr 24, 2019 at 11:57 PM Robert Metzger 
> wrote:
>
>> Hey all,
>>
>> quick update on this project: The frontend and backend code have been put
>> together into this repository: https://github.com/sorahn/flink-ecosystem
>> We also just agreed on an API specification, and will now work on
>> finishing the backend.
>>
>> It will probably take a few more weeks for this to finish, but we are
>> making progress :)
>>
>> Best,
>> Robert
>>
>>
>> On Mon, Apr 15, 2019 at 11:18 AM Robert Metzger 
>> wrote:
>>
>>> Hey Daryl,
>>>
>>> thanks a lot for posting a link to this first prototype on the mailing
>>> list! I really like it!
>>>
>>> Becket: Our plan forward is that Congxian is implementing the backend
>>> for the website. He has already started with the work, but needs at least
>>> one more week.
>>>
>>>
>>> [Re-sending this email because the first one was blocked on dev@f.a.o]
>>>
>>>
>>> On Mon, Apr 15, 2019 at 7:59 AM Becket Qin  wrote:
>>>
 Hi Daryl,

 Thanks a lot for the update. The site looks awesome! This is a great
 progress. I really like the conciseness of GUI.

 One minor suggestion is that for the same library, there might be
 multiple versions compatible with different Flink versions. It would be
 good to show that somewhere in the project page as it seems important to
 the users.

 BTW, will you share the plan to move forward? Would additional hands
 help?

 Thanks,

 Jiangjie (Becket) Qin

 On Sat, Apr 13, 2019 at 7:10 PM Daryl Roberts 
 wrote:

> > Shall we add a guide page to show people how to publish their
> projects to the website? The exact rules can be discussed and drafted in a
> separate email thread IMO
>
> This is a good idea. (Both the guise, and separate thread), I think
> once there is an actual packed in place we’ll be in a lot better position
> to discuss this.
>
> > The "Log in with Github" link doesn't seem to work yet. Will it only
> allow login for admins and publishers, or for everyone?
>
> Correct, all the oauth stuff requires a real server. We are currently
> just faking everything.
>
> I will add a mock-login page (username/password that just accepts
> anything and displays whatever username you type in) so we can see the
> add-comment field and add-packages page once they exist.
>
>
>
>

Re: Add relative path support in Savepoint Connector

[bupt_ljy]

Hi Konstantin,
Thank you for your feedback. You’re right that this part belongs to the 
savepoint desrializing.
This is an old issue which should be resolve before 1.3 version according 
to the comments. Anyway, I’m going to keep following this.


Best Regards,
Jiayi Liao


Original Message
Sender:Konstantin knaufkonstan...@ververica.com
Recipient:dev...@flink.apache.org; Stefan richters.rich...@ververica.com
Date:Wednesday, Jul 17, 2019 16:28
Subject:Re: Add relative path support in Savepoint Connector


Hi Jiayi, I think, this is not an issue with the State Processor API 
specifically, but with savepoints in general. The _metadata file of a savepoint 
uses absolute path references. There is a pretty old Jira ticket, which already 
mentioned this limitation [1]. Stefan (cc) might know more about any ongoing 
development in that direction and might have an idea about the effort of making 
savepoints relocatable. Best, Konstantin [1] 
https://issues.apache.org/jira/browse/FLINK-5763 On Wed, Jul 17, 2019 at 8:35 
AM bupt_ljy bupt_...@163.com wrote:  Hi again,  Anyone has any opinion on this 
topic?Best Regards,  Jiayi LiaoOriginal Message  
Sender:bupt_ljybupt_...@163.com  Recipient:dev...@flink.apache.org  Cc:Tzu-Li 
(Gordon) taitzuli...@apache.org  Date:Tuesday, Jul 16, 2019 15:24  Subject:Add 
relative path support in Savepoint ConnectorHi all,  Firstly I appreciate 
Gordon and Seth’s effort on this feature, which  is really helpful to our 
production use. Like you mentioned in the  FLINK-12047, one of the production 
uses is that we use the existing state  to derive new state. However, since the 
state handle is using the absolute  path to get the input stream, we need to 
directly operate the state in  production environment, which is not an 
anxiety-reducing situation, at  least for me.  So I wonder if we can add the 
relative path support in this module  because the files are persisted in a 
directory after we take a savepoint,  which makes it achievable. I’m not sure 
whether my scenario is a common  case or not, but I think I can give my 
contributions if you all are okay  about this.  Best Regards,  Jiayi Liao 
-- Konstantin Knauf | Solutions Architect +49 160 91394525 Planned Absences: 
10.08.2019 - 31.08.2019, 05.09. - 06.09.2010 -- Ververica GmbH | 
Invalidenstrasse 115, 10115 Berlin, Germany -- Ververica GmbH Registered at 
Amtsgericht Charlottenburg: HRB 158244 B Managing Directors: Dr. Kostas 
Tzoumas, Dr. Stephan Ewen

Re: instable checkpointing after migration to flink 1.8

[Congxian Qiu]

Hi Bekir

Sorry for the previous message, I didn't see the second image of your first
message :(

>From the second image of your first message, seems the sync part consumes
too much time.
5715:40:24(acknowledgement Time)   15m53s (End to End Duration)
464m(State size)  15M48s(Checkpoint Duration(Sync))  4s(Checkpoint Duration
(Async)

Do you enable incremental checkpoint or not?
If you enable incremental checkpoint, then In the sync part of a checkpoint
for a RocksDBStateBackend, we'll 1) flush all data from memory to sst
files, 2) snapshot meta, 3) checkpoint the RocksDB, maybe we should check
the disk info during the long checkpoint.

If you disable incremental checkpoint, then in the sync part of a
checkpoint for RocksDBStateBackend, we'll 1) snapshot meta; 2) get a
snapshot from RocksDB

And another question for this is, do you ever change the user jar's logic
when migrating from 1.6 to 1.8?

Best,
Congxian


Bekir Oguz  于2019年7月17日周三 下午5:15写道：

> Sending again with reduced image sizes due to Apache mail server error.
>
> Begin forwarded message:
>
> *From: *Bekir Oguz 
> *Subject: **Re: instable checkpointing after migration to flink 1.8*
> *Date: *17 July 2019 at 11:10:41 CEST
> *To: *Congxian Qiu 
> *Cc: *dev@flink.apache.org
>
> Hi Congxian,
> Thanks for your response. Here are the memory/cpu/network usage of the
> task manager and the job manager pods around that time.
> The vertical line is the moment the checkpoint is triggered (15:24) and
> acknowledgement received on 15:40.
>
> What we see is the memory usage is jumping around +1GB each time a
> checkpoint is triggered. We can also see the network bandwidth usage
> correlates with the checkpointing interval of 5 mins. After the checkpoint
> is triggered on 15:24 we see a normal network bandwidth usage for 5 mins
> and then nothing for about 15 mins which is the checkpoint ack time for
> this task slot.
>
> Regards,
> Bekir
>
>
>
>
>
> On 17 Jul 2019, at 09:16, Congxian Qiu  wrote:
>
> Hi Bekir
>
> First of all, I think there is something wrong.  the state size is almost
> the same,  but the duration is different so much.
>
> The checkpoint for RocksDBStatebackend is dump sst files, then copy the
> needed sst files(if you enable incremental checkpoint, the sst files
> already on remote will not upload), then complete checkpoint. Can you check
> the network bandwidth usage during checkpoint?
>
> Best,
> Congxian
>
>
> Bekir Oguz  于2019年7月16日周二 下午10:45写道：
>
>> Hi all,
>> We have a flink job with user state, checkpointing to RocksDBBackend
>> which is externally stored in AWS S3.
>> After we have migrated our cluster from 1.6 to 1.8, we see occasionally
>> that some slots do to acknowledge the checkpoints quick enough. As an
>> example: All slots acknowledge between 30-50 seconds except only one slot
>> acknowledges in 15 mins. Checkpoint sizes are similar to each other, like
>> 200-400 MB.
>>
>> We did not experience this weird behaviour in Flink 1.6. We have 5 min
>> checkpoint interval and this happens sometimes once in an hour sometimes
>> more but not in all the checkpoint requests. Please see the screenshot
>> below.
>>
>> Also another point: For the faulty slots, the duration is consistently 15
>> mins and some seconds, we couldn’t find out where this 15 mins response
>> time comes from. And each time it is a different task manager, not always
>> the same one.
>>
>> Do you guys aware of any other users having similar issues with the new
>> version and also a suggested bug fix or solution?
>>
>>
>>
>>
>> Thanks in advance,
>> Bekir Oguz
>>
>
>
>

Re: [DISCUSS] Flink project bylaws

[Dawid Wysakowicz]

Hi all,

Sorry for joining late. I just wanted to say that I really like the
proposed bylaws!

One comment, I also have the same concerns as expressed by few people
before that the "committer +1" on code change might be hard to achieve
currently. On the other hand I would say this would be beneficial for
the quality/uniformity of our codebase and knowledge sharing.

I was just wondering what should be the next step for this? I think it
would make sense to already use those bylaws and put them to PMC vote.

Best,

Dawid

On 12/07/2019 13:35, Piotr Nowojski wrote:
> Hi Aljoscha and Becket
>
> Right, 3 days for FLIP voting is fine I think.
>
>>> I’m missing this stated somewhere clearly. If we are stating that a single
>>> committers +1 is good enough for code review, with 0 hours delay (de facto
>>> the current state), we should also write down that this is subject to the
>>> best judgement of the committer to respect the components expertise and
>>> ongoing development plans (also the de facto current state).
>> Adding the statement would help clarify the intention, but it may be a
>> little difficult to enforce and follow..
> I would be fine with that, it’s a soft/vague rule anyway, intended to be used 
> with your “best judgemenet". I would like to just avoid a situation when 
> someone violates current uncodified state and refers to the bylaws which are 
> saying merging with any committer +1 is always fine (like mine +1 for 
> flink-python or flink-ml). 
>
> Piotrek
>
>> On 12 Jul 2019, at 11:29, Aljoscha Krettek  wrote:
>>
>> @Piotr regarding the 3 days voting on the FLIP. This is just about the 
>> voting, before that there needs to be the discussion thread. If three days 
>> have passed on a vote and there is consensus (i.e. 3 committers/PMCs have 
>> voted +1) that seems a high enough bar for me. So far, we have rarely see 
>> any FLIPs pass that formal bar.
>>
>> According to the recent META-FLIP thread, we want to use "lazy majority" for 
>> the FLIP voting process. I think that should be changed from “consensus” in 
>> the proposed bylaws.
>>
>> Regarding the current state: do we have a current state that we all agree 
>> on? I have the feeling that if we try to come up with something that 
>> reflects the common state, according to PMCs/commiters, that might take a 
>> very long time. In that case I think it’s better to adopt something that we 
>> all like, rather than trying to capture how we do it now.
>>
>> Aljoscha
>>
>>> On 12. Jul 2019, at 11:07, Piotr Nowojski  wrote:
>>>
>>> Hi,
>>>
>>> Thanks for the proposal. Generally speaking +1 from my side to the general 
>>> idea and most of the content. I also see merit to the Chesney's proposal to 
>>> start from the current state. I think either would be fine for me.
>>>
>>> Couple of comments:
>>>
>>> 1. 
>>>
>>> I also think that requiring +1 from another committer would slow down and 
>>> put even more strain on the current reviewing bottleneck that we are 
>>> having. Even if the change clear and simple, context switch cost is quite 
>>> high, and that’s just one less PR that the second “cross” committer could 
>>> have reviewed somewhere else in that time. Besides, current setup that we 
>>> have (with no cross +1 from another committer required) works quite well 
>>> and I do not feel that’s causing troubles. On the other hand reviewing 
>>> bottleneck is. 
>>>
>>> 2.
>>>
 I think a committer should know when to ask another committer for feedback 
 or not.
>>> I’m missing this stated somewhere clearly. If we are stating that a single 
>>> committers +1 is good enough for code review, with 0 hours delay (de facto 
>>> the current state), we should also write down that this is subject to the 
>>> best judgement of the committer to respect the components expertise and 
>>> ongoing development plans (also the de facto current state).
>>>
>>> 3.
>>>
>>> Minimum length of 3 days for FLIP I think currently might be 
>>> problematic/too quick and can lead to problems if respected to the letter. 
>>> Again I think it depends highly on whether the committers with highest 
>>> expertise in the affected components managed to respond or not. 
>>>
>>> Piotrek
>>>
 On 12 Jul 2019, at 09:42, Chesnay Schepler  wrote:

 I'm wondering whether we shouldn't first write down Bylaws that reflect 
 the current state, and then have separate discussions for individual 
 amendments. My gut feeling is that this discussion will quickly become a 
 chaotic mess with plenty points being discussed at once.

 On 11/07/2019 20:03, Bowen Li wrote:
> On Thu, Jul 11, 2019 at 10:38 AM Becket Qin  wrote:
>
>> Thanks everyone for all the comments and feedback. Please see the replies
>> below:
>>
>> 
>> Re: Konstantin
>>
>>> * In addition to a simple "Code Change" we could also add a row for 
>>> "Code
>>> Change requiring a FLIP" with a reference to the FLIP 

[jira] [Created] (FLINK-13311) flink-table section outdated in NOTICE-binary

[Chesnay Schepler (JIRA)]

Chesnay Schepler created FLINK-13311:


 Summary: flink-table section outdated in NOTICE-binary
 Key: FLINK-13311
 URL: https://issues.apache.org/jira/browse/FLINK-13311
 Project: Flink
  Issue Type: Bug
  Components: Build System, Table SQL / Planner
Affects Versions: 1.9.0
Reporter: Chesnay Schepler
Assignee: Chesnay Schepler
 Fix For: 1.9.0


The NOTICE-binary section for flink-table hasn't been updated.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

[jira] [Created] (FLINK-13310) Remove shade-plugin configuration in hive-connector

[Chesnay Schepler (JIRA)]

Chesnay Schepler created FLINK-13310:


 Summary: Remove shade-plugin configuration in hive-connector
 Key: FLINK-13310
 URL: https://issues.apache.org/jira/browse/FLINK-13310
 Project: Flink
  Issue Type: Improvement
  Components: Build System, Connectors / Hive
Affects Versions: 1.9.0
Reporter: Chesnay Schepler
 Fix For: 1.9.0


The hive connector has a shade plugin configuration but isn't doing anything 
interesting. We may as well remove it.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

[jira] [Created] (FLINK-13309) flink-shaded-netty-tcnative-dynamic missing from NOTICE-binary

[Chesnay Schepler (JIRA)]

Chesnay Schepler created FLINK-13309:


 Summary: flink-shaded-netty-tcnative-dynamic missing from 
NOTICE-binary
 Key: FLINK-13309
 URL: https://issues.apache.org/jira/browse/FLINK-13309
 Project: Flink
  Issue Type: Bug
  Components: BuildSystem / Shaded
Affects Versions: 1.9.0
Reporter: Chesnay Schepler
Assignee: Chesnay Schepler
 Fix For: 1.9.0


{{flink-shaded-netty-tcnative-dynamic}} is bundled in flink-dist but is not 
mentioned in the notice file.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

[jira] [Created] (FLINK-13308) flink-python releases 2 jars

[Chesnay Schepler (JIRA)]

Chesnay Schepler created FLINK-13308:


 Summary: flink-python releases 2 jars
 Key: FLINK-13308
 URL: https://issues.apache.org/jira/browse/FLINK-13308
 Project: Flink
  Issue Type: Bug
  Components: API / Python, Build System
Affects Versions: 1.9.0
Reporter: Chesnay Schepler
 Fix For: 1.9.0


{{flink-python}} uses a classifier to differentiate itseld from the old python 
API. turns out thsi doesn't work since it still tries to release a normal 
unshaded flink-python jar.

We should drop the classifier, and either stick to flink-python or rename it as 
proposed in FLINK-12776.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

[jira] [Created] (FLINK-13307) SourceStreamTaskTest test instability.

[Kostas Kloudas (JIRA)]

Kostas Kloudas created FLINK-13307:
--

 Summary: SourceStreamTaskTest test instability.
 Key: FLINK-13307
 URL: https://issues.apache.org/jira/browse/FLINK-13307
 Project: Flink
  Issue Type: Improvement
  Components: Tests
Affects Versions: 1.9.0
Reporter: Kostas Kloudas
 Fix For: 1.9.0






--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

[jira] [Created] (FLINK-13306) flink-examples-streaming-gcp-pubsub is missing NOTICE

[Chesnay Schepler (JIRA)]

Chesnay Schepler created FLINK-13306:


 Summary: flink-examples-streaming-gcp-pubsub is missing NOTICE
 Key: FLINK-13306
 URL: https://issues.apache.org/jira/browse/FLINK-13306
 Project: Flink
  Issue Type: Bug
  Components: Connectors / Google Cloud PubSub, Examples
Affects Versions: 1.9.0
Reporter: Chesnay Schepler
Assignee: Chesnay Schepler
 Fix For: 1.9.0


The pubsub example is bundling various dependencies but is missing a NOTICE 
file. Since the example is included in flink-dist the NOTICE-binary must also 
be updated.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

Re: Flink "allow lateness" for tables

[Jark Wu]

Hi Ramya,

Are you looking for "allow lateness" for tumbling window in Flink SQL?

If yes, Flink SQL doesn't provide such things currently.
However, blink planner (the new runner of Flink SQL) will support this
maybe in the next release (v1.10).
Actually, we already implemented it in blink planner but didn't expose the
feature in v1.9 to let us have enough time to review the API.

For now, you can convert Table to DataStream, and can use tumbling window
functionality on the DataStream which supports allow lateness.
Or you can increase the watermark delay to wait more late records, but the
window output delay will get larger.

Best,
Jark

On Wed, 17 Jul 2019 at 17:39, Ramya Ramamurthy  wrote:

> Hi,
>
> I would like to know if there is some configuration which enabled to
> configure allow lateness in table. The documentation mentions about streams
> and not tables.
> If this is not present, is there a way to collect it on a side output for
> tables.
>
> Today, we see some late packet drops in Flink, where my tumbling window is
> of 1 second and check-pointing every 5 mins.
>
> Thanks.
>

Flink "allow lateness" for tables

[Ramya Ramamurthy]

Hi,

I would like to know if there is some configuration which enabled to
configure allow lateness in table. The documentation mentions about streams
and not tables.
If this is not present, is there a way to collect it on a side output for
tables.

Today, we see some late packet drops in Flink, where my tumbling window is
of 1 second and check-pointing every 5 mins.

Thanks.

[jira] [Created] (FLINK-13305) YARN rest commands not documented

[Chesnay Schepler (JIRA)]

Chesnay Schepler created FLINK-13305:


 Summary: YARN rest commands not documented
 Key: FLINK-13305
 URL: https://issues.apache.org/jira/browse/FLINK-13305
 Project: Flink
  Issue Type: Bug
  Components: Documentation, Runtime / REST
Affects Versions: 1.8.1, 1.9.0
Reporter: Chesnay Schepler


The YARN stop/cancel commands aren't documented. The generated excludes them 
since the headers don't implement the {{MessageHeaders}} interface.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

Re: Add relative path support in Savepoint Connector

[Konstantin Knauf]

Hi Jiayi,

I think, this is not an issue with the State Processor API specifically,
but with savepoints in general. The _metadata file of a savepoint uses
absolute path references. There is a pretty old Jira ticket, which already
mentioned this limitation [1].

Stefan (cc) might know more about any ongoing development in that direction
and might have an idea about the effort of making savepoints relocatable.

Best,

Konstantin

[1] https://issues.apache.org/jira/browse/FLINK-5763

On Wed, Jul 17, 2019 at 8:35 AM bupt_ljy  wrote:

> Hi again,
> Anyone has any opinion on this topic?
>
>
> Best Regards,
> Jiayi Liao
>
>
> Original Message
> Sender:bupt_ljybupt_...@163.com
> Recipient:dev...@flink.apache.org
> Cc:Tzu-Li (Gordon) taitzuli...@apache.org
> Date:Tuesday, Jul 16, 2019 15:24
> Subject:Add relative path support in Savepoint Connector
>
>
> Hi all,
> Firstly I appreciate Gordon and Seth’s effort on this feature, which
> is really helpful to our production use. Like you mentioned in the
> FLINK-12047, one of the production uses is that we use the existing state
> to derive new state. However, since the state handle is using the absolute
> path to get the input stream, we need to directly operate the state in
> production environment, which is not an anxiety-reducing situation, at
> least for me.
> So I wonder if we can add the relative path support in this module
> because the files are persisted in a directory after we take a savepoint,
> which makes it achievable. I’m not sure whether my scenario is a common
> case or not, but I think I can give my contributions if you all are okay
> about this.
>
>
>
>
> Best Regards,
> Jiayi Liao



-- 

Konstantin Knauf | Solutions Architect

+49 160 91394525


Planned Absences: 10.08.2019 - 31.08.2019, 05.09. - 06.09.2010


--

Ververica GmbH | Invalidenstrasse 115, 10115 Berlin, Germany

--

Ververica GmbH
Registered at Amtsgericht Charlottenburg: HRB 158244 B
Managing Directors: Dr. Kostas Tzoumas, Dr. Stephan Ewen

[jira] [Created] (FLINK-13304) Fix implementation of getString and getBinary method in NestedRow

[Caizhi Weng (JIRA)]

Caizhi Weng created FLINK-13304:
---

 Summary: Fix implementation of getString and getBinary method in 
NestedRow
 Key: FLINK-13304
 URL: https://issues.apache.org/jira/browse/FLINK-13304
 Project: Flink
  Issue Type: Bug
  Components: Table SQL / Runtime
Reporter: Caizhi Weng
Assignee: Caizhi Weng
 Fix For: 1.9.0, 1.10.0


The `getString` and `getBinary` method in `NestedRow` are not implemented 
correctly. Also there is no tests guarding these complex data formats.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

[jira] [Created] (FLINK-13303) Add e2e test for hive connector

[Rui Li (JIRA)]

Rui Li created FLINK-13303:
--

 Summary: Add e2e test for hive connector
 Key: FLINK-13303
 URL: https://issues.apache.org/jira/browse/FLINK-13303
 Project: Flink
  Issue Type: Bug
  Components: Connectors / Hive, Tests
Reporter: Rui Li
Assignee: Rui Li






--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

Re: instable checkpointing after migration to flink 1.8

[Congxian Qiu]

Hi Bekir

First of all, I think there is something wrong.  the state size is almost
the same,  but the duration is different so much.

The checkpoint for RocksDBStatebackend is dump sst files, then copy the
needed sst files(if you enable incremental checkpoint, the sst files
already on remote will not upload), then complete checkpoint. Can you check
the network bandwidth usage during checkpoint?

Best,
Congxian


Bekir Oguz  于2019年7月16日周二 下午10:45写道：

> Hi all,
> We have a flink job with user state, checkpointing to RocksDBBackend which
> is externally stored in AWS S3.
> After we have migrated our cluster from 1.6 to 1.8, we see occasionally
> that some slots do to acknowledge the checkpoints quick enough. As an
> example: All slots acknowledge between 30-50 seconds except only one slot
> acknowledges in 15 mins. Checkpoint sizes are similar to each other, like
> 200-400 MB.
>
> We did not experience this weird behaviour in Flink 1.6. We have 5 min
> checkpoint interval and this happens sometimes once in an hour sometimes
> more but not in all the checkpoint requests. Please see the screenshot
> below.
>
> Also another point: For the faulty slots, the duration is consistently 15
> mins and some seconds, we couldn’t find out where this 15 mins response
> time comes from. And each time it is a different task manager, not always
> the same one.
>
> Do you guys aware of any other users having similar issues with the new
> version and also a suggested bug fix or solution?
>
>
>
>
> Thanks in advance,
> Bekir Oguz
>

[jira] [Created] (FLINK-13302) DateTimeUtils.unixDateCeil returns the same value as unixDateFloor does

[Zhenghua Gao (JIRA)]

Zhenghua Gao created FLINK-13302:


 Summary: DateTimeUtils.unixDateCeil returns the same value as 
unixDateFloor does
 Key: FLINK-13302
 URL: https://issues.apache.org/jira/browse/FLINK-13302
 Project: Flink
  Issue Type: Bug
  Components: Table SQL / Legacy Planner
Affects Versions: 1.9.0, 1.10.0
Reporter: Zhenghua Gao
Assignee: Zhenghua Gao
 Fix For: 1.9.0, 1.10.0


Internally, unixDateCeil & unixDateFloor call julianDateFloor and pass a 
boolean parameter to differentiate them. But unixDateCeil passes wrong 
parameter value and returns the same value as unixDateFloor does.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

Application for contributor permission

[1530130567]

Hi??


I want to conrtibute to Apache Flink.
Would you please give me the contributor permission?
My JIRA ID is zhiyezou.

[jira] [Created] (FLINK-13301) Some PlannerExpression resultType is not consistent with Calcite Type inference

[Jing Zhang (JIRA)]

Jing Zhang created FLINK-13301:
--

 Summary: Some PlannerExpression resultType is not consistent with 
Calcite Type inference
 Key: FLINK-13301
 URL: https://issues.apache.org/jira/browse/FLINK-13301
 Project: Flink
  Issue Type: Task
  Components: Table SQL / API
Reporter: Jing Zhang


Some PlannerExpression resultType is not consistent with Calcite Type 
inference. The problem could be happened when run  the following example: 

{code:java}
// prepare source Data
val testData = new mutable.MutableList[(Int)]
testData.+=((3))
val t = env.fromCollection(testData).toTable(tEnv).as('a)

// register a TableSink
val fieldNames = Array("f0")
val fieldTypes: Array[TypeInformation[_]] = Array(Types.INT())
//val fieldTypes: Array[TypeInformation[_]] = Array(Types.LONG())
val sink = new MemoryTableSourceSinkUtil.UnsafeMemoryAppendTableSink
tEnv.registerTableSink("targetTable", sink.configure(fieldNames, 
fieldTypes))

t.select('a.floor()).insertInto("targetTable")

env.execute()
{code}

The cause is ResultType of `floor` is LONG_TYPE_INFO, while in Calcite 
`SqlFloorFunction` infers returnType is the type of the first argument(INT in 
the above case).
If I change `fieldTypes` to Array(Types.INT()), the following error will be 
thrown in compile phase.

{code:java}
org.apache.flink.table.api.ValidationException: Field types of query result and 
registered TableSink [targetTable] do not match.
Query result schema: [_c0: Long]
TableSink schema:[f0: Integer]

at 
org.apache.flink.table.sinks.TableSinkUtils$.validateSink(TableSinkUtils.scala:59)
at 
org.apache.flink.table.planner.StreamPlanner$$anonfun$2.apply(StreamPlanner.scala:158)
at 
org.apache.flink.table.planner.StreamPlanner$$anonfun$2.apply(StreamPlanner.scala:157)
at scala.Option.map(Option.scala:146)
at 
org.apache.flink.table.planner.StreamPlanner.org$apache$flink$table$planner$StreamPlanner$$translate(StreamPlanner.scala:157)
at 
org.apache.flink.table.planner.StreamPlanner$$anonfun$translate$1.apply(StreamPlanner.scala:129)
at 
org.apache.flink.table.planner.StreamPlanner$$anonfun$translate$1.apply(StreamPlanner.scala:129)
at 
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
at 
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
at scala.collection.Iterator$class.foreach(Iterator.scala:891)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1334)
at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
{code}

And If I change `fieldTypes` to Array(Types.LONG()), the other error will be 
thrown in runtime.

{code:java}
org.apache.flink.table.api.TableException: Result field does not match 
requested type. Requested: Long; Actual: Integer

at 
org.apache.flink.table.planner.Conversions$$anonfun$generateRowConverterFunction$2.apply(Conversions.scala:103)
at 
org.apache.flink.table.planner.Conversions$$anonfun$generateRowConverterFunction$2.apply(Conversions.scala:98)
at 
scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:186)
at 
org.apache.flink.table.planner.Conversions$.generateRowConverterFunction(Conversions.scala:98)
at 
org.apache.flink.table.planner.DataStreamConversions$.getConversionMapper(DataStreamConversions.scala:135)
at 
org.apache.flink.table.planner.DataStreamConversions$.convert(DataStreamConversions.scala:91)
{code}

Above inconsistent problem also exists in `Floor`, `Ceil`, `Mod` and so on.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

Re: Add relative path support in Savepoint Connector

[bupt_ljy]

Hi again,
Anyone has any opinion on this topic?


Best Regards,
Jiayi Liao


Original Message
Sender:bupt_ljybupt_...@163.com
Recipient:dev...@flink.apache.org
Cc:Tzu-Li (Gordon) taitzuli...@apache.org
Date:Tuesday, Jul 16, 2019 15:24
Subject:Add relative path support in Savepoint Connector


Hi all,
Firstly I appreciate Gordon and Seth’s effort on this feature, which is 
really helpful to our production use. Like you mentioned in the FLINK-12047, 
one of the production uses is that we use the existing state to derive new 
state. However, since the state handle is using the absolute path to get the 
input stream, we need to directly operate the state in production environment, 
which is not an anxiety-reducing situation, at least for me.
So I wonder if we can add the relative path support in this module because 
the files are persisted in a directory after we take a savepoint, which makes 
it achievable. I’m not sure whether my scenario is a common case or not, but I 
think I can give my contributions if you all are okay about this.




Best Regards,
Jiayi Liao