[jira] [Created] (FLINK-13185) Bump Calcite dependency to 1.20.0 in sql parser & flink planner

[godfrey he (JIRA)]

godfrey he created FLINK-13185:
--

 Summary: Bump Calcite dependency to 1.20.0 in sql parser & flink 
planner
 Key: FLINK-13185
 URL: https://issues.apache.org/jira/browse/FLINK-13185
 Project: Flink
  Issue Type: Improvement
  Components: Table SQL / Legacy Planner
Reporter: godfrey he
Assignee: godfrey he


blink planner had upgraded calcite version to 1.20.0 (before version is 
1.19.0), and blink planner will support DDL in FLINK-1.9 which depends on 
flink-sql-parser. so calcite version in flink-sql-parser should also be upgrade 
to 1.20.0.

[~walterddr], [FLINK-11935|https://issues.apache.org/jira/browse/FLINK-11935] 
will not be fixed in this issue, because supporting DDL in blink planner is 
blocked by this.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Best configurations for bounded and unbounded streams pipelines

[Cam Mach]

Hello Flink experts,
 
I believe the question below has been already asked, but since I couldn't find 
my answer from internet, I'd love to reach out the community for help. 

We basically want to find out the best configurations for Flink that running on 
Kubernetes to achieve the best performance. Thinks like what are the parameters 
to tun e.g. number of Task Manager? number of task slot? parallelism?  

Our use case:
We have around one terabyte of data from legacy systems, and want to stream 
them to cloud. Our pipeline has 2 sources (one from Kinesis, and the other from 
SQL), one operator (that join the two sources by key), and a sink  
We like to enable RocksDb and checkpointing to S3. We're also looking for what 
is the best windowing strategy that can be applied in this scenario?

Assuming resources is not a constraints (since we can scale out easily in AWS's 
Kubernetes)

Appreciate if you can help or give us some pointers.

Thanks,

[jira] [Created] (FLINK-13184) Support launching task executors with multi-thread on YARN.

[Xintong Song (JIRA)]

Xintong Song created FLINK-13184:


 Summary: Support launching task executors with multi-thread on 
YARN.
 Key: FLINK-13184
 URL: https://issues.apache.org/jira/browse/FLINK-13184
 Project: Flink
  Issue Type: Bug
  Components: Deployment / YARN
Affects Versions: 1.8.1, 1.9.0
Reporter: Xintong Song
Assignee: Xintong Song


Currently, YarnResourceManager starts all task executors in main thread. This 
could cause RM thread becomes unresponsive when launching a large number of TEs 
(e.g. > 1000), leading to TE registration/heartbeat timeouts.

 

In Blink, we have a thread pool that RM starts TEs through the YARN NMClient in 
separated threads. I think we should add this feature to the Flink master 
branch as well.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (FLINK-13183) Add PrintTableSink for Table & SQL APi

[hehuiyuan (JIRA)]

hehuiyuan created FLINK-13183:
-

 Summary: Add PrintTableSink for Table & SQL APi
 Key: FLINK-13183
 URL: https://issues.apache.org/jira/browse/FLINK-13183
 Project: Flink
  Issue Type: Wish
  Components: Table SQL / API
Reporter: hehuiyuan






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (FLINK-13182) fix spelling mistake in flink-table/flink-sql-client

[Lun Zhang (JIRA)]

Lun Zhang created FLINK-13182:
-

 Summary: fix spelling mistake in flink-table/flink-sql-client
 Key: FLINK-13182
 URL: https://issues.apache.org/jira/browse/FLINK-13182
 Project: Flink
  Issue Type: Bug
  Components: Table SQL / Client
Affects Versions: 1.9.0
Reporter: Lun Zhang


fix:

EXECUTION_CURRNET_DATABASE ->  EXECUTION_CURRENT_DATABASE

 

EXECUTION_CURRNET_CATALOG ->  EXECUTION_CURRENT_CATALOG

 

and rename the usage of code reference.

 

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (FLINK-13181) Add a constructor function to CsvTableSink

[hehuiyuan (JIRA)]

hehuiyuan created FLINK-13181:
-

 Summary: Add a constructor function to CsvTableSink
 Key: FLINK-13181
 URL: https://issues.apache.org/jira/browse/FLINK-13181
 Project: Flink
  Issue Type: Improvement
  Components: Table SQL / API
Reporter: hehuiyuan


Add a constructor function for parameters :
  @param path   The output path to write the Table to.
  @param fieldDelim The field delimiter
  @param writeMode  The write mode to specify whether existing files 
are overwritten or not.
 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (FLINK-13180) Add a constructor function to CsvTableSink

[hehuiyuan (JIRA)]

hehuiyuan created FLINK-13180:
-

 Summary: Add a constructor function to CsvTableSink
 Key: FLINK-13180
 URL: https://issues.apache.org/jira/browse/FLINK-13180
 Project: Flink
  Issue Type: Improvement
  Components: Table SQL / API
Reporter: hehuiyuan


Add a constructor function to CsvTableSink.

For parameter :
 @param path   The output path to write the Table to.
 @param fieldDelim The field delimiter
 @param writeMode  The write mode to specify whether existing files are 
overwritten or not.
 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (FLINK-13179) Add document on how to run examples.

[Jiangjie Qin (JIRA)]

Jiangjie Qin created FLINK-13179:


 Summary: Add document on how to run examples.
 Key: FLINK-13179
 URL: https://issues.apache.org/jira/browse/FLINK-13179
 Project: Flink
  Issue Type: Improvement
  Components: Documentation, Examples
Reporter: Jiangjie Qin


Some of the examples in Flink do not have sufficient document so users may not 
able to run them easily. We can probably have a shell script to run the 
examples. The shell script should include necessary libraries and maybe log4j 
configs. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [FLIP-47] Savepoints vs Checkpoints

[Becket Qin]

Hi Kostas,

It makes a lot of sense to just have one underlying mechanism (snapshot) to
save the state of a Flink job. And we can use that mechanism in different
scenarios, including checkpoint and user-triggered savepoint.

To facilitate the discussion, maybe it is useful to clarify a few design
goals, for example:

1. one unified snapshot format that supports
 - both incremental and global state saving
 - rescaling on recovery
 - compatibility check / migration across different Flink versions?
2. The snapshot can easily be managed by users.


And I have two questions regarding the FLIP.

1. What are the side-effects when taking a snapshot? Do you mean taking
snapshot may triggers some action other than saving the state of the Job.
Technically speaking, taking snapshot should be a "read-only" action to the
Flink jobs. So I assume by side-effects, you meant it's no-longer
read-only. If so, can you be more specific on what are the side-effects you
are referring to?

2. In the rejected alternative, you mentioned a scenario of AB testing. It
seems that if execution A and execution B runs different configurations
after the savepoints, the history of the two jobs will always be different
after that, right?

Thanks,

Jiangjie (Becket) Qin

On Mon, Jul 8, 2019 at 9:53 PM Kostas Kloudas  wrote:

> Hi Devs,
>
> Currently there is a number of efforts around checkpoints/savepoints, as
> reflected by the number of FLIPs. From a quick look FLIP-34, FLIP-41,
> FLIP-43, and FLIP-45 are all directly related to these topics. This
> reflects the importance of these two notions/features to the users of the
> framework.
>
> Although many efforts are centred around these notions, their semantics and
> the interplay between them is not always clearly defined. This makes them
> difficult to explain them to the users (all the different combinations of
> state-backends, formats and tradeoffs) and in some cases it may have
> negative effects to the users (e.g. the already-fixed-some-time-ago issue
> of savepoints not being considered for recovery although they committed
> side-effects).
>
> FLIP-47 [1] and the related Document [2] is aiming at starting a discussion
> around the semantics of savepoints/checkpoints and their interplay, and to
> some extent help us fix the future steps concerning these notions. As an
> example, should we work towards bringing them closer, or moving them
> further apart.
>
> This is not a complete proposal (by no means), as many of the practical
> implications can only be fleshed out after we agree on the basic semantics
> and the general frame around these notions. To that end, there are no
> concrete implementation steps and the FLIP is going to be updated as the
> discussion continues.
>
> I am really looking forward to your opinions on the topic.
>
> Cheers,
> Kostas
>
> [1]
>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-47%3A+Checkpoints+vs.+Savepoints
> [2]
>
> https://docs.google.com/document/d/1_1FF8D3u0tT_zHWtB-hUKCP_arVsxlmjwmJ-TvZd4fs/edit?usp=sharing
>

Re: [RESULT][VOTE] Migrate to sponsored Travis account

[Kurt Young]

Thanks for all your efforts Chesnay, it indeed improves a lot for our
develop experience. BTW, do you know how to find the master branch
information which the CI runs with?

For example, like this one:
https://travis-ci.com/flink-ci/flink/jobs/214542568
It shows pass with the commits, which rebased on the master when the CI
is triggered. But it's both possible that the master branch CI runs on is
the
same or different with current master. If it's the same, I can simply rely
on the
passed information to push commits, but if it's not, I think i should find
another
way to re-trigger tests based on the newest master.

Do you know where can I get such information?

Best,
Kurt


On Tue, Jul 9, 2019 at 3:27 AM Chesnay Schepler  wrote:

> The kinks have been worked out; the bot is running again and pr builds
> are yet again no longer running on ASF resources.
>
> PRs are mirrored to: https://github.com/flink-ci/flink
> Bot source: https://github.com/flink-ci/ci-bot
>
> On 08/07/2019 17:14, Chesnay Schepler wrote:
> > I have temporarily re-enabled running PR builds on the ASF account;
> > migrating to the Travis subscription caused some issues in the bot
> > that I have to fix first.
> >
> > On 07/07/2019 23:01, Chesnay Schepler wrote:
> >> The vote has passed unanimously in favor of migrating to a separate
> >> Travis account.
> >>
> >> I will now set things up such that no PullRequest is no longer run on
> >> the ASF servers.
> >> This is a major setup in reducing our usage of ASF resources.
> >> For the time being we'll use free Travis plan for flink-ci (i.e. 5
> >> workers, which is the same the ASF gives us). Over the course of the
> >> next week we'll setup the Ververica subscription to increase this limit.
> >>
> >> From now now, a bot will mirror all new and updated PullRequests to a
> >> mirror repository (https://github.com/flink-ci/flink-ci) and write an
> >> update into the PR once the build is complete.
> >> I have ran the bots for the past 3 days in parallel to our existing
> >> Travis and it was working without major issues.
> >>
> >> The biggest change that contributors will see is that there's no
> >> longer a icon next to each commit. We may revisit this in the future.
> >>
> >> I'll setup a repo with the source of the bot later.
> >>
> >> On 04/07/2019 10:46, Chesnay Schepler wrote:
> >>> I've raised a JIRA
> >>> with INFRA to
> >>> inquire whether it would be possible to switch to a different Travis
> >>> account, and if so what steps would need to be taken.
> >>> We need a proper confirmation from INFRA since we are not in full
> >>> control of the flink repository (for example, we cannot access the
> >>> settings page).
> >>>
> >>> If this is indeed possible, Ververica is willing sponsor a Travis
> >>> account for the Flink project.
> >>> This would provide us with more than enough resources than we need.
> >>>
> >>> Since this makes the project more reliant on resources provided by
> >>> external companies I would like to vote on this.
> >>>
> >>> Please vote on this proposal, as follows:
> >>> [ ] +1, Approve the migration to a Ververica-sponsored Travis
> >>> account, provided that INFRA approves
> >>> [ ] -1, Do not approach the migration to a Ververica-sponsored
> >>> Travis account
> >>>
> >>> The vote will be open for at least 24h, and until we have
> >>> confirmation from INFRA. The voting period may be shorter than the
> >>> usual 3 days since our current is effectively not working.
> >>>
> >>> On 04/07/2019 06:51, Bowen Li wrote:
>  Re: > Are they using their own Travis CI pool, or did the switch to
>  an entirely different CI service?
> 
>  I reached out to Wes and Krisztián from Apache Arrow PMC. They are
>  currently moving away from ASF's Travis to their own in-house metal
>  machines at [1] with custom CI application at [2]. They've seen
>  significant improvement w.r.t both much higher performance and
>  basically no resource waiting time, "night-and-day" difference
>  quoting Wes.
> 
>  Re: > If we can just switch to our own Travis pool, just for our
>  project, then this might be something we can do fairly quickly?
> 
>  I believe so, according to [3] and [4]
> 
> 
>  [1] https://ci.ursalabs.org/ 
>  [2] https://github.com/ursa-labs/ursabot
>  [3]
> 
> https://docs.travis-ci.com/user/migrate/open-source-repository-migration
> 
>  [4]
>  https://docs.travis-ci.com/user/migrate/open-source-on-travis-ci-com
> 
> 
> 
>  On Wed, Jul 3, 2019 at 12:01 AM Chesnay Schepler
>  mailto:ches...@apache.org>> wrote:
> 
>  Are they using their own Travis CI pool, or did the switch to an
>  entirely different CI service?
> 
>  If we can just switch to our own Travis pool, just for our
>  project, then
>  this might be something we can do fairly quickly?
> 

[jira] [Created] (FLINK-13178) persist Flink's functions to catalog

[Bowen Li (JIRA)]

Bowen Li created FLINK-13178:


 Summary: persist Flink's functions to catalog
 Key: FLINK-13178
 URL: https://issues.apache.org/jira/browse/FLINK-13178
 Project: Flink
  Issue Type: Sub-task
  Components: Table SQL / API
Reporter: Bowen Li






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (FLINK-13177) make FunctionCatalog able to reference functions across catalogs

[Bowen Li (JIRA)]

Bowen Li created FLINK-13177:


 Summary: make FunctionCatalog able to reference functions across 
catalogs
 Key: FLINK-13177
 URL: https://issues.apache.org/jira/browse/FLINK-13177
 Project: Flink
  Issue Type: Sub-task
  Components: Table SQL / API
Reporter: Bowen Li
Assignee: Bowen Li






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Flink 1.8.1 release tag missing?

[Bekir Oguz]

Thanks a lot guys for your quick response and fix.
I am running a local release on that tag now.

Best regards,
Bekir

> On 9 Jul 2019, at 21:19, jincheng sun  wrote:
> 
> Thanks Bekir Oguz and Chesnay!
> 
> Sorry for that, I forgot push the tag, I've pushed the tag to the repo
> now.  https://github.com/apache/flink/tree/release-1.8.1
> Thanks again, and I'm very sorry for my negligence has caused confusion in
> your use.
> 
> Thanks,
> Jincheng
> 
> Bekir Oguz  于2019年7月10日周三 上午12:50写道：
> 
>> Hi,
>> I would like to build the 1.8.1 version of the flink-connector-kinesis
>> module but cannot find the release tag in GitHub repo.
>> I see release candidate 1 (release-1.8.1-rc1) tag, but not sure whether
>> this consists of all the 40 bug fixes in 1.8.1 or not.
>> 
>> Which hash or tag should I use to release the flink-connector-kinesis
>> module?
>> 
>> Regards,
>> Bekir Oguz
>> 
>> 

[jira] [Created] (FLINK-13176) remember current catalog and database in SQL CLI SessionContext

[Bowen Li (JIRA)]

Bowen Li created FLINK-13176:


 Summary: remember current catalog and database in SQL CLI 
SessionContext
 Key: FLINK-13176
 URL: https://issues.apache.org/jira/browse/FLINK-13176
 Project: Flink
  Issue Type: Sub-task
  Components: Table SQL / Client
Reporter: Bowen Li
Assignee: Bowen Li
 Fix For: 1.9.0


currently the EnvironmentInstance/TableEnvironment in SQL CLI is not reused, 
it's recreated for all SQL commands. The resulting problem is that we lost 
state of the user configured current catalog/database. We believe users should 
be able to run 'USE CATALOG/DATABASE xxx' to change their current 
catalog/database despite the ones set in yaml files.

The core design was that the SQL Client "Gateway" is stateless and the SQL 
Client "CLI" knows everything that is required to submit a SQL job. So only one 
request is sent with all information necessary. `USE CATALOG/DATABASE xxx` 
should be executed in the CLI and stored in the CLI's session context. The 
session context has higher priority than the YAML file. 

For `USE CATALOG/DATABASE xxx` use case the current design should be sufficient 
as the `USE` would just modify an `execution` property in the session context



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (FLINK-13175) FLINK-12951 breaks SQL CLI's ExecutionContextTest

[Bowen Li (JIRA)]

Bowen Li created FLINK-13175:


 Summary: FLINK-12951 breaks SQL CLI's ExecutionContextTest
 Key: FLINK-13175
 URL: https://issues.apache.org/jira/browse/FLINK-13175
 Project: Flink
  Issue Type: Bug
  Components: Table SQL / Client
Affects Versions: 1.9.0
Reporter: Bowen Li
Assignee: Danny Chan
 Fix For: 1.9.0


https://github.com/apache/flink/pull/8844 breaks SQL CLI's ExecutionContextTest

Errors from it's CI in https://travis-ci.com/flink-ci/flink/jobs/214370966

{code:java}
14:23:25.985 [ERROR] Tests run: 6, Failures: 0, Errors: 2, Skipped: 0, Time 
elapsed: 19.518 s <<< FAILURE! - in 
org.apache.flink.table.client.gateway.local.ExecutionContextTest
14:23:25.985 [ERROR] 
testDatabases(org.apache.flink.table.client.gateway.local.ExecutionContextTest) 
 Time elapsed: 10.807 s  <<< ERROR!
org.apache.flink.table.client.gateway.SqlExecutionException: Could not create 
environment instance.
at 
org.apache.flink.table.client.gateway.local.ExecutionContextTest.testDatabases(ExecutionContextTest.java:128)
Caused by: org.apache.flink.table.client.gateway.SqlExecutionException: 
Invalid view 'TestView1' with query:
SELECT scalarUDF(IntegerField1) FROM 
default_catalog.default_database.TableNumber1
Cause: SQL validation failed. From line 1, column 38 to line 1, column 82: 
Object 'TableNumber1' not found within 'default_catalog.default_database'
at 
org.apache.flink.table.client.gateway.local.ExecutionContextTest.testDatabases(ExecutionContextTest.java:128)
14:23:25.985 [ERROR] 
testCatalogs(org.apache.flink.table.client.gateway.local.ExecutionContextTest)  
Time elapsed: 5.295 s  <<< ERROR!
org.apache.flink.table.client.gateway.SqlExecutionException: Could not create 
environment instance.
at 
org.apache.flink.table.client.gateway.local.ExecutionContextTest.testCatalogs(ExecutionContextTest.java:87)
Caused by: org.apache.flink.table.client.gateway.SqlExecutionException: 
Invalid view 'TestView1' with query:
SELECT scalarUDF(IntegerField1) FROM 
default_catalog.default_database.TableNumber1
Cause: SQL validation failed. From line 1, column 38 to line 1, column 82: 
Object 'TableNumber1' not found within 'default_catalog.default_database'
at 
org.apache.flink.table.client.gateway.local.ExecutionContextTest.testCatalogs(ExecutionContextTest.java:87)
{code}




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Flink 1.8.1 release tag missing?

[jincheng sun]

Thanks Bekir Oguz and Chesnay!

Sorry for that, I forgot push the tag, I've pushed the tag to the repo
now.  https://github.com/apache/flink/tree/release-1.8.1
Thanks again, and I'm very sorry for my negligence has caused confusion in
your use.

Thanks,
Jincheng

Bekir Oguz  于2019年7月10日周三 上午12:50写道：

> Hi,
> I would like to build the 1.8.1 version of the flink-connector-kinesis
> module but cannot find the release tag in GitHub repo.
> I see release candidate 1 (release-1.8.1-rc1) tag, but not sure whether
> this consists of all the 40 bug fixes in 1.8.1 or not.
>
> Which hash or tag should I use to release the flink-connector-kinesis
> module?
>
> Regards,
> Bekir Oguz
>
>

[jira] [Created] (FLINK-13174) Changes of HiveTableSinkTest in FLINK-13068 introduces compilation error

[Yu Li (JIRA)]

Yu Li created FLINK-13174:
-

 Summary: Changes of HiveTableSinkTest in FLINK-13068 introduces 
compilation error
 Key: FLINK-13174
 URL: https://issues.apache.org/jira/browse/FLINK-13174
 Project: Flink
  Issue Type: Bug
  Components: Connectors / Hive, Tests
Reporter: Yu Li
Assignee: Rui Li


>From the latest run of our 
>[flink-benchmark|http://codespeed.dak8s.net:8080/job/flink-master-benchmarks/4075/console],
> we could see error like below:
{noformat}
[ERROR] Failed to execute goal 
org.apache.maven.plugins:maven-compiler-plugin:3.8.0:testCompile 
(default-testCompile) on project flink-connector-hive_2.11: Compilation 
failure: Compilation failure: 
[ERROR] 
/home/jenkins/workspace/flink-master-benchmarks/flink/flink-connectors/flink-connector-hive/src/test/java/org/apache/flink/batch/connectors/hive/HiveTableSinkTest.java:[230,17]
 cannot find symbol
[ERROR]   symbol:   class CatalogPartition
[ERROR]   location: class 
org.apache.flink.batch.connectors.hive.HiveTableSinkTest
[ERROR] 
/home/jenkins/workspace/flink-master-benchmarks/flink/flink-connectors/flink-connector-hive/src/test/java/org/apache/flink/batch/connectors/hive/HiveTableSinkTest.java:[232,81]
 cannot find symbol
[ERROR]   symbol:   variable HiveCatalogConfig
[ERROR]   location: class 
org.apache.flink.batch.connectors.hive.HiveTableSinkTest
[ERROR] 
/home/jenkins/workspace/flink-master-benchmarks/flink/flink-connectors/flink-connector-hive/src/test/java/org/apache/flink/batch/connectors/hive/HiveTableSinkTest.java:[233,39]
 cannot find symbol
[ERROR]   symbol:   class Path
[ERROR]   location: class 
org.apache.flink.batch.connectors.hive.HiveTableSinkTest
[ERROR] -> [Help 1]
{noformat}
And checking the commit history this issue is introduced by FLINK-13068



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Flink 1.8.1 release tag missing?

[Bekir Oguz]

Hi,
I would like to build the 1.8.1 version of the flink-connector-kinesis module 
but cannot find the release tag in GitHub repo. 
I see release candidate 1 (release-1.8.1-rc1) tag, but not sure whether this 
consists of all the 40 bug fixes in 1.8.1 or not. 

Which hash or tag should I use to release the flink-connector-kinesis module?

Regards,
Bekir Oguz

Re: Flink 1.8.1 release tag missing?

[Chesnay Schepler]

Yes, it appears the 1.8.1 tag is missing. The 1.8.1-rc1 tag is an 
equivalent that you can use for the time being.


Alternatively you could of course also use the source release to build 
the connector.


@jincheng Could you add the proper 1.8.1 tag?

On 09/07/2019 16:38, Bekir Oguz wrote:

Hi,
I would like to build the 1.8.1 version of the flink-connector-kinesis module 
but cannot find the release tag in GitHub repo.
I see release candidate 1 (release-1.8.1-rc1) tag, but not sure whether this 
consists of all the 40 bug fixes in 1.8.1 or not.

Which hash or tag should I use to release the flink-connector-kinesis module?

Regards,
Bekir Oguz

[jira] [Created] (FLINK-13173) Only run openSSL tests if desired

[Nico Kruber (JIRA)]

Nico Kruber created FLINK-13173:
---

 Summary: Only run openSSL tests if desired
 Key: FLINK-13173
 URL: https://issues.apache.org/jira/browse/FLINK-13173
 Project: Flink
  Issue Type: Sub-task
  Components: Runtime / Network, Tests, Travis
Affects Versions: 1.9.0
Reporter: Nico Kruber
Assignee: Nico Kruber
 Fix For: 1.9.0


Rename {{flink.tests.force-openssl}} to {{flink.tests.with-openssl}} and only 
run openSSL-based unit tests if this is set. This way, we avoid systems where 
the bundled dynamic libraries do not work. Travis seems to run fine and will 
have this property set.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (FLINK-13172) JVM crash with dynamic netty-tcnative wrapper to openSSL on some OS

[Nico Kruber (JIRA)]

Nico Kruber created FLINK-13172:
---

 Summary: JVM crash with dynamic netty-tcnative wrapper to openSSL 
on some OS
 Key: FLINK-13172
 URL: https://issues.apache.org/jira/browse/FLINK-13172
 Project: Flink
  Issue Type: Bug
  Components: Runtime / Network, Tests
Affects Versions: 1.9.0
Reporter: Nico Kruber
Assignee: Nico Kruber


The dynamically-linked wrapper library in 
{{flink-shaded-netty-tcnative-dynamic}} may not work on all systems, depending 
on how the system-provided openSSL library is built.
As a result, when trying to run Flink with {{security.ssl.provider: OPENSSL}} 
or running a test based on {{SSLUtilsTest}} (which checks for openSSL 
availability), the JVM will crash, e.g. with
- on SUSE-based systems:
{code}
/usr/lib64/jvm/java-openjdk/bin/java: relocation error: 
/tmp/liborg_apache_flink_shaded_netty4_netty_tcnative_linux_x86_644115489043239307863.so:
 symbol TLSv1_2_server_method version OPENSSL_1.0.1 not defined in file 
libssl.so.1.0.0 with link time reference
{code}
- on Arch Linux:
{code}
/usr/lib/jvm/default/bin/java: relocation error: 
/tmp/liborg_apache_flink_shaded_netty4_netty_tcnative_linux_x86_648476498532937980008.so:
 symbol SSLv3_method version OPENSSL_1.0.0 not defined in file libssl.so.1.0.0 
with link time reference
{code}

Possible solutions:
# build your own OS-dependent dynamically-linked {{netty-tcnative}} library and 
shade it in your own build of {{flink-shaded-netty-tcnative-dynamic}}, or
# use {{flink-shaded-netty-tcnative-static}}:
{code}
git clone https://github.com/apache/flink-shaded.git
cd flink-shaded
mvn clean package -Pinclude-netty-tcnative-static -pl 
flink-shaded-netty-tcnative-static
{code}
# get your OS-dependent build into netty-tcnative as a special branch similar 
to what they currently do with Fedora-based systems



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (FLINK-13171) Add withParameters(Configuration) method to DataStream operators

[Gyula Fora (JIRA)]

Gyula Fora created FLINK-13171:
--

 Summary: Add withParameters(Configuration) method to DataStream 
operators
 Key: FLINK-13171
 URL: https://issues.apache.org/jira/browse/FLINK-13171
 Project: Flink
  Issue Type: New Feature
  Components: API / DataStream
Reporter: Gyula Fora


The DataStream API doesn't have the appropriate methods to configure user 
functions so Rich functions cannot leverage the parameters passed in the open 
methods similar to the DataSet API.

 

I think this would be a simple but valuable addition.

 

[https://ci.apache.org/projects/flink/flink-docs-release-1.8/dev/batch/#via-withparametersconfiguration]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Flink 1.8.1 release tag missing?

[Bekir Oguz]

Hi,
I would like to build the 1.8.1 version of the flink-connector-kinesis module 
but cannot find the release tag in GitHub repo. 
I see release candidate 1 (release-1.8.1-rc1) tag, but not sure whether this 
consists of all the 40 bug fixes in 1.8.1 or not. 

Which hash or tag should I use to release the flink-connector-kinesis module?

Regards,
Bekir Oguz

Re: [DISCUSS] META-FLIP: Sticking (or not) to a strict FLIP voting process

[Becket Qin]

Hi Aljoscha,

Thanks for the quick response. Yes, you are right. I meant "Voted and
accepted FLIPs should be immutable". Sorry for the confusion.

Thanks,

Jiangjie (Becket) Qin

On Tue, Jul 9, 2019 at 10:09 PM Aljoscha Krettek 
wrote:

> +1 to what Becket said.
>
> I have one comment on 5.: I think you meant that they should be immutable
> once they have been voted on and accepted. During the initial proposal and
> discussion they will change, of course. At least that’s what I think
>
> Aljoscha
>
> > On 9. Jul 2019, at 11:29, Becket Qin  wrote:
> >
> > This discussion thread has been quiet for some time. It looks most people
> > think sticking to a strict voting process is a good idea.
> >
> > In addition to that, there are a few related details that are also
> > discussed, I listed them below and personally I am +1 on all of them.
> >
> > 1. Stick to the current definition of major changes.
> > 2. Committers have binding votes on FLIPs.
> > 3. In general FLIPs should be completed in a reasonable amount of time,
> > rather than lasting for many releases.
> > 4. Always discuss FLIPs based on FLIP wiki page. Google docs can be used
> as
> > handy tools to explain ideas, but FLIP ideas should always be documented
> as
> > a FLIP wiki, regardless whether they are accepted or not.
> > 5. FLIPs should be immutable. Changes to FLIPs need a new vote processes.
> > 6. FLIPs should be concrete in terms of interfaces, semantic and
> behaviors,
> > rather than conceptual.
> >
> > It'll be good to hear what do people think. If there is no further
> > objection, I'd suggest to conclude this discussion in 72 hours and move
> to
> > a lazy majority voting. (For decisions like this, maybe only votes from
> > PMCs should be considered as binding?)
> >
> > Thanks,
> >
> > Jiangjie (Becket) Qin
> >
> > On Fri, Jun 28, 2019 at 10:33 AM Congxian Qiu 
> > wrote:
> >
> >> Thanks a lot for bringing this up, Aljoscha.
> >>
> >> +1 for sticking to the "lazy majority" vote process.
> >>
> >> In my opinion, after the "lazy majority" vote process, we will have a
> >> community consensus about the accepted FLIP.
> >>
> >> Best,
> >> Congxian
> >>
> >>
> >> Becket Qin  于2019年6月28日周五 上午10:06写道：
> >>
> >>> Thanks a lot for bringing this up, Aljoscha.
> >>>
> >>> Big +1 to the following:
> >>>
> >>> 1. Stick to a strict FLIP voting process.
> >>> In practice, I rarely see a FLIP with a voting thread. In fact, the
> >> search
> >>> in mail archive
> >>> <
> >>>
> >>
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/template/NamlServlet.jtp?macro=search_page=1=subject%3AVOTE%2CFLIP=0
> 
> >>> gives
> >>> only 3 FLIPs with voting thread, and unfortunately none of them has met
> >> the
> >>> lazy majority requirements, which needs 3 binding votes. However, we
> have
> >>> 11 adopted-but-unreleased FLIPs and 16 released FLIPs.
> >>> Even though we claimed *"These proposals are more serious than code
> >> changes
> >>> and more serious even than release votes.", *we did not really treat
> them
> >>> seriously. The missing voting process effectively put the efforts of
> FLIP
> >>> in vain. This leads to a few consequences:
> >>> a) The conclusion of the FLIP is never really finalized. People may
> >> change
> >>> the FLIP at wish during the implementation.
> >>> b) Some "adopted" FLIPs only have conceptual ideas instead of necessary
> >>> concrete interfaces, which leaves a lot of problems in the
> implementation
> >>> phase.
> >>> c) New contributors are completely confused on how to contribute. The
> >>> voting threads seems died, and magically someone else's code got
> checked
> >> in
> >>> without a passed FLIP. These "good citizens" may feel excluded and
> simply
> >>> leave the chaos.
> >>> d) API changes / user sensible behavior changes may be checked in
> without
> >>> being carefully inspected. To fix them, hacky tricks has to be made in
> >>> order to keep backwards compatibility.
> >>>
> >>> So a huge +1 to stick to the FLIP voting process.
> >>>
> >>> 2. Stick to the definition of major changes. Generally speaking any
> user
> >>> sensible changes should go through a FLIP.
> >>>- Some changes may be small from the size of patch perspective, but
> >> the
> >>> impact could be huge. Take metric as an example, imagine a cloud
> service
> >>> provider who relies on a metric to do alerting or bill their customer.
> >> Any
> >>> change to such metrics will have huge impact on them.
> >>>- Sometimes there might be no "interface" change per se, but the
> >>> behavior of a method is slightly changed. Even that can be very
> annoying
> >> to
> >>> some users. So I think any user sensible changes should go through a
> >> FLIP.
> >>>
> >>> 3. Generally speaking, make each FLIP completable in a reasonable
> amount
> >> of
> >>> time. Some large changes may need multiple FLIPs.
> >>>   - I agree with David that a long lasting FLIP can be problematic as
> it
> >>> could become obsolete before the work is done. And might 

Re: [ANNOUNCE] Feature freeze for Apache Flink 1.9.0 release

[罗齐]

Hi Gordon,

Will branch 1.9 be cut out today? We're really looking forward to the blink
features in 1.9.

Thanks,
Qi

On Wed, Jun 26, 2019 at 7:18 PM Tzu-Li (Gordon) Tai 
wrote:

> Thanks for the updates so far everyone!
>
> Since support for the new Blink-based Table / SQL runner and fine-grained
> recovery are quite prominent features for 1.9.0,
> and developers involved in these topics have already expressed that these
> could make good use for another week,
> I think it definitely makes sense to postpone the feature freeze.
>
> The new date for feature freeze and feature branch cut for 1.9.0 will be
> *July
> 5*.
>
> Please update on this thread if there are any further concerns!
>
> Cheers,
> Gordon
>
> On Tue, Jun 25, 2019 at 9:05 PM Chesnay Schepler 
> wrote:
>
> > On the fine-grained recovery / batch scheduling side we could make good
> > use of another week.
> > Currently we are on track to have the _feature_ merged, but without
> > having done a great deal of end-to-end testing.
> >
> > On 25/06/2019 15:01, Kurt Young wrote:
> > > Hi Aljoscha,
> > >
> > > I also feel an additional week can make the remaining work more easy.
> At
> > > least
> > > we don't have to check in lots of commits in both branches (master &
> > > release-1.9).
> > >
> > > Best,
> > > Kurt
> > >
> > >
> > > On Tue, Jun 25, 2019 at 8:27 PM Aljoscha Krettek 
> > > wrote:
> > >
> > >> A few threads are converging around supporting the new Blink-based
> Table
> > >> API Runner/Planner. I think hitting the currently proposed feature
> > freeze
> > >> date is hard, if not impossible, and that the work would benefit from
> an
> > >> additional week to get everything in with good quality.
> > >>
> > >> What do the others involved in the topic think?
> > >>
> > >> Aljoscha
> > >>
> > >>> On 24. Jun 2019, at 19:42, Bowen Li  wrote:
> > >>>
> > >>> Hi Gordon,
> > >>>
> > >>> Thanks for driving this effort.
> > >>>
> > >>> Xuefu responded to the discussion thread [1] and I want to bring that
> > to
> > >>> our attention here:
> > >>>
> > >>> Hive integration depends on a few features that are actively
> developed.
> > >> If
> > >>> the completion of those features don't leave enough time for us to
> > >>> integrate, then our work can potentially go beyond the proposed date.
> > >>>
> > >>> Just wanted to point out such a dependency adds uncertainty.
> > >>>
> > >>> [1]
> > >>>
> > >>
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Features-for-Apache-Flink-1-9-0-td28701.html
> > >>> On Thu, Jun 20, 2019 at 1:05 AM Tzu-Li (Gordon) Tai <
> > tzuli...@apache.org
> > >>>
> > >>> wrote:
> > >>>
> >  Hi devs,
> > 
> >  Per the feature discussions for 1.9.0 [1], I hereby announce the
> > >> official
> >  feature freeze for Flink 1.9.0 to be on June 28. A release feature
> > >> branch
> >  for 1.9 will be cut following that date.
> > 
> >  We’re roughly one week away from this date, but please keep in mind
> > >> that we
> >  still shouldn’t rush things. If you feel that there may be problems
> > with
> >  this schedule for the things you are working on, please let us know
> > >> here.
> >  Cheers,
> >  Gordon
> > 
> >  [1]
> > 
> > 
> > >>
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Features-for-Apache-Flink-1-9-0-td28701.html
> > >>
> >
> >
>

Re: [DISCUSS] META-FLIP: Sticking (or not) to a strict FLIP voting process

[Aljoscha Krettek]

+1 to what Becket said.

I have one comment on 5.: I think you meant that they should be immutable once 
they have been voted on and accepted. During the initial proposal and 
discussion they will change, of course. At least that’s what I think

Aljoscha

> On 9. Jul 2019, at 11:29, Becket Qin  wrote:
> 
> This discussion thread has been quiet for some time. It looks most people
> think sticking to a strict voting process is a good idea.
> 
> In addition to that, there are a few related details that are also
> discussed, I listed them below and personally I am +1 on all of them.
> 
> 1. Stick to the current definition of major changes.
> 2. Committers have binding votes on FLIPs.
> 3. In general FLIPs should be completed in a reasonable amount of time,
> rather than lasting for many releases.
> 4. Always discuss FLIPs based on FLIP wiki page. Google docs can be used as
> handy tools to explain ideas, but FLIP ideas should always be documented as
> a FLIP wiki, regardless whether they are accepted or not.
> 5. FLIPs should be immutable. Changes to FLIPs need a new vote processes.
> 6. FLIPs should be concrete in terms of interfaces, semantic and behaviors,
> rather than conceptual.
> 
> It'll be good to hear what do people think. If there is no further
> objection, I'd suggest to conclude this discussion in 72 hours and move to
> a lazy majority voting. (For decisions like this, maybe only votes from
> PMCs should be considered as binding?)
> 
> Thanks,
> 
> Jiangjie (Becket) Qin
> 
> On Fri, Jun 28, 2019 at 10:33 AM Congxian Qiu 
> wrote:
> 
>> Thanks a lot for bringing this up, Aljoscha.
>> 
>> +1 for sticking to the "lazy majority" vote process.
>> 
>> In my opinion, after the "lazy majority" vote process, we will have a
>> community consensus about the accepted FLIP.
>> 
>> Best,
>> Congxian
>> 
>> 
>> Becket Qin  于2019年6月28日周五 上午10:06写道：
>> 
>>> Thanks a lot for bringing this up, Aljoscha.
>>> 
>>> Big +1 to the following:
>>> 
>>> 1. Stick to a strict FLIP voting process.
>>> In practice, I rarely see a FLIP with a voting thread. In fact, the
>> search
>>> in mail archive
>>> <
>>> 
>> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/template/NamlServlet.jtp?macro=search_page=1=subject%3AVOTE%2CFLIP=0
 
>>> gives
>>> only 3 FLIPs with voting thread, and unfortunately none of them has met
>> the
>>> lazy majority requirements, which needs 3 binding votes. However, we have
>>> 11 adopted-but-unreleased FLIPs and 16 released FLIPs.
>>> Even though we claimed *"These proposals are more serious than code
>> changes
>>> and more serious even than release votes.", *we did not really treat them
>>> seriously. The missing voting process effectively put the efforts of FLIP
>>> in vain. This leads to a few consequences:
>>> a) The conclusion of the FLIP is never really finalized. People may
>> change
>>> the FLIP at wish during the implementation.
>>> b) Some "adopted" FLIPs only have conceptual ideas instead of necessary
>>> concrete interfaces, which leaves a lot of problems in the implementation
>>> phase.
>>> c) New contributors are completely confused on how to contribute. The
>>> voting threads seems died, and magically someone else's code got checked
>> in
>>> without a passed FLIP. These "good citizens" may feel excluded and simply
>>> leave the chaos.
>>> d) API changes / user sensible behavior changes may be checked in without
>>> being carefully inspected. To fix them, hacky tricks has to be made in
>>> order to keep backwards compatibility.
>>> 
>>> So a huge +1 to stick to the FLIP voting process.
>>> 
>>> 2. Stick to the definition of major changes. Generally speaking any user
>>> sensible changes should go through a FLIP.
>>>- Some changes may be small from the size of patch perspective, but
>> the
>>> impact could be huge. Take metric as an example, imagine a cloud service
>>> provider who relies on a metric to do alerting or bill their customer.
>> Any
>>> change to such metrics will have huge impact on them.
>>>- Sometimes there might be no "interface" change per se, but the
>>> behavior of a method is slightly changed. Even that can be very annoying
>> to
>>> some users. So I think any user sensible changes should go through a
>> FLIP.
>>> 
>>> 3. Generally speaking, make each FLIP completable in a reasonable amount
>> of
>>> time. Some large changes may need multiple FLIPs.
>>>   - I agree with David that a long lasting FLIP can be problematic as it
>>> could become obsolete before the work is done. And might need to make
>>> changes to the original proposal multiple times. It might be a little
>>> difficult to have a standard to say what kind of FLIP is a long lasting
>>> FLIP.
>>>   - Sometimes long lasting FLIP may be necessary, e.g. a big new module
>> /
>>> functionality, etc. Those FLIPs are rare and usually more independent. We
>>> may need to treat them case by case.
>>> 
>>> 4. Take the votes from both committers and PMCs as binding.
>>> 
>>> 
>>> In 

Apply for being a contributor

[胡伟华]

Hi, 

 I want to contribute to Apache Flink.
 Would you please give me the contributor permission?
 My JIRA ID is huwh, my fullname is HuWeihua.

Thanks a lot.

[jira] [Created] (FLINK-13170) Planner should get table factory from catalog when creating sink for CatalogTable

[Rui Li (JIRA)]

Rui Li created FLINK-13170:
--

 Summary: Planner should get table factory from catalog when 
creating sink for CatalogTable
 Key: FLINK-13170
 URL: https://issues.apache.org/jira/browse/FLINK-13170
 Project: Flink
  Issue Type: Bug
  Components: Table SQL / Planner
Reporter: Rui Li
Assignee: Rui Li






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (FLINK-13169) IT test for fine-grained recovery (task executor failures)

[Andrey Zagrebin (JIRA)]

Andrey Zagrebin created FLINK-13169:
---

 Summary: IT test for fine-grained recovery (task executor failures)
 Key: FLINK-13169
 URL: https://issues.apache.org/jira/browse/FLINK-13169
 Project: Flink
  Issue Type: Sub-task
  Components: Runtime / Coordination
Reporter: Andrey Zagrebin
Assignee: Andrey Zagrebin
 Fix For: 1.9.0


The BatchFineGrainedRecoveryITCase can be extended with an additional test 
failure strategy which abruptly shuts down the task executor. This leads to the 
loss of all previously completed and the in-progress mapper result partitions. 
The fail-over strategy should subsequently restart the current in-progress 
mapper and all previous mappers because the previous result is unavailable. 
When the source is recomputed, all mappers has to be restarted again to 
recalculate their lost results.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (FLINK-13168) clarify isBatch/isStreaming/isBounded flag in flink planner and blink planner

[godfrey he (JIRA)]

godfrey he created FLINK-13168:
--

 Summary: clarify isBatch/isStreaming/isBounded flag in flink 
planner and blink planner
 Key: FLINK-13168
 URL: https://issues.apache.org/jira/browse/FLINK-13168
 Project: Flink
  Issue Type: Improvement
  Components: Table SQL / Legacy Planner, Table SQL / Planner
Reporter: godfrey he
Assignee: godfrey he


in blink planner & flink planner, there are many `isBatch` and `isStreaming` 
flags, they have different meaning in different place. which makes reader and 
coder crazy. especially in blink planner, Only `StreamTableSource` could be 
used for both batch and stream. is `bounded StreamTableSource` means batch, 
`unbounded` means stream ? 

we should make it clear:
1. `isBatch` in `ConnectorCatalogTable`, which tells if the 
tableSource/tableSink is BatchTableSource/BatchTableSink
2. `isStreaming` in `TableSourceTable`, which tells if  if the current table is 
on stream planner
3. `bounded StreamTableSource` could be used for both batch and stream, while 
`unbounded StreamTableSource` could only be used for stream.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (FLINK-13167) Introduce FlinkML import/export framework

[Luo Gen (JIRA)]

Luo Gen created FLINK-13167:
---

 Summary: Introduce FlinkML import/export framework
 Key: FLINK-13167
 URL: https://issues.apache.org/jira/browse/FLINK-13167
 Project: Flink
  Issue Type: Sub-task
  Components: Library / Machine Learning
Reporter: Luo Gen
Assignee: Luo Gen


Import/export a trained model is an important part in machine learning 
lifecycle, especially for serving purpose, hence it triggered a heatedly 
discussion in the previous pull request for FlinkML basic interfaces. Some 
thoughts about this are summarized in the following doc.

https://docs.google.com/document/d/1B84b-1CvOXtwWQ6_tQyiaHwnSeiRqh-V96Or8uHqCp8/edit?usp=sharing

The jira will introduce the import/export framework based on these discussion. 
Import and export framework have almost the same structure so we take export 
for example. Export framework has 3 basic components, including 2 interfaces 
for exporter authors to implement, and one util class for end-users to acquire 
an Exporter.

Interfaces:
 - Exporter: Interface of model exporters. Implementations can export supported 
models to a specific target model type, such as PMML.

- ExporterFactory: Interface of factories to create Exporters. Each 
implementation supports only one model type. Typically an ExporterFactory is 
given a model instance with a property map if needed, and returns an Exporter 
that supports the model to export to target model type. ExporterFactories are 
loaded with ServiceLoader framework of Java, and should be registered in the 
file 
"META-INF/services/org.apache.flink.ml.api.misc.exportation.ExporterFactory".

Utility:
 - ExporterLoader: Loader to get an Exporter for a specific model format and a 
model instance. Users would get Exporters via this class rather than directly 
creating with constructors. This is a concrete util class and no one needs to 
override it.

Besides, a new enumeration named ModelType is also introduced, listing all 
types FlinkML may support. This would be extended when a new exporter support a 
new type.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [DISCUSS] META-FLIP: Sticking (or not) to a strict FLIP voting process

[Becket Qin]

This discussion thread has been quiet for some time. It looks most people
think sticking to a strict voting process is a good idea.

In addition to that, there are a few related details that are also
discussed, I listed them below and personally I am +1 on all of them.

1. Stick to the current definition of major changes.
2. Committers have binding votes on FLIPs.
3. In general FLIPs should be completed in a reasonable amount of time,
rather than lasting for many releases.
4. Always discuss FLIPs based on FLIP wiki page. Google docs can be used as
handy tools to explain ideas, but FLIP ideas should always be documented as
a FLIP wiki, regardless whether they are accepted or not.
5. FLIPs should be immutable. Changes to FLIPs need a new vote processes.
6. FLIPs should be concrete in terms of interfaces, semantic and behaviors,
rather than conceptual.

It'll be good to hear what do people think. If there is no further
objection, I'd suggest to conclude this discussion in 72 hours and move to
a lazy majority voting. (For decisions like this, maybe only votes from
PMCs should be considered as binding?)

Thanks,

Jiangjie (Becket) Qin

On Fri, Jun 28, 2019 at 10:33 AM Congxian Qiu 
wrote:

> Thanks a lot for bringing this up, Aljoscha.
>
> +1 for sticking to the "lazy majority" vote process.
>
> In my opinion, after the "lazy majority" vote process, we will have a
> community consensus about the accepted FLIP.
>
> Best,
> Congxian
>
>
> Becket Qin  于2019年6月28日周五 上午10:06写道：
>
> > Thanks a lot for bringing this up, Aljoscha.
> >
> > Big +1 to the following:
> >
> > 1. Stick to a strict FLIP voting process.
> > In practice, I rarely see a FLIP with a voting thread. In fact, the
> search
> > in mail archive
> > <
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/template/NamlServlet.jtp?macro=search_page=1=subject%3AVOTE%2CFLIP=0
> > >
> > gives
> > only 3 FLIPs with voting thread, and unfortunately none of them has met
> the
> > lazy majority requirements, which needs 3 binding votes. However, we have
> > 11 adopted-but-unreleased FLIPs and 16 released FLIPs.
> > Even though we claimed *"These proposals are more serious than code
> changes
> > and more serious even than release votes.", *we did not really treat them
> > seriously. The missing voting process effectively put the efforts of FLIP
> > in vain. This leads to a few consequences:
> > a) The conclusion of the FLIP is never really finalized. People may
> change
> > the FLIP at wish during the implementation.
> > b) Some "adopted" FLIPs only have conceptual ideas instead of necessary
> > concrete interfaces, which leaves a lot of problems in the implementation
> > phase.
> > c) New contributors are completely confused on how to contribute. The
> > voting threads seems died, and magically someone else's code got checked
> in
> > without a passed FLIP. These "good citizens" may feel excluded and simply
> > leave the chaos.
> > d) API changes / user sensible behavior changes may be checked in without
> > being carefully inspected. To fix them, hacky tricks has to be made in
> > order to keep backwards compatibility.
> >
> > So a huge +1 to stick to the FLIP voting process.
> >
> > 2. Stick to the definition of major changes. Generally speaking any user
> > sensible changes should go through a FLIP.
> > - Some changes may be small from the size of patch perspective, but
> the
> > impact could be huge. Take metric as an example, imagine a cloud service
> > provider who relies on a metric to do alerting or bill their customer.
> Any
> > change to such metrics will have huge impact on them.
> > - Sometimes there might be no "interface" change per se, but the
> > behavior of a method is slightly changed. Even that can be very annoying
> to
> > some users. So I think any user sensible changes should go through a
> FLIP.
> >
> > 3. Generally speaking, make each FLIP completable in a reasonable amount
> of
> > time. Some large changes may need multiple FLIPs.
> >- I agree with David that a long lasting FLIP can be problematic as it
> > could become obsolete before the work is done. And might need to make
> > changes to the original proposal multiple times. It might be a little
> > difficult to have a standard to say what kind of FLIP is a long lasting
> > FLIP.
> >- Sometimes long lasting FLIP may be necessary, e.g. a big new module
> /
> > functionality, etc. Those FLIPs are rare and usually more independent. We
> > may need to treat them case by case.
> >
> > 4. Take the votes from both committers and PMCs as binding.
> >
> >
> > In addition, I'd like to propose the following:
> >
> > 1. Always discuss the FLIP based on a FLIP wiki page instead of a Google
> > doc. It is perfectly fine to use google doc to explain stuff, but the
> FLIP
> > wiki page is the official source for the proposal. The discussion and
> vote
> > needs to be based on that.
> >
> > According to the process of FLIP
> > <
> >
> 

[jira] [Created] (FLINK-13166) Support batch slot requests

[Till Rohrmann (JIRA)]

Till Rohrmann created FLINK-13166:
-

 Summary: Support batch slot requests
 Key: FLINK-13166
 URL: https://issues.apache.org/jira/browse/FLINK-13166
 Project: Flink
  Issue Type: Sub-task
  Components: Runtime / Coordination
Affects Versions: 1.9.0
Reporter: Till Rohrmann
 Fix For: 1.9.0


In order to support the execution of batch jobs with fewer slots than requested 
we need to introduce a special slot request notion which does not register an 
eager timeout. Moreover, this slot request should not react to failure signals 
from the {{ResourceManager}} and only time out if there is not available or 
allocated slot which can fulfill the requested {{ResourceProfile}}.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (FLINK-13165) Complete requested slots in request order

[Till Rohrmann (JIRA)]

Till Rohrmann created FLINK-13165:
-

 Summary: Complete requested slots in request order
 Key: FLINK-13165
 URL: https://issues.apache.org/jira/browse/FLINK-13165
 Project: Flink
  Issue Type: Sub-task
  Components: Runtime / Coordination
Affects Versions: 1.9.0
Reporter: Till Rohrmann
 Fix For: 1.9.0


When executing batch jobs with fewer slots than requested we should make sure 
that the slot requests are being completed in the order in which they were 
enqueued into the {{SlotPool}}. Otherwise we might risk that a consumer task 
gets deployed before a producer causing a resource deadlock situation.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (FLINK-13164) Add a putAndGet benchmark for StateBenchmark

[Congxian Qiu(klion26) (JIRA)]

Congxian Qiu(klion26) created FLINK-13164:
-

 Summary: Add a putAndGet benchmark for StateBenchmark
 Key: FLINK-13164
 URL: https://issues.apache.org/jira/browse/FLINK-13164
 Project: Flink
  Issue Type: Improvement
  Components: Benchmarks
Reporter: Congxian Qiu(klion26)
Assignee: Congxian Qiu(klion26)


As discussed in the 
[thread|https://github.com/dataArtisans/flink-benchmarks/issues/19], we make 
sure that there only one sst file in all list state benchmark, so there will 
not cover the compaction scenario, this ticket wants to add a benchmark for it.

Note: perIteration result may be not stable for the newly introduced benchmark, 
but we have to make sure that the final result stable.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (FLINK-13163) Default value of slot.request.timeout is too small for batch job

[Jeff Zhang (JIRA)]

Jeff Zhang created FLINK-13163:
--

 Summary: Default value of slot.request.timeout is too small for 
batch job
 Key: FLINK-13163
 URL: https://issues.apache.org/jira/browse/FLINK-13163
 Project: Flink
  Issue Type: Bug
  Components: Runtime / Configuration, Runtime / Coordination
Affects Versions: 1.9.0
Reporter: Jeff Zhang


The default value of slot.request.timeout is 5 minutes. It will cause the flink 
job fail if downstream vertex can not get resources in 5 minutes. Ideally, for 
batch job, it should wait there indefinitely. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)