Re: [Discuss] Update the pull request description template.

[Yangze Guo]

In my experience, the template is helpful. Especially for the people
just joined the community and give their first PR. I don't know how
many people have read the contributor guide entirely before they
commit their first PR, but I should admit that I did not read it word
by word for the first time, since not all of the items related to my
work. However, the template forces me to check the basic rules and
guidelines of the community.
Another benefit I can think of is to remind people who touch the code
path they aren't familiar with. If that needs a special test flow, the
template forces them to follow it.

Best,
Yangze Guo

On Fri, Feb 21, 2020 at 2:39 PM Yang Wang  wrote:
>
> I second xintong's suggestion. When i open a PR, i also check the item list
> in the template. It help to
> know whether i should test the PR in a real cluster(Yarn/K8s/Mesos). Or i
> should be more careful
> when touching the per-record code paths.If we have some dependencies
> changes, i will need to check
> the generated jar as expected.
>
>
> Best,
> Yang
>
> Xintong Song  于2020年2月20日周四 上午10:33写道：
>
> > Thanks for the feedbacks, Chesnay and Till. And thanks for the pointer,
> > Congxian.
> >
> > I don't know how often committers and reviewers checks and benefits from
> > the PR description. From your feedbacks and the number of responses to this
> > discussion, it's probably not often.
> >
> > However, as a contributor and speaking only for myself, I actually find the
> > PR template very helpful. I use it as a checking list for opening a PR.
> > Filling in the template forces me to revisit the important things, e.g.,
> > have I added enough test cases to cover the all the important changes, does
> > this change need to be validated with a real deployment (if it touches the
> > deployment and recovery). An experienced developer might be able to check
> > these things without such a checking list, but there might be more primary
> > developers that can benefit from it.
> >
> >
> > Therefore, if we agree that PR template is less useful for reviewers, I
> > would like to propose to reposition it as a contributor checking list. The
> > following are some examples of how the existing items might be
> > repositioned.
> >
> >
> > - The runtime per-record code paths (performance sensitive): (yes / no /
> > don't know). If yes, please check the following items.
> >
> > - Is there a good reason to do that?
> > - Is there an alternative non pre-record approach?
> >
> > - Is Java stream or Optional used in the per-recode code path? (Those
> > should be avoid according to the code style and quality guide[1])
> >
> > - Do we know the exact impact on performance? (Maybe point to the
> > performance benchmarks)
> >
> >
> > - Anything that affects deployment or recovery: JobManager (and its
> > components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: (yes / no /
> > don't know). If yes, please check the following items.
> >
> > - Has this PR been validated with a real deployment?
> >
> > - Has this PR been validated with the failover scenarios?
> >
> > - Does this PR requires any specific version or configuration of an
> > external system? E.g., Kubernetes/Yarn/Mesos/ZooKeeper APIs not supported
> > by all the versions that Flink claims to be compatible with.
> >
> >
> > WDYT?
> >
> >
> > Thank you~
> >
> > Xintong Song
> >
> >
> > [1]https://flink.apache.org/contributing/code-style-and-quality-java.html
> >
> > On Wed, Feb 19, 2020 at 9:24 PM Till Rohrmann 
> > wrote:
> >
> > > I actually wanted to second Chesnay but apparently my impression is a bit
> > > wrong. Out of the last 10 closed PRs (admittedly a small sample size)
> > only
> > > 2 did not fill out the template. I did not check for correctness though.
> > >
> > > Assuming that people use the template, I believe it is a good idea to
> > > update it. One thing to consider is whether we wanna keep the S3 item or
> > > want to generalize it. I think there was some reason why we explicitly
> > > added it to the template but I cannot really remember.
> > >
> > > Cheers,
> > > Till
> > >
> > > On Mon, Feb 17, 2020 at 3:02 PM Congxian Qiu 
> > > wrote:
> > >
> > > > JFYI, there is an issue[1] which I think is related to this thread
> > > > [1] https://issues.apache.org/jira/browse/FLINK-15977
> > > >
> > > > Best,
> > > > Congxian
> > > >
> > > >
> > > > Chesnay Schepler  于2020年2月17日周一 下午9:08写道：
> > > >
> > > > > I think it should just be removed since 99% of pull requests ignore
> > it
> > > > > anyway.
> > > > >
> > > > > On 17/02/2020 13:31, Xintong Song wrote:
> > > > > > Hi all,
> > > > > >
> > > > > > It seems our PR description template is a bit outdated, and I would
> > > > like
> > > > > to
> > > > > > propose updating it.
> > > > > >
> > > > > > I was working on a Kubernetes related PR, and realized that our PR
> > > > > > description does not mention the new Kubernetes integration
> > > questioning
> > > > > > about deployment related changes. Currently is is as follows:
> > > 

[jira] [Created] (FLINK-16197) Failed to query partitioned table when partition folder is removed

[Rui Li (Jira)]

Rui Li created FLINK-16197:
--

 Summary: Failed to query partitioned table when partition folder 
is removed
 Key: FLINK-16197
 URL: https://issues.apache.org/jira/browse/FLINK-16197
 Project: Flink
  Issue Type: Bug
  Components: Connectors / Hive
Reporter: Rui Li






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSS] Improve TableFactory

[Jark Wu]

Hi all,

I would like to add an additional method `getClassloader()` into the
context.
Because a TableFactory may require this classloader to find another
TableFactory,
e.g. we will find format factory in KafkaTableSourceSinkFactory.
See FLINK-15992.

I don't think we need a new VOTE for this, I just want to make this
discussion more publicly.
What do you think?

Best,
Jark

On Wed, 5 Feb 2020 at 16:05, Rui Li  wrote:

> +1, thanks for the efforts.
>
> On Wed, Feb 5, 2020 at 4:00 PM Jingsong Li  wrote:
>
> > Hi all,
> >
> > As Jark suggested in VOTE thread.
> > JIRA created: https://issues.apache.org/jira/browse/FLINK-15912
> >
> > Best,
> > Jingsong Lee
> >
> > On Wed, Feb 5, 2020 at 10:57 AM Jingsong Li 
> > wrote:
> >
> > > Hi Timo,
> > >
> > > G ood catch!
> > >
> > > I really love the idea 2, a full Flink config looks very good to me.
> > >
> > > Try to understand your first one, actually we don't have
> > `TableIdentifier`
> > > class now. But TableFactory already indicate table. So I am OK.
> > >
> > > New Context should be:
> > >
> > >/**
> > > * Context of table source creation. Contains table information and
> > environment information.
> > > */
> > >interface Context {
> > >   /**
> > >* @return full identifier of the given {@link CatalogTable}.
> > >*/
> > >   ObjectIdentifier getObjectIdentifier();
> > >   /**
> > >* @return table {@link CatalogTable} instance.
> > >*/
> > >   CatalogTable getTable();
> > >   /**
> > >* @return readable config of this table environment.
> > >*/
> > >   ReadableConfig getConfiguration();
> > >}
> > >
> > >
> > > Best,
> > > Jingsong Lee
> > >
> > > On Tue, Feb 4, 2020 at 8:51 PM Timo Walther 
> wrote:
> > >
> > >> Hi Jingsong,
> > >>
> > >> some last minute changes from my side:
> > >>
> > >> 1. rename `getTableIdentifier` to `getObjectIdentifier` to keep the
> API
> > >> obvious. Otherwise people expect a `TableIdentifier` class being
> > >> returned here.
> > >>
> > >> 2. rename `getTableConfig` to `getConfiguration()` in the future this
> > >> will not only be a "table" config but might give access to the full
> > >> Flink config
> > >>
> > >> Thanks,
> > >> Timo
> > >>
> > >>
> > >> On 04.02.20 06:27, Jingsong Li wrote:
> > >> > So the interface will be:
> > >> >
> > >> > public interface TableSourceFactory extends TableFactory {
> > >> > ..
> > >> >
> > >> > /**
> > >> >  * Creates and configures a {@link TableSource} based on the
> given
> > >> > {@link Context}.
> > >> >  *
> > >> >  * @param context context of this table source.
> > >> >  * @return the configured table source.
> > >> >  */
> > >> > default TableSource createTableSource(Context context) {
> > >> >ObjectIdentifier tableIdentifier =
> > context.getTableIdentifier();
> > >> >return createTableSource(
> > >> >  new ObjectPath(tableIdentifier.getDatabaseName(),
> > >> > tableIdentifier.getObjectName()),
> > >> >  context.getTable());
> > >> > }
> > >> > /**
> > >> >  * Context of table source creation. Contains table information
> > and
> > >> > environment information.
> > >> >  */
> > >> > interface Context {
> > >> >/**
> > >> > * @return full identifier of the given {@link CatalogTable}.
> > >> > */
> > >> >ObjectIdentifier getTableIdentifier();
> > >> >/**
> > >> > * @return table {@link CatalogTable} instance.
> > >> > */
> > >> >CatalogTable getTable();
> > >> >/**
> > >> > * @return readable config of this table environment.
> > >> > */
> > >> >ReadableConfig getTableConfig();
> > >> > }
> > >> > }
> > >> >
> > >> > public interface TableSinkFactory extends TableFactory {
> > >> > ..
> > >> > /**
> > >> >  * Creates and configures a {@link TableSink} based on the given
> > >> > {@link Context}.
> > >> >  *
> > >> >  * @param context context of this table sink.
> > >> >  * @return the configured table sink.
> > >> >  */
> > >> > default TableSink createTableSink(Context context) {
> > >> >ObjectIdentifier tableIdentifier =
> > context.getTableIdentifier();
> > >> >return createTableSink(
> > >> >  new ObjectPath(tableIdentifier.getDatabaseName(),
> > >> > tableIdentifier.getObjectName()),
> > >> >  context.getTable());
> > >> > }
> > >> > /**
> > >> >  * Context of table sink creation. Contains table information
> and
> > >> > environment information.
> > >> >  */
> > >> > interface Context {
> > >> >/**
> > >> > * @return full identifier of the given {@link CatalogTable}.
> > >> > */
> > >> >ObjectIdentifier getTableIdentifier();
> > >> >/**
> > >> > * @return table {@link CatalogTable} instance.
> > >> > */
> > >> >CatalogTable getTable();
> > 

[jira] [Created] (FLINK-16196) FlinkStandaloneHiveRunner leaks HMS process

[Rui Li (Jira)]

Rui Li created FLINK-16196:
--

 Summary: FlinkStandaloneHiveRunner leaks HMS process
 Key: FLINK-16196
 URL: https://issues.apache.org/jira/browse/FLINK-16196
 Project: Flink
  Issue Type: Test
  Components: Connectors / Hive, Tests
Reporter: Rui Li






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Flink CEP greedy match of single pattern

[Dominik Wosiński]

Hey,
I have a question regarding CEP, assume I have a stream of readings from
various sensors. The application is running in EventTime, so according to
the CEP docs the events are buffered and sorted by timestamp ascending.

So, I want to record the situations when reading from the sensor goes above
some threshold. But what I am interested in is to have a whole match for
the period when the event was above the threshold.

I tried to implement a single pattern that was more or less something:


Pattern.begin[Reading]("beginning")
  .where(_.data() <  Threshold)

  .oneOrMore

  .greedy

  .consecutive



But now it produces multiple partial matches that I can't eliminate. For
example for threshold = 350, I have a stream:

300, 400, 500, 300

And then I get the following lists of events [400], [400, 500], [500].

Is there a way to eliminate those partial matches ??

Best Regards,
Dom.

Re: [VOTE] FLIP-104: Add More Metrics to Jobmanager

[Xintong Song]

FYI, there's an effort planned for 1.11 to improve the memory configuration
of the Flink master process, similar to FLIP-49 but definitely less
complexity.

I would not consider the memory configuration improvement as a blocker for
this effort. As far as I can see, there's nothing in conflict. Just after
the memory configuration improvement, we might be able to present more
information on the JM metrics page, which are tightly corresponding to the
configuration options, like what we planned for the TM metrics page in
FLIP-102. Therefore, it might make sense to proceed this FLIP afterwards.

I'm neutral on this, and would leave the call to Yandong and Lining.

Thank you~

Xintong Song



On Fri, Feb 21, 2020 at 2:47 PM Jark Wu  wrote:

> Thanks Yadong,
>
> I think we can use different color to distinguish the memory usage (from
> green to red?).
> Besides, I think we should add an unit on the "Garbage Collection" ->
> "Time", it's hard to know what the value mean.
> Would be better to display the value like "10ms", "5ns".
>
> Best,
> Jark
>
> On Thu, 20 Feb 2020 at 17:58, Yadong Xie  wrote:
>
> > Hi all
> >
> > I want to start the vote for FLIP-104, which proposes to add more metrics
> > to job manager.
> >
> > To help everyone better understand the proposal, we spent some efforts on
> > making an online POC
> >
> > previous web: http://101.132.122.69:8081/#/job-manager/config
> > POC web: http://101.132.122.69:8081/web/#/job-manager/metrics
> >
> >
> > The vote will last for at least 72 hours, following the consensus voting
> > process.
> >
> > FLIP wiki:
> >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-104%3A+Add+More+Metrics+to+Jobmanager
> >
> > Discussion thread:
> >
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-75-Flink-Web-UI-Improvement-Proposal-td33540.html
> >
> > Thanks,
> >
> > Yadong
> >
>

Adding a new "Docker Images" component to Jira

[Patrick Lucas]

Hi,

Could someone with permissions add a new component to the FLINK project in
Jira for the Docker images ?

There is already a "Deployment / Docker" component, but that's not quite
the same as maintenance/improvements on the flink-docker images.

Either top-level "Docker Images" or perhaps "Release / Docker Images" would
be fine.

Thanks,
Patrick

Re: Adding a new "Docker Images" component to Jira

[Patrick Lucas]

Thanks, Chesnay!

On Fri, Feb 21, 2020 at 11:26 AM Chesnay Schepler 
wrote:

> I've added a "Release System / Docker" component.
>
> On 21/02/2020 11:19, Patrick Lucas wrote:
> > Hi,
> >
> > Could someone with permissions add a new component to the FLINK project
> in
> > Jira for the Docker images ?
> >
> > There is already a "Deployment / Docker" component, but that's not quite
> > the same as maintenance/improvements on the flink-docker images.
> >
> > Either top-level "Docker Images" or perhaps "Release / Docker Images"
> would
> > be fine.
> >
> > Thanks,
> > Patrick
> >
>
>

[jira] [Created] (FLINK-16206) Support JSON_ARRAYAGG for blink planner

[Zili Chen (Jira)]

Zili Chen created FLINK-16206:
-

 Summary: Support JSON_ARRAYAGG for blink planner
 Key: FLINK-16206
 URL: https://issues.apache.org/jira/browse/FLINK-16206
 Project: Flink
  Issue Type: Sub-task
  Components: Table SQL / Planner
Reporter: Zili Chen
 Fix For: 1.11.0






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Adding a new "Docker Images" component to Jira

[Chesnay Schepler]

I've added a "Release System / Docker" component.

On 21/02/2020 11:19, Patrick Lucas wrote:

Hi,

Could someone with permissions add a new component to the FLINK project in
Jira for the Docker images ?

There is already a "Deployment / Docker" component, but that's not quite
the same as maintenance/improvements on the flink-docker images.

Either top-level "Docker Images" or perhaps "Release / Docker Images" would
be fine.

Thanks,
Patrick

[jira] [Created] (FLINK-16199) Support IS JSON predicate for blink planner

[Zili Chen (Jira)]

Zili Chen created FLINK-16199:
-

 Summary: Support IS JSON predicate for blink planner
 Key: FLINK-16199
 URL: https://issues.apache.org/jira/browse/FLINK-16199
 Project: Flink
  Issue Type: Sub-task
  Components: Table SQL / Planner
Reporter: Zili Chen
Assignee: Zili Chen
 Fix For: 1.11.0






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-16200) Support JSON_EXISTS for blink planner

[Zili Chen (Jira)]

Zili Chen created FLINK-16200:
-

 Summary: Support JSON_EXISTS for blink planner
 Key: FLINK-16200
 URL: https://issues.apache.org/jira/browse/FLINK-16200
 Project: Flink
  Issue Type: Sub-task
  Components: Table SQL / Planner
Reporter: Zili Chen
 Fix For: 1.11.0






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-16201) Support JSON_VALUE for blink planner

[Zili Chen (Jira)]

Zili Chen created FLINK-16201:
-

 Summary: Support JSON_VALUE for blink planner
 Key: FLINK-16201
 URL: https://issues.apache.org/jira/browse/FLINK-16201
 Project: Flink
  Issue Type: Sub-task
  Components: Table SQL / Planner
Reporter: Zili Chen
 Fix For: 1.11.0






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-16198) FileUtilsTest fails on Mac OS

[Andrey Zagrebin (Jira)]

Andrey Zagrebin created FLINK-16198:
---

 Summary: FileUtilsTest fails on Mac OS
 Key: FLINK-16198
 URL: https://issues.apache.org/jira/browse/FLINK-16198
 Project: Flink
  Issue Type: Bug
  Components: FileSystems, Tests
Reporter: Andrey Zagrebin


The following tests fail if run on Mac OS (IDE/maven).

 

FileUtilsTest.testCompressionOnRelativePath:

 
{code:java}
java.nio.file.NoSuchFileException: 
../../../../../var/folders/67/v4yp_42d21j6_n8k1h556h0cgn/T/junit6496651678375117676/compressDir/rootDirjava.nio.file.NoSuchFileException:
 
../../../../../var/folders/67/v4yp_42d21j6_n8k1h556h0cgn/T/junit6496651678375117676/compressDir/rootDir
 at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86) at 
sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) at 
sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107) at 
sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:384)
 at java.nio.file.Files.createDirectory(Files.java:674) at 
org.apache.flink.util.FileUtilsTest.verifyDirectoryCompression(FileUtilsTest.java:440)
 at 
org.apache.flink.util.FileUtilsTest.testCompressionOnRelativePath(FileUtilsTest.java:261)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) 
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 at java.lang.reflect.Method.invoke(Method.java:498) at 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
 at 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
 at 
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
 at 
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
 at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) at 
org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55) at 
org.junit.rules.RunRules.evaluate(RunRules.java:20) at 
org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
 at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
 at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at 
org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at 
org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at 
org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at 
org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at 
org.junit.runners.ParentRunner.run(ParentRunner.java:363) at 
org.junit.runner.JUnitCore.run(JUnitCore.java:137) at 
com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68)
 at 
com.intellij.rt.execution.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:47)
 at 
com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:242)
 at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:70)
{code}
 

FileUtilsTest.testDeleteDirectoryConcurrently

 

 
{code:java}
java.nio.file.FileSystemException: 
/var/folders/67/v4yp_42d21j6_n8k1h556h0cgn/T/junit7558825557740784886/junit3566161583262218465/ab1fa0bde8b22cad58b717508c7a7300/121fdf5f7b057183843ed2e1298f9b66/6598025f390d3084d69c98b36e542fe2/8db7cd9c063396a19a86f5b63ce53f66:
 Invalid argument at 
sun.nio.fs.UnixException.translateToIOException(UnixException.java:91)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
at 
sun.nio.fs.UnixFileSystemProvider.implDelete(UnixFileSystemProvider.java:244)
at 
sun.nio.fs.AbstractFileSystemProvider.deleteIfExists(AbstractFileSystemProvider.java:108)
at java.nio.file.Files.deleteIfExists(Files.java:1165)
at 
org.apache.flink.util.FileUtils.deleteFileOrDirectoryInternal(FileUtils.java:324)
at org.apache.flink.util.FileUtils.guardIfWindows(FileUtils.java:391)
at 
org.apache.flink.util.FileUtils.deleteFileOrDirectory(FileUtils.java:258)
at 
org.apache.flink.util.FileUtils.cleanDirectoryInternal(FileUtils.java:376)
at 
org.apache.flink.util.FileUtils.deleteDirectoryInternal(FileUtils.java:335)
at 
org.apache.flink.util.FileUtils.deleteFileOrDirectoryInternal(FileUtils.java:320)
at org.apache.flink.util.FileUtils.guardIfWindows(FileUtils.java:391)
at 
org.apache.flink.util.FileUtils.deleteFileOrDirectory(FileUtils.java:258)
at 
org.apache.flink.util.FileUtils.cleanDirectoryInternal(FileUtils.java:376)
at 
org.apache.flink.util.FileUtils.deleteDirectoryInternal(FileUtils.java:335)
at 

[jira] [Created] (FLINK-16205) Support JSON_OBJECTAGG for blink planner

[Zili Chen (Jira)]

Zili Chen created FLINK-16205:
-

 Summary: Support JSON_OBJECTAGG for blink planner
 Key: FLINK-16205
 URL: https://issues.apache.org/jira/browse/FLINK-16205
 Project: Flink
  Issue Type: Sub-task
  Components: Table SQL / Planner
Reporter: Zili Chen
 Fix For: 1.11.0






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-16203) Support JSON_OBJECT for blink planner

[Zili Chen (Jira)]

Zili Chen created FLINK-16203:
-

 Summary: Support JSON_OBJECT for blink planner
 Key: FLINK-16203
 URL: https://issues.apache.org/jira/browse/FLINK-16203
 Project: Flink
  Issue Type: Sub-task
Reporter: Zili Chen
 Fix For: 1.11.0






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-16202) Support JSON_QUERY for blink planner

[Zili Chen (Jira)]

Zili Chen created FLINK-16202:
-

 Summary: Support JSON_QUERY for blink planner
 Key: FLINK-16202
 URL: https://issues.apache.org/jira/browse/FLINK-16202
 Project: Flink
  Issue Type: Sub-task
  Components: Table SQL / Planner
Reporter: Zili Chen
 Fix For: 1.11.0






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-16204) Support JSON_ARRAY for blink planner

[Zili Chen (Jira)]

Zili Chen created FLINK-16204:
-

 Summary: Support JSON_ARRAY for blink planner
 Key: FLINK-16204
 URL: https://issues.apache.org/jira/browse/FLINK-16204
 Project: Flink
  Issue Type: Sub-task
  Components: Table SQL / Planner
Reporter: Zili Chen
 Fix For: 1.11.0






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-16207) In stream processing concepts section, rework distribution patterns description

[Aljoscha Krettek (Jira)]

Aljoscha Krettek created FLINK-16207:


 Summary: In stream processing concepts section, rework 
distribution patterns description
 Key: FLINK-16207
 URL: https://issues.apache.org/jira/browse/FLINK-16207
 Project: Flink
  Issue Type: Sub-task
  Components: Documentation
Reporter: Aljoscha Krettek


Currently, we only distinguish between one-to-one and redistribution. We should 
instead describe it as

- Forward
- Broadcast
- Random
- Keyed



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-16209) Add Latency and Completeness section in timely stream processing concepts

[Aljoscha Krettek (Jira)]

Aljoscha Krettek created FLINK-16209:


 Summary: Add Latency and Completeness section in timely stream 
processing concepts
 Key: FLINK-16209
 URL: https://issues.apache.org/jira/browse/FLINK-16209
 Project: Flink
  Issue Type: Sub-task
  Components: Documentation
Reporter: Aljoscha Krettek






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-16211) Add introduction to stream processing concepts documentation

[Aljoscha Krettek (Jira)]

Aljoscha Krettek created FLINK-16211:


 Summary: Add introduction to stream processing concepts 
documentation
 Key: FLINK-16211
 URL: https://issues.apache.org/jira/browse/FLINK-16211
 Project: Flink
  Issue Type: Sub-task
  Components: Documentation
Reporter: Aljoscha Krettek






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-16212) Describe how Flink is a unified batch/stream processing system in concepts documentation

[Aljoscha Krettek (Jira)]

Aljoscha Krettek created FLINK-16212:


 Summary: Describe how Flink is a unified batch/stream processing 
system in concepts documentation
 Key: FLINK-16212
 URL: https://issues.apache.org/jira/browse/FLINK-16212
 Project: Flink
  Issue Type: Sub-task
  Components: Documentation
Reporter: Aljoscha Krettek






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-16214) Describe how state is different for stream/batch programs in concepts documentation

[Aljoscha Krettek (Jira)]

Aljoscha Krettek created FLINK-16214:


 Summary: Describe how state is different for stream/batch programs 
in concepts documentation
 Key: FLINK-16214
 URL: https://issues.apache.org/jira/browse/FLINK-16214
 Project: Flink
  Issue Type: Sub-task
  Components: Documentation
Reporter: Aljoscha Krettek


This should go into {{stateful-stream-processing.md}}.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSS] Improve TableFactory

[Jingsong Li]

Hi Jark,

I think user ClassLoader is clear in SQL-CLI. I agree with you that we can
add ClassLoader to Context.

But how to implement user ClassLoader in TableEnvironment, there is no
ClassLoader in TableEnvironment. (Maybe EnvironmentSettings could contains
user ClassLoader in the future)

So I recommend maybe we can have a whole story design about ClassLoader in
Table.

Guowei and Yingjie is working on supporting Table API connectors in
Plugins, it is ClassLoader thing, I think they can have some inputs.

Best,
Jingsong Lee

On Fri, Feb 21, 2020 at 4:02 PM Jark Wu  wrote:

> Hi all,
>
> I would like to add an additional method `getClassloader()` into the
> context.
> Because a TableFactory may require this classloader to find another
> TableFactory,
> e.g. we will find format factory in KafkaTableSourceSinkFactory.
> See FLINK-15992.
>
> I don't think we need a new VOTE for this, I just want to make this
> discussion more publicly.
> What do you think?
>
> Best,
> Jark
>
> On Wed, 5 Feb 2020 at 16:05, Rui Li  wrote:
>
> > +1, thanks for the efforts.
> >
> > On Wed, Feb 5, 2020 at 4:00 PM Jingsong Li 
> wrote:
> >
> > > Hi all,
> > >
> > > As Jark suggested in VOTE thread.
> > > JIRA created: https://issues.apache.org/jira/browse/FLINK-15912
> > >
> > > Best,
> > > Jingsong Lee
> > >
> > > On Wed, Feb 5, 2020 at 10:57 AM Jingsong Li 
> > > wrote:
> > >
> > > > Hi Timo,
> > > >
> > > > G ood catch!
> > > >
> > > > I really love the idea 2, a full Flink config looks very good to me.
> > > >
> > > > Try to understand your first one, actually we don't have
> > > `TableIdentifier`
> > > > class now. But TableFactory already indicate table. So I am OK.
> > > >
> > > > New Context should be:
> > > >
> > > >/**
> > > > * Context of table source creation. Contains table information
> and
> > > environment information.
> > > > */
> > > >interface Context {
> > > >   /**
> > > >* @return full identifier of the given {@link CatalogTable}.
> > > >*/
> > > >   ObjectIdentifier getObjectIdentifier();
> > > >   /**
> > > >* @return table {@link CatalogTable} instance.
> > > >*/
> > > >   CatalogTable getTable();
> > > >   /**
> > > >* @return readable config of this table environment.
> > > >*/
> > > >   ReadableConfig getConfiguration();
> > > >}
> > > >
> > > >
> > > > Best,
> > > > Jingsong Lee
> > > >
> > > > On Tue, Feb 4, 2020 at 8:51 PM Timo Walther 
> > wrote:
> > > >
> > > >> Hi Jingsong,
> > > >>
> > > >> some last minute changes from my side:
> > > >>
> > > >> 1. rename `getTableIdentifier` to `getObjectIdentifier` to keep the
> > API
> > > >> obvious. Otherwise people expect a `TableIdentifier` class being
> > > >> returned here.
> > > >>
> > > >> 2. rename `getTableConfig` to `getConfiguration()` in the future
> this
> > > >> will not only be a "table" config but might give access to the full
> > > >> Flink config
> > > >>
> > > >> Thanks,
> > > >> Timo
> > > >>
> > > >>
> > > >> On 04.02.20 06:27, Jingsong Li wrote:
> > > >> > So the interface will be:
> > > >> >
> > > >> > public interface TableSourceFactory extends TableFactory {
> > > >> > ..
> > > >> >
> > > >> > /**
> > > >> >  * Creates and configures a {@link TableSource} based on the
> > given
> > > >> > {@link Context}.
> > > >> >  *
> > > >> >  * @param context context of this table source.
> > > >> >  * @return the configured table source.
> > > >> >  */
> > > >> > default TableSource createTableSource(Context context) {
> > > >> >ObjectIdentifier tableIdentifier =
> > > context.getTableIdentifier();
> > > >> >return createTableSource(
> > > >> >  new ObjectPath(tableIdentifier.getDatabaseName(),
> > > >> > tableIdentifier.getObjectName()),
> > > >> >  context.getTable());
> > > >> > }
> > > >> > /**
> > > >> >  * Context of table source creation. Contains table
> information
> > > and
> > > >> > environment information.
> > > >> >  */
> > > >> > interface Context {
> > > >> >/**
> > > >> > * @return full identifier of the given {@link
> CatalogTable}.
> > > >> > */
> > > >> >ObjectIdentifier getTableIdentifier();
> > > >> >/**
> > > >> > * @return table {@link CatalogTable} instance.
> > > >> > */
> > > >> >CatalogTable getTable();
> > > >> >/**
> > > >> > * @return readable config of this table environment.
> > > >> > */
> > > >> >ReadableConfig getTableConfig();
> > > >> > }
> > > >> > }
> > > >> >
> > > >> > public interface TableSinkFactory extends TableFactory {
> > > >> > ..
> > > >> > /**
> > > >> >  * Creates and configures a {@link TableSink} based on the
> given
> > > >> > {@link Context}.
> > > >> >  *
> > > >> >  * @param context context of this table sink.
> > > >> >  * @return the configured table sink.
> > > >> >  

Re: [ANNOUNCE] Jingsong Lee becomes a Flink committer

[Jingsong Li]

Thanks everyone~

It's my pleasure to be part of the community. I hope I can make a better
contribution in future.

Best,
Jingsong Lee

On Fri, Feb 21, 2020 at 2:48 PM Hequn Cheng  wrote:

> Congratulations Jingsong! Well deserved.
>
> Best,
> Hequn
>
> On Fri, Feb 21, 2020 at 2:42 PM Yang Wang  wrote:
>
>> Congratulations！Jingsong. Well deserved.
>>
>>
>> Best,
>> Yang
>>
>> Zhijiang  于2020年2月21日周五 下午1:18写道：
>>
>>> Congrats Jingsong! Welcome on board!
>>>
>>> Best,
>>> Zhijiang
>>>
>>> --
>>> From:Zhenghua Gao 
>>> Send Time:2020 Feb. 21 (Fri.) 12:49
>>> To:godfrey he 
>>> Cc:dev ; user 
>>> Subject:Re: [ANNOUNCE] Jingsong Lee becomes a Flink committer
>>>
>>> Congrats Jingsong！
>>>
>>>
>>> *Best Regards,*
>>> *Zhenghua Gao*
>>>
>>>
>>> On Fri, Feb 21, 2020 at 11:59 AM godfrey he  wrote:
>>> Congrats Jingsong! Well deserved.
>>>
>>> Best,
>>> godfrey
>>>
>>> Jeff Zhang  于2020年2月21日周五 上午11:49写道：
>>> Congratulations！Jingsong. You deserve it
>>>
>>> wenlong.lwl  于2020年2月21日周五 上午11:43写道：
>>> Congrats Jingsong!
>>>
>>> On Fri, 21 Feb 2020 at 11:41, Dian Fu  wrote:
>>>
>>> > Congrats Jingsong!
>>> >
>>> > > 在 2020年2月21日，上午11:39，Jark Wu  写道：
>>> > >
>>> > > Congratulations Jingsong! Well deserved.
>>> > >
>>> > > Best,
>>> > > Jark
>>> > >
>>> > > On Fri, 21 Feb 2020 at 11:32, zoudan  wrote:
>>> > >
>>> > >> Congratulations! Jingsong
>>> > >>
>>> > >>
>>> > >> Best,
>>> > >> Dan Zou
>>> > >>
>>> >
>>> >
>>>
>>>
>>> --
>>> Best Regards
>>>
>>> Jeff Zhang
>>>
>>>
>>>

-- 
Best, Jingsong Lee

[jira] [Created] (FLINK-16217) SQL Client crashed when any uncatched exception is thrown

[Jark Wu (Jira)]

Jark Wu created FLINK-16217:
---

 Summary: SQL Client crashed when any uncatched exception is thrown
 Key: FLINK-16217
 URL: https://issues.apache.org/jira/browse/FLINK-16217
 Project: Flink
  Issue Type: Bug
  Components: Table SQL / Client
Reporter: Jark Wu
 Fix For: 1.10.1
 Attachments: 15821324825831.jpg

Currently, SQL CLI doesn't catch all the exceptions, for example, Calcite 
exceptions. 

We should catch any possible exceptions thrown by the planner, and print the 
root cause. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-16208) Add introduction to timely stream processing concepts documentation

[Aljoscha Krettek (Jira)]

Aljoscha Krettek created FLINK-16208:


 Summary: Add introduction to timely stream processing concepts 
documentation
 Key: FLINK-16208
 URL: https://issues.apache.org/jira/browse/FLINK-16208
 Project: Flink
  Issue Type: Sub-task
  Components: Documentation
Reporter: Aljoscha Krettek






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-16210) Add section about applications and clusters/session in concepts documentation

[Aljoscha Krettek (Jira)]

Aljoscha Krettek created FLINK-16210:


 Summary: Add section about applications and clusters/session in 
concepts documentation
 Key: FLINK-16210
 URL: https://issues.apache.org/jira/browse/FLINK-16210
 Project: Flink
  Issue Type: Sub-task
Reporter: Aljoscha Krettek


This can either go into the existing _Flink Architecture_ 
({{flink-architecture.md}}) documentation or be a new section. We can possibly 
remove the old _Flink Architecture_ section then.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-16213) Add "What Is State" section in concepts documentation

[Aljoscha Krettek (Jira)]

Aljoscha Krettek created FLINK-16213:


 Summary: Add "What Is State" section in concepts documentation
 Key: FLINK-16213
 URL: https://issues.apache.org/jira/browse/FLINK-16213
 Project: Flink
  Issue Type: Sub-task
Reporter: Aljoscha Krettek


This should go into {{stateful-stream-processing.md}}.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-16215) Start redundant TaskExecutor when JM failed

[YufeiLiu (Jira)]

YufeiLiu created FLINK-16215:


 Summary: Start redundant TaskExecutor when JM failed
 Key: FLINK-16215
 URL: https://issues.apache.org/jira/browse/FLINK-16215
 Project: Flink
  Issue Type: Bug
  Components: Runtime / Coordination
Affects Versions: 1.10.0
Reporter: YufeiLiu


TaskExecutor will reconnect to the new ResourceManager leader when JM failed, 
and JobMaster will restart and reschedule job. If job slot request arrive 
earlier than TM registration, RM will start new workers rather than reuse the 
existing TMs.
It‘s hard to reproduce becasue TM registration usually come first, and timeout 
check will stop redundant TMs. 
But I think it would be better if we make the {{recoverWokerNode}} to 
interface, and put recovered slots in {{pendingSlots}} wait for TM reconnection.




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-16216) Describe end-to-end exactly once programs in stateful stream processing concepts documentation

[Aljoscha Krettek (Jira)]

Aljoscha Krettek created FLINK-16216:


 Summary: Describe end-to-end exactly once programs in stateful 
stream processing concepts documentation
 Key: FLINK-16216
 URL: https://issues.apache.org/jira/browse/FLINK-16216
 Project: Flink
  Issue Type: Sub-task
  Components: Documentation
Reporter: Aljoscha Krettek


This should go into {{stateful-stream-processing.md}}.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-16220) JsonRowSerializationSchema throws cast exception : NullNode cannot be cast to ArrayNode

[Benchao Li (Jira)]

Benchao Li created FLINK-16220:
--

 Summary: JsonRowSerializationSchema throws cast exception : 
NullNode cannot be cast to ArrayNode
 Key: FLINK-16220
 URL: https://issues.apache.org/jira/browse/FLINK-16220
 Project: Flink
  Issue Type: Bug
  Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile)
Reporter: Benchao Li


It's because the object reuse. For the below schema:
{code:java}
create table sink {
  col1 int,
  col2 array
}{code}
if col2 is null, then the reused object will be {{NullNode}}. for the next 
record, if it's not null, we will cast the reused object {{NullNode}} to 
{{ArrayNode}}, which will throw cast exception.

 

cc [~jark] [~twalthr] 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [ANNOUNCE] Jingsong Lee becomes a Flink committer

[Yun Gao]

Congratulations Jingsong!

   Best,
   Yun


--
From:Jingsong Li 
Send Time:2020 Feb. 21 (Fri.) 21:42
To:Hequn Cheng 
Cc:Yang Wang ; Zhijiang ; 
Zhenghua Gao ; godfrey he ; dev 
; user 
Subject:Re: [ANNOUNCE] Jingsong Lee becomes a Flink committer

Thanks everyone~

It's my pleasure to be part of the community. I hope I can make a better 
contribution in future.

Best,
Jingsong Lee
On Fri, Feb 21, 2020 at 2:48 PM Hequn Cheng  wrote:

Congratulations Jingsong! Well deserved.

Best, 
Hequn 
On Fri, Feb 21, 2020 at 2:42 PM Yang Wang  wrote:
Congratulations！Jingsong. Well deserved.


Best,
Yang

Zhijiang  于2020年2月21日周五 下午1:18写道：
Congrats Jingsong! Welcome on board!

Best,
Zhijiang

--
From:Zhenghua Gao 
Send Time:2020 Feb. 21 (Fri.) 12:49
To:godfrey he 
Cc:dev ; user 
Subject:Re: [ANNOUNCE] Jingsong Lee becomes a Flink committer

Congrats Jingsong！


Best Regards,
Zhenghua Gao

On Fri, Feb 21, 2020 at 11:59 AM godfrey he  wrote:
Congrats Jingsong! Well deserved.

Best,
godfrey
Jeff Zhang  于2020年2月21日周五 上午11:49写道：
Congratulations！Jingsong. You deserve it 

wenlong.lwl  于2020年2月21日周五 上午11:43写道：
Congrats Jingsong!

 On Fri, 21 Feb 2020 at 11:41, Dian Fu  wrote:

 > Congrats Jingsong!
 >
 > > 在 2020年2月21日，上午11:39，Jark Wu  写道：
 > >
 > > Congratulations Jingsong! Well deserved.
 > >
 > > Best,
 > > Jark
 > >
 > > On Fri, 21 Feb 2020 at 11:32, zoudan  wrote:
 > >
 > >> Congratulations! Jingsong
 > >>
 > >>
 > >> Best,
 > >> Dan Zou
 > >>
 >
 >


-- 
Best Regards

Jeff Zhang



-- 
Best, Jingsong Lee

Re: [ANNOUNCE] Jingsong Lee becomes a Flink committer

[Bowen Li]

Congrats, Jingsong!

On Fri, Feb 21, 2020 at 7:28 AM Till Rohrmann  wrote:

> Congratulations Jingsong!
>
> Cheers,
> Till
>
> On Fri, Feb 21, 2020 at 4:03 PM Yun Gao  wrote:
>
>>   Congratulations Jingsong!
>>
>>Best,
>>Yun
>>
>> --
>> From:Jingsong Li 
>> Send Time:2020 Feb. 21 (Fri.) 21:42
>> To:Hequn Cheng 
>> Cc:Yang Wang ; Zhijiang <
>> wangzhijiang...@aliyun.com>; Zhenghua Gao ; godfrey he
>> ; dev ; user <
>> u...@flink.apache.org>
>> Subject:Re: [ANNOUNCE] Jingsong Lee becomes a Flink committer
>>
>> Thanks everyone~
>>
>> It's my pleasure to be part of the community. I hope I can make a better
>> contribution in future.
>>
>> Best,
>> Jingsong Lee
>>
>> On Fri, Feb 21, 2020 at 2:48 PM Hequn Cheng  wrote:
>> Congratulations Jingsong! Well deserved.
>>
>> Best,
>> Hequn
>>
>> On Fri, Feb 21, 2020 at 2:42 PM Yang Wang  wrote:
>> Congratulations！Jingsong. Well deserved.
>>
>>
>> Best,
>> Yang
>>
>> Zhijiang  于2020年2月21日周五 下午1:18写道：
>> Congrats Jingsong! Welcome on board!
>>
>> Best,
>> Zhijiang
>>
>> --
>> From:Zhenghua Gao 
>> Send Time:2020 Feb. 21 (Fri.) 12:49
>> To:godfrey he 
>> Cc:dev ; user 
>> Subject:Re: [ANNOUNCE] Jingsong Lee becomes a Flink committer
>>
>> Congrats Jingsong！
>>
>>
>> *Best Regards,*
>> *Zhenghua Gao*
>>
>>
>> On Fri, Feb 21, 2020 at 11:59 AM godfrey he  wrote:
>> Congrats Jingsong! Well deserved.
>>
>> Best,
>> godfrey
>>
>> Jeff Zhang  于2020年2月21日周五 上午11:49写道：
>> Congratulations！Jingsong. You deserve it
>>
>> wenlong.lwl  于2020年2月21日周五 上午11:43写道：
>> Congrats Jingsong!
>>
>> On Fri, 21 Feb 2020 at 11:41, Dian Fu  wrote:
>>
>> > Congrats Jingsong!
>> >
>> > > 在 2020年2月21日，上午11:39，Jark Wu  写道：
>> > >
>> > > Congratulations Jingsong! Well deserved.
>> > >
>> > > Best,
>> > > Jark
>> > >
>> > > On Fri, 21 Feb 2020 at 11:32, zoudan  wrote:
>> > >
>> > >> Congratulations! Jingsong
>> > >>
>> > >>
>> > >> Best,
>> > >> Dan Zou
>> > >>
>> >
>> >
>>
>>
>> --
>> Best Regards
>>
>> Jeff Zhang
>>
>>
>>
>> --
>> Best, Jingsong Lee
>>
>>
>>

Re: Flink CEP greedy match of single pattern

[Till Rohrmann]

Hi Dominik,

you can control FlinkCEP's consumption behaviour via the after match skip
strategies [1]. They allow you to control how Flink treats events after a
match has occurred.

If you are interested in the longest possible window of events exceeding
your threshold, then you could also add terminating event which is below
the threshold. Only then you can be sure that any following event won't
continue the window.

[1]
https://ci.apache.org/projects/flink/flink-docs-stable/dev/libs/cep.html#after-match-skip-strategy

Cheers,
Till

On Fri, Feb 21, 2020 at 10:56 AM Dominik Wosiński  wrote:

> Hey,
> I have a question regarding CEP, assume I have a stream of readings from
> various sensors. The application is running in EventTime, so according to
> the CEP docs the events are buffered and sorted by timestamp ascending.
>
> So, I want to record the situations when reading from the sensor goes above
> some threshold. But what I am interested in is to have a whole match for
> the period when the event was above the threshold.
>
> I tried to implement a single pattern that was more or less something:
>
>
> Pattern.begin[Reading]("beginning")
>   .where(_.data() <  Threshold)
>
>   .oneOrMore
>
>   .greedy
>
>   .consecutive
>
>
>
> But now it produces multiple partial matches that I can't eliminate. For
> example for threshold = 350, I have a stream:
>
> 300, 400, 500, 300
>
> And then I get the following lists of events [400], [400, 500], [500].
>
> Is there a way to eliminate those partial matches ??
>
> Best Regards,
> Dom.
>

Re: [DISCUSS] FLINK-16194: Refactor the Kubernetes architecture design

[Till Rohrmann]

Thanks for starting this discussion Canbin. If I understand your proposal
correctly, then you would like to evolve the existing decorator approach so
that decorators are monadic and smaller in size and functionality. The
latter aspect will allow to reuse them between the client and the cluster.
Just to make sure, it is not a fundamentally different approach compared to
what we have right now, is it?

If this is the case, then I think it makes sense to reuse code as much as
possible and to create small code units which are easier to test.

Cheers,
Till

On Fri, Feb 21, 2020 at 4:41 PM felixzheng zheng 
wrote:

> Thanks for the feedback @Yang Wang. I would like to discuss some of the
> details in depth about why I am confused about the existing design.
>
> Question 1: How do we mount a configuration file?
>
> For the existing design,
>
>1.
>
>We need several classes to finish it:
>1.
>
>   InitializerDecorator
>   2.
>
>   OwnerReferenceDecorator
>   3.
>
>   ConfigMapDecorator
>   4.
>
>   KubernetesUtils: providing the getConfigMapVolume method to share for
>   the FlinkMasterDeploymentDecorator and the TaskManagerPodDecorator.
>   5.
>
>   FlinkMasterDeploymentDecorator: mounts the ConfigMap volume.
>   6.
>
>   TaskManagerPodDecorator: mounts the ConfigMap volume.
>   7.
>
>   If in the future, someone would like to introduce an init Container,
>   the InitContainerDecorator has to mount the ConfigMap volume too.
>
>
> I am confused about the current solution to mounting a configuration file:
>
>1.
>
>Actually, we do not need so many Decorators for mounting a file.
>2.
>
>If we would like to mount a new file, we have no choice but to repeat
>the same tedious and scattered routine.
>3.
>
>There’s no easy way to test the file mounting functionality alone; we
>have to construct the ConfigMap, the Deployment or the TaskManagerPod
> first
>and then do a final test.
>
>
> The reason why it is so complex to mount a configuration file is that we
> don’t fully consider the internal connections among those resources in the
> existing design.
>
> The new abstraction we proposed could solve such a kind of problem, the new
> Decorator object is as follows:
>
> public interface KubernetesStepDecorator {
>
>   /**
>
>* Apply transformations to the given FlinkPod in accordance with this
> feature. This can include adding
>
>* labels/annotations, mounting volumes, and setting startup command or
> parameters, etc.
>
>*/
>
>   FlinkPod decorateFlinkPod(FlinkPod flinkPod);
>
>   /**
>
>* Build the accompanying Kubernetes resources that should be introduced
> to support this feature. This could
>
>* only applicable to the client-side submission process.
>
>*/
>
>   List buildAccompanyingKubernetesResources() throws
> IOException;
>
> }
>
> The FlinkPod is a composition of the Pod, the main Container, the init
> Container, and the sidecar Container.
>
> Next, we introduce a KubernetesStepDecorator implementation, the method of
> buildAccompanyingKubernetesResources creates the corresponding ConfigMap,
> and the method of decorateFlinkPod configures the Volume for the Pod and
> all the Containers.
>
> So, for the scenario of mounting a configuration file, the advantages of
> this new architecture are as follows:
>
>1.
>
>One dedicated KubernetesStepDecorator implementation is enough to finish
>all the mounting work, meanwhile, this class would be shared between the
>client-side and the master-side. Besides that, the number of lines of
> code
>in that class will not exceed 300 lines which facilitate code
> readability
>and maintenance.
>2.
>
>Testing becomes an easy thing now, as we can add a dedicated test class
>for only this class, the test would never rely on the construction of
> other
>components such as the Deployment.
>3.
>
>It’s quite convenient to mount a new configuration file via the newly
>dedicated KubernetesStepDecorator implementation.
>
>
> Question 2: How do we construct the Pod?
>
> For the existing design,
>
>1.
>
>The FlinkMasterDeploymentDecorator is responsible for building the
>Deployment and configuring the Pod and the Containers, while the
>TaskManagerPodDecorator is responsible for building the TaskManager Pod.
>Take FlinkMasterDeploymentDecorator as an example, let’s see what it has
>done.
>1.
>
>   Configure the main Container, including the name, command, args,
>   image, image pull policy, resource requirements, ports, all kinds of
>   environment variables, all kinds of volume mounts, etc.
>   2.
>
>   Configure the Pod, including service account, all kinds of volumes,
>   and attach all kinds of Container, including the main Container, the
> init
>   Container, and the sidecar Container.
>   3.
>
>   Configure the Deployment.
>
>
>
>1.
>
>The 

[jira] [Created] (FLINK-16223) Flat aggregate misuses key group ranges

[Seth Wiesman (Jira)]

Seth Wiesman created FLINK-16223:


 Summary: Flat aggregate misuses key group ranges
 Key: FLINK-16223
 URL: https://issues.apache.org/jira/browse/FLINK-16223
 Project: Flink
  Issue Type: Bug
  Components: Table SQL / Planner
Affects Versions: 1.10.0, 1.11.0
Reporter: Seth Wiesman


The implementation of flat aggregate appears to misuse key group ranges. When 
we add a check in AbstractStreamOperator that the current key belongs to the 
key group assigned to that subtask tests in TableAggregateITCase begin to fail. 

This patch can be used to reproduce the issue[1]. 

https://github.com/sjwiesman/flink/tree/keygrouprangecheck

cc [~jincheng] [~hequn8128]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [VOTE] FLIP-101: Add Pending Slots Detail

[Zhijiang]

Thanks Yadong for this FLIP!

If I understood correctly, the motivation is based on the scenario of scheduled 
job state for long time in the well resourced cluster.

It is meaningful for me if we can provide some useful infos to help users 
analyze the reason. I see from PoC that we can get the
respective vertex which slot is pending and the detailed slot id info.
I am curious that is it feasible to do some valid analysis from logs based on 
the slot id? Or what's the expectation to use these infos?

Best,
Zhijiang
--
From:Kurt Young 
Send Time:2020 Feb. 21 (Fri.) 14:38
To:dev 
Subject:Re: [VOTE] FLIP-101: Add Pending Slots Detail

I agree with Jark, even if we have pending slots now, a dedicated tab seems
to be too much.

Best,
Kurt


On Fri, Feb 21, 2020 at 2:12 PM Jark Wu  wrote:

> Thanks Yadong,
>
> I think a pending slot view will be helpful. But will it be verbose when
> there is no pending slot, but a "pending slot" in the tab?
> What do you think to show the pending slot page when click the "?" on the
> vertex status?
>
> Best,
> Jark
>
> On Thu, 20 Feb 2020 at 17:50, Yadong Xie  wrote:
>
> > Hi all
> >
> > I want to start the vote for FLIP-101, which proposes to add pending
> slots
> > information to help users check which vertex/subtask is blocked.
> >
> > To help everyone better understand the proposal, we spent some efforts on
> > making an online POC
> >
> > previous web:
> >
> http://101.132.122.69:8081/#/job/b88840a1e71a0535e1556b52c4c12fcc/overview
> > POC web:
> >
> >
> http://101.132.122.69:8081/web/#/job/b88840a1e71a0535e1556b52c4c12fcc/pending-slots
> >
> >
> > The vote will last for at least 72 hours, following the consensus voting
> > process.
> >
> > FLIP wiki:
> >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-101%3A+Add+Pending+Slots+Detail
> >
> > Discussion thread:
> >
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-75-Flink-Web-UI-Improvement-Proposal-td33540.html
> >
> > Thanks,
> >
> > Yadong
> >
>

Re: [Discuss] Update the pull request description template.

[Zhijiang]

Thanks for launching this discussion and all the involved feedbacks!

Since there are still many users relying on the template to raise attentions 
and guide review, it sounds reasonable to update the template based on demands.

In my experience, I was always filling the template when submitting PRs before. 
But I found a bit trouble for filling the last two sections
 "Does this pull request potentially affect one of the following parts:" and 
"Documentation", because I had to either remove one from "yes|no" or highlight 
one
manually for every listed item. And I guess for most of PRs, the results of 
these items should be "no" by default.  

If we can refactor to another description here, E.g. "Selecting the following 
parts which this pull request potentially affects", and further make every item 
selectable instead.
Then most of the users do not need to touch these sections by default , which 
means without implication.  I guess it would save some efforts.

My above concern is tiny and might not be the key motivation of this 
discussion. Just share my thoughts by this chance. :)

Best,
Zhijiang


--
From:Yangze Guo 
Send Time:2020 Feb. 21 (Fri.) 16:16
To:dev 
Subject:Re: [Discuss] Update the pull request description template.

In my experience, the template is helpful. Especially for the people
just joined the community and give their first PR. I don't know how
many people have read the contributor guide entirely before they
commit their first PR, but I should admit that I did not read it word
by word for the first time, since not all of the items related to my
work. However, the template forces me to check the basic rules and
guidelines of the community.
Another benefit I can think of is to remind people who touch the code
path they aren't familiar with. If that needs a special test flow, the
template forces them to follow it.

Best,
Yangze Guo

On Fri, Feb 21, 2020 at 2:39 PM Yang Wang  wrote:
>
> I second xintong's suggestion. When i open a PR, i also check the item list
> in the template. It help to
> know whether i should test the PR in a real cluster(Yarn/K8s/Mesos). Or i
> should be more careful
> when touching the per-record code paths.If we have some dependencies
> changes, i will need to check
> the generated jar as expected.
>
>
> Best,
> Yang
>
> Xintong Song  于2020年2月20日周四 上午10:33写道：
>
> > Thanks for the feedbacks, Chesnay and Till. And thanks for the pointer,
> > Congxian.
> >
> > I don't know how often committers and reviewers checks and benefits from
> > the PR description. From your feedbacks and the number of responses to this
> > discussion, it's probably not often.
> >
> > However, as a contributor and speaking only for myself, I actually find the
> > PR template very helpful. I use it as a checking list for opening a PR.
> > Filling in the template forces me to revisit the important things, e.g.,
> > have I added enough test cases to cover the all the important changes, does
> > this change need to be validated with a real deployment (if it touches the
> > deployment and recovery). An experienced developer might be able to check
> > these things without such a checking list, but there might be more primary
> > developers that can benefit from it.
> >
> >
> > Therefore, if we agree that PR template is less useful for reviewers, I
> > would like to propose to reposition it as a contributor checking list. The
> > following are some examples of how the existing items might be
> > repositioned.
> >
> >
> > - The runtime per-record code paths (performance sensitive): (yes / no /
> > don't know). If yes, please check the following items.
> >
> > - Is there a good reason to do that?
> > - Is there an alternative non pre-record approach?
> >
> > - Is Java stream or Optional used in the per-recode code path? (Those
> > should be avoid according to the code style and quality guide[1])
> >
> > - Do we know the exact impact on performance? (Maybe point to the
> > performance benchmarks)
> >
> >
> > - Anything that affects deployment or recovery: JobManager (and its
> > components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: (yes / no /
> > don't know). If yes, please check the following items.
> >
> > - Has this PR been validated with a real deployment?
> >
> > - Has this PR been validated with the failover scenarios?
> >
> > - Does this PR requires any specific version or configuration of an
> > external system? E.g., Kubernetes/Yarn/Mesos/ZooKeeper APIs not supported
> > by all the versions that Flink claims to be compatible with.
> >
> >
> > WDYT?
> >
> >
> > Thank you~
> >
> > Xintong Song
> >
> >
> > [1]https://flink.apache.org/contributing/code-style-and-quality-java.html
> >
> > On Wed, Feb 19, 2020 at 9:24 PM Till Rohrmann 
> > wrote:
> >
> > > I actually wanted to second Chesnay but apparently my impression is a bit
> > > wrong. Out of the last 10 closed PRs (admittedly a small sample size)
> > only
> > > 2 did not 

Re: [ANNOUNCE] Jingsong Lee becomes a Flink committer

[Rong Rong]

Congratulations Jingsong!!

Cheers,
Rong

On Fri, Feb 21, 2020 at 8:45 AM Bowen Li  wrote:

> Congrats, Jingsong!
>
> On Fri, Feb 21, 2020 at 7:28 AM Till Rohrmann 
> wrote:
>
>> Congratulations Jingsong!
>>
>> Cheers,
>> Till
>>
>> On Fri, Feb 21, 2020 at 4:03 PM Yun Gao  wrote:
>>
>>>   Congratulations Jingsong!
>>>
>>>Best,
>>>Yun
>>>
>>> --
>>> From:Jingsong Li 
>>> Send Time:2020 Feb. 21 (Fri.) 21:42
>>> To:Hequn Cheng 
>>> Cc:Yang Wang ; Zhijiang <
>>> wangzhijiang...@aliyun.com>; Zhenghua Gao ; godfrey
>>> he ; dev ; user <
>>> u...@flink.apache.org>
>>> Subject:Re: [ANNOUNCE] Jingsong Lee becomes a Flink committer
>>>
>>> Thanks everyone~
>>>
>>> It's my pleasure to be part of the community. I hope I can make a better
>>> contribution in future.
>>>
>>> Best,
>>> Jingsong Lee
>>>
>>> On Fri, Feb 21, 2020 at 2:48 PM Hequn Cheng  wrote:
>>> Congratulations Jingsong! Well deserved.
>>>
>>> Best,
>>> Hequn
>>>
>>> On Fri, Feb 21, 2020 at 2:42 PM Yang Wang  wrote:
>>> Congratulations！Jingsong. Well deserved.
>>>
>>>
>>> Best,
>>> Yang
>>>
>>> Zhijiang  于2020年2月21日周五 下午1:18写道：
>>> Congrats Jingsong! Welcome on board!
>>>
>>> Best,
>>> Zhijiang
>>>
>>> --
>>> From:Zhenghua Gao 
>>> Send Time:2020 Feb. 21 (Fri.) 12:49
>>> To:godfrey he 
>>> Cc:dev ; user 
>>> Subject:Re: [ANNOUNCE] Jingsong Lee becomes a Flink committer
>>>
>>> Congrats Jingsong！
>>>
>>>
>>> *Best Regards,*
>>> *Zhenghua Gao*
>>>
>>>
>>> On Fri, Feb 21, 2020 at 11:59 AM godfrey he  wrote:
>>> Congrats Jingsong! Well deserved.
>>>
>>> Best,
>>> godfrey
>>>
>>> Jeff Zhang  于2020年2月21日周五 上午11:49写道：
>>> Congratulations！Jingsong. You deserve it
>>>
>>> wenlong.lwl  于2020年2月21日周五 上午11:43写道：
>>> Congrats Jingsong!
>>>
>>> On Fri, 21 Feb 2020 at 11:41, Dian Fu  wrote:
>>>
>>> > Congrats Jingsong!
>>> >
>>> > > 在 2020年2月21日，上午11:39，Jark Wu  写道：
>>> > >
>>> > > Congratulations Jingsong! Well deserved.
>>> > >
>>> > > Best,
>>> > > Jark
>>> > >
>>> > > On Fri, 21 Feb 2020 at 11:32, zoudan  wrote:
>>> > >
>>> > >> Congratulations! Jingsong
>>> > >>
>>> > >>
>>> > >> Best,
>>> > >> Dan Zou
>>> > >>
>>> >
>>> >
>>>
>>>
>>> --
>>> Best Regards
>>>
>>> Jeff Zhang
>>>
>>>
>>>
>>> --
>>> Best, Jingsong Lee
>>>
>>>
>>>

Re: [DISCUSS] FLINK-16194: Refactor the Kubernetes architecture design

[felixzheng zheng]

Thanks for the feedback @Yang Wang. I would like to discuss some of the
details in depth about why I am confused about the existing design.

Question 1: How do we mount a configuration file?

For the existing design,

   1.

   We need several classes to finish it:
   1.

  InitializerDecorator
  2.

  OwnerReferenceDecorator
  3.

  ConfigMapDecorator
  4.

  KubernetesUtils: providing the getConfigMapVolume method to share for
  the FlinkMasterDeploymentDecorator and the TaskManagerPodDecorator.
  5.

  FlinkMasterDeploymentDecorator: mounts the ConfigMap volume.
  6.

  TaskManagerPodDecorator: mounts the ConfigMap volume.
  7.

  If in the future, someone would like to introduce an init Container,
  the InitContainerDecorator has to mount the ConfigMap volume too.


I am confused about the current solution to mounting a configuration file:

   1.

   Actually, we do not need so many Decorators for mounting a file.
   2.

   If we would like to mount a new file, we have no choice but to repeat
   the same tedious and scattered routine.
   3.

   There’s no easy way to test the file mounting functionality alone; we
   have to construct the ConfigMap, the Deployment or the TaskManagerPod first
   and then do a final test.


The reason why it is so complex to mount a configuration file is that we
don’t fully consider the internal connections among those resources in the
existing design.

The new abstraction we proposed could solve such a kind of problem, the new
Decorator object is as follows:

public interface KubernetesStepDecorator {

  /**

   * Apply transformations to the given FlinkPod in accordance with this
feature. This can include adding

   * labels/annotations, mounting volumes, and setting startup command or
parameters, etc.

   */

  FlinkPod decorateFlinkPod(FlinkPod flinkPod);

  /**

   * Build the accompanying Kubernetes resources that should be introduced
to support this feature. This could

   * only applicable to the client-side submission process.

   */

  List buildAccompanyingKubernetesResources() throws
IOException;

}

The FlinkPod is a composition of the Pod, the main Container, the init
Container, and the sidecar Container.

Next, we introduce a KubernetesStepDecorator implementation, the method of
buildAccompanyingKubernetesResources creates the corresponding ConfigMap,
and the method of decorateFlinkPod configures the Volume for the Pod and
all the Containers.

So, for the scenario of mounting a configuration file, the advantages of
this new architecture are as follows:

   1.

   One dedicated KubernetesStepDecorator implementation is enough to finish
   all the mounting work, meanwhile, this class would be shared between the
   client-side and the master-side. Besides that, the number of lines of code
   in that class will not exceed 300 lines which facilitate code readability
   and maintenance.
   2.

   Testing becomes an easy thing now, as we can add a dedicated test class
   for only this class, the test would never rely on the construction of other
   components such as the Deployment.
   3.

   It’s quite convenient to mount a new configuration file via the newly
   dedicated KubernetesStepDecorator implementation.


Question 2: How do we construct the Pod?

For the existing design,

   1.

   The FlinkMasterDeploymentDecorator is responsible for building the
   Deployment and configuring the Pod and the Containers, while the
   TaskManagerPodDecorator is responsible for building the TaskManager Pod.
   Take FlinkMasterDeploymentDecorator as an example, let’s see what it has
   done.
   1.

  Configure the main Container, including the name, command, args,
  image, image pull policy, resource requirements, ports, all kinds of
  environment variables, all kinds of volume mounts, etc.
  2.

  Configure the Pod, including service account, all kinds of volumes,
  and attach all kinds of Container, including the main Container, the init
  Container, and the sidecar Container.
  3.

  Configure the Deployment.



   1.

   The InitializerDecorator and the OwnerReferenceDecorator have basic
   logic so that the most complex work is completed in the
   FlinkMasterDeploymentDecorator and the TaskManagerPodDecorator. With the
   introduction of new features for the Pod, such as customized volume mounts,
   Hadoop configuration support, Kerberized HDFS support, secret mounts,
   Python support, etc. the construction process could become far more
   complicated, and the functionality of a single class could explode, which
   hurts code readability, writability, and testability. Besides that, both
   the client-side and the master-side shares much of the Pod construction
   logic.


So the problems are as follows:

   1.

   We don’t have a consistent abstraction that is applicable to both the
   client-side and the master-side for Pod construction (including the
   Containers) so 

Re: [ANNOUNCE] Jingsong Lee becomes a Flink committer

[Fabian Hueske]

Congrats Jingsong!

Cheers, Fabian

Am Fr., 21. Feb. 2020 um 17:49 Uhr schrieb Rong Rong :

> Congratulations Jingsong!!
>
> Cheers,
> Rong
>
> On Fri, Feb 21, 2020 at 8:45 AM Bowen Li  wrote:
>
> > Congrats, Jingsong!
> >
> > On Fri, Feb 21, 2020 at 7:28 AM Till Rohrmann 
> > wrote:
> >
> >> Congratulations Jingsong!
> >>
> >> Cheers,
> >> Till
> >>
> >> On Fri, Feb 21, 2020 at 4:03 PM Yun Gao  wrote:
> >>
> >>>   Congratulations Jingsong!
> >>>
> >>>Best,
> >>>Yun
> >>>
> >>> --
> >>> From:Jingsong Li 
> >>> Send Time:2020 Feb. 21 (Fri.) 21:42
> >>> To:Hequn Cheng 
> >>> Cc:Yang Wang ; Zhijiang <
> >>> wangzhijiang...@aliyun.com>; Zhenghua Gao ; godfrey
> >>> he ; dev ; user <
> >>> u...@flink.apache.org>
> >>> Subject:Re: [ANNOUNCE] Jingsong Lee becomes a Flink committer
> >>>
> >>> Thanks everyone~
> >>>
> >>> It's my pleasure to be part of the community. I hope I can make a
> better
> >>> contribution in future.
> >>>
> >>> Best,
> >>> Jingsong Lee
> >>>
> >>> On Fri, Feb 21, 2020 at 2:48 PM Hequn Cheng  wrote:
> >>> Congratulations Jingsong! Well deserved.
> >>>
> >>> Best,
> >>> Hequn
> >>>
> >>> On Fri, Feb 21, 2020 at 2:42 PM Yang Wang 
> wrote:
> >>> Congratulations！Jingsong. Well deserved.
> >>>
> >>>
> >>> Best,
> >>> Yang
> >>>
> >>> Zhijiang  于2020年2月21日周五 下午1:18写道：
> >>> Congrats Jingsong! Welcome on board!
> >>>
> >>> Best,
> >>> Zhijiang
> >>>
> >>> --
> >>> From:Zhenghua Gao 
> >>> Send Time:2020 Feb. 21 (Fri.) 12:49
> >>> To:godfrey he 
> >>> Cc:dev ; user 
> >>> Subject:Re: [ANNOUNCE] Jingsong Lee becomes a Flink committer
> >>>
> >>> Congrats Jingsong！
> >>>
> >>>
> >>> *Best Regards,*
> >>> *Zhenghua Gao*
> >>>
> >>>
> >>> On Fri, Feb 21, 2020 at 11:59 AM godfrey he 
> wrote:
> >>> Congrats Jingsong! Well deserved.
> >>>
> >>> Best,
> >>> godfrey
> >>>
> >>> Jeff Zhang  于2020年2月21日周五 上午11:49写道：
> >>> Congratulations！Jingsong. You deserve it
> >>>
> >>> wenlong.lwl  于2020年2月21日周五 上午11:43写道：
> >>> Congrats Jingsong!
> >>>
> >>> On Fri, 21 Feb 2020 at 11:41, Dian Fu  wrote:
> >>>
> >>> > Congrats Jingsong!
> >>> >
> >>> > > 在 2020年2月21日，上午11:39，Jark Wu  写道：
> >>> > >
> >>> > > Congratulations Jingsong! Well deserved.
> >>> > >
> >>> > > Best,
> >>> > > Jark
> >>> > >
> >>> > > On Fri, 21 Feb 2020 at 11:32, zoudan  wrote:
> >>> > >
> >>> > >> Congratulations! Jingsong
> >>> > >>
> >>> > >>
> >>> > >> Best,
> >>> > >> Dan Zou
> >>> > >>
> >>> >
> >>> >
> >>>
> >>>
> >>> --
> >>> Best Regards
> >>>
> >>> Jeff Zhang
> >>>
> >>>
> >>>
> >>> --
> >>> Best, Jingsong Lee
> >>>
> >>>
> >>>
>

Re: [ANNOUNCE] Jingsong Lee becomes a Flink committer

[Till Rohrmann]

Congratulations Jingsong!

Cheers,
Till

On Fri, Feb 21, 2020 at 4:03 PM Yun Gao  wrote:

>   Congratulations Jingsong!
>
>Best,
>Yun
>
> --
> From:Jingsong Li 
> Send Time:2020 Feb. 21 (Fri.) 21:42
> To:Hequn Cheng 
> Cc:Yang Wang ; Zhijiang ;
> Zhenghua Gao ; godfrey he ; dev <
> dev@flink.apache.org>; user 
> Subject:Re: [ANNOUNCE] Jingsong Lee becomes a Flink committer
>
> Thanks everyone~
>
> It's my pleasure to be part of the community. I hope I can make a better
> contribution in future.
>
> Best,
> Jingsong Lee
>
> On Fri, Feb 21, 2020 at 2:48 PM Hequn Cheng  wrote:
> Congratulations Jingsong! Well deserved.
>
> Best,
> Hequn
>
> On Fri, Feb 21, 2020 at 2:42 PM Yang Wang  wrote:
> Congratulations！Jingsong. Well deserved.
>
>
> Best,
> Yang
>
> Zhijiang  于2020年2月21日周五 下午1:18写道：
> Congrats Jingsong! Welcome on board!
>
> Best,
> Zhijiang
>
> --
> From:Zhenghua Gao 
> Send Time:2020 Feb. 21 (Fri.) 12:49
> To:godfrey he 
> Cc:dev ; user 
> Subject:Re: [ANNOUNCE] Jingsong Lee becomes a Flink committer
>
> Congrats Jingsong！
>
>
> *Best Regards,*
> *Zhenghua Gao*
>
>
> On Fri, Feb 21, 2020 at 11:59 AM godfrey he  wrote:
> Congrats Jingsong! Well deserved.
>
> Best,
> godfrey
>
> Jeff Zhang  于2020年2月21日周五 上午11:49写道：
> Congratulations！Jingsong. You deserve it
>
> wenlong.lwl  于2020年2月21日周五 上午11:43写道：
> Congrats Jingsong!
>
> On Fri, 21 Feb 2020 at 11:41, Dian Fu  wrote:
>
> > Congrats Jingsong!
> >
> > > 在 2020年2月21日，上午11:39，Jark Wu  写道：
> > >
> > > Congratulations Jingsong! Well deserved.
> > >
> > > Best,
> > > Jark
> > >
> > > On Fri, 21 Feb 2020 at 11:32, zoudan  wrote:
> > >
> > >> Congratulations! Jingsong
> > >>
> > >>
> > >> Best,
> > >> Dan Zou
> > >>
> >
> >
>
>
> --
> Best Regards
>
> Jeff Zhang
>
>
>
> --
> Best, Jingsong Lee
>
>
>

[jira] [Created] (FLINK-16222) Use plugins mechanism for initializing MetricReporters

[Alexander Fedulov (Jira)]

Alexander Fedulov created FLINK-16222:
-

 Summary: Use plugins mechanism for initializing MetricReporters
 Key: FLINK-16222
 URL: https://issues.apache.org/jira/browse/FLINK-16222
 Project: Flink
  Issue Type: Improvement
  Components: Runtime / Metrics
Reporter: Alexander Fedulov


https://issues.apache.org/jira/browse/FLINK-11952 introduced Plugins mechanism 
into Flink. Metrics reporters initialization mechanism can profit from using 
this new functionality. Instead of placing MetricsReporters JARs into /libs, it 
should be additionally possible (and encouraged) to convert them into plugins 
and use the /plugins folder for initialization via independent plugin 
classloaders. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-16226) Add back pressure to HttpFunction.

[Igal Shilman (Jira)]

Igal Shilman created FLINK-16226:


 Summary: Add back pressure to HttpFunction.
 Key: FLINK-16226
 URL: https://issues.apache.org/jira/browse/FLINK-16226
 Project: Flink
  Issue Type: Task
  Components: Stateful Functions
Reporter: Igal Shilman
Assignee: Igal Shilman


Recently a simple back pressure mechanism was introduced to stateful functions.
Now it can be used from the HttpFunction to keep the request backlog under 
control.




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [ANNOUNCE] Jingsong Lee becomes a Flink committer

[Peter Huang]

Congrats Jingsong!


On Fri, Feb 21, 2020 at 8:49 AM Rong Rong  wrote:

> Congratulations Jingsong!!
>
> Cheers,
> Rong
>
> On Fri, Feb 21, 2020 at 8:45 AM Bowen Li  wrote:
>
>> Congrats, Jingsong!
>>
>> On Fri, Feb 21, 2020 at 7:28 AM Till Rohrmann 
>> wrote:
>>
>>> Congratulations Jingsong!
>>>
>>> Cheers,
>>> Till
>>>
>>> On Fri, Feb 21, 2020 at 4:03 PM Yun Gao  wrote:
>>>
   Congratulations Jingsong!

Best,
Yun

 --
 From:Jingsong Li 
 Send Time:2020 Feb. 21 (Fri.) 21:42
 To:Hequn Cheng 
 Cc:Yang Wang ; Zhijiang <
 wangzhijiang...@aliyun.com>; Zhenghua Gao ; godfrey
 he ; dev ; user <
 u...@flink.apache.org>
 Subject:Re: [ANNOUNCE] Jingsong Lee becomes a Flink committer

 Thanks everyone~

 It's my pleasure to be part of the community. I hope I can make a
 better contribution in future.

 Best,
 Jingsong Lee

 On Fri, Feb 21, 2020 at 2:48 PM Hequn Cheng  wrote:
 Congratulations Jingsong! Well deserved.

 Best,
 Hequn

 On Fri, Feb 21, 2020 at 2:42 PM Yang Wang 
 wrote:
 Congratulations！Jingsong. Well deserved.


 Best,
 Yang

 Zhijiang  于2020年2月21日周五 下午1:18写道：
 Congrats Jingsong! Welcome on board!

 Best,
 Zhijiang

 --
 From:Zhenghua Gao 
 Send Time:2020 Feb. 21 (Fri.) 12:49
 To:godfrey he 
 Cc:dev ; user 
 Subject:Re: [ANNOUNCE] Jingsong Lee becomes a Flink committer

 Congrats Jingsong！


 *Best Regards,*
 *Zhenghua Gao*


 On Fri, Feb 21, 2020 at 11:59 AM godfrey he 
 wrote:
 Congrats Jingsong! Well deserved.

 Best,
 godfrey

 Jeff Zhang  于2020年2月21日周五 上午11:49写道：
 Congratulations！Jingsong. You deserve it

 wenlong.lwl  于2020年2月21日周五 上午11:43写道：
 Congrats Jingsong!

 On Fri, 21 Feb 2020 at 11:41, Dian Fu  wrote:

 > Congrats Jingsong!
 >
 > > 在 2020年2月21日，上午11:39，Jark Wu  写道：
 > >
 > > Congratulations Jingsong! Well deserved.
 > >
 > > Best,
 > > Jark
 > >
 > > On Fri, 21 Feb 2020 at 11:32, zoudan  wrote:
 > >
 > >> Congratulations! Jingsong
 > >>
 > >>
 > >> Best,
 > >> Dan Zou
 > >>
 >
 >


 --
 Best Regards

 Jeff Zhang



 --
 Best, Jingsong Lee

[jira] [Created] (FLINK-16221) Execution::transitionState() should log an error when error parameter is not null

[yazgoo (Jira)]

yazgoo created FLINK-16221:
--

 Summary: Execution::transitionState() should log an error when 
error parameter is not null 
 Key: FLINK-16221
 URL: https://issues.apache.org/jira/browse/FLINK-16221
 Project: Flink
  Issue Type: Improvement
  Components: Runtime / Coordination
Affects Versions: 1.10.0, 1.9.2
Reporter: yazgoo
 Attachments: info_to_error.patch

When execution state transitions with an error, an INFO is logged n 
Execution::transitionState().
I think an ERROR should be logged.
This is especially usefull when states transitions to failing, to be able to 
retrieve the error causing the failure.
So:
|LOG.error("{} ({}) switched from {} to {}.", 
getVertex().getTaskNameWithSubtaskIndex(), getAttemptId(), currentState, 
targetState, error);|


should become
|LOG.info("{} ({}) switched from {} to {}.", 
getVertex().getTaskNameWithSubtaskIndex(), getAttemptId(), currentState, 
targetState, error);|



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-16225) Metaspace Out Of Memory should be handled as Fatal Error in TaskManager

[Stephan Ewen (Jira)]

Stephan Ewen created FLINK-16225:


 Summary: Metaspace Out Of Memory should be handled as Fatal Error 
in TaskManager
 Key: FLINK-16225
 URL: https://issues.apache.org/jira/browse/FLINK-16225
 Project: Flink
  Issue Type: Improvement
  Components: Runtime / Task
Reporter: Stephan Ewen
 Fix For: 1.11.0


When an {{OutOfMemory (Metaspace)}} exception happens, there is usually no way 
to recover. This is often the result of user code or libraries that have subtle 
class loading leaks.

The one way to recover is to kill the TaskManagers and to let the resource 
orchestrators (K8s, Yarn, Mesos) restart them. Flink's fault tolerance should 
then be able to recover the job.

I would suggest to implement this the following way:
* The user code ClassLoader takes an "OOM Handler", which is called when class 
loading causes an OOM exception.
* The handler wraps this into an Exception with a good error message (see 
below) and invokes the TaskManager's {{FatalErrorHandler}}.
* The {{FatalErrorHandler}} in turn should attempt to cancel everything and 
notify the JM before shutting down. That way, we get decent error reporting and 
users can see what is going on.


The error message should describe the following:
* If user sees the error consistently on the first deploy, then the metaspace 
is simply too small for their application, and they need to explicitly increase 
it in the configuration
* If the user sees occasionally TaskManagers in a session cluster failing with 
that exception when deploying new jobs, then some user code or library probably 
has a class leak. The TM failure / restart is done in order to forcefully clean 
up.




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-16224) Refine Hadoop Delegation Token based testing framework

[Rong Rong (Jira)]

Rong Rong created FLINK-16224:
-

 Summary: Refine Hadoop Delegation Token based testing framework
 Key: FLINK-16224
 URL: https://issues.apache.org/jira/browse/FLINK-16224
 Project: Flink
  Issue Type: Sub-task
  Components: Deployment / YARN
Reporter: Rong Rong
Assignee: Rong Rong


Currently the SecureTestEnvironment doesn't support Hadoop delegation token, 
which makes the E2E testing of delegation-token-based YARN application 
impossible.

Propose to enhance the testing framework to support delegation token based 
launch.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-16230) Use LinkedHashMap instead of HashMap for a deterministic order when testing serialization

[cpugputpu (Jira)]

cpugputpu created FLINK-16230:
-

 Summary: Use LinkedHashMap instead of HashMap for a deterministic 
order when testing serialization
 Key: FLINK-16230
 URL: https://issues.apache.org/jira/browse/FLINK-16230
 Project: Flink
  Issue Type: Bug
 Environment: TEST: 
org.apache.flink.api.java.typeutils.runtime.kryo.KryoGenericTypeSerializerTest#testJavaSet

StackTrace:

java.util.HashMap$HashIterator$HashIteratorShuffler.
java.util.HashMap$HashIterator.(HashMap.java:1435)
java.util.HashMap$KeyIterator.(HashMap.java:1467)
java.util.HashMap$KeySet.iterator(HashMap.java:917)
java.util.HashSet.iterator(HashSet.java:173)
org.apache.flink.testutils.DeeplyEqualsChecker.deepEqualsIterable(DeeplyEqualsChecker.java:107)
org.apache.flink.testutils.DeeplyEqualsChecker.deepEquals0(DeeplyEqualsChecker.java:94)
org.apache.flink.testutils.DeeplyEqualsChecker.lambda$deepEquals$0(DeeplyEqualsChecker.java:79)
java.util.Optional.orElseGet(Optional.java:267)
org.apache.flink.testutils.DeeplyEqualsChecker.deepEquals(DeeplyEqualsChecker.java:79)
org.apache.flink.testutils.CustomEqualityMatcher.matches(CustomEqualityMatcher.java:63)
org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:12)
org.junit.Assert.assertThat(Assert.java:956)
org.apache.flink.api.common.typeutils.SerializerTestBase.deepEquals(SerializerTestBase.java:493)
org.apache.flink.api.common.typeutils.SerializerTestBase.testSerializedCopyIndividually(SerializerTestBase.java:379)
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
java.lang.reflect.Method.invoke(Method.java:498)
org.apache.flink.api.common.typeutils.SerializerTestInstance.testAll(SerializerTestInstance.java:92)
org.apache.flink.api.java.typeutils.runtime.AbstractGenericTypeSerializerTest.runTests(AbstractGenericTypeSerializerTest.java:155)
org.apache.flink.api.java.typeutils.runtime.kryo.KryoGenericTypeSerializerTest.testJavaSet(KryoGenericTypeSerializerTest.java:59)
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
java.lang.reflect.Method.invoke(Method.java:498)
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
org.junit.runners.ParentRunner.run(ParentRunner.java:363)
org.junit.runners.Suite.runChild(Suite.java:128)
org.junit.runners.Suite.runChild(Suite.java:27)
org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
org.junit.runners.ParentRunner.run(ParentRunner.java:363)
org.apache.maven.surefire.junitcore.JUnitCore.run(JUnitCore.java:55)
org.apache.maven.surefire.junitcore.JUnitCoreWrapper.createRequestAndRun(JUnitCoreWrapper.java:137)
org.apache.maven.surefire.junitcore.JUnitCoreWrapper.executeEager(JUnitCoreWrapper.java:107)
org.apache.maven.surefire.junitcore.JUnitCoreWrapper.execute(JUnitCoreWrapper.java:83)
org.apache.maven.surefire.junitcore.JUnitCoreWrapper.execute(JUnitCoreWrapper.java:75)
org.apache.maven.surefire.junitcore.JUnitCoreProvider.invoke(JUnitCoreProvider.java:158)
org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
Reporter: cpugputpu


The test in 
org.apache.flink.api.java.typeutils.runtime.kryo.KryoGenericTypeSerializerTest.testJavaSet(KryoGenericTypeSerializerTest.java:59)
 may fail due to a different iteration order of HashSet. The test aims to check 
the 

Re: [DISCUSS] FLINK-16194: Refactor the Kubernetes architecture design

[felixzheng zheng]

Great thanks for the quick feedback Till. You are right; it is not a
fundamentally different approach compared to
what we have right now, all the Kubernetes resources created are the same,
we aim to evolve the existing decorator approach so that,
1. the decorators are monadic and smaller in size and functionality.
2. the new decorator design allows reusing the decorators between the
client and the cluster as much as possible.
3. all the decorators are independent with each other, and they could have
arbitrary order in the chain, they share the same APIs and follow a unified
orchestrator architecture so that new developers could quickly understand
what should be done to introduce a new feature.

Besides that, the new approach allows us adding tests for every decorator
alone instead of doing a final test of all the decorators in the
Fabric8ClientTest.java.

Cheers,
Canbin Zheng

Till Rohrmann  于2020年2月22日周六 上午12:28写道：

> Thanks for starting this discussion Canbin. If I understand your proposal
> correctly, then you would like to evolve the existing decorator approach so
> that decorators are monadic and smaller in size and functionality. The
> latter aspect will allow to reuse them between the client and the cluster.
> Just to make sure, it is not a fundamentally different approach compared to
> what we have right now, is it?
>
> If this is the case, then I think it makes sense to reuse code as much as
> possible and to create small code units which are easier to test.
>
> Cheers,
> Till
>
> On Fri, Feb 21, 2020 at 4:41 PM felixzheng zheng 
> wrote:
>
> > Thanks for the feedback @Yang Wang. I would like to discuss some of the
> > details in depth about why I am confused about the existing design.
> >
> > Question 1: How do we mount a configuration file?
> >
> > For the existing design,
> >
> >1.
> >
> >We need several classes to finish it:
> >1.
> >
> >   InitializerDecorator
> >   2.
> >
> >   OwnerReferenceDecorator
> >   3.
> >
> >   ConfigMapDecorator
> >   4.
> >
> >   KubernetesUtils: providing the getConfigMapVolume method to share
> for
> >   the FlinkMasterDeploymentDecorator and the TaskManagerPodDecorator.
> >   5.
> >
> >   FlinkMasterDeploymentDecorator: mounts the ConfigMap volume.
> >   6.
> >
> >   TaskManagerPodDecorator: mounts the ConfigMap volume.
> >   7.
> >
> >   If in the future, someone would like to introduce an init
> Container,
> >   the InitContainerDecorator has to mount the ConfigMap volume too.
> >
> >
> > I am confused about the current solution to mounting a configuration
> file:
> >
> >1.
> >
> >Actually, we do not need so many Decorators for mounting a file.
> >2.
> >
> >If we would like to mount a new file, we have no choice but to repeat
> >the same tedious and scattered routine.
> >3.
> >
> >There’s no easy way to test the file mounting functionality alone; we
> >have to construct the ConfigMap, the Deployment or the TaskManagerPod
> > first
> >and then do a final test.
> >
> >
> > The reason why it is so complex to mount a configuration file is that we
> > don’t fully consider the internal connections among those resources in
> the
> > existing design.
> >
> > The new abstraction we proposed could solve such a kind of problem, the
> new
> > Decorator object is as follows:
> >
> > public interface KubernetesStepDecorator {
> >
> >   /**
> >
> >* Apply transformations to the given FlinkPod in accordance with this
> > feature. This can include adding
> >
> >* labels/annotations, mounting volumes, and setting startup command or
> > parameters, etc.
> >
> >*/
> >
> >   FlinkPod decorateFlinkPod(FlinkPod flinkPod);
> >
> >   /**
> >
> >* Build the accompanying Kubernetes resources that should be
> introduced
> > to support this feature. This could
> >
> >* only applicable to the client-side submission process.
> >
> >*/
> >
> >   List buildAccompanyingKubernetesResources() throws
> > IOException;
> >
> > }
> >
> > The FlinkPod is a composition of the Pod, the main Container, the init
> > Container, and the sidecar Container.
> >
> > Next, we introduce a KubernetesStepDecorator implementation, the method
> of
> > buildAccompanyingKubernetesResources creates the corresponding ConfigMap,
> > and the method of decorateFlinkPod configures the Volume for the Pod and
> > all the Containers.
> >
> > So, for the scenario of mounting a configuration file, the advantages of
> > this new architecture are as follows:
> >
> >1.
> >
> >One dedicated KubernetesStepDecorator implementation is enough to
> finish
> >all the mounting work, meanwhile, this class would be shared between
> the
> >client-side and the master-side. Besides that, the number of lines of
> > code
> >in that class will not exceed 300 lines which facilitate code
> > readability
> >and maintenance.
> >2.
> >
> >Testing becomes an easy thing now, as we can add a 

[jira] [Created] (FLINK-16227) Streaming bucketing end-to-end test / test_streaming_bucketing.sh unstable

[Robert Metzger (Jira)]

Robert Metzger created FLINK-16227:
--

 Summary: Streaming bucketing end-to-end test / 
test_streaming_bucketing.sh unstable
 Key: FLINK-16227
 URL: https://issues.apache.org/jira/browse/FLINK-16227
 Project: Flink
  Issue Type: Bug
  Components: API / DataStream, Tests
Affects Versions: 1.11.0
Reporter: Robert Metzger


This nightly cron job has failed: 
https://travis-ci.org/apache/flink/jobs/653454540

{code}
==
Running 'Streaming bucketing end-to-end test'
==
TEST_DATA_DIR: 
/home/travis/build/apache/flink/flink-end-to-end-tests/test-scripts/temp-test-directory-05739414867
Flink dist directory: 
/home/travis/build/apache/flink/flink-dist/target/flink-1.11-SNAPSHOT-bin/flink-1.11-SNAPSHOT
Setting up SSL with: internal JDK dynamic
Using SAN 
dns:travis-job-b9e26d64-0a62-42c7-9802-6c49defb4ad7,ip:10.20.0.145,ip:172.17.0.1
Certificate was added to keystore
Certificate was added to keystore
Certificate reply was installed in keystore
MAC verified OK
Setting up SSL with: rest JDK dynamic
Using SAN 
dns:travis-job-b9e26d64-0a62-42c7-9802-6c49defb4ad7,ip:10.20.0.145,ip:172.17.0.1
Certificate was added to keystore
Certificate was added to keystore
Certificate reply was installed in keystore
MAC verified OK
Mutual ssl auth: false
Starting cluster.
Starting standalonesession daemon on host 
travis-job-b9e26d64-0a62-42c7-9802-6c49defb4ad7.
Starting taskexecutor daemon on host 
travis-job-b9e26d64-0a62-42c7-9802-6c49defb4ad7.
Waiting for Dispatcher REST endpoint to come up...
Waiting for Dispatcher REST endpoint to come up...
Waiting for Dispatcher REST endpoint to come up...
Waiting for Dispatcher REST endpoint to come up...
Waiting for Dispatcher REST endpoint to come up...
Dispatcher REST endpoint is up.
[INFO] 1 instance(s) of taskexecutor are already running on 
travis-job-b9e26d64-0a62-42c7-9802-6c49defb4ad7.
Starting taskexecutor daemon on host 
travis-job-b9e26d64-0a62-42c7-9802-6c49defb4ad7.
[INFO] 2 instance(s) of taskexecutor are already running on 
travis-job-b9e26d64-0a62-42c7-9802-6c49defb4ad7.
Starting taskexecutor daemon on host 
travis-job-b9e26d64-0a62-42c7-9802-6c49defb4ad7.
[INFO] 3 instance(s) of taskexecutor are already running on 
travis-job-b9e26d64-0a62-42c7-9802-6c49defb4ad7.
Starting taskexecutor daemon on host 
travis-job-b9e26d64-0a62-42c7-9802-6c49defb4ad7.
Number of running task managers 1 is not yet 4.
Number of running task managers 2 is not yet 4.
Number of running task managers has reached 4.
java.lang.NoClassDefFoundError: org/apache/hadoop/conf/Configuration
at java.lang.Class.getDeclaredMethods0(Native Method)
at java.lang.Class.privateGetDeclaredMethods(Class.java:2701)
at java.lang.Class.getDeclaredMethod(Class.java:2128)
at 
org.apache.flink.api.java.ClosureCleaner.usesCustomSerialization(ClosureCleaner.java:164)
at 
org.apache.flink.api.java.ClosureCleaner.clean(ClosureCleaner.java:89)
at 
org.apache.flink.api.java.ClosureCleaner.clean(ClosureCleaner.java:71)
at 
org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.clean(StreamExecutionEnvironment.java:1820)
at 
org.apache.flink.streaming.api.datastream.DataStream.clean(DataStream.java:188)
at 
org.apache.flink.streaming.api.datastream.DataStream.addSink(DataStream.java:1328)
at 
org.apache.flink.streaming.tests.BucketingSinkTestProgram.main(BucketingSinkTestProgram.java:84)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:321)
at 
org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:205)
at 
org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:138)
at 
org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:664)
at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:213)
at 
org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:895)
at 
org.apache.flink.client.cli.CliFrontend.lambda$main$10(CliFrontend.java:968)
at 
org.apache.flink.runtime.security.contexts.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:30)
at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:968)
Caused by: java.lang.ClassNotFoundException: 
org.apache.hadoop.conf.Configuration
at 

[jira] [Created] (FLINK-16229) testCompressionOnRelativePath() fails due to Files.createDirectory() in macOS

[Liu (Jira)]

Liu created FLINK-16229:
---

 Summary: testCompressionOnRelativePath() fails due to 
Files.createDirectory() in macOS
 Key: FLINK-16229
 URL: https://issues.apache.org/jira/browse/FLINK-16229
 Project: Flink
  Issue Type: Bug
Reporter: Liu


      I am using flink 1.10.0. In macOS, cd flink/flink-core  and execute "mvn 
-Dtest=org.apache.flink.util.FileUtilsTest#testCompressionOnRelativePath test". 
It reports the following error.

      In linux, the test is ok. So I think that Files.createDirectory()  can 
not work well in mac. Should flink ignore this test or something better to do?

 

java.nio.file.NoSuchFileException: 
../../../../../var/folders/7h/3lhbyjl15m93hz9vpx303jvhgn/T/junit761366460676035615/compressDir/rootDirjava.nio.file.NoSuchFileException:
 
../../../../../var/folders/7h/3lhbyjl15m93hz9vpx303jvhgn/T/junit761366460676035615/compressDir/rootDir
 at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86) at 
sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) at 
sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107) at 
sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:384)
 at java.nio.file.Files.createDirectory(Files.java:674) at 
org.apache.flink.util.FileUtilsTest.verifyDirectoryCompression(FileUtilsTest.java:445)
 at 
org.apache.flink.util.FileUtilsTest.testCompressionOnRelativePath(FileUtilsTest.java:265)



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-16228) test_mesos_wordcount.sh fails

[Robert Metzger (Jira)]

Robert Metzger created FLINK-16228:
--

 Summary: test_mesos_wordcount.sh fails
 Key: FLINK-16228
 URL: https://issues.apache.org/jira/browse/FLINK-16228
 Project: Flink
  Issue Type: Bug
  Components: Deployment / Mesos, Tests
Affects Versions: 1.11.0
Reporter: Robert Metzger


In a recent cron build, the mesos wordcount test failed: 
https://travis-ci.org/apache/flink/jobs/653454544

{code}
2020-02-21 20:37:44,334 INFO  
org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Shutting 
MesosSessionClusterEntrypoint down with application status FAILED. Diagnostics 
java.lang.NoClassDefFoundError: org/apache/hadoop/security/UserGroupInformation
at 
org.apache.flink.runtime.clusterframework.overlays.HadoopUserOverlay$Builder.fromEnvironment(HadoopUserOverlay.java:74)
at 
org.apache.flink.mesos.util.MesosUtils.applyOverlays(MesosUtils.java:152)
at 
org.apache.flink.mesos.util.MesosUtils.createContainerSpec(MesosUtils.java:131)
at 
org.apache.flink.mesos.runtime.clusterframework.MesosResourceManagerFactory.createActiveResourceManager(MesosResourceManagerFactory.java:81)
at 
org.apache.flink.runtime.resourcemanager.ActiveResourceManagerFactory.createResourceManager(ActiveResourceManagerFactory.java:57)
at 
org.apache.flink.runtime.entrypoint.component.DefaultDispatcherResourceManagerComponentFactory.create(DefaultDispatcherResourceManagerComponentFactory.java:170)
at 
org.apache.flink.runtime.entrypoint.ClusterEntrypoint.runCluster(ClusterEntrypoint.java:215)
at 
org.apache.flink.runtime.entrypoint.ClusterEntrypoint.lambda$startCluster$0(ClusterEntrypoint.java:169)
at 
org.apache.flink.runtime.security.contexts.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:30)
at 
org.apache.flink.runtime.entrypoint.ClusterEntrypoint.startCluster(ClusterEntrypoint.java:168)
at 
org.apache.flink.runtime.entrypoint.ClusterEntrypoint.runClusterEntrypoint(ClusterEntrypoint.java:518)
at 
org.apache.flink.mesos.entrypoint.MesosSessionClusterEntrypoint.main(MesosSessionClusterEntrypoint.java:126)
Caused by: java.lang.ClassNotFoundException: 
org.apache.hadoop.security.UserGroupInformation
at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
at java.lang.ClassLoader.loadClass(ClassLoader.java:419)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
at java.lang.ClassLoader.loadClass(ClassLoader.java:352)
... 12 more
.
{code}




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-16218) Chinese character is garbled when reading from MySQL using JDBC source

[Jark Wu (Jira)]

Jark Wu created FLINK-16218:
---

 Summary: Chinese character is garbled when reading from MySQL 
using JDBC source
 Key: FLINK-16218
 URL: https://issues.apache.org/jira/browse/FLINK-16218
 Project: Flink
  Issue Type: Bug
  Components: Connectors / JDBC, Table SQL / Planner
Reporter: Jark Wu
 Fix For: 1.10.1
 Attachments: 15821322356269.jpg

I have set the database and table to use UTF8 collections, and use 
{{jdbc:mysql://localhost:3306/db_test?useUnicode=true=utf-8}} 
as the connection jdbc url. However, the Chinese characters are still garbled.

Btw, we should have a test to cover this.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-16219) Make AsyncWaitOperator chainable again

[Arvid Heise (Jira)]

Arvid Heise created FLINK-16219:
---

 Summary: Make AsyncWaitOperator chainable again
 Key: FLINK-16219
 URL: https://issues.apache.org/jira/browse/FLINK-16219
 Project: Flink
  Issue Type: Improvement
  Components: Runtime / Task
Affects Versions: 1.11.0
Reporter: Arvid Heise
 Fix For: 1.11.0


With the yield to downstream fixes, we may reenable chaining.

Chaining of yielding operators should be disallowed with 
`SourceFunction`/`SourceStreamTask`.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)