date:20200929

[jira] [Created] (FLINK-19471) CVE-2020-7712 is reported for flink-streaming-java_2.11:jar:1.11.1

2020-09-29 Thread Jeff Hu (Jira)

Jeff Hu created FLINK-19471:
---

 Summary: CVE-2020-7712 is reported for 
flink-streaming-java_2.11:jar:1.11.1
 Key: FLINK-19471
 URL: https://issues.apache.org/jira/browse/FLINK-19471
 Project: Flink
  Issue Type: Bug
  Components: API / DataStream
Affects Versions: 1.11.1
Reporter: Jeff Hu


flink-shaded-zookeeper-3-3.4.14-11.0.jar 
(pkg:maven/org.apache.flink/flink-shaded-zookeeper-3@3.4.14-11.0, 
cpe:2.3:a:apache:flink:3.4.14.11.0:*:*:*:*:*:*:*, 
cpe:2.3:a:apache:zookeeper:3.4.14.11.0:*:*:*:*:*:*:*) : CVE-2020-7712
zookeeper-3.4.14.jar (pkg:maven/org.apache.zookeeper/zookeeper@3.4.14, 
cpe:2.3:a:apache:zookeeper:3.4.14:*:*:*:*:*:*:*) : CVE-2020-7712

 

 

 

[INFO] +- org.apache.flink:flink-streaming-java_2.11:jar:1.11.1:provided
[INFO] | +- org.apache.flink:flink-runtime_2.11:jar:1.11.1:compile
[INFO] | | +- org.apache.flink:flink-hadoop-fs:jar:1.11.1:compile
[INFO] | | +- org.apache.flink:flink-shaded-jackson:jar:2.10.1-11.0:compile
[INFO] | | +- org.apache.flink:flink-shaded-zookeeper-3:jar:3.4.14-11.0:compile

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Need help in creating Flink Streaming s3 Job for multiple path reader one by one

2020-09-29 Thread Satyaa Dixit

Hi Guys,

Sorry to bother you again, but someone could help me here? Any help in this
regard will be much appreciated.

Regards,
Satya

On Tue, Sep 29, 2020 at 2:57 PM Satyaa Dixit  wrote:

> Hi Guys,
> I need one help, any leads will be highly appreciated.I have written a
> flink streaming job to read the data from s3 bucket and push to kafka.
> Below is the working source that deal with single s3 path:
> TextInputFormat format = new TextInputFormat(new
> org.apache.flink.core.fs.Path("s3a://directory/2020-09-03/"));
> format.setNestedFileEnumeration(true);
> DataStream inputStream = environment.readFile(format,
> "s3a://directory/2020-09-03/", FileProcessingMode.PROCESS_ONCE, -1,
> FilePathFilter.createDefaultFilter());
> inputStream.addSink(kafka);
>
> But my requirement is get the list of paths and pass them one by one to
> this environment.readFile() method.How we can achieve this.
>
> Thanks,
> Satya
>


-- 
--
Best Regards
Satya Prakash
(M)+91-9845111913

[jira] [Created] (FLINK-19470) ParquetColumnarRowSplitReader::reachEnd returns false after it has reached end

2020-09-29 Thread Rui Li (Jira)

Rui Li created FLINK-19470:
--

 Summary: ParquetColumnarRowSplitReader::reachEnd returns false 
after it has reached end
 Key: FLINK-19470
 URL: https://issues.apache.org/jira/browse/FLINK-19470
 Project: Flink
  Issue Type: Bug
  Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile)
Reporter: Rui Li
 Fix For: 1.12.0


After a {{ParquetColumnarRowSplitReader}} has reached its end, calling 
{{reachEnd}} again gets false.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [VOTE] FLIP-143: Unified Sink API

2020-09-29 Thread Guowei Ma

Hi all,

The voting time for FLIP-143 [1] has passed. I'm closing the vote now.

There were 5 votes, 4 of which are binding:

- Aljoscha Krettek (binding)
- Steven Wu
- Kostas Kloudas (binding)
- Jingsong Li (binding)
- Jiangang Liu (binding)

There were no -1 votes.

Thus, changes have been accepted. I'll update the FLIP doc accordingly.

There is still a discussion about the `CommitResult/GlobaCommitter
interface, which we could do more discussion at the implementation phase.

Thanks everyone for participating and discussing

[1] https://cwiki.apache.org/confluence/x/KEJ4CQ

Best,
Guowei


On Tue, Sep 29, 2020 at 6:32 PM 刘建刚  wrote:

> +1 (binding)
>
> Best,
> Liu Jiangang
>
> Jingsong Li  于2020年9月29日周二 下午1:36写道：
>
> > +1 (binding)
> >
> > Best,
> > Jingsong
> >
> > On Mon, Sep 28, 2020 at 3:21 AM Kostas Kloudas 
> wrote:
> >
> > > +1 (binding)
> > >
> > > @Steven Wu I think there will be opportunities to fine tune the API
> > > during the implementation.
> > >
> > > Cheers,
> > > Kostas
> > >
> > > On Sun, Sep 27, 2020 at 7:56 PM Steven Wu 
> wrote:
> > > >
> > > > +1 (non-binding)
> > > >
> > > > Although I would love to continue the discussion for tweaking the
> > > > CommitResult/GlobaCommitter interface maybe during the implementation
> > > phase.
> > > >
> > > > On Fri, Sep 25, 2020 at 5:35 AM Aljoscha Krettek <
> aljos...@apache.org>
> > > > wrote:
> > > >
> > > > > +1 (binding)
> > > > >
> > > > > Aljoscha
> > > > >
> > > > > On 25.09.20 14:26, Guowei Ma wrote:
> > > > > >  From the discussion[1] we could find that FLIP focuses on
> > providing
> > > an
> > > > > > unified transactional sink API. So I updated the FLIP's title to
> > > "Unified
> > > > > > Transactional Sink API". But I found that the old link could not
> be
> > > > > opened
> > > > > > again.
> > > > > >
> > > > > > I would update the link[2] here. Sorry for the inconvenience.
> > > > > >
> > > > > > [1]
> > > > > >
> > > > >
> > >
> >
> https://lists.apache.org/thread.html/rf09dfeeaf35da5ee98afe559b5a6e955c9f03ade0262727f6b5c4c1e%40%3Cdev.flink.apache.org%3E
> > > > > > [2] https://cwiki.apache.org/confluence/x/KEJ4CQ
> > > > > >
> > > > > > Best,
> > > > > > Guowei
> > > > > >
> > > > > >
> > > > > > On Thu, Sep 24, 2020 at 8:13 PM Guowei Ma 
> > > wrote:
> > > > > >
> > > > > >> Hi, all
> > > > > >>
> > > > > >> After the discussion in [1], I would like to open a voting
> thread
> > > for
> > > > > >> FLIP-143 [2], which proposes a unified sink api.
> > > > > >>
> > > > > >> The vote will be open until September 29th (72h + weekend),
> unless
> > > there
> > > > > >> is an objection or not enough votes.
> > > > > >>
> > > > > >> [1]
> > > > > >>
> > > > >
> > >
> >
> https://lists.apache.org/thread.html/rf09dfeeaf35da5ee98afe559b5a6e955c9f03ade0262727f6b5c4c1e%40%3Cdev.flink.apache.org%3E
> > > > > >> [2]
> > > > > >>
> > > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-143%3A+Unified+Sink+API
> > > > > >>
> > > > > >> Best,
> > > > > >> Guowei
> > > > > >>
> > > > > >
> > > > >
> > > > >
> > >
> >
> >
> > --
> > Best, Jingsong Lee
> >
>

[jira] [Created] (FLINK-19469) HBase connector 2.2 failed to download dependencies "org.glassfish:javax.el:jar:3.0.1-b06-SNAPSHOT"

2020-09-29 Thread Dian Fu (Jira)

Dian Fu created FLINK-19469:
---

 Summary: HBase connector 2.2 failed to download dependencies 
"org.glassfish:javax.el:jar:3.0.1-b06-SNAPSHOT" 
 Key: FLINK-19469
 URL: https://issues.apache.org/jira/browse/FLINK-19469
 Project: Flink
  Issue Type: Bug
  Components: Connectors / HBase
Affects Versions: 1.12.0
Reporter: Dian Fu


https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=7093=logs=d44f43ce-542c-597d-bf94-b0718c71e5e8=03dca39c-73e8-5aaf-601d-328ae5c35f20

{code}
2020-09-29T20:59:24.8085970Z [ERROR] Failed to execute goal on project 
flink-connector-hbase-2.2_2.11: Could not resolve dependencies for project 
org.apache.flink:flink-connector-hbase-2.2_2.11:jar:1.12-SNAPSHOT: Failed to 
collect dependencies at org.apache.hbase:hbase-server:jar:tests:2.2.3 -> 
org.glassfish.web:javax.servlet.jsp:jar:2.3.2 -> 
org.glassfish:javax.el:jar:3.0.1-b06-SNAPSHOT: Failed to read artifact 
descriptor for org.glassfish:javax.el:jar:3.0.1-b06-SNAPSHOT: Could not 
transfer artifact org.glassfish:javax.el:pom:3.0.1-b06-SNAPSHOT from/to 
jvnet-nexus-snapshots (https://maven.java.net/content/repositories/snapshots): 
Failed to transfer file: 
https://maven.java.net/content/repositories/snapshots/org/glassfish/javax.el/3.0.1-b06-SNAPSHOT/javax.el-3.0.1-b06-SNAPSHOT.pom.
 Return code is: 503 , ReasonPhrase:Service Unavailable: Back-end server is at 
capacity. -> [Help 1]
{code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-19468) Metrics not returned when data stream / operator name contains "+"

2020-09-29 Thread Boyang Jerry Peng (Jira)

Boyang Jerry Peng created FLINK-19468:
-

 Summary: Metrics not returned when data stream / operator name 
contains "+"
 Key: FLINK-19468
 URL: https://issues.apache.org/jira/browse/FLINK-19468
 Project: Flink
  Issue Type: Bug
  Components: API / DataStream
Affects Versions: 1.9.3, 1.9.2, 1.9.0, 2.0.0
Reporter: Boyang Jerry Peng


There is an issue in which the special character "+" is not removed from the 
data stream / operator name which causes metrics for the operator to not be 
properly returned. Code Reference:

[https://github.com/apache/flink/blob/master/flink-runtime/src/main/java/org/apache/flink/runtime/metrics/dump/MetricQueryService.java#L208]

 

For example if the operator name is:

pulsar(url: pulsar+ssl://192.168.1.198:56014)

Metrics for an operator with the above name will always return empty.

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-19467) Implement HashMapStateBackend and EmbeddedRocksDBStateBackend

2020-09-29 Thread Seth Wiesman (Jira)

Seth Wiesman created FLINK-19467:


 Summary: Implement HashMapStateBackend and 
EmbeddedRocksDBStateBackend
 Key: FLINK-19467
 URL: https://issues.apache.org/jira/browse/FLINK-19467
 Project: Flink
  Issue Type: Sub-task
Reporter: Seth Wiesman






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-19466) Implement JobManagerCheckpointStorage and FileSystemCheckpointStorage

2020-09-29 Thread Seth Wiesman (Jira)

Seth Wiesman created FLINK-19466:


 Summary: Implement JobManagerCheckpointStorage and 
FileSystemCheckpointStorage
 Key: FLINK-19466
 URL: https://issues.apache.org/jira/browse/FLINK-19466
 Project: Flink
  Issue Type: Sub-task
Reporter: Seth Wiesman
Assignee: Seth Wiesman






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-19465) Add CheckpointStorage interface

2020-09-29 Thread Seth Wiesman (Jira)

Seth Wiesman created FLINK-19465:


 Summary: Add CheckpointStorage interface
 Key: FLINK-19465
 URL: https://issues.apache.org/jira/browse/FLINK-19465
 Project: Flink
  Issue Type: Sub-task
Reporter: Seth Wiesman
Assignee: Seth Wiesman


Add checkpoint storage interface and wire it through the runtime



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-19464) Rename CheckpointStorage interface to CheckpointStorageAccess

2020-09-29 Thread Seth Wiesman (Jira)

Seth Wiesman created FLINK-19464:


 Summary: Rename CheckpointStorage interface to 
CheckpointStorageAccess
 Key: FLINK-19464
 URL: https://issues.apache.org/jira/browse/FLINK-19464
 Project: Flink
  Issue Type: Sub-task
Reporter: Seth Wiesman
Assignee: Seth Wiesman






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-19463) Disentangle StateBackends from Checkpointing

2020-09-29 Thread Seth Wiesman (Jira)

Seth Wiesman created FLINK-19463:


 Summary: Disentangle StateBackends from Checkpointing
 Key: FLINK-19463
 URL: https://issues.apache.org/jira/browse/FLINK-19463
 Project: Flink
  Issue Type: Improvement
Reporter: Seth Wiesman
Assignee: Seth Wiesman


This is an umbrella issue for tracking the implementation of FLIP-142

More details can be found on the wiki[1]. 

[1] 
https://cwiki.apache.org/confluence/display/FLINK/FLIP-142%3A+Disentangle+StateBackends+from+Checkpointing



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[RESULT][VOTE] FLIP-142: Disentangle StateBackends from Checkpointing

2020-09-29 Thread Seth Wiesman

Hi all,

The voting time for FLIP-142 has passed. I'm closing the vote now.

- Konstantin (binding)
- David Anderson (binding)
- Gordon (binding)
- Congxian
- David Wysakowicz (binding)
- Aljoscha (binding)
- Yu (binding)
- Kostas (binding)

Including myself, there were 9 votes, 8 binding. There were no disapproving
votes.

Thus, FLIP-142 has been accepted.

Thanks, everyone for joining the discussion and giving feedback!

Best,
Seth

[jira] [Created] (FLINK-19462) Checkpoint statistics for unfinished task snapshots

2020-09-29 Thread Nico Kruber (Jira)

Nico Kruber created FLINK-19462:
---

 Summary: Checkpoint statistics for unfinished task snapshots
 Key: FLINK-19462
 URL: https://issues.apache.org/jira/browse/FLINK-19462
 Project: Flink
  Issue Type: Improvement
  Components: Runtime / Checkpointing, Runtime / Metrics
Reporter: Nico Kruber


If a checkpoint times out, there are currently no stats on the not-yet-finished 
tasks in the Web UI, so you have to crawl into (debug?) logs.

It would be nice to have these incomplete stats in there instead so that you 
know quickly what was going on. I could think of these ways to accomplish this:
 * the checkpoint coordinator could ask the TMs for it after failing the 
checkpoint or
 * the TMs could send the stats when they notice that the checkpoint is aborted

Maybe there are more options, but I think, this improvement in general would 
benefit debugging checkpoints.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-19461) yarn-sesson.sh -jm -tm arguments have no effect

2020-09-29 Thread Robert Metzger (Jira)

Robert Metzger created FLINK-19461:
--

 Summary: yarn-sesson.sh -jm -tm arguments have no effect
 Key: FLINK-19461
 URL: https://issues.apache.org/jira/browse/FLINK-19461
 Project: Flink
  Issue Type: Bug
  Components: Deployment / YARN
Affects Versions: 1.12.0
Reporter: Robert Metzger


It seems that I can set arbitrary values for the documented {{-jm}} and {{-tm}} 
arguments, not leading to any effects.

Example: {{./bin/yarn-session -jm 512m -tm 512m}} should fail in my opinion, 
but it starts with the default memory configuration (1280mb / 1200mb? or so).



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-19460) AWS Kinesis Producer EXACTLY_ONCE semantic

2020-09-29 Thread Chris Slotterback (Jira)

Chris Slotterback created FLINK-19460:
-

 Summary: AWS Kinesis Producer EXACTLY_ONCE semantic 
 Key: FLINK-19460
 URL: https://issues.apache.org/jira/browse/FLINK-19460
 Project: Flink
  Issue Type: Improvement
Reporter: Chris Slotterback


Wanted to create a ticket to discuss adding EXACTLY_ONCE semantics to the AWS 
Kinesis producer, similar to how the Kafka producer functions.

The kinesis producer would need to be modified to participate in commits, per 
kinesis:
{noformat}
Each PutRecords request can support up to 500 records. Each record in the 
request can be as large as 1 MiB, up to a limit of 5 MiB for the entire 
request, including partition keys. Each shard can support writes up to 1,000 
records per second, up to a maximum data write total of 1 MiB per second.
{noformat}

Order is not guaranteed when writing to kinesis. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-19459) flink-dist won't build locally with newer (3.3+) maven versions

2020-09-29 Thread Robert Metzger (Jira)

Robert Metzger created FLINK-19459:
--

 Summary: flink-dist won't build locally with newer (3.3+) maven 
versions
 Key: FLINK-19459
 URL: https://issues.apache.org/jira/browse/FLINK-19459
 Project: Flink
  Issue Type: Bug
  Components: Build System
Affects Versions: 1.12.0
Reporter: Robert Metzger


flink-dist will fail on non Maven 3.2.5 versions because of banned dependencies.

These are the messages you'll see:
{code}
[INFO] --- maven-enforcer-plugin:3.0.0-M1:enforce (ban-unsafe-snakeyaml) @ 
flink-dist_2.11 ---
[WARNING] Rule 0: org.apache.maven.plugins.enforcer.BannedDependencies failed 
with message:
Found Banned Dependency: org.yaml:snakeyaml:jar:1.24
Use 'mvn dependency:tree' to locate the source of the banned dependencies.
[INFO] 
[INFO] Reactor Summary for Flink : 1.12-SNAPSHOT:

...

[ERROR] Failed to execute goal 
org.apache.maven.plugins:maven-enforcer-plugin:3.0.0-M1:enforce 
(ban-unsafe-snakeyaml) on project flink-dist_2.11: Some Enforcer rules have 
failed. Look above for specific messages explaining why the rule failed. -> 
[Help 1]
{code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSS] Release flink-shaded 12.0

2020-09-29 Thread Robert Metzger

It seems that we have consensus to create a flink-shaded release.

I'll soon propose a RC.

On Fri, Sep 25, 2020 at 9:11 AM Konstantin Knauf  wrote:

> +1
>
>
>
> On Wed, Sep 23, 2020 at 9:13 AM Yu Li  wrote:
>
> > +1
> >
> > Best Regards,
> > Yu
> >
> >
> > On Tue, 22 Sep 2020 at 17:49, Robert Metzger 
> wrote:
> >
> > > No concerns from my side.
> > >
> > > On Fri, Sep 18, 2020 at 8:25 AM Chesnay Schepler 
> > > wrote:
> > >
> > > > Hello,
> > > >
> > > > I'd like to kickoff the next release of flink-shaded, which will
> > contain
> > > > a bump to netty (4.1.49) and snakeyaml (1.27).
> > > >
> > > > Any concerns? Any other dependency  people want upgrade for the 1.12?
> > > >
> > > >
> > >
> >
>
>
> --
>
> Konstantin Knauf
>
> https://twitter.com/snntrable
>
> https://github.com/knaufk
>

[jira] [Created] (FLINK-19458) ZooKeeperLeaderElectionITCase.testJobExecutionOnClusterWithLeaderChange: ZooKeeper unexpectedly modified

2020-09-29 Thread Robert Metzger (Jira)

Robert Metzger created FLINK-19458:
--

 Summary: 
ZooKeeperLeaderElectionITCase.testJobExecutionOnClusterWithLeaderChange: 
ZooKeeper unexpectedly modified
 Key: FLINK-19458
 URL: https://issues.apache.org/jira/browse/FLINK-19458
 Project: Flink
  Issue Type: Bug
  Components: Runtime / Coordination
Affects Versions: 1.12.0
Reporter: Robert Metzger


https://dev.azure.com/rmetzger/Flink/_build/results?buildId=8422=logs=70ad9b63-500e-5dc9-5a3c-b60356162d7e=944c7023-8984-5aa2-b5f8-54922bd90d3a

{code}
2020-09-29T13:34:18.1803081Z [ERROR] 
testJobExecutionOnClusterWithLeaderChange(org.apache.flink.test.runtime.leaderelection.ZooKeeperLeaderElectionITCase)
  Time elapsed: 23.524 s  <<< ERROR!
2020-09-29T13:34:18.1803707Z java.util.concurrent.ExecutionException: 
org.apache.flink.runtime.client.JobSubmissionException: Failed to submit job.
2020-09-29T13:34:18.1804343Zat 
java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
2020-09-29T13:34:18.1804738Zat 
java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1908)
2020-09-29T13:34:18.1805274Zat 
org.apache.flink.test.runtime.leaderelection.ZooKeeperLeaderElectionITCase.testJobExecutionOnClusterWithLeaderChange(ZooKeeperLeaderElectionITCase.java:117)
2020-09-29T13:34:18.1805772Zat 
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
2020-09-29T13:34:18.1806136Zat 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
2020-09-29T13:34:18.1806555Zat 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
2020-09-29T13:34:18.1806936Zat 
java.lang.reflect.Method.invoke(Method.java:498)
2020-09-29T13:34:18.1807313Zat 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
2020-09-29T13:34:18.1807731Zat 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
2020-09-29T13:34:18.1808341Zat 
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
2020-09-29T13:34:18.1808973Zat 
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
2020-09-29T13:34:18.1809376Zat 
org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
2020-09-29T13:34:18.1809851Zat 
org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
2020-09-29T13:34:18.1810201Zat 
org.junit.rules.RunRules.evaluate(RunRules.java:20)
2020-09-29T13:34:18.1810632Zat 
org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
2020-09-29T13:34:18.1811035Zat 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
2020-09-29T13:34:18.1811700Zat 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
2020-09-29T13:34:18.1812082Zat 
org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
2020-09-29T13:34:18.1812447Zat 
org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
2020-09-29T13:34:18.1812824Zat 
org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
2020-09-29T13:34:18.1813190Zat 
org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
2020-09-29T13:34:18.1813565Zat 
org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
2020-09-29T13:34:18.1813964Zat 
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
2020-09-29T13:34:18.1814364Zat 
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
2020-09-29T13:34:18.1814752Zat 
org.junit.runners.ParentRunner.run(ParentRunner.java:363)
2020-09-29T13:34:18.1815298Zat 
org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
2020-09-29T13:34:18.1816096Zat 
org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
2020-09-29T13:34:18.1816552Zat 
org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
2020-09-29T13:34:18.1816984Zat 
org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
2020-09-29T13:34:18.1817421Zat 
org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
2020-09-29T13:34:18.1817894Zat 
org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
2020-09-29T13:34:18.1818318Zat 
org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
2020-09-29T13:34:18.181Zat 
org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
2020-09-29T13:34:18.1819294ZSuppressed: 
org.apache.flink.util.FlinkException: Could not close resource.
2020-09-29T13:34:18.1819698Zat 
org.apache.flink.util.AutoCloseableAsync.close(AutoCloseableAsync.java:42)
2020-09-29T13:34:18.1820260Zat

[jira] [Created] (FLINK-19457) Port NumberSequenceSource to FLIP-27 source interface

2020-09-29 Thread Stephan Ewen (Jira)

Stephan Ewen created FLINK-19457:


 Summary: Port NumberSequenceSource to FLIP-27 source interface
 Key: FLINK-19457
 URL: https://issues.apache.org/jira/browse/FLINK-19457
 Project: Flink
  Issue Type: Improvement
  Components: API / Core
Reporter: Stephan Ewen
Assignee: Stephan Ewen
 Fix For: 1.12.0


Both {{DataStream}} and {{DataSet}} APIs have a source generating a sequence of 
numbers.
This is useful for debugging and testing.

We should port this source to the FLIP-27 interface, to support testing 
programs with the new source API.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [VOTE] FLIP-142: Disentangle StateBackends from Checkpointing

2020-09-29 Thread Kostas Kloudas

+1 (binding)

Kostas

On Tue, Sep 29, 2020 at 4:57 PM Yu Li  wrote:
>
> +1 (binding)
>
> Thanks all for the patience of answering / addressing my questions in the
> discussion thread.
>
> Best Regards,
> Yu
>
>
> On Thu, 17 Sep 2020 at 14:39, Dawid Wysakowicz 
> wrote:
>
> > +1 (binding)
> >
> > On 17/09/2020 07:19, Congxian Qiu wrote:
> > > +1 (non-binding)
> > >
> > > Best,
> > > Congxian
> > >
> > >
> > > David Anderson  于2020年9月15日周二 下午11:38写道：
> > >
> > >> +1 (binding)
> > >>
> > >> David
> > >>
> > >> On Tue, Sep 15, 2020 at 10:25 AM Tzu-Li (Gordon) Tai <
> > tzuli...@apache.org>
> > >> wrote:
> > >>
> > >>> +1 (binding)
> > >>>
> > >>> On Tue, Sep 15, 2020 at 3:26 PM Aljoscha Krettek 
> > >>> wrote:
> > >>>
> >  +1 (binding)
> > 
> >  Aljoscha
> > 
> >
> >

Re: [DISCUSS] FLIP-142: Disentangle StateBackends from Checkpointing

2020-09-29 Thread Yu Li

Yes let's move on, already cast my vote in the voting thread.

Thanks all for the patience answering / addressing my belated questions!

Best Regards,
Yu


On Sun, 27 Sep 2020 at 20:00, Stephan Ewen  wrote:

> Good to see this FLIP moving.
>
> From what I understand, the remaining questions are mainly about how to
> express the roles of the CheckpointStorage and specifically the behavior of
> JMCheckpointStorage and FsCheckpointStorage in the docs.
> This sounds like details we should discuss over the concrete text proposals
> in the PR.
>
> On Sun, Sep 27, 2020 at 5:38 AM Yu Li  wrote:
>
> > Thanks Seth, the updated FLIP overall LGTM, and I've left some inline
> > comments in the doc.
> >
> > Best Regards,
> > Yu
> >
> >
> > On Fri, 25 Sep 2020 at 20:58, Seth Wiesman  wrote:
> >
> > > Done
> > >
> > > Seth
> > >
> > > On Fri, Sep 25, 2020 at 2:47 AM Yu Li  wrote:
> > >
> > > > *bq. I think it might help to highlight specific stumbling blocks
> users
> > > > have today and why I believe this change addresses those issues.*
> > > > Thanks for adding more details, I believe adding these blocks to the
> > FLIP
> > > > doc could make the motivation more vivid and convincing.
> > > >
> > > > *bq. To be concrete I think the JavaDoc for setCheckpointStorage
> would
> > be
> > > > something like...*
> > > > I could see this definition extracts the existing description from
> the
> > > > current `StateBackend` interface, it's a valid option, and let me
> quote
> > > it
> > > > again:
> > > > - CheckpointStorage defines how checkpoint snapshots are persisted
> for
> > > > fault tolerance. Various implementations store their checkpoints in
> > > > different fashions and have different requirements and availability
> > > > guarantees.
> > > > - JobManagerCheckpointStorage stores checkpoints in the memory of the
> > > > JobManager. It is lightweight and without additional dependencies but
> > is
> > > > not highly available.
> > > > - FileSystemCheckpointStorage stores checkpoints in a file system.
> For
> > > > systems like HDFS, NFS Drives, S3, and GCS, this storage policy
> > supports
> > > > large state size, in the magnitude of many terabytes while providing
> a
> > > > highly available foundation for stateful applications. This
> checkpoint
> > > > storage policy is recommended for most production deployments.
> > > >
> > > > Sticking to this definition, I think we should have the below
> migration
> > > > plans for existing backends:
> > > > - `MemoryStateBackend(null, String savepointPath)` to
> > > > `HashMapStateBackend() + JobManagerCheckpointStorage()`
> > > > - `MemoryStateBackend(, String
> > savepointPath)`
> > > to
> > > > `HashMapStateBackend() + FileSystemCheckpointStorage()`
> > > > in addition of the existing:
> > > > - `MemoryStateBackend()` to `HashMapStateBackend() +
> > > > JobManagerCheckpointStorage()`
> > > > and could be summarized as:
> > > > - MemoryStateBackend with checkpoint path: `HashMapStateBackend() +
> > > > FileSystemCheckpointStorage()`
> > > > - MemoryStateBackend w/o checkpoint path: `HashMapStateBackend() +
> > > > JobManagerCheckpointStorage()`
> > > >
> > > > And I believe adding the above highlighted blocks to the FLIP doc
> (the
> > > "New
> > > > StateBackend User API" and "Migration Plan" sections, separately)
> could
> > > > make it more complete.
> > > >
> > > > PS. Please note that although the current javadoc of `StateBackend`
> > > states
> > > > "MemoryStateBackend is not highly available", it actually supports
> > > > persisting the checkpoint data to DFS when checkpoint path is given,
> so
> > > the
> > > > mapping between old and new APIs are not that straight-forward and
> need
> > > > some clear clarifications, from my point of view.
> > > >
> > > > Best Regards,
> > > > Yu
> > > >
> > > >
> > > > On Fri, 25 Sep 2020 at 08:33, Seth Wiesman 
> > wrote:
> > > >
> > > > > Hi Yu,
> > > > >
> > > > > bq* I thought the FLIP aims at resolving some *existing* confusion,
> > > i.e.
> > > > > the durability mystery to users.
> > > > >
> > > > > I think it might help to highlight specific stumbling blocks users
> > have
> > > > > today and why I believe this change addresses those issues. Some
> > > frequent
> > > > > things I've heard over the past several years include:
> > > > >
> > > > > 1) "We use RocksDB because we don't need fault tolerance."
> > > > > 2) "We don't use RocksDB because we don't want to manage an
> external
> > > > > database."
> > > > > 3) Believing RocksDB is reading and writing directly with S3 or
> HDFS
> > > (vs.
> > > > > local disk)
> > > > > 4) Believing FsStateBackend spills to disk or has anything to do
> with
> > > the
> > > > > local filesystem
> > > > > 5) Pointing RocksDB at network-attached storage, believing that the
> > > state
> > > > > backend needs to be fault-tolerant
> > > > >
> > > > > This question from the ml is very representative of where users are
> > > > > struggling[1]. Many of these questions were not from new users

Re: [VOTE] FLIP-142: Disentangle StateBackends from Checkpointing

2020-09-29 Thread Yu Li

+1 (binding)

Thanks all for the patience of answering / addressing my questions in the
discussion thread.

Best Regards,
Yu


On Thu, 17 Sep 2020 at 14:39, Dawid Wysakowicz 
wrote:

> +1 (binding)
>
> On 17/09/2020 07:19, Congxian Qiu wrote:
> > +1 (non-binding)
> >
> > Best,
> > Congxian
> >
> >
> > David Anderson  于2020年9月15日周二 下午11:38写道：
> >
> >> +1 (binding)
> >>
> >> David
> >>
> >> On Tue, Sep 15, 2020 at 10:25 AM Tzu-Li (Gordon) Tai <
> tzuli...@apache.org>
> >> wrote:
> >>
> >>> +1 (binding)
> >>>
> >>> On Tue, Sep 15, 2020 at 3:26 PM Aljoscha Krettek 
> >>> wrote:
> >>>
>  +1 (binding)
> 
>  Aljoscha
> 
>
>

[jira] [Created] (FLINK-19456) sql client execute insert sql with comment ahead

2020-09-29 Thread ledong Lin (Jira)

ledong Lin created FLINK-19456:
--

 Summary: sql client execute insert sql with comment ahead
 Key: FLINK-19456
 URL: https://issues.apache.org/jira/browse/FLINK-19456
 Project: Flink
  Issue Type: Bug
  Components: Table SQL / Client
Affects Versions: 1.11.2
 Environment: flink-1.11.2-bin-scala_2.11

standalone cluster

bin/sql-client.sh embedded
Reporter: ledong Lin


*Environment*: standalone cluster
 *Step in sql client*:
{code:java}

Flink SQL> create table a( a string) with ( 'connector'='print');
 [INFO] Table has been created.
Flink SQL> –- test
 > insert into a
 > select date_format(current_timestamp, 'MMdd');
 [INFO] Submitting SQL update statement to the cluster...
Exception in thread "main" org.apache.flink.table.client.SqlClientException: 
Unexpected exception. This is a bug. Please consider filing an issue.Exception 
in thread "main" org.apache.flink.table.client.SqlClientException: Unexpected 
exception. This is a bug. Please consider filing an issue. at 
org.apache.flink.table.client.SqlClient.main(SqlClient.java:213)Caused by: 
java.lang.NullPointerException at 
org.apache.flink.table.client.cli.CliClient.callInsert(CliClient.java:598) at 
org.apache.flink.table.client.cli.CliClient.callCommand(CliClient.java:315) at 
java.util.Optional.ifPresent(Optional.java:159) at 
org.apache.flink.table.client.cli.CliClient.open(CliClient.java:212) at 
org.apache.flink.table.client.SqlClient.openCli(SqlClient.java:142) at 
org.apache.flink.table.client.SqlClient.start(SqlClient.java:114) at 
org.apache.flink.table.client.SqlClient.main(SqlClient.java:201)
Shutting down the session...done.
{code}
And the `_-- test_` is the cause of the problem. 

*error info in _flink-llin-sql-client-bigdata00.log_*
{code:java}
//代码占位符
2020-09-29 22:10:54,325 ERROR org.apache.flink.table.client.SqlClient           
           [] - SQL Client must stop. Unexpected exception. This is a bug. 
Please consider filing an issue.2020-09-29 22:10:54,325 ERROR 
org.apache.flink.table.client.SqlClient                      [] - SQL Client 
must stop. Unexpected exception. This is a bug. Please consider filing an 
issue.java.lang.NullPointerException: null at 
org.apache.flink.table.client.cli.CliClient.callInsert(CliClient.java:598) 
~[flink-sql-client_2.11-1.11.2.jar:1.11.2] at 
org.apache.flink.table.client.cli.CliClient.callCommand(CliClient.java:315) 
~[flink-sql-client_2.11-1.11.2.jar:1.11.2] at 
java.util.Optional.ifPresent(Optional.java:159) ~[?:1.8.0_202] at 
org.apache.flink.table.client.cli.CliClient.open(CliClient.java:212) 
~[flink-sql-client_2.11-1.11.2.jar:1.11.2] at 
org.apache.flink.table.client.SqlClient.openCli(SqlClient.java:142) 
~[flink-sql-client_2.11-1.11.2.jar:1.11.2] at 
org.apache.flink.table.client.SqlClient.start(SqlClient.java:114) 
~[flink-sql-client_2.11-1.11.2.jar:1.11.2] at 
org.apache.flink.table.client.SqlClient.main(SqlClient.java:201) 
[flink-sql-client_2.11-1.11.2.jar:1.11.2]
{code}
 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSS] FLIP-144: Native Kubernetes HA for Flink

2020-09-29 Thread Till Rohrmann

For 1. I was wondering whether we can't write the leader connection
information directly when trying to obtain the leadership (trying to update
the leader key with one's own value)? This might be a little detail, though.

2. Alright, so we are having a similar mechanism as we have in ZooKeeper
with the ephemeral lock nodes. I guess that this complicates the
implementation a bit, unfortunately.

3. Wouldn't the StatefulSet solution also work without a PV? One could
configure a different persistent storage like HDFS or S3 for storing the
checkpoints and job blobs like in the ZooKeeper case. The current benefit I
see is that we avoid having to implement this multi locking mechanism in
the ConfigMaps using the annotations because we can be sure that there is
only a single leader at a time if I understood the guarantees of K8s
correctly.

Cheers,
Till

On Tue, Sep 29, 2020 at 8:10 AM Yang Wang  wrote:

> Hi Till, thanks for your valuable feedback.
>
> 1. Yes, leader election and storing leader information will use a same
> ConfigMap. When a contender successfully performs a versioned annotation
> update operation to the ConfigMap, it means that it has been elected as the
> leader. And it will write the leader information in the callback of leader
> elector[1]. The Kubernetes resource version will help us to avoid the
> leader ConfigMap is wrongly updated.
>
> 2. The lock and release is really a valid concern. Actually in current
> design, we could not guarantee that the node who tries to write his
> ownership is the real leader. Who writes later, who is the owner. To
> address this issue, we need to store all the owners of the key. Only when
> the owner is empty, the specific key(means a checkpoint or job graph) could
> be deleted. However, we may have a residual checkpoint or job graph when
> the old JobManager crashed exceptionally and do not release the lock. To
> solve this problem completely, we need a timestamp renew mechanism
> for CompletedCheckpointStore and JobGraphStore, which could help us to the
> check the JobManager timeout and then clean up the residual keys.
>
> 3. Frankly speaking, I am not against with this solution. However, in my
> opinion, it is more like a temporary proposal. We could use StatefulSet to
> avoid leader election and leader retrieval. But I am not sure whether
> TaskManager could properly handle the situation that same hostname with
> different IPs, because the JobManager failed and relaunched. Also we may
> still have two JobManagers running in some corner cases(e.g. kubelet is
> down but the pod is running). Another concern is we have a strong
> dependency on the PersistentVolume(aka PV) in FileSystemHAService. But it
> is not always true especially in self-build Kubernetes cluster. Moreover,
> PV provider should guarantee that each PV could only be mounted once. Since
> the native HA proposal could cover all the functionality of StatefulSet
> proposal, that's why I prefer the former.
>
>
> [1].
> https://github.com/fabric8io/kubernetes-client/blob/6d83d41d50941bf8f2d4e0c859951eb10f617df6/kubernetes-client/src/main/java/io/fabric8/kubernetes/client/extended/leaderelection/LeaderElector.java#L70
>
> Best,
> Yang
>
> Till Rohrmann  于2020年9月28日周一 下午9:29写道：
>
>> Thanks for creating this FLIP Yang Wang. I believe that many of our users
>> will like a ZooKeeper-less HA setup.
>>
>> +1 for not separating the leader information and the leader election if
>> possible. Maybe it is even possible that the contender writes his leader
>> information directly when trying to obtain the leadership by performing a
>> versioned write operation.
>>
>> Concerning the lock and release operation I have a question: Can there be
>> multiple owners for a given key-value pair in a ConfigMap? If not, how can
>> we ensure that the node which writes his ownership is actually the leader
>> w/o transactional support from K8s? In ZooKeeper we had the same problem
>> (we should probably change it at some point to simply use a
>> transaction which checks whether the writer is still the leader) and
>> therefore introduced the ephemeral lock nodes. What they allow is that
>> there can be multiple owners of a given ZNode at a time. The last owner
>> will then be responsible for the cleanup of the node.
>>
>> I see the benefit of your proposal over the stateful set proposal because
>> it can support multiple standby JMs. Given the problem of locking key-value
>> pairs it might be simpler to start with this approach where we only have
>> single JM. This might already add a lot of benefits for our users. Was
>> there a specific reason why you discarded this proposal (other than
>> generality)?
>>
>> @Uce it would be great to hear your feedback on the proposal since you
>> already implemented a K8s based HA service.
>>
>> Cheers,
>> Till
>>
>> On Thu, Sep 17, 2020 at 5:06 AM Yang Wang  wrote:
>>
>>> Hi Xintong and Stephan,
>>>
>>> Thanks a lot for your attention on this FLIP. I will address the
>>> comments

Re: CheckpointedFunction initialization during checkpoint

2020-09-29 Thread Aljoscha Krettek


Hi Teng,

I think if the system is slowed down enough it can happen that some 
parts of the graph are still restoring while others are already taking a 
checkpoint. By virtue of how checkpointing works (by sending barriers 
along the network connections between tasks) this should not be a 
problem, though.


It would be good to check in the logs if for all individual tasks it 
holds that "restoring" comes before "checkpointing".


Best,
Aljoscha

On 29.09.20 04:00, Teng Fei Liao wrote:

Hey all,

I've been trying to debug a job recovery performance issue and I'm noticing
some interesting events in the timeline that seem unexpected to me. Here's
a brief outline of the first checkpoint following a job restart:

1. All tasks are deployed and transition into the RUNNING state.
2. I see logs for a subset of initializeState calls ("{} - restoring state"
from TwoPhaseCommitSinkFunction)
3. A checkpoint gets triggered "Triggering checkpoint {} @ {} for job {}."
4. I see more "{} - restoring state" logs.
5. Checkpoint completes "Completed checkpoint {} for job {} ({} bytes in {}
ms)."

The 2 questions I have are:
Are the initializations in 4) in the middle of a checkpoint expected? Since
all the tasks transition in 1) I would think that they are initialized
there as well.

Are the initializations in 4) causing the checkpoint to take longer to
complete? During the checkpoint, I do see "{} - checkpoint {} complete,
committing transaction {} from checkpoint {}" logs
(TwoPhaseCommitSinkFunction's notifyCheckpointComplete method) which
suggests that the kafka producers in 2) and 4) are contributing to the
checkpoint.

Thanks!

-Teng

Re: [DISCUSS] FLIP-146: Improve new TableSource and TableSink interfaces

2020-09-29 Thread Aljoscha Krettek


Hi,

I'll only respond regarding the parallelism for now because I need to 
think some more about DataStream.


What I'm saying is that exposing a parallelism only for Table Connectors 
is not the right thing. If we want to allow sources to tell the 
system/framework what would be a good parallelism it would be at the 
underlying level.


I'll explain with the SourceFunction. A Table API Source connector is 
basically a factory that will give you a SourceFunction that corresponds 
to whatever the user configured via properties and other means. If the 
Table Connector somehow happens to know what would be a good parallelism 
for the source it could "tell" the source when creating it, i.e.


  new MyRealSource(path, and, whatnot, parallelismHint)

Then the source could either work with that information it got, by 
shutting down (at runtime) some of its parallel instances. Or we could 
extend the Source (SourceFunction) API to expose a "parallelism hint" to 
the system.


The basic thing is that Table Connectors are not the real connectors, 
they just delegate to underlying real connectors. So those underlying 
connectors are where we need to change things. Otherwise we would just 
have special-case solutions for the Table API.


Best,
Aljoscha

On 25.09.20 14:30, admin wrote:

Hi everyone,
Thanks for the proposal.

In our company,we meet the same situation as @liu shouwei.
We developed some features base on flink.Such as parallelism of sql source/sink 
 connector, and kafka delay consumer which is adding a flatmap and a keyby 
transformation after the source Datastream.
What make us embarrassing is that when we migrate this features to Flink 
1.11,we found that the DataSteam is missing,So we modify the blink’s code to 
support parallelism.But kafka delay comsumer is unsolved until now.

 From user’s perspective,it necessary to manipulate DataStream or have the 
interoperability between Table API and DataStream.

Best




2020年9月25日 下午4:18，Rui Li  写道：

Hi Jingsong,

Thanks for driving this effort. I have two minor comments.


   1. IMHO, parallelism is a concept that applies to all ScanTableSource.
   So instead of defining a new interface, is it more natural to incorporate
   parallel inference to existing interfaces, e.g. ScanTableSource
   or ScanRuntimeProvider?
   2. `scan.infer-parallelism.enabled` doesn't seem very useful to me. From
   a user's perspective, parallelism is either set by `scan.parallelism`, or
   automatically decided by Flink. If a user doesn't want the connector to
   infer parallelism, he/she can simply set `scan.parallelism`, no?


On Fri, Sep 25, 2020 at 3:33 PM Jingsong Li  wrote:


Hi Aljoscha,

Thank you for your feedback,

## Connector parallelism

Requirements:
Set parallelism by user specified or inferred by connector.

How to configure parallelism in DataStream:
In the DataStream world, the only way to configure parallelism is
`SingleOutputStreamOperator.setParallelism`. Actually, users need to have
access to DataStream when using a connector, not just the `SourceFunction`
/ `Source` interface.
Is parallelism related to connectors? I think yes, there are many
connectors that can support obtaining parallelism related information from
them, and users do exactly that. This means parallelism inference (From
connectors).
The key is that `DataStream` is an open programming API, and users can
freely program to set parallelism.

How to configure parallelism in Table/SQL:
But Table/SQL is not an open programming API, every feature needs a
corresponding mechanism, because the user is no longer able to program. Our
current connector interface: SourceFunctionProvider, SinkFunctionProvider,
through these interfaces, there is no ability to generate connector related
parallelism.
Back to our original intention: to avoid users directly manipulating
`DataStream`. Since we want to avoid it, we need to provide corresponding
features.

And parallelism is the runtime information of connectors, It fits the name
of `ScanRuntimeProvider`.


If we wanted to add a "get parallelism" it would be in those underlying

connectors but I'm also skeptical about adding such a method there because
it is a static assignment and would preclude clever optimizations about the
parallelism of a connector at runtime.

I think that when a job is submitted, it is in compile time. It should only
provide static parallelism.

## DataStream in table connector

As I said before, if we want to completely cancel DataStream in the table
connector, we need to provide corresponding functions in
`xxRuntimeProvider`.
Otherwise, we and users may not be able to migrate the old connectors.
Including Hive/FileSystem connectors and the user cases I mentioned above.
CC: @liu shouwei

We really need to consider these cases.
If there is no alternative in a short period of time, for a long
time, users need to continue to use the old table connector API, which has
been deprecated.

Why not use StreamTableEnvironment

[jira] [Created] (FLINK-19455) Module 'flink-sql-connector-hive-2.3.6' build fail by maven-enforcer-plugin

2020-09-29 Thread hailong wang (Jira)

hailong wang created FLINK-19455:


 Summary: Module 'flink-sql-connector-hive-2.3.6' build fail by 
maven-enforcer-plugin
 Key: FLINK-19455
 URL: https://issues.apache.org/jira/browse/FLINK-19455
 Project: Flink
  Issue Type: Improvement
  Components: Connectors / Hive
Affects Versions: 1.11.0
Reporter: hailong wang
 Fix For: 1.12.0, 1.11.2


I run command 'mvn clean package' in flink-sql-connector-hive-2.3.6, and it 
failed,
{code:java}
 Rule 0: org.apache.maven.plugins.enforcer.BannedDependencies failed with 
message:
Found Banned Dependency: org.apache.kafka:kafka_2.10:jar:0.10.2.0
Use 'mvn dependency:tree' to locate the source of the banned dependencies.
{code}
For root pom has a root as follow:
{code:java}
bannedDependencies>
   
  *:*_2.12
  *:*_2.10
   

{code}
As for kafka_2.10 dependency is useless, So we can exclude it as follow:
{code:java}

   org.apache.hive
   hive-exec
   2.3.6
   
  
 log4j
 log4j
  
  
 org.slf4j
 slf4j-log4j12
  
  
 org.pentaho
 pentaho-aggdesigner-algorithm
  
  
 org.apache.kafka
 kafka_2.10
  
   
{code}
 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-19454) Translate page 'Importing Flink into an IDE' into Chinese

2020-09-29 Thread wulei0302 (Jira)

wulei0302 created FLINK-19454:
-

 Summary: Translate page 'Importing Flink into an IDE' into Chinese
 Key: FLINK-19454
 URL: https://issues.apache.org/jira/browse/FLINK-19454
 Project: Flink
  Issue Type: Improvement
  Components: chinese-translation, Documentation
Affects Versions: 1.11.2
Reporter: wulei0302


The page url is 
[https://ci.apache.org/projects/flink/flink-docs-release-1.11/flinkDev/ide_setup.html]

The markdown file is located in {{flink/docs/flinkDev/ide_setup.zh.md}}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [VOTE] FLIP-143: Unified Sink API

2020-09-29 Thread 刘建刚

+1 (binding)

Best,
Liu Jiangang

Jingsong Li  于2020年9月29日周二 下午1:36写道：

> +1 (binding)
>
> Best,
> Jingsong
>
> On Mon, Sep 28, 2020 at 3:21 AM Kostas Kloudas  wrote:
>
> > +1 (binding)
> >
> > @Steven Wu I think there will be opportunities to fine tune the API
> > during the implementation.
> >
> > Cheers,
> > Kostas
> >
> > On Sun, Sep 27, 2020 at 7:56 PM Steven Wu  wrote:
> > >
> > > +1 (non-binding)
> > >
> > > Although I would love to continue the discussion for tweaking the
> > > CommitResult/GlobaCommitter interface maybe during the implementation
> > phase.
> > >
> > > On Fri, Sep 25, 2020 at 5:35 AM Aljoscha Krettek 
> > > wrote:
> > >
> > > > +1 (binding)
> > > >
> > > > Aljoscha
> > > >
> > > > On 25.09.20 14:26, Guowei Ma wrote:
> > > > >  From the discussion[1] we could find that FLIP focuses on
> providing
> > an
> > > > > unified transactional sink API. So I updated the FLIP's title to
> > "Unified
> > > > > Transactional Sink API". But I found that the old link could not be
> > > > opened
> > > > > again.
> > > > >
> > > > > I would update the link[2] here. Sorry for the inconvenience.
> > > > >
> > > > > [1]
> > > > >
> > > >
> >
> https://lists.apache.org/thread.html/rf09dfeeaf35da5ee98afe559b5a6e955c9f03ade0262727f6b5c4c1e%40%3Cdev.flink.apache.org%3E
> > > > > [2] https://cwiki.apache.org/confluence/x/KEJ4CQ
> > > > >
> > > > > Best,
> > > > > Guowei
> > > > >
> > > > >
> > > > > On Thu, Sep 24, 2020 at 8:13 PM Guowei Ma 
> > wrote:
> > > > >
> > > > >> Hi, all
> > > > >>
> > > > >> After the discussion in [1], I would like to open a voting thread
> > for
> > > > >> FLIP-143 [2], which proposes a unified sink api.
> > > > >>
> > > > >> The vote will be open until September 29th (72h + weekend), unless
> > there
> > > > >> is an objection or not enough votes.
> > > > >>
> > > > >> [1]
> > > > >>
> > > >
> >
> https://lists.apache.org/thread.html/rf09dfeeaf35da5ee98afe559b5a6e955c9f03ade0262727f6b5c4c1e%40%3Cdev.flink.apache.org%3E
> > > > >> [2]
> > > > >>
> > > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-143%3A+Unified+Sink+API
> > > > >>
> > > > >> Best,
> > > > >> Guowei
> > > > >>
> > > > >
> > > >
> > > >
> >
>
>
> --
> Best, Jingsong Lee
>

[jira] [Created] (FLINK-19453) Deprecate old source and sink interfaces

2020-09-29 Thread Timo Walther (Jira)

Timo Walther created FLINK-19453:


 Summary: Deprecate old source and sink interfaces
 Key: FLINK-19453
 URL: https://issues.apache.org/jira/browse/FLINK-19453
 Project: Flink
  Issue Type: Sub-task
  Components: Table SQL / API
Reporter: Timo Walther
Assignee: Timo Walther


Deprecate all interfaces and classes that are not necessary anymore with 
FLIP-95.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSS][Release 1.12] Stale blockers and build instabilities

2020-09-29 Thread Dian Fu

Hi all,

I'd like to update the status about the blocker issues and build instabilities 
as there is only one month left and the number of blocker issues increases a 
lot compared to last week.

== Blockers: https://issues.apache.org/jira/browse/FLINK-18682?filter=12349334 


Currently there are 10 blocker issues 
- 3 performance regression (https://issues.apache.org/jira/browse/FLINK-19439 
, 
https://issues.apache.org/jira/browse/FLINK-19440 
, 
https://issues.apache.org/jira/browse/FLINK-19441 
)
- 3 Runtime (https://issues.apache.org/jira/browse/FLINK-19264 
, 
https://issues.apache.org/jira/browse/FLINK-19388 
, 
https://issues.apache.org/jira/browse/FLINK-19249 
)
- 1 HBase connector (https://issues.apache.org/jira/browse/FLINK-19445 
)
- 1 Application mode (https://issues.apache.org/jira/browse/FLINK-19154 
)
- 1 New source API (https://issues.apache.org/jira/browse/FLINK-19384 
)
- 1 Kinesis (https://issues.apache.org/jira/browse/FLINK-19332 
)

== Recent notable build instabilities which still have no owners:
- New source API
   https://issues.apache.org/jira/browse/FLINK-19253 
  
SourceReaderTestBase.testAddSplitToExistingFetcher hangs
   https://issues.apache.org/jira/browse/FLINK-19370 
  
FileSourceTextLinesITCase.testContinuousTextFileSource failed as results 
mismatch
   https://issues.apache.org/jira/browse/FLINK-19427 
  
SplitFetcherTest.testNotifiesWhenGoingIdleConcurrent is instable, 
   https://issues.apache.org/jira/browse/FLINK-19437 
  
FileSourceTextLinesITCase.testContinuousTextFileSource failed with 
"SimpleStreamFormat is not splittable, but found split end (0) different from 
file length (198)"
   https://issues.apache.org/jira/browse/FLINK-19448 
  
CoordinatedSourceITCase.testEnumeratorReaderCommunication hangs
- Runtime/Network 
   https://issues.apache.org/jira/browse/FLINK-19426 
  End-to-end test sometimes 
fails with PartitionConnectionException
- Unaligned Checkpoint
   https://issues.apache.org/jira/browse/FLINK-19027 
  
UnalignedCheckpointITCase.shouldPerformUnalignedCheckpointOnParallelRemoteChannel
 failed because of test timeout
- Table 
   https://issues.apache.org/jira/browse/FLINK-19340 
 
AggregateITCase.testListAggWithDistinct failed with "expected: but was:"
- HBase connector
   https://issues.apache.org/jira/browse/FLINK-18570 
  
SQLClientHBaseITCase.testHBase fails on azure
https://issues.apache.org/jira/browse/FLINK-19447 
  
HBaseConnectorITCase.HBaseTestingClusterAutoStarter failed with "Master not 
initialized after 20ms"
- Avro
   https://issues.apache.org/jira/browse/FLINK-19422 
  Avro Confluent Schema 
Registry nightly end-to-end test failed with "Register operation timed out; 
error code: 50002"

Regards,
Dian

> 在 2020年9月21日，下午2:32，Robert Metzger  写道：
> 
> Hi all,
> 
> An update on the release status:
> 1. We have 35 days = *5 weeks left until feature freeze*
> 2. There are currently 2 blockers for Flink
> , all
> making progress
> 3. We have 72 test instabilities
>  (down 7 from 2 weeks
> ago). I have pinged people to help addressing frequent or critical issues.
> 
> Best,
> Robert
> 
> 
> On Mon, Sep 7, 2020 at 10:37 AM Robert Metzger  wrote:
> 
>> Hi all,
>> 
>> another two weeks have passed. We now have 5 blockers
>>  (Up
>> 3 from 2 weeks ago), but they are all making progress.
>> 
>> We currently have 79 test-instabilities
>> ,
>> since the last report, a few have been resolved, and some others have been
>> added.
>> I have checked the tickets, closed some old ones and pinged people to help
>> resolve new or frequent ones.
>> Except for Kafka, there are no major

[jira] [Created] (FLINK-19452) statistics of group by CDC data is always 1

2020-09-29 Thread Zhengchao Shi (Jira)

Zhengchao Shi created FLINK-19452:
-

 Summary: statistics of group by CDC data is always 1
 Key: FLINK-19452
 URL: https://issues.apache.org/jira/browse/FLINK-19452
 Project: Flink
  Issue Type: Bug
  Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile)
Affects Versions: 1.11.1
Reporter: Zhengchao Shi
 Fix For: 1.12.0


When using CDC to do count statistics, if only updates are made to the source 
table(mysql table), then the value of count is always 1.
{code:sql}
CREATE TABLE orders (
  order_number int,
  product_id   int
) with (
  'connector' = 'kafka-0.11',
  'topic' = 'Topic',
  'properties.bootstrap.servers' = 'localhost:9092',
  'properties.group.id' = 'GroupId',
  'scan.startup.mode' = 'latest-offset',
  'format' = 'canal-json'
);

CREATE TABLE order_test (
  order_number int,
  order_cnt bigint
) WITH (
  'connector' = 'print'
);

INSERT INTO order_test
SELECT order_number, count(1) FROM orders GROUP BY order_number;
{code}
3 records in  “orders” :
||order_number||product_id||
|10001|1|
|10001|2|
|10001|3|

 now update orders table:
{code:sql}
update orders set product_id = 5 where order_number = 10001;
{code}
the output of is :

-D(10001,1)
 +I(10001,1)
 -D(10001,1)
 +I(10001,1)
 -D(10001,1)
 +I(10001,1)

i think, the final result is +I(10001, 3)



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-19451) Add HELM chart distribution to StateFun release process

2020-09-29 Thread Igal Shilman (Jira)

Igal Shilman created FLINK-19451:


 Summary: Add HELM chart distribution to StateFun release process
 Key: FLINK-19451
 URL: https://issues.apache.org/jira/browse/FLINK-19451
 Project: Flink
  Issue Type: Task
  Components: Stateful Functions
Reporter: Igal Shilman


Helm charts can be packaged and hosted publicly, see here: 
[https://helm.sh/docs/topics/chart_repository/#create-a-chart-repository]

 

As part of our release process we release a source distribution that includes 
the Helm charts, but we can also add a step that hosts them in an Apache 
compatible way.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Need help in creating Flink Streaming s3 Job for multiple path reader one by one

2020-09-29 Thread Satyaa Dixit

Hi Guys,
I need one help, any leads will be highly appreciated.I have written a
flink streaming job to read the data from s3 bucket and push to kafka.
Below is the working source that deal with single s3 path:
TextInputFormat format = new TextInputFormat(new
org.apache.flink.core.fs.Path("s3a://directory/2020-09-03/"));
format.setNestedFileEnumeration(true);
DataStream inputStream = environment.readFile(format,
"s3a://directory/2020-09-03/", FileProcessingMode.PROCESS_ONCE, -1,
FilePathFilter.createDefaultFilter());
inputStream.addSink(kafka);

But my requirement is get the list of paths and pass them one by one to
this environment.readFile() method.How we can achieve this.

Thanks,
Satya

[jira] [Created] (FLINK-19450) Optimize the Python CI Test

2020-09-29 Thread Huang Xingbo (Jira)

Huang Xingbo created FLINK-19450:


 Summary: Optimize the Python CI Test
 Key: FLINK-19450
 URL: https://issues.apache.org/jira/browse/FLINK-19450
 Project: Flink
  Issue Type: Improvement
  Components: API / Python, Build System / Azure Pipelines
Reporter: Huang Xingbo
 Fix For: 1.12.0


Currently, the CI test of PyFlink will run 4 versions of Python, which takes a 
lot of time. We will optimize CI test to run only one version of Python. And 
then nightly test will run all versions of Python.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-19449) LEAD/LAG cannot work correctly in streaming mode

2020-09-29 Thread Benchao Li (Jira)

Benchao Li created FLINK-19449:
--

 Summary: LEAD/LAG cannot work correctly in streaming mode
 Key: FLINK-19449
 URL: https://issues.apache.org/jira/browse/FLINK-19449
 Project: Flink
  Issue Type: Bug
  Components: Table SQL / API, Table SQL / Runtime
Affects Versions: 1.11.2, 1.10.2
Reporter: Benchao Li






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Need help in setting up flink 1.10

2020-09-29 Thread David Anderson

Ravi,

Note that questions like this are better suited for the user mailing list.

According to [1], google cloud storage is supported under the gcs: url
scheme. Also, since Flink 1.10, most filesystems must be loaded as plugins,
rather than from the lib directory [2].

I don't have experience with GCS, but I hope this helps.

[1]
https://ci.apache.org/projects/flink/flink-docs-release-1.10/internals/filesystems.html#implementations
[2]
https://ci.apache.org/projects/flink/flink-docs-release-1.10/ops/filesystems/#pluggable-file-systems

Regards,
David

On Tue, Sep 29, 2020 at 3:04 AM Ravi Sankar Reddy Sangana 
wrote:

>
> Hi Team,
>
> We are using flink 1.7 currently in prod. We want to move to latest
> release and trying 1.10. We are using google storage for our checkpoints.
>
> In these release we need Hadoop to the integrated from our side. So
> following this link<
> https://ci.apache.org/projects/flink/flink-docs-release-1.10/ops/deployment/hadoop.html>
> we copied the flink-shaded-hadoop-2-uber-2.4.1-10.0.jar to the lib folder.
> Now I an getting this error while submitting the job.
>
> Caused by: java.io.IOException: No FileSystem for scheme: gs
>
> Can you please explain or show me a doc or any blog to solve this error.
>
>
> flink-conf.yml
> state.backend: rocksdb
> state.checkpoints.dir: gs://xyz/flink/pacifier/flink-checkpoints
>
> Regards,
> Ravi Sankar Reddy
>
> Sent from Mail for
> Windows 10
>
>

[jira] [Created] (FLINK-19448) CoordinatedSourceITCase.testEnumeratorReaderCommunication hangs

2020-09-29 Thread Dian Fu (Jira)

Dian Fu created FLINK-19448:
---

 Summary: CoordinatedSourceITCase.testEnumeratorReaderCommunication 
hangs
 Key: FLINK-19448
 URL: https://issues.apache.org/jira/browse/FLINK-19448
 Project: Flink
  Issue Type: Bug
  Components: Connectors / Common, Tests
Affects Versions: 1.12.0
Reporter: Dian Fu


https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=7042=logs=5cae8624-c7eb-5c51-92d3-4d2dacedd221=420bd9ec-164e-562e-8947-0dacde3cec91

{code}
2020-09-28T21:40:41.0736918Z [INFO] Running 
org.apache.flink.connector.base.source.reader.CoordinatedSourceITCase
2020-09-28T21:57:23.4590733Z 
==
2020-09-28T21:57:23.4591238Z Process produced no output for 900 seconds.
2020-09-28T21:57:23.4591593Z 
==
2020-09-28T21:57:23.4595995Z 
==
2020-09-28T21:57:23.4596439Z The following Java processes are running (JPS)
2020-09-28T21:57:23.4596789Z 
==
2020-09-28T21:57:23.4638075Z Picked up JAVA_TOOL_OPTIONS: 
-XX:+HeapDumpOnOutOfMemoryError
2020-09-28T21:57:23.6127853Z 21907 surefirebooter2023202237772619676.jar
2020-09-28T21:57:23.6128185Z 534 Launcher
2020-09-28T21:57:23.6128381Z 24630 Jps
2020-09-28T21:57:23.6159852Z 
==
2020-09-28T21:57:23.6160256Z Printing stack trace of Java process 21907
2020-09-28T21:57:23.6160806Z 
==
2020-09-28T21:57:23.6203860Z Picked up JAVA_TOOL_OPTIONS: 
-XX:+HeapDumpOnOutOfMemoryError
2020-09-28T21:57:23.9470219Z 2020-09-28 21:57:23
2020-09-28T21:57:23.9471512Z Full thread dump OpenJDK 64-Bit Server VM 
(25.242-b08 mixed mode):
2020-09-28T21:57:23.9472274Z 
2020-09-28T21:57:23.9472805Z "Attach Listener" #215 daemon prio=9 os_prio=0 
tid=0x7f13c8074800 nid=0x6052 waiting on condition [0x]
2020-09-28T21:57:23.9473343Zjava.lang.Thread.State: RUNNABLE
2020-09-28T21:57:23.9473660Z 
2020-09-28T21:57:23.9474554Z "flink-akka.actor.default-dispatcher-103" #214 
prio=5 os_prio=0 tid=0x7f13cc1b5000 nid=0x6018 waiting on condition 
[0x7f13bb4f5000]
2020-09-28T21:57:23.9475189Zjava.lang.Thread.State: WAITING (parking)
2020-09-28T21:57:23.9475815Zat sun.misc.Unsafe.park(Native Method)
2020-09-28T21:57:23.9476662Z- parking to wait for  <0x87a80408> (a 
akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
2020-09-28T21:57:23.9477295Zat 
akka.dispatch.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
2020-09-28T21:57:23.9477871Zat 
akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
2020-09-28T21:57:23.9480210Zat 
akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
2020-09-28T21:57:23.9480723Z 
2020-09-28T21:57:23.9481669Z "flink-taskexecutor-io-thread-8" #125 daemon 
prio=5 os_prio=0 tid=0x7f13e401d000 nid=0x571b waiting on condition 
[0x7f13e84f8000]
2020-09-28T21:57:23.9482321Zjava.lang.Thread.State: WAITING (parking)
2020-09-28T21:57:23.9482727Zat sun.misc.Unsafe.park(Native Method)
2020-09-28T21:57:23.9483562Z- parking to wait for  <0x87a80860> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
2020-09-28T21:57:23.9484241Zat 
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
2020-09-28T21:57:23.9484899Zat 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
2020-09-28T21:57:23.9485585Zat 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
2020-09-28T21:57:23.9486194Zat 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
2020-09-28T21:57:23.9486818Zat 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
2020-09-28T21:57:23.9487440Zat 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
2020-09-28T21:57:23.9487970Zat java.lang.Thread.run(Thread.java:748)
2020-09-28T21:57:23.9488278Z 
2020-09-28T21:57:23.9489120Z "flink-taskexecutor-io-thread-7" #124 daemon 
prio=5 os_prio=0 tid=0x7f13e401c800 nid=0x571a waiting on condition 
[0x7f13b8f46000]
2020-09-28T21:57:23.9489760Zjava.lang.Thread.State: WAITING (parking)
2020-09-28T21:57:23.9490190Zat sun.misc.Unsafe.park(Native Method)
2020-09-28T21:57:23.9491003Z- parking to wait for  <0x87a80860> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
2020-09-28T21:57:23.9491667Zat 
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

[jira] [Created] (FLINK-19447) HBaseConnectorITCase.HBaseTestingClusterAutoStarter failed with "Master not initialized after 200000ms"

2020-09-29 Thread Dian Fu (Jira)

Dian Fu created FLINK-19447:
---

 Summary: HBaseConnectorITCase.HBaseTestingClusterAutoStarter 
failed with "Master not initialized after 20ms"
 Key: FLINK-19447
 URL: https://issues.apache.org/jira/browse/FLINK-19447
 Project: Flink
  Issue Type: Bug
  Components: Connectors / HBase
Affects Versions: 1.12.0
Reporter: Dian Fu


https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=7042=logs=961f8f81-6b52-53df-09f6-7291a2e4af6a=60581941-0138-53c0-39fe-86d62be5f407

{code}
2020-09-28T21:52:21.2146147Z 
org.apache.flink.connector.hbase2.HBaseConnectorITCase  Time elapsed: 208.382 
sec  <<< ERROR!
2020-09-28T21:52:21.2146638Z java.io.IOException: Shutting down
2020-09-28T21:52:21.2147004Zat 
org.apache.hadoop.hbase.MiniHBaseCluster.init(MiniHBaseCluster.java:266)
2020-09-28T21:52:21.2147637Zat 
org.apache.hadoop.hbase.MiniHBaseCluster.(MiniHBaseCluster.java:116)
2020-09-28T21:52:21.2148120Zat 
org.apache.hadoop.hbase.HBaseTestingUtility.startMiniHBaseCluster(HBaseTestingUtility.java:1142)
2020-09-28T21:52:21.2148831Zat 
org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:1107)
2020-09-28T21:52:21.2149347Zat 
org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:1061)
2020-09-28T21:52:21.2149896Zat 
org.apache.flink.connector.hbase2.util.HBaseTestingClusterAutoStarter.setUp(HBaseTestingClusterAutoStarter.java:122)
2020-09-28T21:52:21.2150721Zat 
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
2020-09-28T21:52:21.2151136Zat 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
2020-09-28T21:52:21.2151609Zat 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
2020-09-28T21:52:21.2152039Zat 
java.lang.reflect.Method.invoke(Method.java:498)
2020-09-28T21:52:21.2152462Zat 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
2020-09-28T21:52:21.2152941Zat 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
2020-09-28T21:52:21.2153489Zat 
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
2020-09-28T21:52:21.2153962Zat 
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24)
2020-09-28T21:52:21.2154406Zat 
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
2020-09-28T21:52:21.2154828Zat 
org.junit.runners.ParentRunner.run(ParentRunner.java:363)
2020-09-28T21:52:21.2155381Zat 
org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:367)
2020-09-28T21:52:21.2155864Zat 
org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:274)
2020-09-28T21:52:21.2156378Zat 
org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
2020-09-28T21:52:21.2156865Zat 
org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:161)
2020-09-28T21:52:21.2157458Zat 
org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:290)
2020-09-28T21:52:21.2157993Zat 
org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:242)
2020-09-28T21:52:21.2158470Zat 
org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:121)
2020-09-28T21:52:21.2158890Z Caused by: java.lang.RuntimeException: Master not 
initialized after 20ms
2020-09-28T21:52:21.2159350Zat 
org.apache.hadoop.hbase.util.JVMClusterUtil.waitForEvent(JVMClusterUtil.java:229)
2020-09-28T21:52:21.2159823Zat 
org.apache.hadoop.hbase.util.JVMClusterUtil.startup(JVMClusterUtil.java:197)
2020-09-28T21:52:21.2160270Zat 
org.apache.hadoop.hbase.LocalHBaseCluster.startup(LocalHBaseCluster.java:413)
2020-09-28T21:52:21.2160800Zat 
org.apache.hadoop.hbase.MiniHBaseCluster.init(MiniHBaseCluster.java:259)
2020-09-28T21:52:21.2161096Z... 22 more
{code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-19446) canal-json has a situation that -U and +U are equal, when updating the null field to be non-null

2020-09-29 Thread Zhengchao Shi (Jira)

Zhengchao Shi created FLINK-19446:
-

 Summary: canal-json has a situation that -U and +U are equal, when 
updating the null field to be non-null
 Key: FLINK-19446
 URL: https://issues.apache.org/jira/browse/FLINK-19446
 Project: Flink
  Issue Type: Bug
  Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile)
Affects Versions: 1.11.1
Reporter: Zhengchao Shi
 Fix For: 1.12.0


line 118 in CanalJsonDeserializationSchema#deserialize method:

{code:java}
GenericRowData after = (GenericRowData) data.getRow(i, fieldCount);
GenericRowData before = (GenericRowData) old.getRow(i, fieldCount);
for (int f = 0; f < fieldCount; f++) {
if (before.isNullAt(f)) {
// not null fields in "old" (before) means the fields are 
changed
// null/empty fields in "old" (before) means the fields are not 
changed
// so we just copy the not changed fields into before
before.setField(f, after.getField(f));
}
}
before.setRowKind(RowKind.UPDATE_BEFORE);
after.setRowKind(RowKind.UPDATE_AFTER);
{code}

if a field is null before update,it will cause -U and +U to be equal



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-19445) Several tests for HBase connector 1.4 failed with "NoSuchMethodError: com.google.common.base.Preconditions.checkArgument(ZLjava/lang/String;Ljava/lang/Object;)V"

2020-09-29 Thread Dian Fu (Jira)

Dian Fu created FLINK-19445:
---

 Summary: Several tests for HBase connector 1.4 failed with 
"NoSuchMethodError: 
com.google.common.base.Preconditions.checkArgument(ZLjava/lang/String;Ljava/lang/Object;)V"
 Key: FLINK-19445
 URL: https://issues.apache.org/jira/browse/FLINK-19445
 Project: Flink
  Issue Type: Bug
  Components: Connectors / HBase
Affects Versions: 1.12.0
Reporter: Dian Fu
 Fix For: 1.12.0


https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=7042=logs=ba53eb01-1462-56a3-8e98-0dd97fbcaab5=bfbc6239-57a0-5db0-63f3-41551b4f7d51

{code}
2020-09-28T21:28:29.4171075Z Running 
org.apache.flink.connector.hbase1.HBaseTablePlanTest
2020-09-28T21:28:31.0367584Z Tests run: 5, Failures: 0, Errors: 1, Skipped: 0, 
Time elapsed: 1.62 sec <<< FAILURE! - in 
org.apache.flink.connector.hbase1.HBaseTablePlanTest
2020-09-28T21:28:31.0368925Z 
testProjectionPushDown(org.apache.flink.connector.hbase1.HBaseTablePlanTest)  
Time elapsed: 0.031 sec  <<< ERROR!
2020-09-28T21:28:31.0369805Z org.apache.flink.table.api.ValidationException: 
2020-09-28T21:28:31.0370409Z Unable to create a source for reading table 
'default_catalog.default_database.hTable'.
2020-09-28T21:28:31.0370707Z 
2020-09-28T21:28:31.0370976Z Table options are:
2020-09-28T21:28:31.0371204Z 
2020-09-28T21:28:31.0371528Z 'connector'='hbase-1.4'
2020-09-28T21:28:31.0371871Z 'table-name'='my_table'
2020-09-28T21:28:31.0372255Z 'zookeeper.quorum'='localhost:2021'
2020-09-28T21:28:31.0372812Zat 
org.apache.flink.table.factories.FactoryUtil.createTableSource(FactoryUtil.java:125)
2020-09-28T21:28:31.0373359Zat 
org.apache.flink.table.planner.plan.schema.CatalogSourceTable.buildTableScan(CatalogSourceTable.scala:135)
2020-09-28T21:28:31.0373905Zat 
org.apache.flink.table.planner.plan.schema.CatalogSourceTable.toRel(CatalogSourceTable.scala:78)
2020-09-28T21:28:31.0374390Zat 
org.apache.calcite.sql2rel.SqlToRelConverter.toRel(SqlToRelConverter.java:3492)
2020-09-28T21:28:31.0375224Zat 
org.apache.calcite.sql2rel.SqlToRelConverter.convertIdentifier(SqlToRelConverter.java:2415)
2020-09-28T21:28:31.0375867Zat 
org.apache.calcite.sql2rel.SqlToRelConverter.convertFrom(SqlToRelConverter.java:2102)
2020-09-28T21:28:31.0376479Zat 
org.apache.flink.table.planner.calcite.FlinkPlannerImpl$$anon$1.convertFrom(FlinkPlannerImpl.scala:181)
2020-09-28T21:28:31.0377077Zat 
org.apache.calcite.sql2rel.SqlToRelConverter.convertFrom(SqlToRelConverter.java:2051)
2020-09-28T21:28:31.0377593Zat 
org.apache.flink.table.planner.calcite.FlinkPlannerImpl$$anon$1.convertFrom(FlinkPlannerImpl.scala:181)
2020-09-28T21:28:31.0378114Zat 
org.apache.calcite.sql2rel.SqlToRelConverter.convertSelectImpl(SqlToRelConverter.java:661)
2020-09-28T21:28:31.0378622Zat 
org.apache.calcite.sql2rel.SqlToRelConverter.convertSelect(SqlToRelConverter.java:642)
2020-09-28T21:28:31.0379132Zat 
org.apache.calcite.sql2rel.SqlToRelConverter.convertQueryRecursive(SqlToRelConverter.java:3345)
2020-09-28T21:28:31.0379872Zat 
org.apache.calcite.sql2rel.SqlToRelConverter.convertQuery(SqlToRelConverter.java:568)
2020-09-28T21:28:31.0380477Zat 
org.apache.flink.table.planner.calcite.FlinkPlannerImpl.org$apache$flink$table$planner$calcite$FlinkPlannerImpl$$rel(FlinkPlannerImpl.scala:196)
2020-09-28T21:28:31.0381128Zat 
org.apache.flink.table.planner.calcite.FlinkPlannerImpl.rel(FlinkPlannerImpl.scala:154)
2020-09-28T21:28:31.0381666Zat 
org.apache.flink.table.planner.operations.SqlToOperationConverter.toQueryOperation(SqlToOperationConverter.java:823)
2020-09-28T21:28:31.0382264Zat 
org.apache.flink.table.planner.operations.SqlToOperationConverter.convertSqlQuery(SqlToOperationConverter.java:795)
2020-09-28T21:28:31.0382968Zat 
org.apache.flink.table.planner.operations.SqlToOperationConverter.convert(SqlToOperationConverter.java:250)
2020-09-28T21:28:31.0383550Zat 
org.apache.flink.table.planner.delegation.ParserImpl.parse(ParserImpl.java:78)
2020-09-28T21:28:31.0384172Zat 
org.apache.flink.table.api.internal.TableEnvironmentImpl.sqlQuery(TableEnvironmentImpl.java:640)
2020-09-28T21:28:31.0384700Zat 
org.apache.flink.table.planner.utils.TableTestUtilBase.doVerifyPlan(TableTestBase.scala:346)
2020-09-28T21:28:31.0385201Zat 
org.apache.flink.table.planner.utils.TableTestUtilBase.verifyPlan(TableTestBase.scala:271)
2020-09-28T21:28:31.0385717Zat 
org.apache.flink.connector.hbase1.HBaseTablePlanTest.testProjectionPushDown(HBaseTablePlanTest.java:124)
2020-09-28T21:28:31.0386166Zat 
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
2020-09-28T21:28:31.0386575Zat 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
2020-09-28T21:28:31.0387257Zat 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

Re: [DISCUSS] FLIP-144: Native Kubernetes HA for Flink

2020-09-29 Thread Yang Wang

Hi Till, thanks for your valuable feedback.

1. Yes, leader election and storing leader information will use a same
ConfigMap. When a contender successfully performs a versioned annotation
update operation to the ConfigMap, it means that it has been elected as the
leader. And it will write the leader information in the callback of leader
elector[1]. The Kubernetes resource version will help us to avoid the
leader ConfigMap is wrongly updated.

2. The lock and release is really a valid concern. Actually in current
design, we could not guarantee that the node who tries to write his
ownership is the real leader. Who writes later, who is the owner. To
address this issue, we need to store all the owners of the key. Only when
the owner is empty, the specific key(means a checkpoint or job graph) could
be deleted. However, we may have a residual checkpoint or job graph when
the old JobManager crashed exceptionally and do not release the lock. To
solve this problem completely, we need a timestamp renew mechanism
for CompletedCheckpointStore and JobGraphStore, which could help us to the
check the JobManager timeout and then clean up the residual keys.

3. Frankly speaking, I am not against with this solution. However, in my
opinion, it is more like a temporary proposal. We could use StatefulSet to
avoid leader election and leader retrieval. But I am not sure whether
TaskManager could properly handle the situation that same hostname with
different IPs, because the JobManager failed and relaunched. Also we may
still have two JobManagers running in some corner cases(e.g. kubelet is
down but the pod is running). Another concern is we have a strong
dependency on the PersistentVolume(aka PV) in FileSystemHAService. But it
is not always true especially in self-build Kubernetes cluster. Moreover,
PV provider should guarantee that each PV could only be mounted once. Since
the native HA proposal could cover all the functionality of StatefulSet
proposal, that's why I prefer the former.

[1].
https://github.com/fabric8io/kubernetes-client/blob/6d83d41d50941bf8f2d4e0c859951eb10f617df6/kubernetes-client/src/main/java/io/fabric8/kubernetes/client/extended/leaderelection/LeaderElector.java#L70

Best,
Yang

Till Rohrmann  于2020年9月28日周一 下午9:29写道：

> Thanks for creating this FLIP Yang Wang. I believe that many of our users
> will like a ZooKeeper-less HA setup.
>
> +1 for not separating the leader information and the leader election if
> possible. Maybe it is even possible that the contender writes his leader
> information directly when trying to obtain the leadership by performing a
> versioned write operation.
>
> Concerning the lock and release operation I have a question: Can there be
> multiple owners for a given key-value pair in a ConfigMap? If not, how can
> we ensure that the node which writes his ownership is actually the leader
> w/o transactional support from K8s? In ZooKeeper we had the same problem
> (we should probably change it at some point to simply use a
> transaction which checks whether the writer is still the leader) and
> therefore introduced the ephemeral lock nodes. What they allow is that
> there can be multiple owners of a given ZNode at a time. The last owner
> will then be responsible for the cleanup of the node.
>
> I see the benefit of your proposal over the stateful set proposal because
> it can support multiple standby JMs. Given the problem of locking key-value
> pairs it might be simpler to start with this approach where we only have
> single JM. This might already add a lot of benefits for our users. Was
> there a specific reason why you discarded this proposal (other than
> generality)?
>
> @Uce it would be great to hear your feedback on the proposal since you
> already implemented a K8s based HA service.
>
> Cheers,
> Till
>
> On Thu, Sep 17, 2020 at 5:06 AM Yang Wang  wrote:
>
>> Hi Xintong and Stephan,
>>
>> Thanks a lot for your attention on this FLIP. I will address the comments
>> inline.
>>
>> # Architecture -> One or two ConfigMaps
>>
>> Both of you are right. One ConfigMap will make the design and
>> implementation easier. Actually, in my POC codes,
>> I am using just one ConfigMap(e.g. "k8s-ha-app1-restserver" for rest
>> server component) for the leader election
>> and storage. Once a JobManager win the election, it will update the
>> ConfigMap with leader address and periodically
>> renew the lock annotation to keep as the active leader. I will update the
>> FLIP document, including the architecture diagram,
>> to avoid the misunderstanding.
>>
>>
>> # HA storage > Lock and release
>>
>> This is a valid concern. Since for Zookeeper ephemeral nodes, it will be
>> deleted by the ZK server automatically when
>> the client is timeout. It could happen in a bad network environment or
>> the ZK client crashed exceptionally. For Kubernetes,
>> we need to implement a similar mechanism. First, when we want to lock a
>> specific key in ConfigMap, we will put the owner identify,
>> lease

[jira] [Created] (FLINK-19444) flink 1.11 sql group by tumble Window aggregate can only be defined over a time attribute column, but TIMESTAMP(3) encountered

2020-09-29 Thread panxiaohu (Jira)

panxiaohu created FLINK-19444:
-

 Summary: flink 1.11 sql group by tumble Window aggregate can only 
be defined over a time attribute column, but TIMESTAMP(3) encountered
 Key: FLINK-19444
 URL: https://issues.apache.org/jira/browse/FLINK-19444
 Project: Flink
  Issue Type: Bug
  Components: Table SQL / API
Affects Versions: 1.11.2
Reporter: panxiaohu


Here's the code:

String createSql = "CREATE TABLE clicks (\n" +
 " `user` STRING,\n" +
 " create_time TIMESTAMP(3),\n" +
 " PRIMARY KEY (`user`) NOT ENFORCED\n" +
 ") WITH (\n" +
 " 'connector' = 'jdbc',\n" +
 " 'url' = 'jdbc:mysql://localhost:3306/learning',\n" +
 " 'username' = 'root',\n" +
 " 'password' = 'john123',\n" +
 " 'table-name' = 'clicks'\n" +
 ")";

Table table = tableEnv.sqlQuery("select user,TUMBLE_START(create_time, INTERVAL 
'1' DAY),count(user) from clicks group by TUMBLE(create_time, INTERVAL '1' 
DAY),user" );

 

then exception occurs as follows:

org.apache.flink.table.api.TableException: Window aggregate can only be defined 
over a time attribute column, but TIMESTAMP(3) 
encountered.org.apache.flink.table.api.TableException: Window aggregate can 
only be defined over a time attribute column, but TIMESTAMP(3) encountered.
 at 
org.apache.flink.table.planner.plan.rules.logical.StreamLogicalWindowAggregateRule.getInAggregateGroupExpression(StreamLogicalWindowAggregateRule.scala:50)
 at 
org.apache.flink.table.planner.plan.rules.logical.LogicalWindowAggregateRuleBase.onMatch(LogicalWindowAggregateRuleBase.scala:79)
 at 
org.apache.calcite.plan.AbstractRelOptPlanner.fireRule(AbstractRelOptPlanner.java:328)
 at org.apache.calcite.plan.hep.HepPlanner.applyRule(HepPlanner.java:562) at 
org.apache.calcite.plan.hep.HepPlanner.applyRules(HepPlanner.java:427) at 
org.apache.calcite.plan.hep.HepPlanner.executeInstruction(HepPlanner.java:264) 
at 
org.apache.calcite.plan.hep.HepInstruction$RuleInstance.execute(HepInstruction.java:127)



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

43 matches

Mail list logo