Flink Kubernetes Session sample from Documentation

2023-04-04 Thread Le Xu 
Hello!

I'm trying out the Kubernetes sample

described in the official doc but I am not able to submit job with the
following error:

-
org.apache.flink.client.program.ProgramInvocationException: The main method
caused an error:
org.apache.flink.client.deployment.ClusterRetrieveException: Could not
create the RestClusterClient.
at
org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:372)
at
org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:222)
at
org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:105)
at
org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:851)
at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:245)
at
org.apache.flink.client.cli.CliFrontend.parseAndRun(CliFrontend.java:1095)
at
org.apache.flink.client.cli.CliFrontend.lambda$mainInternal$9(CliFrontend.java:1189)
at
org.apache.flink.runtime.security.contexts.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:28)
at
org.apache.flink.client.cli.CliFrontend.mainInternal(CliFrontend.java:1189)
at
org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1157)
Caused by: java.lang.RuntimeException:
org.apache.flink.client.deployment.ClusterRetrieveException: Could not
create the RestClusterClient.
at
org.apache.flink.kubernetes.KubernetesClusterDescriptor.lambda$createClusterClientProvider$1(KubernetesClusterDescriptor.java:121)
at
org.apache.flink.kubernetes.KubernetesClusterDescriptor.retrieve(KubernetesClusterDescriptor.java:148)
at
org.apache.flink.kubernetes.KubernetesClusterDescriptor.retrieve(KubernetesClusterDescriptor.java:69)
at
org.apache.flink.client.deployment.executors.AbstractSessionClusterExecutor.execute(AbstractSessionClusterExecutor.java:80)
at
org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.executeAsync(StreamExecutionEnvironment.java:2197)
at
org.apache.flink.client.program.StreamContextEnvironment.executeAsync(StreamContextEnvironment.java:189)
at
org.apache.flink.client.program.StreamContextEnvironment.execute(StreamContextEnvironment.java:118)
at
org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.execute(StreamExecutionEnvironment.java:2058)
at
org.apache.flink.streaming.examples.windowing.TopSpeedWindowing.main(TopSpeedWindowing.java:154)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:355)
... 9 more
Caused by: org.apache.flink.client.deployment.ClusterRetrieveException:
Could not create the RestClusterClient.
... 23 more
Caused by: java.net.UnknownHostException:
my-first-flink-cluster-rest.default: Name or service not known
at java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method)
at java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:929)
at
java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1324)
at java.net.InetAddress.getAllByName0(InetAddress.java:1277)
at java.net.InetAddress.getAllByName(InetAddress.java:1193)
at java.net.InetAddress.getAllByName(InetAddress.java:1127)
at java.net.InetAddress.getByName(InetAddress.java:1077)
at
org.apache.flink.runtime.highavailability.HighAvailabilityServicesUtils.getWebMonitorAddress(HighAvailabilityServicesUtils.java:229)
at
org.apache.flink.kubernetes.KubernetesClusterDescriptor.getWebMonitorAddress(KubernetesClusterDescriptor.java:140)
at
org.apache.flink.kubernetes.KubernetesClusterDescriptor.lambda$null$0(KubernetesClusterDescriptor.java:119)
at
org.apache.flink.client.program.rest.RestClusterClient.(RestClusterClient.java:237)
at
org.apache.flink.client.program.rest.RestClusterClient.(RestClusterClient.java:197)
at
org.apache.flink.kubernetes.KubernetesClusterDescriptor.lambda$createClusterClientProvider$1(KubernetesClusterDescriptor.java:114)
... 22 more
"

-


My kubernetes service does have DNS running (see the following):


-
root@node0:/mydata/flink-1.17.0# kubectl get pods -n

Failed docker-entrypoint.sh in Flink-Operator Logs

2023-04-04 Thread Ivan Webber via user 

I’ve noticed that all jobs I start with `flink-operator` have the following 
message at the top of their logs for both jobmanager and taskmanager pods:

```
sed: couldn't open temporary file /opt/flink/conf/sedRTwsr1: Read-only file 
system
sed: couldn't open temporary file /opt/flink/conf/sedcDS30D: Read-only file 
system
/docker-entrypoint.sh: line 73: /opt/flink/conf/flink-conf.yaml: Read-only file 
system
/docker-entrypoint.sh: line 89: /opt/flink/conf/flink-conf.yaml.tmp: Read-only 
file system
Starting kubernetes-application as a console application on host 
test-replay-run-b6458d699-nmfvf.
```

It seems these failures are due to the flink Docker images’ entrypoint being 
run by a user without permissions to write to `/opt/flink/conf` (as part of 
`sed -i`) or pipe to files in that folder. However, I’ve made my own container 
based on the docker scripts and even ensuring that all files are owned by 
`flink:flink` with full read-write permissions when running with Flink-operator 
these messages show up. Accordingly, I’m wondering if this is a bug or just 
something to ignore (e.g. flink-operator initialized the files and locked them 
to prevent further changes). If they are just something to ignore, it might be 
good to add an argument to `/docker-entrypoint.sh` to skip running it so there 
aren’t confusing error messages.

Thanks,

Ivan

[SUMMARY] Flink 1.18 Release Sync 4/4/2023

2023-04-04 Thread Jing Ge via user 
Dear devs and users,

Today was the kickoff meeting for Flink 1.18 release cycle. I'd like to
share the info synced in the meeting.

Meeting schedule:

Zoom will be used with a 40 mins limit for each meeting. That should be
fine for now. We will check it again if we have an issue with the time
limit later.

Release cycle will start bi-weekly and switch to weekly after the feature
freeze.

Feature freezing date:

July 11, 2023

Retrospective of 1.17 release:

There are many valuable thoughts and suggestions from previous release
managers[1]. Some of them are summarized as following:

- [Attention] Backports/merges without PRs will break master/release
branches. Kindly remind, every Flink developer, please pay attention to
avoid doing it.
- It is encouraged to create release testing tasks in advance, label them
properly, and finish them earlier, not necessarily wait to do it at the end
of the release cycle.
- A non-votable rc0 will be released in the future for developers to
validate the release.
- Some jira tickets have been created for 1.17 release that could be used
as the starting point to build a standard release pipeline. The release
process documented on the wiki page could be turned into a list of Jira
tasks (Jira template) in the future.

Daily work divisions:

In general, every release manager will be working on all daily issues. For
some major tasks, in order to make sure there will at least always be
someone to take care of them, they have been assigned to specific release
managers[2]. If you need support in each of these areas, please don't
hesitate to contact us.

1.18 Release page:

Flink 1.18 release has been kicked off today. We'd like to invite you to
update your development plan on the release page[2].

The next release sync up meeting will be on April 18, 2023. Please feel
free to join us!

Zoom meeting:
https://us04web.zoom.us/j/79158702091?pwd=8CXPqxMzbabWkma5b0qFXI1IcLbxBh.1

Best regards,
Konstantin, Sergey, Qingsheng, and Jing

[1] https://cwiki.apache.org/confluence/display/FLINK/1.17+Release
[2] https://cwiki.apache.org/confluence/display/FLINK/1.18+Release

Re:Re: How to set reblance in Flink Sql like Streaming api?

2023-04-04 Thread hjw 
Hi Shammon
Are you suggesting that I use over and partition by , right? if it is like 
this, I must define a agg_func on a specific column.
For Example,I have a product table.


Before partition by :
select user,product,amount
FROM product


After partition by :
select user,product,amount, FIRST_VALUE(user) OVER( partition by product) AS 
meanless_col FROM product


it will cause state too large problem possible.Do I need to define a Range like 
this?
select user,product,amount, FIRST_VALUE(user) OVER( partition by product) AS 
meanless_col
RANGEBETWEENINTERVAL'1'HOURPRECEDINGANDCURRENTROW FROM product







--

Best，
Hjw




At 2023-04-04 12:23:26, "Shammon FY"  wrote:

Hi hjw


To rescale data for dim join, I think you can use `partition by` in sql before 
`dim join` which will redistribute data by specific column. In addition, you 
can add cache for `dim table` to improve performance too.


Best,
Shammon FY




On Tue, Apr 4, 2023 at 10:28 AM Hang Ruan  wrote:

Hi, hiw,


IMO, I think the parallelism 1 is enough for you job if we do not consider the 
sink. I do not know why you need set the lookup join operator's parallelism to 
6.
The SQL planner will help us to decide the type of the edge and we can not 
change it.
Maybe you could share the Execution graph to provide more information.


Best,
Hang


hjw  于2023年4月4日周二 00:37写道：

For example. I create a kafka source to subscribe  the topic that have one 
partition and set the default parallelism of the job to 6.The next operator of 
kafka source is that  lookup join a mysql table.However, the relationship 
between the kafka Source and the Lookup join operator is Forward, so only one 
subtask in the Lookup join operator can receive data.I want to set the 
relationship between the kafka Source and the Lookup join operator is reblance 
so that all subtask in Lookup join operator can recevie data.


Env:
Flink version:1.15.1




--

Best，
Hjw

Re: 退订

2023-04-04 Thread Shammon FY 
Hi

发送任意邮件到 user-zh-unsubscr...@flink.apache.org
 退订

[1]
https://flink.apache.org/community.html#how-to-subscribe-to-a-mailing-list


On Tue, Apr 4, 2023 at 1:20 PM 柳懿珊  wrote:

> 退订