Re: MODERATE for d...@flink.apache.org
Hi Sri, Thank you for your interest with Apache Flink. To continue to interact with people in the mailing list, please subscribe to the list [1] to make sure your posts are delivered to the right list. Thanks, Henry [1] https://flink.apache.org/community.html#how-to-subscribe-to-a-mailing-list > > -- Forwarded message -- > From: sri hari kali charan Tummala > To: user , d...@flink.apache.org > Cc: > Bcc: > Date: Mon, 24 Feb 2020 16:25:59 -0600 > Subject: Batch Flink Job S3 write performance vs Spark > Hi All, > > have a question did anyone compared the performance of Flink batch job > writing to s3 vs spark writing to s3? > > -- > Thanks & Regards > Sri Tummala > >
Re: MODERATE for d...@flink.apache.org
Have you tried removing the "cygdrive" portion from the path? Something along the lines of “state.backend.fs.checkpointdir: file:///Y:/flink-checkpoint-dir state.checkpoints.dir: Y:/flink-checkpoints state.backend: filesystem high-availability.storageDir: file:///Y:/flink-recovery " On 07.11.2017 12:36, Jordan Kuan wrote: Dear Ufuk, Sorry, I still can’t get that work, I have double checked all the nodes can access the mapped drive Y. Here is my state backend configuration, and it writes files to local C drive. “state.backend.fs.checkpointdir: file:///cygdrive/Y/flink-checkpoint-dir state.checkpoints.dir: /cygdrive/Y/flink-checkpoints state.backend: filesystem high-availability.storageDir: file:///cygdrive/Y/flink-recovery " I have attached the flink-conf.yaml and screenshots, Is here any wrong setting? Thanks, Jordan On 7 Nov 2017, at 6:49 PM, Ufuk Celebi <u...@apache.org <mailto:u...@apache.org>> wrote: Hey Jordan, yeah, that should just work. Check out the state backend configuration here: https://ci.apache.org/projects/flink/flink-docs-release-1.4/ops/state/state_backends.html <https://ci.apache.org/projects/flink/flink-docs-release-1.4/ops/state/state_backends.html> – Ufuk On Tue, Nov 7, 2017 at 11:44 AM, Jordan Kuan <jordan.k...@gmail.com <mailto:jordan.k...@gmail.com>> wrote: Dear Ufuk, Thank you for your reply. All the cluster Flink servers are able to access network drive, and it mapped as drive Y in all nodes. Do I need to provide more information? Thanks, Jordan On 7 Nov 2017, at 6:36 PM, Ufuk Celebi <u...@apache.org <mailto:u...@apache.org>> wrote: As answered by David on SO, the files need to be accessible by all nodes. In your setup this seems not to be the case, therefore it won't work. You need a distributed file system (e.g. NFS or HDFS) or object store (e.g. S3) that is accessible from all nodes. – Ufuk On Tue, Nov 7, 2017 at 3:34 AM, Jordan Kuan <jordan.k...@gmail.com <mailto:jordan.k...@gmail.com>> wrote: Dear Flink Dev Team I have encountered a problem and can't find any solution in Google. And I have created a thread in stackoverflow.com <http://stackoverflow.com> but no response. https://stackoverflow.com/questions/47123371/flink-windows-ha <https://stackoverflow.com/questions/47123371/flink-windows-ha> I would really appreciate it if you could give some suggestions to me. Thanks, Jordan On Tue, Nov 7, 2017 at 1:53 AM, Robert Metzger <rmetz...@apache.org <mailto:rmetz...@apache.org>> wrote: Hi, I would suggest to send your question to the user@flink.apache.org <mailto:user@flink.apache.org> list (make sure to subscribe first) -- Forwarded message -- From: <dev-reject-1509985381.14933.cigohlkhhjaclimeg...@flink.apache.org <mailto:dev-reject-1509985381.14933.cigohlkhhjaclimeg...@flink.apache.org>> Date: Mon, Nov 6, 2017 at 5:23 PM Subject: MODERATE for d...@flink.apache.org <mailto:d...@flink.apache.org> To: Cc: dev-allow-tc.1509985381.dcccgaimcaiiefbkiapi-jordan.kuan=gmail@flink.apache.org <mailto:gmail@flink.apache.org> To approve: dev-accept-1509985381.14933.cigohlkhhjaclimeg...@flink.apache.org <mailto:dev-accept-1509985381.14933.cigohlkhhjaclimeg...@flink.apache.org> To reject: dev-reject-1509985381.14933.cigohlkhhjaclimeg...@flink.apache.org <mailto:dev-reject-1509985381.14933.cigohlkhhjaclimeg...@flink.apache.org> To give a reason to reject: %%% Start comment %%% End comment -- Forwarded message -- From: Jordan Kuan <jordan.k...@gmail.com <mailto:jordan.k...@gmail.com>> To: d...@flink.apache.org <mailto:d...@flink.apache.org> Cc: Bcc: Date: Tue, 7 Nov 2017 00:22:53 +0800 Subject: Flink Windows HA Issue. Dear Flink Dev Team I have encountered a problem and can't find any solution in Google. And I have created a thread in stackoverflow.com <http://stackoverflow.com> but no response. https://stackoverflow.com/questions/47123371/flink-windows-ha <https://stackoverflow.com/questions/47123371/flink-windows-ha> I would really appreciate it if you could give some suggestions to me. Thanks, Jordan -- Best Regards, Jordan Kuan
Re: MODERATE for d...@flink.apache.org
Dear Ufuk, Sorry, I still can’t get that work, I have double checked all the nodes can access the mapped drive Y. Here is my state backend configuration, and it writes files to local C drive. “state.backend.fs.checkpointdir: file:///cygdrive/Y/flink-checkpoint-dir state.checkpoints.dir: /cygdrive/Y/flink-checkpoints state.backend: filesystem high-availability.storageDir: file:///cygdrive/Y/flink-recovery " I have attached the flink-conf.yaml and screenshots, Is here any wrong setting? Thanks, Jordan On 7 Nov 2017, at 6:49 PM, Ufuk Celebi <u...@apache.org> wrote: Hey Jordan, yeah, that should just work. Check out the state backend configuration here: https://ci.apache.org/projects/flink/flink-docs-release-1.4/ ops/state/state_backends.html – Ufuk On Tue, Nov 7, 2017 at 11:44 AM, Jordan Kuan <jordan.k...@gmail.com> wrote: Dear Ufuk, Thank you for your reply. All the cluster Flink servers are able to access network drive, and it mapped as drive Y in all nodes. Do I need to provide more information? Thanks, Jordan On 7 Nov 2017, at 6:36 PM, Ufuk Celebi <u...@apache.org> wrote: As answered by David on SO, the files need to be accessible by all nodes. In your setup this seems not to be the case, therefore it won't work. You need a distributed file system (e.g. NFS or HDFS) or object store (e.g. S3) that is accessible from all nodes. – Ufuk On Tue, Nov 7, 2017 at 3:34 AM, Jordan Kuan <jordan.k...@gmail.com> wrote: Dear Flink Dev Team I have encountered a problem and can't find any solution in Google. And I have created a thread in stackoverflow.com but no response. https://stackoverflow.com/questions/47123371/flink-windows-ha I would really appreciate it if you could give some suggestions to me. Thanks, Jordan On Tue, Nov 7, 2017 at 1:53 AM, Robert Metzger <rmetz...@apache.org> wrote: Hi, I would suggest to send your question to the user@flink.apache.org list (make sure to subscribe first) -- Forwarded message -- From: <dev-reject-1509985381.14933.cigohlkhhjaclimeg...@flink.apache.org> Date: Mon, Nov 6, 2017 at 5:23 PM Subject: MODERATE for d...@flink.apache.org To: Cc: dev-allow-tc.1509985381.dcccgaimcaiiefbkiapi-jordan.kuan=gma il@flink.apache.org To approve: dev-accept-1509985381.14933.cigohlkhhjaclimeg...@flink.apache.org To reject: dev-reject-1509985381.14933.cigohlkhhjaclimeg...@flink.apache.org To give a reason to reject: %%% Start comment %%% End comment -- Forwarded message -- From: Jordan Kuan <jordan.k...@gmail.com> To: d...@flink.apache.org Cc: Bcc: Date: Tue, 7 Nov 2017 00:22:53 +0800 Subject: Flink Windows HA Issue. Dear Flink Dev Team I have encountered a problem and can't find any solution in Google. And I have created a thread in stackoverflow.com but no response. https://stackoverflow.com/questions/47123371/flink-windows-ha I would really appreciate it if you could give some suggestions to me. Thanks, Jordan -- Best Regards, Jordan Kuan flink-conf.yaml Description: Binary data
Re: MODERATE for d...@flink.apache.org
Hey Jordan, yeah, that should just work. Check out the state backend configuration here: https://ci.apache.org/projects/flink/flink-docs-release-1.4/ops/state/state_backends.html – Ufuk On Tue, Nov 7, 2017 at 11:44 AM, Jordan Kuan <jordan.k...@gmail.com> wrote: > Dear Ufuk, > > Thank you for your reply. > > All the cluster Flink servers are able to access network drive, and it mapped > as drive Y in all nodes. > Do I need to provide more information? > > Thanks, > Jordan > > >> On 7 Nov 2017, at 6:36 PM, Ufuk Celebi <u...@apache.org> wrote: >> >> As answered by David on SO, the files need to be accessible by all >> nodes. In your setup this seems not to be the case, therefore it won't >> work. You need a distributed file system (e.g. NFS or HDFS) or object >> store (e.g. S3) that is accessible from all nodes. >> >> – Ufuk >> >> >> On Tue, Nov 7, 2017 at 3:34 AM, Jordan Kuan <jordan.k...@gmail.com> wrote: >>> Dear Flink Dev Team >>> >>> I have encountered a problem and can't find any solution in Google. >>> And I have created a thread in stackoverflow.com but no response. >>> https://stackoverflow.com/questions/47123371/flink-windows-ha >>> >>> I would really appreciate it if you could give some suggestions to me. >>> >>> Thanks, >>> >>> Jordan >>> >>> On Tue, Nov 7, 2017 at 1:53 AM, Robert Metzger <rmetz...@apache.org> wrote: >>>> >>>> Hi, >>>> I would suggest to send your question to the user@flink.apache.org list >>>> (make sure to subscribe first) >>>> >>>> -- Forwarded message -- >>>> From: <dev-reject-1509985381.14933.cigohlkhhjaclimeg...@flink.apache.org> >>>> Date: Mon, Nov 6, 2017 at 5:23 PM >>>> Subject: MODERATE for d...@flink.apache.org >>>> To: >>>> Cc: >>>> dev-allow-tc.1509985381.dcccgaimcaiiefbkiapi-jordan.kuan=gmail@flink.apache.org >>>> >>>> >>>> >>>> To approve: >>>> dev-accept-1509985381.14933.cigohlkhhjaclimeg...@flink.apache.org >>>> To reject: >>>> dev-reject-1509985381.14933.cigohlkhhjaclimeg...@flink.apache.org >>>> To give a reason to reject: >>>> %%% Start comment >>>> %%% End comment >>>> >>>> >>>> >>>> -- Forwarded message -- >>>> From: Jordan Kuan <jordan.k...@gmail.com> >>>> To: d...@flink.apache.org >>>> Cc: >>>> Bcc: >>>> Date: Tue, 7 Nov 2017 00:22:53 +0800 >>>> Subject: Flink Windows HA Issue. >>>> Dear Flink Dev Team >>>> >>>> I have encountered a problem and can't find any solution in Google. >>>> And I have created a thread in stackoverflow.com but no response. >>>> https://stackoverflow.com/questions/47123371/flink-windows-ha >>>> >>>> I would really appreciate it if you could give some suggestions to me. >>>> >>>> Thanks, >>>> >>>> Jordan >>>> >>>> >>>> >>> >>> >>> >>> -- >>> Best Regards, >>> Jordan Kuan >
Re: MODERATE for d...@flink.apache.org
Dear Ufuk, Thank you for your reply. All the cluster Flink servers are able to access network drive, and it mapped as drive Y in all nodes. Do I need to provide more information? Thanks, Jordan > On 7 Nov 2017, at 6:36 PM, Ufuk Celebi <u...@apache.org> wrote: > > As answered by David on SO, the files need to be accessible by all > nodes. In your setup this seems not to be the case, therefore it won't > work. You need a distributed file system (e.g. NFS or HDFS) or object > store (e.g. S3) that is accessible from all nodes. > > – Ufuk > > > On Tue, Nov 7, 2017 at 3:34 AM, Jordan Kuan <jordan.k...@gmail.com> wrote: >> Dear Flink Dev Team >> >> I have encountered a problem and can't find any solution in Google. >> And I have created a thread in stackoverflow.com but no response. >> https://stackoverflow.com/questions/47123371/flink-windows-ha >> >> I would really appreciate it if you could give some suggestions to me. >> >> Thanks, >> >> Jordan >> >> On Tue, Nov 7, 2017 at 1:53 AM, Robert Metzger <rmetz...@apache.org> wrote: >>> >>> Hi, >>> I would suggest to send your question to the user@flink.apache.org list >>> (make sure to subscribe first) >>> >>> -- Forwarded message -- >>> From: <dev-reject-1509985381.14933.cigohlkhhjaclimeg...@flink.apache.org> >>> Date: Mon, Nov 6, 2017 at 5:23 PM >>> Subject: MODERATE for d...@flink.apache.org >>> To: >>> Cc: >>> dev-allow-tc.1509985381.dcccgaimcaiiefbkiapi-jordan.kuan=gmail@flink.apache.org >>> >>> >>> >>> To approve: >>> dev-accept-1509985381.14933.cigohlkhhjaclimeg...@flink.apache.org >>> To reject: >>> dev-reject-1509985381.14933.cigohlkhhjaclimeg...@flink.apache.org >>> To give a reason to reject: >>> %%% Start comment >>> %%% End comment >>> >>> >>> >>> -- Forwarded message -- >>> From: Jordan Kuan <jordan.k...@gmail.com> >>> To: d...@flink.apache.org >>> Cc: >>> Bcc: >>> Date: Tue, 7 Nov 2017 00:22:53 +0800 >>> Subject: Flink Windows HA Issue. >>> Dear Flink Dev Team >>> >>> I have encountered a problem and can't find any solution in Google. >>> And I have created a thread in stackoverflow.com but no response. >>> https://stackoverflow.com/questions/47123371/flink-windows-ha >>> >>> I would really appreciate it if you could give some suggestions to me. >>> >>> Thanks, >>> >>> Jordan >>> >>> >>> >> >> >> >> -- >> Best Regards, >> Jordan Kuan
Re: MODERATE for d...@flink.apache.org
As answered by David on SO, the files need to be accessible by all nodes. In your setup this seems not to be the case, therefore it won't work. You need a distributed file system (e.g. NFS or HDFS) or object store (e.g. S3) that is accessible from all nodes. – Ufuk On Tue, Nov 7, 2017 at 3:34 AM, Jordan Kuan <jordan.k...@gmail.com> wrote: > Dear Flink Dev Team > > I have encountered a problem and can't find any solution in Google. > And I have created a thread in stackoverflow.com but no response. > https://stackoverflow.com/questions/47123371/flink-windows-ha > > I would really appreciate it if you could give some suggestions to me. > > Thanks, > > Jordan > > On Tue, Nov 7, 2017 at 1:53 AM, Robert Metzger <rmetz...@apache.org> wrote: >> >> Hi, >> I would suggest to send your question to the user@flink.apache.org list >> (make sure to subscribe first) >> >> -- Forwarded message -- >> From: <dev-reject-1509985381.14933.cigohlkhhjaclimeg...@flink.apache.org> >> Date: Mon, Nov 6, 2017 at 5:23 PM >> Subject: MODERATE for d...@flink.apache.org >> To: >> Cc: >> dev-allow-tc.1509985381.dcccgaimcaiiefbkiapi-jordan.kuan=gmail@flink.apache.org >> >> >> >> To approve: >>dev-accept-1509985381.14933.cigohlkhhjaclimeg...@flink.apache.org >> To reject: >>dev-reject-1509985381.14933.cigohlkhhjaclimeg...@flink.apache.org >> To give a reason to reject: >> %%% Start comment >> %%% End comment >> >> >> >> -- Forwarded message -- >> From: Jordan Kuan <jordan.k...@gmail.com> >> To: d...@flink.apache.org >> Cc: >> Bcc: >> Date: Tue, 7 Nov 2017 00:22:53 +0800 >> Subject: Flink Windows HA Issue. >> Dear Flink Dev Team >> >> I have encountered a problem and can't find any solution in Google. >> And I have created a thread in stackoverflow.com but no response. >> https://stackoverflow.com/questions/47123371/flink-windows-ha >> >> I would really appreciate it if you could give some suggestions to me. >> >> Thanks, >> >> Jordan >> >> >> > > > > -- > Best Regards, > Jordan Kuan
Re: MODERATE for d...@flink.apache.org
Dear Flink Dev Team I have encountered a problem and can't find any solution in Google. And I have created a thread in stackoverflow.com but no response. https://stackoverflow.com/questions/47123371/flink-windows-ha I would really appreciate it if you could give some suggestions to me. Thanks, Jordan On Tue, Nov 7, 2017 at 1:53 AM, Robert Metzger <rmetz...@apache.org> wrote: > Hi, > I would suggest to send your question to the user@flink.apache.org list > (make sure to subscribe first) > > -- Forwarded message -- > From: <dev-reject-1509985381.14933.cigohlkhhjaclimeg...@flink.apache.org> > Date: Mon, Nov 6, 2017 at 5:23 PM > Subject: MODERATE for d...@flink.apache.org > To: > Cc: dev-allow-tc.1509985381.dcccgaimcaiiefbkiapi-jordan.kuan= > gmail@flink.apache.org > > > > To approve: >dev-accept-1509985381.14933.cigohlkhhjaclimeg...@flink.apache.org > To reject: >dev-reject-1509985381.14933.cigohlkhhjaclimeg...@flink.apache.org > To give a reason to reject: > %%% Start comment > %%% End comment > > > > -- Forwarded message -- > From: Jordan Kuan <jordan.k...@gmail.com> > To: d...@flink.apache.org > Cc: > Bcc: > Date: Tue, 7 Nov 2017 00:22:53 +0800 > Subject: Flink Windows HA Issue. > Dear Flink Dev Team > > I have encountered a problem and can't find any solution in Google. > And I have created a thread in stackoverflow.com but no response. > https://stackoverflow.com/questions/47123371/flink-windows-ha > > I would really appreciate it if you could give some suggestions to me. > > Thanks, > > Jordan > > > > -- Best Regards, Jordan Kuan flink-conf.yaml Description: Binary data
