Re: Ceph configuration for checkpoints?

[Martin Pompéry]

Hello Julio,

how we deploy Ceph:

1) make sure Ceph is mounted through all the nodes that make up your Flink
cluster; ensure path(s) are the same for each node
2) simply reference the Ceph mountpoint(s) via file://


Martin


On Tue, Feb 13, 2018 at 10:18 AM Piotr Nowojski 
wrote:

> Hi,
>
> Have you tried to refer to ceph documentation?
> http://docs.ceph.com/docs/jewel/cephfs/hadoop/ It claims to be:
>
> > a drop-in replacement for the Hadoop File System (HDFS)
>
> So I would first try to configure ceph according to their documentation
> and try to use Flink’s built in Hadoop connector.
>
> Piotrek
>
> On 12 Feb 2018, at 20:11, Julio Biason  wrote:
>
> Hello people,
>
> I'm looking for the configuration options required to use Ceph as a
> backend for checkpoints.
>
> So far, I can see that `state.backend.fs.checkpointdir` and
> `state.checkpoints.dir` should be something like `
> ceph:///my-checkpoint-bucket`. But where do I define server and acess and
> secret keys? I couldn't find anything remotely related to that in the
> docs...
>
> --
> *Julio Biason*, Sofware Engineer
> *AZION*  |  Deliver. Accelerate. Protect.
> Office: +55 51 3083 8101   |  Mobile: +55 51
> *99907 0554*
>
>
>

Re: Ceph configuration for checkpoints?

[Piotr Nowojski]

Hi,

Have you tried to refer to ceph documentation? 
http://docs.ceph.com/docs/jewel/cephfs/hadoop/ 
 It claims to be:

> a drop-in replacement for the Hadoop File System (HDFS)

So I would first try to configure ceph according to their documentation and try 
to use Flink’s built in Hadoop connector.

Piotrek

> On 12 Feb 2018, at 20:11, Julio Biason  wrote:
> 
> Hello people,
> 
> I'm looking for the configuration options required to use Ceph as a backend 
> for checkpoints.
> 
> So far, I can see that `state.backend.fs.checkpointdir` and 
> `state.checkpoints.dir` should be something like 
> `ceph:///my-checkpoint-bucket`. But where do I define server and acess and 
> secret keys? I couldn't find anything remotely related to that in the docs...
> 
> -- 
> Julio Biason, Sofware Engineer
> AZION  |  Deliver. Accelerate. Protect.
> Office: +55 51 3083 8101   |  Mobile: +55 51  
> 99907 0554

Ceph configuration for checkpoints?

[Julio Biason]

Hello people,

I'm looking for the configuration options required to use Ceph as a backend
for checkpoints.

So far, I can see that `state.backend.fs.checkpointdir` and
`state.checkpoints.dir` should be something like
`ceph:///my-checkpoint-bucket`. But where do I define server and acess and
secret keys? I couldn't find anything remotely related to that in the
docs...

-- 
*Julio Biason*, Sofware Engineer
*AZION*  |  Deliver. Accelerate. Protect.
Office: +55 51 3083 8101   |  Mobile: +55 51
*99907 0554*