[Csync2] How to contribute to csync2 [Was: Using Csync2 when hosts are not resolved by dns]

[Lars Ellenberg]

On Mon, Oct 22, 2012 at 01:26:26PM -0700, Michi Mutsuzaki wrote:
 Yeah I think adding a documentation would be great. I spent a lot of
 time figuring out the config file syntax, right options to use, etc..
 I don't know what the process is for contributing to csync2. I hope
 someone can help you with that.

git clone git://git.linbit.com/csync2
cd csync2/doc

edit csync2_paper.tex, and check the results:
pdflatex --interaction=batchmode csync2_paper.tex
pdflatex --interaction=batchmode csync2_paper.tex

now check csync2_paper.pdf ...

Iterate until you are satisfied.

Then, git diff  my-doc-improvement.diff,
and attach that to an email to this list.

(works of course the same for non-documentation, but code improvements)


Alternatively, commit locally:
git commit -a
  write a nice commit message describing what you did

And then post your commits to this mailing list,
either by
 git format-patch -k origin/master
   and then attaching the generated diffs to some email yourself.
or by
 git send-email --dry-run --to csync2@lists.linbit.com origin/master

(leave off the dry-run, if that in fact did what you intended)


Thanks,

Lars

-- 
: Lars Ellenberg
: LINBIT | Your Way to High Availability
: DRBD/HA support and consulting http://www.linbit.com
___
Csync2 mailing list
Csync2@lists.linbit.com
http://lists.linbit.com/mailman/listinfo/csync2

Re: [Csync2] prefix works except when file is specified

[Lars Ellenberg]

On Thu, Jul 10, 2014 at 01:53:20AM -0400, dlo wrote:
 Lars- I see it's not so far off, but definitely not a drop-in replacement
 (already ran into some problems).  It worries me to make so many changes to
 a system that's been working so well for so long... is there really no hope
 for getting file-specific syncs working with prefixes in 1.34?

*I* won't work on anything 1.x,
and I have no idea what would need to be done there.
Chances are that what ever needs to be done has already been done
on the way to 2.0.

If anything needs to be fixed,
then please fix it on top of the current git.

Lars

 I'm also worried that it still might not work even if I do... I took a good
 long look at the source code for both versions.  In csync_update_host(),
 there's a line that seems to me will always be false, leading to no update:
 
 if ( compare_files
 http://sourcecodebrowser.com/csync2/1.34/update_8c.html#a405f4be2b99d5da4363abb4b8348c16b(filename,
 patlist[i], recursive) ) use_this = 1;
 
 
 My understanding is that filename is as it's listed in the database, which
 for a config with prefix, will include the prefix (eg: %FSROOT%/test1).
 
 Isn't patlist[i] simply the filename as passed on the command line? (eg:
 /path/to/test1)
 
 It doesn't appear that compare_files() takes into account the prefix.  If
 so, won't such a compare_files() call always return 0? (resulting in no
 update) [http://codepad.org/mYtUB4o7]
 
 btw- the relevant code appears to be the same in 2.0 as in 1.34.  If so,
 specifying the file on the command line in a configuration with a prefix
 still won't work... can someone on this list having a prefix config (using
 either version) confirm that specifying the filename on the command line
 triggers an update?
 
 Note that csync_check() clearly substitutes the prefix, but I don't see
 where such a substitution happens for updates.
 
 Am I wading in too deep?

___
Csync2 mailing list
Csync2@lists.linbit.com
http://lists.linbit.com/mailman/listinfo/csync2

Re: [Csync2] Younger/older test tied

[Lars Ellenberg]

On Tue, Oct 27, 2015 at 02:25:20PM +, Marcus Downing wrote:
> When synchronising between two masters and one slave (client sensitive
> information sanitised, of course):
> 
> group cluster {
> host server1 server2 (backup);
> key /etc/csync2.key;
> include /var/www/htdocs;
> auto younger;
> }
> 
> 
> I've been seeing a great many errors like this on both sides:
> 
> While syncing file
> /var/www/htdocs/wp-content/themes/wireframe/widgets/search_widget.php:
> ERROR from peer server1: File is also marked dirty here!
> Do not auto-resolve conflict: Lost 'younger/older' test.

So why is that file generated simultaneously on both nodes?

> I understand that csync2 is push only, and this error is expect whenever
> the other server has more recent changes than we do. However, I'm seeing it
> on both sides, and I'm seeing it for files that haven't changed recently. I
> believe the cause is that the files' timestamps one one server have been
> updated by an external rsync, but the contents weren't changed at all.
> 
> I suggest two changes to fix this. First, csync2 needs to recognise that
> the content, timestamp and other attributes of a file are identical, ie
> that there's nothing to sync, and clear the dirty flag for that file in its
> local database.

If the stat information (as recorded by csync2) is identical,
*and* the content is identical, even if both sides are "marked dirty",
You will get "File is already up to date on peer.",
and the dirty flag is cleared (on both sides, for this host pair).

At least that is how it *should* be, and how it works for me.
So if that does not work for you, you may need to upgrade,
or come back with a test case to show how to reproduce the misbehaviour.

> 
> Second, I suggest the following line be changed:
> http://git.linbit.com/csync2.git/blob/HEAD:/update.c#l553
> 
> from
> 
> if ((localdata > remotedata) ==
> 
> 
> to:
> 
> if ((localdata >= remotedata) ==
> 
> 
> which is to say, if the timestamps on the two sides are the same it
> considers itself to have won the conflict and proceeds to sync successfully.

younger or older is exactly *not* "... or same timestamp"

IIUC, semantically you are proposing that "younger" would be changed to
"younger", but if same time stamp, "first"
  (the guy who first called csync2 --update wins)

maybe that should be made explicit, though, rather than implicitly by
only changing less-than to less-than-or-equal.  At least it deserves
some change in the documentation and a code comment.


-- 
: Lars Ellenberg
: http://www.LINBIT.com | Your Way to High Availability
: DRBD, Linux-HA  and  Pacemaker support and consulting

DRBD® and LINBIT® are registered trademarks of LINBIT, Austria.
___
Csync2 mailing list
Csync2@lists.linbit.com
http://lists.linbit.com/mailman/listinfo/csync2

Re: [Csync2] Patchwork to running on musl libc and some cleanups

[Lars Ellenberg]

On Fri, Oct 09, 2015 at 01:22:03PM +0300, Valery Kartel wrote:
> I have a strange runtime error on my Alpinelinux box.
> While debugging I found that snprintf("%Ld"...) format for (long
> long) is invalid.
> The correct one is "%lld"
> 
> The first patch fixes this.
> 
> The second patch is a cleanup to not compile a 'strlcmp' function if
> we have system one.
> 
> The third patch is to disable ssl by default, because many distros
> do not generate ssl certificate and key.
> 
> Don't know where to send patches. So I place it here.

Best place to send patches for csync2.
Thanks, looks innocent enough ;-)

> -   xxprintf(":type=reg:size=%Ld", (long long)st->st_size);
> +       xxprintf(":type=reg:size=%lld", (long long)st->st_size);

Cheers,

-- 
: Lars Ellenberg
: http://www.LINBIT.com | Your Way to High Availability
: DRBD, Linux-HA  and  Pacemaker support and consulting

DRBD® and LINBIT® are registered trademarks of LINBIT, Austria.
___
Csync2 mailing list
Csync2@lists.linbit.com
http://lists.linbit.com/mailman/listinfo/csync2

Re: [Csync2] Logfile Issues

[Lars Ellenberg]

On Sat, Sep 26, 2015 at 09:28:16PM -0400, Brian Blater wrote:
> I've got csync2 installed on a couple of Ubuntu servers that together with
> watcher.py will be keeping a large file system synchronized. All the
> files/directories are under the path /data/ and there is about 5
> directories there, but under those directories could be hundreds if not
> thousands of other files and directories.
> 
> I'm basically getting everything setup right now and doing some tests. The
> sync part works great together with watcher.py to maintain an almost
> realtime update.
> 
> The problem I'm having is getting logging setup and working for csync2.
> From what I can gather I have to setup an action under the group for the
> logging. This is what I have in the csync2.cfg:
> 
> nossl * *;
> 
> group production {
> key /etc/csync-production-group.key;
> host server1;
> host server2;
> include /data;
> exclude /data/conflict;
> exclude *~ .*;
> action {
> pattern /data/*;
> logfile "/var/log/csync2/csync2.log";

That would log the output of the *action*
(but you did not define any).

action being some command to be called
if csync2 updated a file matching pattern.

intended use case would be a pattern matching some apache conf files,
and an action that basically does "apachectl graceful", once that
conf file has been copied over.

> do-local;
> }
> auto none;
> }
> 
> I'm not getting any logging at all. I can't figure out if the pattern is
> wrong or what. I basically want csync2 to log what it does in the log file,
> but I can't get it working.
> 
> Any help would be greatly appreciated.

If you want csync2 to log what it does,
you have to increase its "verbosity" (add -v...),
and possibly tell it to use syslog.
it usually should redirect to syslog anyways,
if it detects that it has no tty.


-- 
: Lars Ellenberg
: http://www.LINBIT.com | Your Way to High Availability
: DRBD, Linux-HA  and  Pacemaker support and consulting

DRBD® and LINBIT® are registered trademarks of LINBIT, Austria.
___
Csync2 mailing list
Csync2@lists.linbit.com
http://lists.linbit.com/mailman/listinfo/csync2

Re: [Csync2] Csync -M doesn't work as espected

[Lars Ellenberg]

On Fri, Nov 20, 2015 at 11:21:31AM -0600, Miguel Bonera TXT wrote:
> Hi.
> 
> I'm really happy with csync2: I could finally have a full
> geo-syncronization of my webservers, improving performances and
> reliability of the hosted sites. I wrote a little daemon that handle
> synchronization management, but there is something unclear and I
> don't know where I'm wrong.
> 
> The problem is that I wanted to use the -M feature to display
> modifications (after a csync2 -cr, of course), but the only result I
> obtain is an empty output and a 2 as exit code, no matter if I have
> or not modification on the path I'm including in sync.
> 
> I tried csync -M with or without -C config file, with version 1.34
> and 2.0, but none of it works. What am I doing wrong?


install csync2 on alice.demo and bob.demo
generate "key" file
edit config file

optionally initialize local database with what is already there
 (csync2 -crvI /)

now, nothing is dirty:
alice.demo# csync2 -M ; echo $?
2

assuming you have "include /csync2/demo;"
alice.demo# touch /csync2/demo/something

but it does not know this yet:
alice.demo# csync2 -M

make it notice:
alice.demo# csync2 -cvr /
Marking file as dirty: /csync2/demo/something

ask it again:
alice.demo# csync2 -M
chary   alice.demo  bob.demo/csync2/demo/something

works as expected (for me).

What did you expect?

-- 
: Lars Ellenberg
: http://www.LINBIT.com | Your Way to High Availability
: DRBD, Linux-HA  and  Pacemaker support and consulting

DRBD® and LINBIT® are registered trademarks of LINBIT, Austria.
___
Csync2 mailing list
Csync2@lists.linbit.com
http://lists.linbit.com/mailman/listinfo/csync2

Re: [Csync2] [patch] document -N

[Lars Ellenberg]

On Wed, Apr 27, 2016 at 11:43:35AM +1000, Aristedes Maniatis wrote:
> Lars Ellenberg wrote:
> 
> > You do realize that there is a git beyond github?
> 
> Yes. I and I know you are only joking. But there are big advantages IMO to 
> using github. Everyone expects you to be there and it is confusing when a 
> project is not. Plus of course the benefit of the simple task tracker, pull 
> requests and the web browsing of a repo.
> 
> Finally, submitting patches by email loses all the commit history of that 
> patch, who made it and why it was made.

Not at all, not if you use git format-patch.
But as you wish. github/LINBIT/csync2 :-)

> Anyhow, I made a simple patch for you for updating the docs:
> 
> https://github.com/ari/csync2/commit/4cf424738840f226ef01492ca3fbceed96cbb62d

Thanks,
Lars
___
Csync2 mailing list
Csync2@lists.linbit.com
http://lists.linbit.com/mailman/listinfo/csync2

Re: [Csync2] ipv6 problems

[Lars Ellenberg]

On Wed, Apr 27, 2016 at 06:49:21PM +1000, Aristedes Maniatis wrote:
> On 27/04/2016 11:45am, Aristedes Maniatis wrote:
> > On 19/04/2016 9:14pm, Lars Ellenberg wrote:
> >> I don't think so.
> >> It's supposed to bind to something that is suitable for *both* IPv6 and 
> >> IPv4.
> > 
> > A patch has been submitted that might fix the IPv6 problem:
> > 
> > https://github.com/ari/csync2/commit/10636a4786485f7a45d1c35a554b9bb57fa52aae
> > 
> > 
> > I haven't yet had a chance to test it, but I'll try and do that in the next 
> > few days. Does it look reasonable to you?
> 
> 
> I've now tested it and it works nicely in my environment (FreeBSD 10.2).

Thanks, pulled.  May need a config option at some point,
if people want to restrict it to either family.

Lars

___
Csync2 mailing list
Csync2@lists.linbit.com
http://lists.linbit.com/mailman/listinfo/csync2

Re: [Csync2] Issue with the auto-backup feature. A bug?

[Lars Ellenberg]

On Fri, Apr 29, 2016 at 07:05:30PM -0400, a...@gluu.org wrote:
> Greetings.
> 
> I'm a novice user of csync2, and just have built my first test
> cluster with it. However, I've encountered issue I haven't been able
> to find solution for yet. I installed csync2 using this package:
> http://oss.linbit.com/csync2/csync2-2.0.tar.gz I have quite a simple
> cluster setup of 2 nodes which try to replicate 1 directory with 1
> file in it. Here is a sample config file:

Most likely, this is again inetd mixing stderr of csync2 into stdout,
thereby "corrupting" the csync2 protocol.

add "-l" to the inetd invocation.
as in
/etc/inetd.conf:
-csync2  stream  tcp nowait  root/usr/sbin/csync2 csync2 -i
+csync2  stream  tcp nowait  root/usr/sbin/csync2 csync2 -i -l


or add up to two -v (the third -v will break protocol again,
when run from inetd, see below).

currently csync2 defaults to using syslog when run from inetd
when "-v" is present only.

I'll try to push a workaround for defaulting to syslog when run from inetd,
disregarding the debug level.

But csync2 still uses stderr (or even stdout!) directly in some cases.
This optional "debug" logging of csync2 is badly grown
$swearword_of_choice; actually fixing that might take some time.


-- 
: Lars Ellenberg
: LINBIT | Keeping the Digital World Running
: DRBD -- Heartbeat -- Corosync -- Pacemaker
: R, Integration, Ops, Consulting, Support

DRBD® and LINBIT® are registered trademarks of LINBIT
___
Csync2 mailing list
Csync2@lists.linbit.com
http://lists.linbit.com/mailman/listinfo/csync2

Re: [Csync2] ipv6 problems

[Lars Ellenberg]

On Wed, Apr 20, 2016 at 04:28:26PM +1000, Aristedes Maniatis wrote:
> On 19/04/2016 9:14pm, Lars Ellenberg wrote:
> > You can tell it to bind to "-N your.node.name".
> > If not specified, it will use the "any" address
> > (pass a NULL node to getaddrinfo)
> 
> Passing -N on the command line works nicely to bind to a specific IP. Thanks.
> 
> When you have a moment, please add that switch to the man page, I'm sure 
> others would like to know about it.
> 

man page is generated (mostly) via help2man,
so the --help message is the relevant source,
feel free to add a paragraph.

> > The relevant function is "csync_server_bind()" in csync2.c
> 
> I'm not a C coder, but I'm confused about
> https://github.com/rvanlaar/csync2/blob/master/csync2.c#L276  I'm
> guessing that (on FreeBSD at least) it binds to an IPv6 socket and
> then exits the loop.

The intended (and, at least on linux, and according to various man
pages, elsewhere, also the actual) result is to bind to one socket that
accepts BOTH ipv4 and IPv6 adresses.

If your platform really does not have that (I don't know,
and I'm not going to check myself), please suggest a workaround.

Lars

___
Csync2 mailing list
Csync2@lists.linbit.com
http://lists.linbit.com/mailman/listinfo/csync2

Re: [Csync2] 2-node masters and root-owned files

[Lars Ellenberg]

On Mon, Aug 14, 2017 at 07:00:23AM +0400, Vadim Abdulayev wrote:
> Hello.

When replying to digests, please change the subject appropriately.
(Digest mode on a list that is so low volume as this one
is not useful anyways, really.)

> All command executed on node1.
> 
> After resync i receive wrong owner on node1.
> On node2, i always have
> correct owner and permission.
>
> node2 is backup node. No one working there.

csync2 is push only.

according to you, you execute all commands on node1.
node1 then is the source.
node2 is the target.

Still, according to you,
the ownership on the source changes.

Which means either your report is inconsistent,
or we are talking about a different "csync2",
or "something else is going on".

csync2 itself will never change ownership
(or anything else) on the source.
It will apply changes on the target,
to try to make the target look like the source.

Hope that helps you drilling down further.

Lars

___
Csync2 mailing list
Csync2@lists.linbit.com
http://lists.linbit.com/mailman/listinfo/csync2

Re: [Csync2] 2-node masters and root-owned files

[Lars Ellenberg]

On Tue, Aug 08, 2017 at 11:04:49AM +1000, Aristedes Maniatis wrote:
> On 8/8/17 10:06AM, Lars Ellenberg wrote:
> > but in this case, until proven otherwise,
> > I suspect you have some git command on the other node
> > running as root, and updating the index file there.
> 
> Hi Lars
> 
> I know you really need some detailed logs to understand the problem,
> however I can confirm anecdotally that it has happened for me
> recently. That is, random files owned by root instead of the correct
> user.

That's why I said "in this case" :-)

"similar" symptoms don't necessarily imply same root causes.

> For me, I have three nodes and syncing happening in all directions and
> use -B. I have probably 10,000 file operations per day and I've seen
> the problem happen maybe twice in four months. So logs are going to be
> very hard to get for you.

Your case sounds more like your csync2 connection is interrupted
at a very "inconvenient" time...

> Sorry I can't be of more direct help, but if I find a way to make it
> happen more frequently then I'll post here.

Thanks,

-- 
: Lars Ellenberg
: LINBIT | Keeping the Digital World Running
: DRBD -- Heartbeat -- Corosync -- Pacemaker
: R, Integration, Ops, Consulting, Support

DRBD® and LINBIT® are registered trademarks of LINBIT
___
Csync2 mailing list
Csync2@lists.linbit.com
http://lists.linbit.com/mailman/listinfo/csync2

Re: [Csync2] 2-node masters and root-owned files

[Lars Ellenberg]

On Mon, Aug 07, 2017 at 11:26:38PM +0400, Vadim Abdulayev wrote:
> Hello.
> 
> Thank you for help.
> Sometimes = one time per day or more often
> 
> example:
> 

would have been useful if you also included the node name in the command
line...

> #ll /home/myuser/.git/index
> -rw--- 1 root root 9955696 авг  7 18:03 /home/myuser/.git/index
> 
> #chown myuser:myuser /home/myuser/.git/index
> 
> …RESYNC...

From where to where?

> # ll /home/myuser/.git/index
> -rw--- 1 root root 9955696 авг  7 22:05 /home/myuser/.git/index

and still the "wrong" uid, even immediately after a "successful" update?

> LOG:

"check" log is boring.

"update" log may be interesting, 
especially on the *receiving* ("server") side
(the side where the "wrong" uid/gid is observed)

This seems to be the log of the "client" side.
which exact command produced this?
Still,

> File /home/myuser/.git/index is different on peer (cktxt char #18).
> >>> PEER:  v1:mtime=0:mode=33204:uid=1000:gid=1000:type=reg:size=9955696
> >>> LOCAL: v1:mtime=0:mode=33152:uid=1000:gid=1000:type=reg:size=9955696

differing permissions trigger the sync,

> SQL: DELETE FROM dirty WHERE filename = '/home/myuser/.git/index' AND 
> peername = 'node2.domain.local'
> SQL: SELECT command, logfile FROM action GROUP BY command, logfile
> SQL Query finished.
> Connection closed.
> Finished with 0 errors.

and to csync2 the sync looks as if it was ok.

debug level logs and maybe even strace from the "receiving" side would be 
interesting.

but in this case, until proven otherwise,
I suspect you have some git command on the other node
running as root, and updating the index file there.


-- 
: Lars Ellenberg
: LINBIT | Keeping the Digital World Running
: DRBD -- Heartbeat -- Corosync -- Pacemaker
: R, Integration, Ops, Consulting, Support

DRBD® and LINBIT® are registered trademarks of LINBIT
___
Csync2 mailing list
Csync2@lists.linbit.com
http://lists.linbit.com/mailman/listinfo/csync2

Re: [Csync2] 3-node masters and root-owned files

[Lars Ellenberg]

On Wed, Aug 02, 2017 at 08:58:05PM -0500, Kevin Cackler wrote:
> For those of you guys who are experiencing this issue: I noticed tonight,
> after this happened yet again, that one of my files was owned by UID 1012 on
> node3. The problem is that that uid does not exist on node3. That uid DOES
> exist on node1, and was the affected user. So when csync2 copied this file
> from node1 to node3, it copied the UID as well, and since that UID doesn't
> exist on node3, the user on node3 did not have permission to access the
> file.
> 
> In other words:
> 
> "user" on node1 has a UID of 1012
> "user" on node3 has a UID of 1007
> "user" updated a file on node1. When that file was copied to node3, the UID
> of the owner and group was 1012 instead of 1007. This meant that "user"
> could not access the file on node3.
> 
> I really hope this can be resolved soon, as it's causing a ton of grief for
> us.

csync2 replicates numerical uid/gid.
that has always been that way.
If you find someone to write a "by name" feature,
we are happy to take it.

when authenticating against /etc/passwd and friends,
you may want to put that into csync2 as well.
otherwise, you have "centralized" authentication anyways (ldap or
similar), and the id <-> name mapping is already covered.

Lars

___
Csync2 mailing list
Csync2@lists.linbit.com
http://lists.linbit.com/mailman/listinfo/csync2

Re: [Csync2] csync_check_del broken under PostgreSQL

[Lars Ellenberg]

On Sun, Oct 22, 2017 at 11:33:55AM +0100, James Le Cuirot wrote:
> When checking for deleted files recursively, csync2 issues a SQL
> statement like this.
> 
> => SELECT filename from file where filename > 
> '/mnt/csync2/portage/app-editors/ng/' and filename < 
> '/mnt/csync2/portage/app-editors/ng0' ORDER BY filename;
>  filename 
> --
> (0 rows)
> 
> This never returns anything under PostgreSQL. The behaviour is actually
> rather odd. Using LIKE gives the result we expect.
> 
> => SELECT filename FROM file WHERE filename LIKE 
> '/mnt/csync2/portage/app-editors/ng/%';
>filename   
> --
>  /mnt/csync2/portage/app-editors/ng/metadata.xml
>  /mnt/csync2/portage/app-editors/ng/files
>  /mnt/csync2/portage/app-editors/ng/files/ng-1.5beta1-ncurses.patch
>  /mnt/csync2/portage/app-editors/ng/files/ng-1.5beta1-configure.patch
>  /mnt/csync2/portage/app-editors/ng/Manifest
>  /mnt/csync2/portage/app-editors/ng/ng-1.5_beta1-r2.ebuild
> (6 rows)
> 
> But watch what happens when we change the 0 to a g.
> 
> => SELECT filename from file where filename > 
> '/mnt/csync2/portage/app-editors/ng/' and filename < 
> '/mnt/csync2/portage/app-editors/ngg' ORDER BY filename;
>filename   
> --
>  /mnt/csync2/portage/app-editors/ng/files
>  /mnt/csync2/portage/app-editors/ng/files/ng-1.5beta1-configure.patch
>  /mnt/csync2/portage/app-editors/ng/files/ng-1.5beta1-ncurses.patch
> (3 rows)
> 
> It's almost like the / in ng/files isn't used in the comparison. I
> couldn't find anything in the PostgreSQL documentation specifically
> about using the <> operators with strings.

What are your locale and collate settings?

afaik, = < > compare "collate character units", while LIKE, appart from
supporting wildcards, compares "byte per byte" when used with literals.
I may be wrong.

Anyways, maybe alter table ... binary ... helps?
Or change "TEXT" to "BLOB"?
Something like that?

I don't think many installations out there use csync2 with mysql or postgres,
so it is very likely that there are a number of shortcomings burried in there,
I'd expect "very long path names" to show problems as well, where "very
long" may be simply "longer than the default supported unique key length
on text fields.

-- 
: Lars Ellenberg
: LINBIT | Keeping the Digital World Running
: DRBD -- Heartbeat -- Corosync -- Pacemaker
: R, Integration, Ops, Consulting, Support

DRBD® and LINBIT® are registered trademarks of LINBIT
___
Csync2 mailing list
Csync2@lists.linbit.com
http://lists.linbit.com/mailman/listinfo/csync2

Re: [Csync2] csync_check_del broken under PostgreSQL

[Lars Ellenberg]

On Fri, Oct 27, 2017 at 11:00:14PM +0100, James Le Cuirot wrote:
> On Mon, 23 Oct 2017 13:23:51 +0200
> Lars Ellenberg <lars.ellenb...@linbit.com> wrote:
> 
> My locale and collation is entirely en_GB.UTF-8.

> > afaik, = < > compare "collate character units", while LIKE, appart from
> > supporting wildcards, compares "byte per byte" when used with literals.
> > I may be wrong.
> > 
> > Anyways, maybe alter table ... binary ... helps?
> > Or change "TEXT" to "BLOB"?
> > Something like that?
> 
> This gives the right result:
> 
> csync2=> SELECT filename from file where filename::bytea > 
> '/mnt/csync2/portage/app-editors/ng/' and filename::bytea < 
> '/mnt/csync2/portage/app-editors/ng0' ORDER BY filename;
> 
> Presumably this is comparing "byte for byte" as you say LIKE does but
> then why not just use LIKE in the first place?

"probably" (code is / was like that since over ten years)
at some point it was thought (or even measured) to help with
performance (presumably back when it was sqlite2 only).
Maybe that can be re-evaluated.

Also we probably should
CREATE TABLE ... ( ... TEXT COLLATE "C" NOT NULL ...)
Or use BYTEA / BLOB. Not sure which option is better.

You want to check what "select 'ABC' = 'abc';"
returns for various locales/collations,
and then double check for your favorite
umlauts or accented characters...

> > I don't think many installations out there use csync2 with mysql or 
> > postgres,
> > so it is very likely that there are a number of shortcomings burried in 
> > there,
> > I'd expect "very long path names" to show problems as well, where "very
> > long" may be simply "longer than the default supported unique key length
> > on text fields.
> 
> I figured that few people use this, especially since you can't specify
> the database in the configuration file.

Supposedly you just put something like this into the config file:
database "pgsql://james:secret@localhost/";
optionally with full database name.
Same string you'd put after the "-D" command line switch.

> I happen to have PostgreSQL running on both nodes anyway and thought
> it might be faster than SQLite.

If you are willing to do some benchmarks,
let us know the results.

In "normal usage" of csync2, I doubt that the database
would ever become the bottleneck.

> I gather the unique key length varies but is generally around 2000-3000
> characters. I don't expect to have paths that long but thanks for the
> heads up. Does SQLite not have such a limit? I couldn't find any
> reference to one.

No, sqlite apparently does not care for text field length there.

-- 
: Lars Ellenberg
: LINBIT | Keeping the Digital World Running
: DRBD -- Heartbeat -- Corosync -- Pacemaker
: R, Integration, Ops, Consulting, Support

DRBD® and LINBIT® are registered trademarks of LINBIT
___
Csync2 mailing list
Csync2@lists.linbit.com
http://lists.linbit.com/mailman/listinfo/csync2

Re: [Csync2] Always marking file as dirty

[Lars Ellenberg]

On Fri, Oct 27, 2017 at 12:29:59PM +0200, Gonzalo Aguilera - Diputación de 
Ávila wrote:
> Hi,
> 
> If there is two files in the same directory with similar name (just differ
> uppercase an lowcase) is always marked as dirty.
> 
> For example:
> 
> csync2 -xv
> 
> Marking file as dirty: /home/joe/hello.pdf
> Marking file as dirty: /home/joe/Hello.pdf
> 
> Updating  /home/joe/Hello.pdf on node01...
> File is already up to date on peer.


Which csync2 version,
which file system,
whic data base backend?

Maybe this is a locale/collation problem with the data base?

Tried to run with "LC_ALL=C"?
Or whatever else is necessary to tell your data base backend that
this is supposed to be a "binary", case sensitive, collation?

-- 
: Lars Ellenberg
: LINBIT | Keeping the Digital World Running
: DRBD -- Heartbeat -- Corosync -- Pacemaker
: R, Integration, Ops, Consulting, Support

DRBD® and LINBIT® are registered trademarks of LINBIT
___
Csync2 mailing list
Csync2@lists.linbit.com
http://lists.linbit.com/mailman/listinfo/csync2

Re: [Csync2] hot add/remove/maintenance host

[Lars Ellenberg]

On Wed, Dec 27, 2017 at 01:43:53PM -0800, E-Blokos wrote:
> On 12/25/2017 5:34 AM, Lars Ellenberg wrote:
> >On Sat, Dec 23, 2017 at 03:38:20AM -0800, E-Blokos wrote:
> > >
> > > is there any API/CLI command to put a host in maintenance/add/remove
> > > as hot plug?
> >
> > I'm not sure I understand what you mean by that.
> > But as every help text and man page says:
> > 
> >Csync2 will refuse to do anything
> >when a /etc/csync2/csync2.lock file is found.
> > 
> > So maybe "touch /etc/csync2/csync2.lock" is what you want?
> > 
> >  Lars

> Thanks for your answer.
> Maybe I didn't describe well.
> let's say there is a group of 4 nodes.
> for various reason the admin wants to just
> put one node in maintenance through the replication
> chain, temporary, so csync2 won't try to connect to it
> until the node is back in "normal" mode.
> Until now I had to remove the node on all csync2 files on all nodes
> to reproduce it. Also with this kind of option it will avoid csync2 to lock
> trying to connect and will avoid thousands email sent by crond too ;)

Nothing exactly like this exists.
Yes, you would need to remove it from the config file,
and later re-add (and re-integrate) it again.

Maybe you can have your cron (or other) jobs simply not do "sync to
everyone", but sync to specific peers instead, and then exclude the
"offline" peer from the list of peers, or list of cronjobs?

csync2 supports this:
csync2 -u -P only,these,peers

You could have one cronjob for each peer,
and disable/comment out the "offline" ones.

Or use a for loop

  SUPPOSEDLY_ONLINE_PEERS="some number of peers"
  for peer in $SUPPOSEDLY_ONLINE_PEERS; do csync2 -u -P $peer ; done

And change that variable as you see fit.


-- 
: Lars Ellenberg
: LINBIT | Keeping the Digital World Running
: DRBD -- Heartbeat -- Corosync -- Pacemaker
: R, Integration, Ops, Consulting, Support

DRBD® and LINBIT® are registered trademarks of LINBIT
___
Csync2 mailing list
Csync2@lists.linbit.com
http://lists.linbit.com/mailman/listinfo/csync2

Re: [Csync2] Issues with synced git clones

[Lars Ellenberg]

On Thu, Dec 21, 2017 at 02:43:58AM +0100, Dominik George wrote:
> What happends is hard to describe, but it looks like the git index gets
> broken. The effect is that all nodes in the cluster end up with a broken
> state, namely with a lot of untracked files in the repository.

Which should be fixable with a simple "git reset", as long as those git
repos are supposed to be identical after that sync.

> Some details that might be of interest:
> 
>  * In the synced directory tree, there are several clones of the same
>git repository
>  * Changes are made only on one node
>  * csync is configured to prefer the younger copy of a file

> > but let me first ask:
> > git is a distributed version control system,
> > so why are you not using *git* to distribute stuff?
> > as in git push/fetch/pull/remote update and so on?
> 
> Because git is only a conincident.
> 
> This syncs home directories across a cluster of PXE servers (over a
> relatively slow ADSL link, so distributed filesystems are not an
> option), and I have no control over what users happen to have inside
> their $HOME.

If in a "healthy" git checkout you do
git ls-files --debug, it shows you what git caches in that index file.
among other things that is the inode number.
Any (even local) copy (or "sync") will change the inode number.

Git uses the ctime,mtime, ownership, some more stat info,
and specifically the inode number only as a "hint" that
the contentent "might" have been changed.

Whenever git reads the "index" ("cache") file, and checks the "stat"
information of the tracked files, for those files where the stat
information indicates a possible content change,
it will check the actual content (re-hash the file),
and record the new stat info in the index.

Which means that even for "innocent" git commands like "status",
the index file will change, if any of the (relevant) stat information of
the tracked files has changed, even if their content and names have not.

If you do multi-directional sync, and the "newer" file wins,
depending on when a git command runs in one of the supposedly
"unchanging" nodes, that index file may still have "old" content,
(from the last sync) but will get a timestamp of now.

If you now sync that in the other direction, because it has the younger
time stamp, it will win over the "correct" index file, overwriting it
with a "stale" version of the file list and file hashes, doing something
equivalent to "git reset some-point-in-the-past" of your repo checkout.

When synchronizing file trees,
you need to use special rules for meta data files containing stat
information about themselves or other parts of the tree.

I suggest you exclude the git index files, and tell users to "git reset
HEAD" when they encounter "strange" behaviour.  Better yet, exclude git
directories, and (have them) use git to distribute those.

You could also hack git, and make it not use "now" for an updated git
index file, but explicitly set utime() of the most recent tracked or
cached file. That way, it would at least not present "stale" content
with "new" timestamp.

Lars

___
Csync2 mailing list
Csync2@lists.linbit.com
http://lists.linbit.com/mailman/listinfo/csync2

Re: [Csync2] hot add/remove/maintenance host

[Lars Ellenberg]

On Sat, Dec 23, 2017 at 03:38:20AM -0800, E-Blokos wrote:
> Hello,
> 
> is there any API/CLI command to put a host in maintenance/add/remove
> as hot plug?

I'm not sure I understand what you mean by that.
But as every help text and man page says:

  Csync2 will refuse to do anything
  when a /etc/csync2/csync2.lock file is found.

So maybe "touch /etc/csync2/csync2.lock" is what you want?

Lars

___
Csync2 mailing list
Csync2@lists.linbit.com
http://lists.linbit.com/mailman/listinfo/csync2

Re: [Csync2] Issues with synced git clones

[Lars Ellenberg]

On Thu, Dec 21, 2017 at 02:43:58AM +0100, Dominik George wrote:
> Hi,
> 
> we are having strange issues with syncing directories containing clones
> of git repositories. We were using unison before, and it suspiciously
> showed the same issues…
> 
> What happends is hard to describe, but it looks like the git index gets
> broken. The effect is that all nodes in the cluster end up with a broken
> state, namely with a lot of untracked files in the repository.
> 
> Some details that might be of interest:
> 
>  * In the synced directory tree, there are several clones of the same
>git repository
>  * Changes are made only on one node
>  * csync is configured to prefer the younger copy of a file
> 
> Any ideas on that?

I would have some ideas on git and git index as well, ...

but let me first ask:
git is a distributed version control system,
so why are you not using *git* to distribute stuff?
as in git push/fetch/pull/remote update and so on?

Cheers,

Lars

___
Csync2 mailing list
Csync2@lists.linbit.com
http://lists.linbit.com/mailman/listinfo/csync2

Re: [Csync2] Official page moved to github?

[Lars Ellenberg]

Yes,
we moved all public git repositories
that used to be hosted on git.linbit.com
over to github.com/LINBIT/
(and some more).

And dropped the ugly "index" of their tarballs
that used to be oss.linbit.com.

Lars

___
Csync2 mailing list
Csync2@lists.linbit.com
http://lists.linbit.com/mailman/listinfo/csync2

Re: [Csync2] Dynamic master ip...

[Lars Ellenberg]

On Mon, Mar 23, 2020 at 11:57:11AM +0100, Marco Gaiarin wrote:
> 
> In this time of Coronavirus, i've switched to 'smart working'.
> 
> I've setup a set of csync2 config file to propagate some sets of config
> files to a bunch of servers. Eg:
> 
> host hermi...@hermione.sv.lnf.it;
> 
> host (t...@tank.sv.lnf.it);
> host (invernom...@invernomuto.sv.lnf.it);
> host (l...@link.sv.lnf.it);
>   [...]
> 
> But now my IP change, or it is not anymore 'hermione.sv.lnf.it'.
> 
> 
> There's some way to have a 'floating master host'? And this can be
> achived NOT passing every 'slave' server to change local csync2 config
> file?

This is how csync2 "authenticates" a peer:
it does a "forward lookup" (getaddrinfo) of the name the peer claims to be,
and checks if the address as returned by getpeername() on the connected
socket is listed in the results (and if that name is listed in the config).

If that IP changes all the time, you probably have to use dynamic DNS.
Or use some VPN, inside of which you then have your own, "static" IPs.

You can try to add the "expected" (by you) info to /etc/hosts,
if that makes sense in your context.

Or, of course, change csync2 to not do this "forward lookup of claimed
name lists peer address" sanity check. Though I'm not convinced that
would be a good idea.

hth,

Lars

___
Csync2 mailing list
Csync2@lists.linbit.com
https://lists.linbit.com/mailman/listinfo/csync2

Re: [Csync2] How does Csync2 decide whether "Identification failed!" [Was: Again CSync2, VPN, strangeness...]

[Lars Ellenberg]

On Tue, Jun 23, 2020 at 01:07:54PM +0200, Marco Gaiarin wrote:
> 
> > What i'm missing? Thanks.
> 
> No, sorry, but there's something in the csync2 logic that realy i don't
> understand.
> 
> As stated in previous email, i was used to update some files via
> csync2, using my portable PC at work. So i had:
> 
>   host hermi...@hermione.sv.lnf.it;
> 
> switching to smartwork, i've simply added the second, VPN, address of
> my portable:
> 
>   host hermi...@hermione.sv.lnf.it;
>   host hermi...@gaio.vpn.sv.lnf.it;
> 
> and found that csync still work as expected.
> 
> 
> Now i'm back to work, and syncing with the 'old' address assigned does
> not work (eg, i can sync only from VPN address); if i comment out the
> VPN address, eg:
> 
>   host hermi...@hermione.sv.lnf.it;
>   #host hermi...@gaio.vpn.sv.lnf.it;
> 
> I can sync from the non-VPN with some server, but not others:
> 
>  Dirty item %wpkghome%/packages/vlc.xml gaio.vpn.sv.lnf.it 0
>  Local> HELLO gaio.vpn.sv.lnf.it\n
>  Peer> Identification failed!\n
>  While syncing file %wpkghome%/packages/vlc.xml:
>  response from peer(%wpkghome%/packages/vlc.xml): vdmpp1.pp.lnf.it [13] <- 
> Identification failed!
>  Local> BYE\n
>  Peer> OK (cu_later).\n
>  response from peer(): vdmpp1.pp.lnf.it [6] <- OK (cu_later).
> 
> 
> I'm really puzzled... please help me...

It all depends on the *forward* name resolution
from the view point of the server.

The server accepts a TCP connection,
calls getpeername() on that socket,
gets the client address.

Later, that client "identifies" with HELLO some.name.there.

The server uses a *forward* lookup on that,
it calls getaddrinfo(some.name.there, ...).

If one of the addresses returned by this getaddrinfo()
matches the client address found earlier with getpeername(),
csync2 is happy.  Otherwise, it is not.

Lars

___
Csync2 mailing list
Csync2@lists.linbit.com
https://lists.linbit.com/mailman/listinfo/csync2

Re: [Csync2] Query on Initializing the file table for csync2 database

[Lars Ellenberg]

On Wed, Feb 03, 2021 at 08:00:59AM +, Ganesh Jayaraman wrote:
> Hi Team,
> 
> This is Ganesh Jayaraman from Gigamon. We use csync2 for synchronising files 
> across group of hosts in a cluster.
> 
> Members to the group will be added sequentially and we need to do the sync 
> whenever we add a member to the cluster.
>
> 
> Note :
> 
>   1.  Cluster will have one master and rest of them will act as standby hosts
>   2.  CSync2 configurations are done as per the guidance given in the 
> doc
>Before starting the first sync (ie. To the first host/peer that
>is getting added to the cluster) we would initialize the
>database on the master node (from where we want to sync to
>peers) using the command csync2 -cIr -N  which
>is in-turn creating the DB and tables

In the doc, as "bootstrappign large setups".
Could do with more obvious examples maybe, but it is all there.

initial setup:
  all-nodes: csync2 -cIr /
master-node: csync2 -TIXF ; csync2 -uv

adding a new node:
   new-node# csync2 -cIr /
master-node# csync2 -TUXIF master-node new-node
master-node# csync2 -uv -P new-node


> Whenever a new member is added to the group we tried to sync the files using 
> the command
> # csync2 -N  -rx -P 

You should not need to use "-N" at all.
That was introduced to be able to _fake_ a node name,
e.g. while trying to do simple test runs with only one node.


Lars

___
Csync2 mailing list
Csync2@lists.linbit.com
https://lists.linbit.com/mailman/listinfo/csync2