Thanks Enrico. I wrote a test case that reproduces the issue, and opened http://tracker.ceph.com/issues/21772 to track the bug. It sounds like this is a regression in luminous.

On 10/11/2017 06:41 PM, Enrico Kern wrote:
or this:

   {
        "shard_id": 22,
        "entries": [
            {
                "id": "1_1507761448.758184_10459.1",
                "section": "data",
"name": "testbucket:6a9448d2-bdba-4bec-aad6-aba72cd8eac6.21344646.3/Wireshark-win64-2.2.7.exe",
                "timestamp": "2017-10-11 22:37:28.758184Z",
                "info": {
                    "source_zone": "6a9448d2-bdba-4bec-aad6-aba72cd8eac6",
                    "error_code": 5,
                    "message": "failed to sync object"
                }
            }
        ]
    },


<http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> Virenfrei. www.avg.com <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail>


On Thu, Oct 12, 2017 at 12:39 AM, Enrico Kern <[email protected] <mailto:[email protected]>> wrote:

    its 45MB, but it happens with all multipart uploads.

    sync error list shows

       {
            "shard_id": 31,
            "entries": [
                {
                    "id": "1_1507761459.607008_8197.1",
                    "section": "data",
                    "name":
    "testbucket:6a9448d2-bdba-4bec-aad6-aba72cd8eac6.21344646.3",
                    "timestamp": "2017-10-11 22:37:39.607008Z",
                    "info": {
                        "source_zone":
    "6a9448d2-bdba-4bec-aad6-aba72cd8eac6",
                        "error_code": 5,
                        "message": "failed to sync bucket instance:
    (5) Input/output error"
                    }
                }
            ]
        }

    for multiple shards not just this one



    On Thu, Oct 12, 2017 at 12:31 AM, Yehuda Sadeh-Weinraub
    <[email protected] <mailto:[email protected]>> wrote:

        What is the size of the object? Is it only this one?

        Try this command: 'radosgw-admin sync error list'. Does it
        show anything related to that object?

        Thanks,
        Yehuda


        On Wed, Oct 11, 2017 at 3:26 PM, Enrico Kern
        <[email protected]
        <mailto:[email protected]>> wrote:

            if i change permissions the sync status shows that it is
            syncing 1 shard, but no files ends up in the pool (testing
            with empty data pool). after a while it shows that data is
            back in sync but there is no file

            On Wed, Oct 11, 2017 at 11:26 PM, Yehuda Sadeh-Weinraub
            <[email protected] <mailto:[email protected]>> wrote:

                Thanks for your report. We're looking into it. You can
                try to see if touching the object (e.g., modifying its
                permissions) triggers the sync.

                Yehuda

                On Wed, Oct 11, 2017 at 1:36 PM, Enrico Kern
                <[email protected]
                <mailto:[email protected]>> wrote:

                    Hi David,

                    yeah seems you are right, they are stored as
                    different filenames in the data bucket when using
                    multisite upload. But anyway it stil doesnt get
                    replicated. As example i have files like

                    
6a9448d2-bdba-4bec-aad6-aba72cd8eac6.21344646.1__multipart_Wireshark-win64-2.2.7.exe.2~0LAfq93OMdk7hrijvyzW_EBRkVQLX37.6

                    in the data pool on one zone. But its not
                    replicated to the other zone. naming is not
                    relevant, the other data bucket doesnt have any
                    file multipart or not.

                    im really missing the file on the other zone.

                    
<http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail>
                        Virenfrei. www.avg.com
                    
<http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail>



                    On Wed, Oct 11, 2017 at 10:25 PM, David Turner
                    <[email protected]
                    <mailto:[email protected]>> wrote:

                        Multipart is a client side setting when
                        uploading. Multisite in and of itself is a
                        client and it doesn't use multipart (at least
                        not by default). I have a Jewel RGW Multisite
                        cluster and one site has the object as
                        multi-part while the second site just has it
                        as a single object.  I had to change from
                        looking at the objects in the pool for
                        monitoring to looking at an ls of the buckets
                        to see if they were in sync.

                        I don't know if multisite has the option to
                        match if an object is multipart between sites,
                        but it definitely doesn't seem to be the
                        default behavior.

                        On Wed, Oct 11, 2017 at 3:56 PM Enrico Kern
                        <[email protected]
                        <mailto:[email protected]>> wrote:

                            Hi all,

                            i just setup multisite replication
                            according to the docs from
                            http://docs.ceph.com/docs/master/radosgw/multisite/
                            
<http://docs.ceph.com/docs/master/radosgw/multisite/>
                            and everything works except that if a
                            client uploads via multipart the files
                            dont get replicated.

                            If i in one zone rename a file that was
                            uploaded via multipart it gets replicated,
                            but not if i left it untouched. Any ideas
                            why? I remember there was a similar bug
                            with jewel a while back.

                            On the slave node i also permanently get
                            this error (unrelated to the replication)
                            in the radosgw log:

                             meta sync: ERROR: failed to read mdlog
                            info with (2) No such file or directory

                            we didnt run radosgw before the luminous
                            upgrade of our clusters.

                            after a finished multipart upload which is
                            only visible at one zone "radosgw-admin
                            sync status" just shows that metadata and
                            data is caught up with the source.



--
                            Enrico Kern
                            *Lead System Engineer*

                            *T*+49 (0) 30 555713017
                            <tel:+49%2030%20555713017>  | *M *+49
                            (0)152 26814501 <tel:+49%201522%206814501>
                            *E*[email protected]
                            <mailto:[email protected]>| *Skype
                            flyersa*| LinkedIn View my Profile
                            <https://www.linkedin.com/in/enricokern/>**
                            *
                            *
                            ***
                            *

                            *Glispa GmbH* - Berlin Office
                            Sonnenburger Str. 73 10437 Berlin, Germany
                            
<https://maps.google.com/?q=Sonnenburger+Str.+73+10437+Berlin,+Germany&entry=gmail&source=g>

                            Managing Director: David Brown, Registered
                            in Berlin, AG Charlottenburg HRB 114678B
                            
<http://www.glispa.com/><mailto:[email protected]><https://www.linkedin.com/company-beta/143634/><https://plus.google.com/u/0/b/116135915389937318808/116135915389937318808><https://twitter.com/glispa><https://www.facebook.com/glispamedia/><https://www.instagram.com/glispaglobalgroup/>

                            
<http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail>
                                Virenfrei. www.avg.com
                            
<http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail>


                            _______________________________________________
                            ceph-users mailing list
                            [email protected]
                            <mailto:[email protected]>
                            
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
                            
<http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com>




--
                    Enrico Kern
                    *Lead System Engineer*

                    *T*+49 (0) 30 555713017
                    <tel:+49%2030%20555713017>  | *M *+49 (0)152
                    26814501 <tel:+49%201522%206814501>
                    *E*[email protected]
                    <mailto:[email protected]>| *Skype flyersa*|
                    LinkedIn View my Profile
                    <https://www.linkedin.com/in/enricokern/>**
                    *
                    *
                    ***
                    *

                    *Glispa GmbH* - Berlin Office
                    Sonnenburger Str. 73 10437 Berlin, Germany
                    
<https://maps.google.com/?q=Sonnenburger+Str.+73+10437+Berlin,+Germany&entry=gmail&source=g>

                    Managing Director: Dina Karol-Gavish, Registered
                    in Berlin, AG Charlottenburg HRB 114678B
                    
<http://www.glispa.com/><mailto:[email protected]><https://www.linkedin.com/company-beta/143634/><https://plus.google.com/u/0/b/116135915389937318808/116135915389937318808><https://twitter.com/glispa><https://www.facebook.com/glispamedia/><https://www.instagram.com/glispaglobalgroup/>

                    _______________________________________________
                    ceph-users mailing list
                    [email protected]
                    <mailto:[email protected]>
                    http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
                    <http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com>





--
            Enrico Kern
            *Lead System Engineer*

            *T*+49 (0) 30 555713017 <tel:+49%2030%20555713017>  | *M
            *+49 (0)152 26814501 <tel:+49%201522%206814501>
            *E*[email protected] <mailto:[email protected]>|
            *Skype flyersa*| LinkedIn View my Profile
            <https://www.linkedin.com/in/enricokern/>**
            *
            *
            ***
            *

            *Glispa GmbH* - Berlin Office
            Sonnenburger Str. 73 10437 Berlin, Germany
            
<https://maps.google.com/?q=Sonnenburger+Str.+73+10437+Berlin,+Germany&entry=gmail&source=g>

            Managing Director: Dina Karol-Gavish, Registered in
            Berlin, AG Charlottenburg HRB 114678B
            
<http://www.glispa.com/><mailto:[email protected]><https://www.linkedin.com/company-beta/143634/><https://plus.google.com/u/0/b/116135915389937318808/116135915389937318808><https://twitter.com/glispa><https://www.facebook.com/glispamedia/><https://www.instagram.com/glispaglobalgroup/>





--
    Enrico Kern
    *Lead System Engineer*

    *T*+49 (0) 30 555713017 <tel:+49%2030%20555713017>  | *M *+49
    (0)152 26814501 <tel:+49%201522%206814501>
    *E*[email protected] <mailto:[email protected]>| *Skype
    flyersa*| LinkedIn View my Profile
    <https://www.linkedin.com/in/enricokern/>**
    *
    *
    ***
    *

    *Glispa GmbH* - Berlin Office
    Sonnenburger Str. 73 10437 Berlin, Germany
    Managing Director: Dina Karol-Gavish, Registered in Berlin, AG
    Charlottenburg HRB 114678B
    
<http://www.glispa.com/><mailto:[email protected]><https://www.linkedin.com/company-beta/143634/><https://plus.google.com/u/0/b/116135915389937318808/116135915389937318808><https://twitter.com/glispa><https://www.facebook.com/glispamedia/><https://www.instagram.com/glispaglobalgroup/>




--

Enrico Kern
*Lead System Engineer*

*T*+49 (0) 30 555713017  | *M *+49 (0)152 26814501
*E*[email protected] <mailto:[email protected]>| *Skype flyersa*| LinkedIn View my Profile <https://www.linkedin.com/in/enricokern/>**
*
*
***
*

*Glispa GmbH* - Berlin Office
Sonnenburger Str. 73 10437 Berlin, Germany
Managing Director: Dina Karol-Gavish, Registered in Berlin, AG Charlottenburg HRB 114678B
<http://www.glispa.com/><mailto:[email protected]><https://www.linkedin.com/company-beta/143634/><https://plus.google.com/u/0/b/116135915389937318808/116135915389937318808><https://twitter.com/glispa><https://www.facebook.com/glispamedia/><https://www.instagram.com/glispaglobalgroup/>


_______________________________________________
ceph-users mailing list
[email protected]
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

_______________________________________________
ceph-users mailing list
[email protected]
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Reply via email to