I made script which will ensure that translated MD files from Crowdin will have correct license headers.
Please review: https://github.com/apache/cordova-docs/pull/264 Usage: from cordova-docs repository root directory invoke ./bin/fixyaml Currently this script only working over cordova-docs repository, not verifying plugin repositories. If I download latest versions seems, that only one issue with license headers we have: docs/<lang>/guide/next/index.md has missing license header. That's because to Crowdin was uploaded file without license header. 2015-02-04 10:11 GMT+06:00 Victor Sosa <sosah.vic...@gmail.com>: > Hello community. > > So that everybody is aware about the proceedings on this topic. > > Andrey and I just had a quick call about this topic and here the agreement > > We want to still continue using Crowdin as our tool for automatic > translation, but we do acknowledge the problems depicted on this thread > (autolinking and formatting). In order to fix these problems, Andrey worked > on a couple of validation scripts that we can integrate in the translation > process, but for this to happen we need to do the following (Andrey, please > keep me honest): > > 1. I'll reach out the Crowdin folks to see whether they have any API to > put translated documents back in CrowdIn. This way we can run the > validation scripts locally and then put the fixed docs back and run the > usual Crowdin build. If no API is available then try to find a workaround > to this problem (as of now, we couldn't find a way to do it). > > 2. As a low priority task for now, I'll refactor the current translation > script (shell script) to something more cross-platform (ideas about the > best solution/technology are welcome) > > 3. Andrey will continue his work making sure the output of the validation > scripts is as expected. > > Hope I didn't miss anything from our chat. > On Feb 3, 2015 8:32 PM, "Victor Sosa" <sosah.vic...@gmail.com> wrote: > >> Hi Andrey >> >> Yes, in about an hour as agreed >> On Feb 3, 2015 8:22 PM, "Andrey Kurdumov" <kant2...@googlemail.com> >> wrote: >> >>> Hi, >>> >>> will you able chat today 9:30 PM CST as we plan earlier? >>> >>> 2015-01-30 8:47 GMT+06:00 Victor Sosa <sosah.vic...@gmail.com>: >>> >>>> Cool! So I'll book it on my calendar, Tuesday 9:30pm. That's Your >>>> Wednesday 9:30am. Talk to you then. >>>> >>>> Have a great weekend! >>>> On Jan 29, 2015 8:39 PM, "Andrey Kurdumov" <kant2...@googlemail.com> >>>> wrote: >>>> >>>>> I'm fine for that time. >>>>> >>>>> 2015-01-30 8:37 GMT+06:00 Victor Sosa <sosah.vic...@gmail.com>: >>>>> >>>>>> So let's schedule a hangout next Wednesday in the morning, which is >>>>>> my Tuesday night. If I recall correctly we are 12 hours apart from each >>>>>> other (I'm in GMT -6). How about 9:30 AM for you? >>>>>> On Jan 29, 2015 7:52 PM, "Andrey Kurdumov" <kant2...@googlemail.com> >>>>>> wrote: >>>>>> >>>>>>> If this is working for you, we could chat. >>>>>>> Morning is start for me, hopefully it is not too late for you. I >>>>>>> could chat in next 4 hours or more without problem. >>>>>>> >>>>>>> 2015-01-30 6:27 GMT+06:00 Victor Sosa <sosah.vic...@gmail.com>: >>>>>>> >>>>>>>> Hi Andrey. Thanks for the reminder. >>>>>>>> >>>>>>>> In the process in step 4 you are correct. This "manual" step is >>>>>>>> actually going to CrowdIn and launch the build from there. >>>>>>>> >>>>>>>> I agree that after 5, we need to establish some kind of mechanism >>>>>>>> to validate the two scenarios you state here (formatting errors and >>>>>>>> autolinking). I presume this translationreport script is the one you >>>>>>>> created and will be useful to validate the autolinking problems. And >>>>>>>> as you >>>>>>>> state, it is quite difficult to maintain each language and therefore we >>>>>>>> have to live with the problems we have for the time being :S >>>>>>>> >>>>>>>> Let me tell you this, why don't you and schedule a hangout to speed >>>>>>>> up this conversation and convey an agreement? I'm fine if you want to >>>>>>>> book >>>>>>>> the hangout in your morning, which is my night ;-), just let me know >>>>>>>> your >>>>>>>> availability (we can have a quick chat on gmail privately to not spam >>>>>>>> the >>>>>>>> list) >>>>>>>> >>>>>>>> 2015-01-29 9:08 GMT-06:00 Andrey Kurdumov <kant2...@googlemail.com> >>>>>>>> : >>>>>>>> >>>>>>>> Hi, want to bump this thread. Do you have any thoughts how I could >>>>>>>>> fix current documentation issues without interfering with you? >>>>>>>>> >>>>>>>>> 2015-01-23 13:46 GMT+06:00 Andrey Kurdumov < >>>>>>>>> kant2...@googlemail.com>: >>>>>>>>> >>>>>>>>>> Thanks for explanation. The process is more like i think. >>>>>>>>>> One thing about which I'm not sure is #4. I assume that this is >>>>>>>>>> some step in CrowdIn which generated ZIP file from the latest >>>>>>>>>> translations. >>>>>>>>>> Correct me if I wrong. >>>>>>>>>> >>>>>>>>>> a) My feeling how I would like to work: >>>>>>>>>> 1. The script downloads the latest updates to your local Got repo. >>>>>>>>>> 2. Adds the docs in English to CrowdIn. >>>>>>>>>> 3. Manually start the translation via a machine in CrowdIn >>>>>>>>>> 4. Once translated, manually run a build of the docs. >>>>>>>>>> 5. Using the script, download the changes in documentation to >>>>>>>>>> your local repo. >>>>>>>>>> >>>>>>>>>> X. Run validation scripts. >>>>>>>>>> X.a. In case of any translation error (autolinking for example) >>>>>>>>>> go to CrowdIn and start from step #3. >>>>>>>>>> X.b In case of some minor formatting errors, correct them >>>>>>>>>> manually and continue to step #6. >>>>>>>>>> >>>>>>>>>> 6. Commit changes to Git. >>>>>>>>>> >>>>>>>>>> I just want to add validation step which will not allow to commit >>>>>>>>>> specifically for Russian language, since this is only language which >>>>>>>>>> I >>>>>>>>>> could support. >>>>>>>>>> Other languages I don't plan to support, and from my perspective >>>>>>>>>> they are broken. >>>>>>>>>> >>>>>>>>>> Here the small Node.js utility which could help you test broken >>>>>>>>>> auto-linking on any language you care. >>>>>>>>>> Run >>>>>>>>>> ./bin/translationreport ru edge -v >>>>>>>>>> from the root folder of [cordova-docs] repository. >>>>>>>>>> >>>>>>>>>> I will never go to step #6 until I fix every errors which I found >>>>>>>>>> on step X. This is how I feel it should be. But I open for >>>>>>>>>> alternatives >>>>>>>>>> which allows me improve quality of Russian translation, >>>>>>>>>> >>>>>>>>>> Regarding make validation step possible in CrowdIn. So far, I >>>>>>>>>> don't see any method which allows verify of translation without >>>>>>>>>> downloading >>>>>>>>>> it first from CrowdIn (see https://crowdin.com/page/api). >>>>>>>>>> >>>>>>>>>> I really try make good translation using just CrowdIn, but don't >>>>>>>>>> see any opportunity to fix broken MD files inside CrowdIn. >>>>>>>>>> If you have any ideas how I could fix errors in the links I show >>>>>>>>>> you using just CrowdIn, I would be all-in for that. >>>>>>>>>> >>>>>>>>>> Best regards, >>>>>>>>>> Andrey >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> 2015-01-22 8:56 GMT+06:00 Victor Sosa <sosah.vic...@gmail.com>: >>>>>>>>>> >>>>>>>>>>> Hi Andrey, sorry my late replay. Here my 2 cents. >>>>>>>>>>> >>>>>>>>>>> This is what the current translation process does: >>>>>>>>>>> >>>>>>>>>>> 1. The script downloads the latest updates to your local Got >>>>>>>>>>> repo. >>>>>>>>>>> 2. Adds the docs in English to CrowdIn. >>>>>>>>>>> 3. Manually start the translation via a machine in CrowdIn >>>>>>>>>>> 4. Once translated, manually run a build of the docs. >>>>>>>>>>> 5. Using the script, download the changes in documentation to >>>>>>>>>>> your local repo. >>>>>>>>>>> 6. Commit changes to Git. >>>>>>>>>>> >>>>>>>>>>> If a translator modifies a doc at some point in CrowdIn, I'll >>>>>>>>>>> take the changes in the next build. >>>>>>>>>>> >>>>>>>>>>> As Lisa stated, any changes made in the translated docs outside >>>>>>>>>>> of CrowdIn will end up ignored in the next build. >>>>>>>>>>> >>>>>>>>>>> If I understood correctly, you propose to validate the generated >>>>>>>>>>> docs using the tool you created. I wonder if we can add the >>>>>>>>>>> validation >>>>>>>>>>> process to modify the docs in CrowdIn instead of directly doing it >>>>>>>>>>> in Git. >>>>>>>>>>> >>>>>>>>>>> Correct me if wrong, but I think this is the way to go in order >>>>>>>>>>> not just to keep the current process but to enhance it. Let me know >>>>>>>>>>> what >>>>>>>>>>> you think. >>>>>>>>>>> >>>>>>>>>>> P.S. I'm in Central Timezone, which means I'm 12 hours away from >>>>>>>>>>> you. Communication is an exciting topic here :-D >>>>>>>>>>> On Jan 20, 2015 9:24 PM, "Andrey Kurdumov" < >>>>>>>>>>> kant2...@googlemail.com> wrote: >>>>>>>>>>> >>>>>>>>>>>> My main concern is that current translation not good. >>>>>>>>>>>> Proof: http://cordova.apache.org/docs/es/edge/index.html or >>>>>>>>>>>> http://cordova.apache.org/docs/ru/edge/index.html >>>>>>>>>>>> >>>>>>>>>>>> These links contain two major issues with translated docs. >>>>>>>>>>>> a) License headers >>>>>>>>>>>> b) Missing links which could lead to missing parts of docs. >>>>>>>>>>>> other issues are also present which also introduced by our >>>>>>>>>>>> tools. >>>>>>>>>>>> >>>>>>>>>>>> ---------------------------------------------------- >>>>>>>>>>>> First kind of errors was introduced as of Cordova 3.4 or like >>>>>>>>>>>> that. >>>>>>>>>>>> There no way fix that using CrowdIn, so I fix that in the Git >>>>>>>>>>>> repo byt sending pull request >>>>>>>>>>>> https://github.com/apache/cordova-docs/pull/250 >>>>>>>>>>>> >>>>>>>>>>>> Example of changes >>>>>>>>>>>> >>>>>>>>>>>> https://github.com/apache/cordova-docs/commit/691f0f016ef4932e9cd224f49bb9641161b726e1 >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> ---------------------------------------------------- >>>>>>>>>>>> Second kind of errors (missing links) could be verified by >>>>>>>>>>>> running new tool >>>>>>>>>>>> >>>>>>>>>>>> ./bin/translationreport ru edge -v >>>>>>>>>>>> or >>>>>>>>>>>> ./bin/translationreport ru edge >>>>>>>>>>>> for more details. >>>>>>>>>>>> >>>>>>>>>>>> ---------------------------------------------------- >>>>>>>>>>>> >>>>>>>>>>>> Really I don't trust output of CrowdIn due to way we generate >>>>>>>>>>>> documentation. It is very easy translate sentence and broke our >>>>>>>>>>>> autolinking in the documentation. Current process is not enough >>>>>>>>>>>> for me, since before commit changes to Git for Russian translation >>>>>>>>>>>> I want to manually generate Russian documentation and verify >>>>>>>>>>>> quality of produced output using automated-tools and by manual >>>>>>>>>>>> spot-checking. >>>>>>>>>>>> If any of tools report about problem with translation I would >>>>>>>>>>>> not commit any changes until I resolve issue. If that mean that >>>>>>>>>>>> I have to manually insert blank lines after pulling changes >>>>>>>>>>>> from CrowdIn, I will do that. >>>>>>>>>>>> >>>>>>>>>>>> I have doubts that this is practical follow this approach for >>>>>>>>>>>> other languages, that's why separation. If you are ready to >>>>>>>>>>>> go with that approach for all other languages, I would be more >>>>>>>>>>>> then happy not to separate Russian. I want find solution/process >>>>>>>>>>>> where all >>>>>>>>>>>> we would work with same set of tools, plus I will use >>>>>>>>>>>> additional tools to raise quality bar for Russian. >>>>>>>>>>>> >>>>>>>>>>>> Want to notice that I don't want to dump CrowdIn from the >>>>>>>>>>>> process, it is very good to for translation, >>>>>>>>>>>> but not for the verification of the translated results. Reason >>>>>>>>>>>> that I made pull request on GitHub directly, >>>>>>>>>>>> it is because CrowdIn does not allow do what I need. >>>>>>>>>>>> >>>>>>>>>>>> Hopefully this is clears picture from my side. >>>>>>>>>>>> >>>>>>>>>>>> BTW: another problem is that I GMT +6 which is 11 hours >>>>>>>>>>>> different from East side of US and 14 from West side of US. >>>>>>>>>>>> Add some communication problems. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> 2015-01-20 22:43 GMT+06:00 Lisa Seacat DeLuca < >>>>>>>>>>>> ldel...@us.ibm.com>: >>>>>>>>>>>> >>>>>>>>>>>>> Andrey~ >>>>>>>>>>>>> >>>>>>>>>>>>> I'm a little confused by your request. Are you saying that >>>>>>>>>>>>> you make changes to the translation outside of the Crowdin tool? >>>>>>>>>>>>> If that's >>>>>>>>>>>>> the case, yes, all of your changes are going to be lost every >>>>>>>>>>>>> time. You'll >>>>>>>>>>>>> need to make the changes within Crowdin. When the script is ran >>>>>>>>>>>>> to make >>>>>>>>>>>>> sure the latest documentation is pushed to the Crowdin tool it >>>>>>>>>>>>> doesn't >>>>>>>>>>>>> override any translations that have already been done. It just >>>>>>>>>>>>> adds >>>>>>>>>>>>> updates to the documentation (new sentences, new code, etc) or >>>>>>>>>>>>> changes that >>>>>>>>>>>>> were made. So unless sections of the english version of the >>>>>>>>>>>>> documentation >>>>>>>>>>>>> has been changes your Russian updates shouldn't be affected. I >>>>>>>>>>>>> see no >>>>>>>>>>>>> reason why you couldn't own the Russian part of the translation >>>>>>>>>>>>> process. >>>>>>>>>>>>> Whenever Victor goes to pull down the translations if the Russian >>>>>>>>>>>>> translations aren't at 100% he can talk to you. Does that sound >>>>>>>>>>>>> reasonable? Did I understand your concern correctly? >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> Lisa >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> [image: Inactive hide details for Andrey Kurdumov >>>>>>>>>>>>> ---01/20/2015 01:22:53 AM---Hi, I start working on the improving >>>>>>>>>>>>> Cordova >>>>>>>>>>>>> documentatio]Andrey Kurdumov ---01/20/2015 01:22:53 AM---Hi, >>>>>>>>>>>>> I start working on the improving Cordova documentation for >>>>>>>>>>>>> Russian language. >>>>>>>>>>>>> >>>>>>>>>>>>> From: Andrey Kurdumov <kant2...@googlemail.com> >>>>>>>>>>>>> To: dev@cordova.apache.org, Victor Adrian Sosa Herrera < >>>>>>>>>>>>> sosah.vic...@gmail.com> >>>>>>>>>>>>> Date: 01/20/2015 01:22 AM >>>>>>>>>>>>> Subject: Crowdin and translation process >>>>>>>>>>>>> ------------------------------ >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> Hi, >>>>>>>>>>>>> >>>>>>>>>>>>> I start working on the improving Cordova documentation for >>>>>>>>>>>>> Russian language. >>>>>>>>>>>>> Earlier I put fix for the YAML license headers directly to >>>>>>>>>>>>> Github for all >>>>>>>>>>>>> translations. >>>>>>>>>>>>> >>>>>>>>>>>>> Right now after I put documentation translation checker tool, >>>>>>>>>>>>> I see issues >>>>>>>>>>>>> which I would like to improve Russian translation. >>>>>>>>>>>>> >>>>>>>>>>>>> I download latest translation from Crowdin and compare that >>>>>>>>>>>>> field with docs >>>>>>>>>>>>> in the source control. What I found is that YAML fixes is not >>>>>>>>>>>>> in CrowdIn. >>>>>>>>>>>>> >>>>>>>>>>>>> 1. This raises question how we could could work on translation >>>>>>>>>>>>> with Victor >>>>>>>>>>>>> together effectively, not impacting each other? >>>>>>>>>>>>> When documentation files is uploaded to the Crowdin? >>>>>>>>>>>>> When translation is collected from Crowdin? >>>>>>>>>>>>> Are there any rules which could be used, so I could prepare to >>>>>>>>>>>>> these cycles? >>>>>>>>>>>>> >>>>>>>>>>>>> 2. Current script just download everything and commit it in the >>>>>>>>>>>>> cordova-docs repository. >>>>>>>>>>>>> What I think is that it could be effective is that I take >>>>>>>>>>>>> responsibility >>>>>>>>>>>>> for maintaining Russian translation and Victor would maintain >>>>>>>>>>>>> other >>>>>>>>>>>>> translations, like it was before. And downloading script could >>>>>>>>>>>>> be modified >>>>>>>>>>>>> to include language which should be downloaded, so this allow >>>>>>>>>>>>> explicit >>>>>>>>>>>>> control which languages is created. I recently cleanup a lot >>>>>>>>>>>>> of files from >>>>>>>>>>>>> probably Crowdin mistakes, and don't want that this would be >>>>>>>>>>>>> introduced >>>>>>>>>>>>> again. >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> Victor Adrian Sosa Herrera >>>>>>>> IBM Software Engineer >>>>>>>> Guadalajara, Jalisco >>>>>>>> >>>>>>> >>>>>>> >>>>> >>>
