Re: [Ledger-smb-devel] Cleaning up the repository. How does the community feel about it?

[David G]

  
  
Hi John,

Thanks for that writeup. 
It is good to get some more background on the issue.

I also find it enlightening that the Drupal and Wordpress packages
(debian) use their own internal versions of Dojo.
This suggests that my assertion that we are not including the dojo
library, but a customized version of dojo, is acceptable within the
debian packaging policy.

Regards
David G

On 03/01/16 13:25, John Locke wrote:


  
  Hi,
  
  On 01/01/2016 09:57 AM, Erik
Huelsmann wrote:
  
  

  
On Fri, Jan 1, 2016 at 6:45 PM,
  Robert James Clay 
  wrote:
  On Thursday, December 31, 2015 09:50:59 AM
  David G wrote:
  >
  > It is worth pointing out that it is undesirable
  to use an OS package to
  > provide dojo as there are known to be issues from
  minor release to minor
  > release that would break LedgerSMB.
  
  What are those issues and how do they break
LedgerSMB? That is;  how is that
being tested and what does that find?  And why are those
minor releases"  being
used, if they cause such issues?
  
  John says he has experience that there are slight
incompatibilities between versions 1.7, 1.8, ...  and
that very careful testing is required to be sure your
code remains working. Assuming that's true, I can be
nothing but extremely disappointed about the development
processes used.
  
  One of the questions that come to mind though is: how
do they keep their own stuff working if the various
versions break expectations? It seems some widgets
haven't been changed since the early days of Dojo.
Surely those would be broken now if this goes on too
long?
  
   
  

  

  
  So I actually have not even used 1.9 or 1.10, my experience is
  based on moving from 1.2 to 1.3, to 1.5, to 1.7... every time we
  upgraded our single-page app, stuff broke and needed to get
  rewritten.
  
  The stuff that broke were largely custom widgets we wrote, using
  underlying Dojo classes for widgets and templates -- they kept
  refactoring these and changing the API in subtle ways that broke
  our custom code.
  
  For LSMB this might not be much of an issue -- yet. At least not
  before LSMB 1.5. But 1.5 relies upon Dojo far more than any
  previous version (is it even in 1.3? And for 1.4, it's only used
  for simple form elements and some layout stuff in the Contacts
  area -- I don't think we've built any actual custom complex
  widgets).
  
  It's certainly possible that things have shaken out and gotten
  more stable -- as it is the jump from Dojo 1.5 to 1.7 (which
  introduced AMD) was so great we still have not completely upgraded
  our internal app, and are still using 1.5.
  
  And then also realize that there's no server-side testing that
  would reveal a problem with Dojo -- it's purely client-side. So
  testing it properly is largely visual regression testing, stuff on
  top of Selenium/PhantomJS or similar, or using writing tests for
  Dojo's own test runner, DOH (if that's still around... looks like
  they now promote "intern" which looks interesting...)
  
  Dojo does have its own test suite. But the problem I've
  experienced with Dojo upgrades is in its base widget classes
  (dijit/_Templated, dijit/_Widget, etc) which have changed far more
  than you would think should be acceptable within a minor release.
  If we're not extending those, then we're not affected -- yet. But
  the ability to extend those to have smart widgets is the main
  reason I was advocating for Dojo in the first place, over simpler
  widget systems like jquery.ui and the like.
  
  And as previously noted, I have not experienced any issues with
  updating a patch release within the same minor version (e.g. 1.5.1
  to 1.5.3), only between the minor versions (1.5 to 1.7). And Dojo
  itself is still releasing patch releases for 1.5 -- if they
  expected everyone could upgrade easily, they would not still be
  issuing patches for 1.4
  
  
  Remember that JS libraries don't need to only consider what else
  is on the Linux server -- it's something that needs to support all
  major browsers on all operating systems. Microsoft Edge, IE11,
  IE10, Safari, Chrome, Firefox on Windows, Mac, 

Re: [Ledger-smb-devel] Cleaning up the repository. How does the community feel about it?

[John Locke]

Hi,

On 01/01/2016 09:57 AM, Erik Huelsmann wrote:



On Fri, Jan 1, 2016 at 6:45 PM, Robert James Clay > wrote:


On Thursday, December 31, 2015 09:50:59 AM David G wrote:
>
> It is worth pointing out that it is undesirable to use an OS
package to
> provide dojo as there are known to be issues from minor release
to minor
> release that would break LedgerSMB.

  What are those issues and how do they break LedgerSMB? That is; 
how is that

being tested and what does that find?  And why are those minor
releases"  being
used, if they cause such issues?

John says he has experience that there are slight incompatibilities 
between versions 1.7, 1.8, ...  and that very careful testing is 
required to be sure your code remains working. Assuming that's true, I 
can be nothing but extremely disappointed about the development 
processes used.
One of the questions that come to mind though is: how do they keep 
their own stuff working if the various versions break expectations? It 
seems some widgets haven't been changed since the early days of Dojo. 
Surely those would be broken now if this goes on too long?


So I actually have not even used 1.9 or 1.10, my experience is based on 
moving from 1.2 to 1.3, to 1.5, to 1.7... every time we upgraded our 
single-page app, stuff broke and needed to get rewritten.


The stuff that broke were largely custom widgets we wrote, using 
underlying Dojo classes for widgets and templates -- they kept 
refactoring these and changing the API in subtle ways that broke our 
custom code.


For LSMB this might not be much of an issue -- yet. At least not before 
LSMB 1.5. But 1.5 relies upon Dojo far more than any previous version 
(is it even in 1.3? And for 1.4, it's only used for simple form elements 
and some layout stuff in the Contacts area -- I don't think we've built 
any actual custom complex widgets).


It's certainly possible that things have shaken out and gotten more 
stable -- as it is the jump from Dojo 1.5 to 1.7 (which introduced AMD) 
was so great we still have not completely upgraded our internal app, and 
are still using 1.5.


And then also realize that there's no server-side testing that would 
reveal a problem with Dojo -- it's purely client-side. So testing it 
properly is largely visual regression testing, stuff on top of 
Selenium/PhantomJS or similar, or using writing tests for Dojo's own 
test runner, DOH (if that's still around... looks like they now promote 
"intern" which looks interesting...)


Dojo does have its own test suite. But the problem I've experienced with 
Dojo upgrades is in its base widget classes (dijit/_Templated, 
dijit/_Widget, etc) which have changed far more than you would think 
should be acceptable within a minor release. If we're not extending 
those, then we're not affected -- yet. But the ability to extend those 
to have smart widgets is the main reason I was advocating for Dojo in 
the first place, over simpler widget systems like jquery.ui and the like.


And as previously noted, I have not experienced any issues with updating 
a patch release within the same minor version (e.g. 1.5.1 to 1.5.3), 
only between the minor versions (1.5 to 1.7). And Dojo itself is still 
releasing patch releases for 1.5 -- if they expected everyone could 
upgrade easily, they would not still be issuing patches for 1.4



Remember that JS libraries don't need to only consider what else is on 
the Linux server -- it's something that needs to support all major 
browsers on all operating systems. Microsoft Edge, IE11, IE10, Safari, 
Chrome, Firefox on Windows, Mac, Linux, Android, iOS, ChromeOS, etc. As 
Javascript itself evolves into ECMAScript 6, with greatly varying 
degrees of OS support, JS libraries need to evolve and change much 
faster than probably any other package on a Linux server. And HTML5 was 
in its infancy 5 years ago -- we were still mostly doing XHTML. And 
unfortunately, sometimes JS libraries have functions that get turned 
into reserved words in newer JS version or particular browsers... jQuery 
has the same issues with upgrading between minor version releases.


Can you point me to any major web application that actually depends upon 
the Debian dojo or jquery packages, instead of bundling the exact 
version they have fully tested? What I see using apt-rdepends are a 
bunch of local applications -- so many desktop toolkits work with 
Javascript, it makes sense to use a system JS library for a desktop app. 
Not for a web app with a huge range of other targets... Drupal and 
WordPress both have Debian packages, and both ship/use jQuery... and do 
not require or reference the Debian libjs-jquery (or any other jquery) 
package.


Cheers,
John
--
___
Ledger-smb-devel mailing list
Ledger-smb-devel@lists.sourceforge.net

Re: [Ledger-smb-devel] Cleaning up the repository. How does the community feel about it?

[Robert James Clay]

On Thursday, December 31, 2015 09:50:59 AM David G wrote:
> 
> It is worth pointing out that it is undesirable to use an OS package to
> provide dojo as there are known to be issues from minor release to minor
> release that would break LedgerSMB. 

  What are those issues and how do they break LedgerSMB? That is;  how is that 
being tested and what does that find?  And why are those minor releases"  being 
used, if they cause such issues?

And as I noted elsewhere; I have so far not seen any issues using the Debian 
packages, although I'll admit that the operational testing I do with it is not 
that  extensive. Nor have there been comments about such issues that I'm aware 
of from anyone using the 1.4.x versions available at apt.ledgersmb.org.

Note that the LedgerSMB source file used for the Debian v1.4.x packages are 
repacked without the dojo related files (but after the archive is checked 
against the gpg key), removing some 10,196 files, and sym links are installed 
pointing to the dojo/dojo, dojo/dijit, & dojo/dojox directories from the dojo 
Debian packages that are installed .  Note also that Debian v8 ('Jessie') has 
dojo v1.10.2 while Debian testing currently has dojo v1.10.4.




Regards,
  RJ Clay








--
___
Ledger-smb-devel mailing list
Ledger-smb-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/ledger-smb-devel

Re: [Ledger-smb-devel] Cleaning up the repository. How does the community feel about it?

[Erik Huelsmann]

On Fri, Jan 1, 2016 at 6:45 PM, Robert James Clay  wrote:

> On Thursday, December 31, 2015 09:50:59 AM David G wrote:
> >
> > It is worth pointing out that it is undesirable to use an OS package to
> > provide dojo as there are known to be issues from minor release to minor
> > release that would break LedgerSMB.
>
>   What are those issues and how do they break LedgerSMB? That is;  how is
> that
> being tested and what does that find?  And why are those minor releases"
> being
> used, if they cause such issues?
>
John says he has experience that there are slight incompatibilities between
versions 1.7, 1.8, ...  and that very careful testing is required to be
sure your code remains working. Assuming that's true, I can be nothing but
extremely disappointed about the development processes used.
One of the questions that come to mind though is: how do they keep their
own stuff working if the various versions break expectations? It seems some
widgets haven't been changed since the early days of Dojo. Surely those
would be broken now if this goes on too long?


> And as I noted elsewhere; I have so far not seen any issues using the
> Debian
> packages, although I'll admit that the operational testing I do with it is
> not
> that  extensive. Nor have there been comments about such issues that I'm
> aware
> of from anyone using the 1.4.x versions available at apt.ledgersmb.org.
>
> We have been using a similar setup (although not based on your packages,
obviously), with the Dojo packages from Debian Wheezy and Debian Jessie. So
far, I've had no complaints. I do know we get operational testing, because
I get more than enough problem reports on 1.4. So far, none of those were
caused by our Dojo versions.


> Note that the LedgerSMB source file used for the Debian v1.4.x packages are
> repacked without the dojo related files (but after the archive is checked
> against the gpg key), removing some 10,196 files, and sym links are
> installed
> pointing to the dojo/dojo, dojo/dijit, & dojo/dojox directories from the
> dojo
> Debian packages that are installed .  Note also that Debian v8 ('Jessie')
> has
> dojo v1.10.2 while Debian testing currently has dojo v1.10.4.
>
Ah. But those are patch releases (not minors). So, from Jessie to testing,
there shouldn't be a problem.


Regards,

Erik.
--
___
Ledger-smb-devel mailing list
Ledger-smb-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/ledger-smb-devel

Re: [Ledger-smb-devel] Cleaning up the repository. How does the community feel about it?

[David G]

Hi Robert,

I personally don't know what issues can be caused.
The fact that Issues are know to be caused by minor revision changes in
dojo was raised by John (freelock) and was based on his extensive
experience with dojo over the years.

Obviously if a distribution provided instance of dojo is use, we
(LedgerSMB) have no control over the use of any minor versions that may
be available.
Yes we could institute checks in our code for a dojo version string,
then use different functions to allow for any know differences, but that
would require us having the manpower to test against all possible
versions of dojo, then write variations of any affected code.

Personally I would rather include our version of the dojo library so we
can free up developer time to work on other things.

I understand that debian policy prevents copies of libraries being
included in packages.
What we are proposing is not to include a copy of dojo, but a customised
version of dojo.
This is necessary to improve performance, and it should also bring some
security benefits as well, just due to the decrease in exposure (less
lines of code in our modified dojo)

Dojo source provides the tools to build the modified version, it is
considered a normal thing to do.

I am sure that John will be able to provide some examples of issues he
has seen in the past.

Of course, if someone wishes to install with a system instance of dojo
it would just be a matter of removing the provided custom version, and
creating a symlink.
At that point any unexpected UI behavior would need to be tracked down
and fixed, unless the owner of the installation is going to provide
resources to trackdown and fix these issues, this places additional load
on the devs.
If we were only targetting a single platform (OS) this would be
acceptable, but we have user installations on quite a variety of
platforms and potentially would need to cater for a different set of
issues on every platform.

In my eyes just the performance gains by using a customized dojo are
enough that I would not want to use a system version.

Regards
David G

On 02/01/16 01:45, Robert James Clay wrote:
> On Thursday, December 31, 2015 09:50:59 AM David G wrote:
>> It is worth pointing out that it is undesirable to use an OS package to
>> provide dojo as there are known to be issues from minor release to minor
>> release that would break LedgerSMB. 
>   What are those issues and how do they break LedgerSMB? That is;  how is 
> that 
> being tested and what does that find?  And why are those minor releases"  
> being 
> used, if they cause such issues?
>
> And as I noted elsewhere; I have so far not seen any issues using the Debian 
> packages, although I'll admit that the operational testing I do with it is 
> not 
> that  extensive. Nor have there been comments about such issues that I'm 
> aware 
> of from anyone using the 1.4.x versions available at apt.ledgersmb.org.
>
> Note that the LedgerSMB source file used for the Debian v1.4.x packages are 
> repacked without the dojo related files (but after the archive is checked 
> against the gpg key), removing some 10,196 files, and sym links are installed 
> pointing to the dojo/dojo, dojo/dijit, & dojo/dojox directories from the dojo 
> Debian packages that are installed .  Note also that Debian v8 ('Jessie') has 
> dojo v1.10.2 while Debian testing currently has dojo v1.10.4.
>
>
>
>
> Regards,
>   RJ Clay
>
>
>
>
>
>
>
>
> --
> ___
> Ledger-smb-devel mailing list
> Ledger-smb-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/ledger-smb-devel
>


--
___
Ledger-smb-devel mailing list
Ledger-smb-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/ledger-smb-devel

[Ledger-smb-devel] Cleaning up the repository. How does the community feel about it?

[David G]

  
  
Hi everyone,
  
Erik (ehuelsmann), John (freelock), and myself (dcg)
  have been discussing the possibility of stripping at least
  dojox (unused) and preferably  all of the dojo source tree
  from the LedgerSMB repository primarily to reduce the download



  size, and disk footprint.

It is worth pointing out that it is undesirable to
  use an OS package to provide dojo as there are known to be
  issues from minor release to minor release that would break
  LedgerSMB. This forces us to use and update our own local
  version of dojo which has historically been the complete
  source held within the UI/lib/dojo/ directory

If we do remove the entire dojo src it would be
  replaced with a "built" version of dojo that only includes the
  parts we are using.
  This has a number of benefits, 
      * reduced download size at install or update,
      * during runtime
          * reduced number of server requests,
      * reduced number of bytes downloaded from the server.

In testing it was found that the following size
  improvements were obtained.


  

  
  
  lsmb-test-full
  
  lsmb-test-shrunk
  lsmb-test-shrunk-2


  
  
  includes
    dojo
    dojox
  
  removed
    dojox
  
  removed
    dojo
    dojox
  


  download size **
  
  43.64 MiB
  
  27.94 MiB  (-15.60 MiB)
  
  21.17 MiB (-22.47 MiB)
  


  total size on disk ***
  
  151 MiB
  
  89 MiB (-62 MiB)
  63 MiB (-88 MiB)
  


  
  
  
  
  
  
  
  

  

*To obtain these gains dojox and or dojo were removed from the entire history of the repository using a git tool called "bfg"
 Once the built version of dojo is added, there will be a small size increase that is yet to be determined but probably about 1-2M (download)
**   taken from git pull progress after completion
***  obtained with "du -sh . " from the top level of the repo


  To allow development changes to the dojo source we use, it
  would be moved to a github submodule that would only need to
  be used by developers working directly on the dojo source.

The benefits are significant, even if we only remove
  dojox.
  Consider the fact that an active developer like Erik may do a
  fresh clone of the repository 10 to 20 times in a day, 
  a saving of 15M to 20M per clone (400M) can save a lot of time
  spent waiting.

There are however some downsides that will only
  affect some users and developers.
  If a you have copy of the repository with local changes there
  will be a one off requirement to either rebase, or merge the
  local changes with the updated repository.
  If we only remove dojox, this should be a simpler task,
  however if we remove the entire dojo source it could be more
  work.

We are still doing tests to verify how much work this
  would be for both cases,
  but decided it was time to ask for comments from the
  community.
The three sample repositories were created as a
  snapshot in time from the normal ledgersmb repository using
  the methods outlined below.


  
https://github.com/ledgersmb/LedgerSMB   Original LedgerSMB repository
  
  
https://github.com/sbts/lsmb-test-full.git   Exact Mirror of the Original
  
  
https://github.com/sbts/lsmb-test-shrunk.git lsmb-test-full with dojox removed

  
  
https://github.com/sbts/lsmb-test-shrunk-2.git   lsmb-test-full with dojox and dojo removed

  

lsmb-test-full          will work to run
  LedgerSMB, but this is NOT RECOMMENDED 
  lsmb-test-shrunk    will in theory work to run LedgerSMB, but
  this has not been tested at the time of writing
  lsmb-test-shrunk-2 will not be usable to run LedgerSMB from as
  dojo source has been removed and there is no built version
  included.

All three lsmb-test-* repositories contain the exact
  same LedgerSMB code.
  The only differences are the removal of dojo/dojox from the
  shrunk repos
These lsmb-test* repositories are for testing what
  will happen if/when we strip dojo/dojox from the repository.
  Please do not