Re: [Ledger-smb-devel] Initial steps for 1.6, and some longer-term questions

[John Locke]

Hi,

On 11/04/2015 08:08 PM, Chris Travers wrote:

Hi John;


Well, the sorts of things that go into the Setup.pl are really not 
going away any time soon (or ever) and the tooling there is less than 
ideal.  I have a partial replacement there.




2.  We could bundle all our tools together.  In this case you
still run the same program which checks your system, sees
whether it can run a web server and if not gives information
as to why not. Such a program would then typically not update
a system but could change file permissions after running and
setting things up.

In both cases, the installer checks external prerequisites,
CPAN dependencies, and can either solve or advise on how to
solve missing dependencies.  I would like LedgerSMB 1.6 to be
easier to install than SQL-Ledger 2.6 was. Mostly this is
hand-holding, providing clear feedback on how to resolve
problems, and a nice interface for showing the information.

I would argue that we should skip all the dependency-checking,
etc, and point people either to packages for Debian/RHEL/etc, or
the Docker images.


For RHEL users, there are currently a lot of extra dependencies to 
track down that are not in the standard repos, and if you are 
installing on a Linux server, you really need to know how to set up 
PostgreSQL etc.  So having a nice installer would help out a lot.   If 
we want to go this route, we really need our own yum/dnf repo.  I 
suppose we could host one at Efficito.


I would prefer to keep you and Erik focused on the core platform, and 
not package management ;-) seems like an opportunity for others to step 
up and contribute, if they really want to go the package route...


Docker solves a lot of the Postgres setup -- the "official" postgres 
container required no configuration to use straight out of the gate. 
Because it's in a container, you have to interact with it via TCP (well, 
if you mount a socket you don't technically "have" to but that's an 
extra step) so Postgres is TCP-enabled in the default container. You set 
the Postgres password when you create the container, and you can log 
into the LSMB setup.pl script with it immediately with no other 
configuration.


It also greatly eases the ability to run different versions of Postgres 
than may happen to get shipped with your installed OS. So perhaps it 
becomes our answer to cutting off support for older Postgres versions -- 
"run a newer Postgres in Docker".




Where we do need to spend time hand-holding is once the system is
up and running -- setting up the Chart of Accounts, providing
guidance on user permissions, how to set up bulk imports, etc.
Nice interfaces for showing best accounting practices, templating
guidance, etc.


Agreed on this.  This also means db creation and management, etc.   In 
other words ,the area that App::LedgerSMB::Admin (and ::Web) have 
focused on regarding a next generation cli and web interface. These 
are the external tools I would like to integrate. It does mean an 
additional couple dependencies which means getting dependency tracking 
is a little more important.





Sure! I think this starts with a list of what is required.

The Dockerfile is a very nice way of documenting that -- it's 
essentially an install script. The Perl:5 image uses Debian Jessie as a 
base, and you can see what I've got installing here:


1.4 - https://github.com/ledgersmb/ledgersmb-docker/blob/1.4/Dockerfile
1.5 - https://github.com/ledgersmb/ledgersmb-docker/blob/master/Dockerfile


... I think we'll need some coordination here between package 
maintainers for whichever distributions we package for, Docker, and 
making sure the list is reflected on the website for those who may want 
to install everything from source.




> I also want to make sure we have both command line and web tools.  
Command line tools give
experienced administrators a productivity edge here (because
it is easier to quickly send complex information to the
program), but web tools give new users an ability to get up
and running with less immediate knowledge.


By "command line" tools, do you mean something more than what's
already there -- the ability to run any of the .pl scripts
directly and pass in info using a Bash script, for example?


The immediate ones that come to my mind are:

1)  Create a database with a specific coa, etc.
2)  Add a user to a database
3)  Update a database within a stable branch
4)  Upgrade a database from one stable branch to another.

These would help out quite a bit where multiple dbs are managed in the 
same organization.   Docker or no, having a good scriptble admin 
interface makes things a lot easier.


Now, for 1 and 3, I already have this fairly well working.  For 2 and 
4, there are some complications.


But they come in really handy in a number of cases, not only hosting 

Re: [Ledger-smb-devel] Initial steps for 1.6, and some longer-term questions

[Chris Travers]

Hi John;

On Sun, Nov 1, 2015 at 7:13 PM, John Locke  wrote:

> Hi,
>
> Love to hear the directions and priorities. Overall, I'm completely
> onboard, and support these. Couple comments:
>
> On 10/31/2015 11:50 PM, Chris Travers wrote:
>
>
>> There are a few things architecturally however that I would hope to
>>> accomplish with 1.6:
>>>
>>> 1.  From 1.2 through 1.4, ease of installation has not been a priority
>>> and has degraded  We need to reverse that.  Ease of installation and
>>> management needs to be a top priority.  I think we have the back-end stuff
>>> down, so the question is front-end and administrator tooling.
>>>
>>
>> I would say this is particularly true: we don't have a web tool which
>> points out which libraries or other dependencies are missing from an
>> installation. Tools like Zabbix have that, allowing an admin to quickly
>> address the issue(s) and assess the level of support for any of the
>> optional features/dependencies. This could seriously improve our "first
>> user experience".
>>
>>
>>
>> Well, I do think that using Docker can alleviate a lot of this pain, in 2
> ways:
>
> 1. Provide a "ready-to-run" installation provisioned in two steps (run a
> Postgres container, run an LSMB container).
>
> 2. Provide a supported installation recipe for anyone who wants to run it
> themselves (the Dockerfile). We could support multiple flavors of this:
> install from git (what the current dockerfile does), install from tarball,
> install from package. And of course, some steps for creating a development
> environment (e.g. use Docker to provide all dependencies but mount a git
> tree into the container that you can develop on directly).
>
> There are some small blockers/disadvantages here, but this is close to
> being ready for use, and I've been using a Dockerized install in production
> for many tasks. The current downsides/blockers:
>
> - Size. Current images are > 1.5GB, largely due to LaTeX and other
> dependencies.
>

You can probably cut down on the size of the LaTeX install.  The problem is
that the TexLive bundles that include everything we use are large, but they
include tons of things we don't and lots of documentation.  I don't know if
you are installing the docs but the basic LaTeX install with the specific
cls and sty files we use wouldnt be very big.


> - Issues #854, #917 -- templates cannot be easily managed in the
> filesystem in Docker, so until all templates can be loaded into the
> database, this is impractical for casual users.
> - Server-side printing -- this might be something relatively easy to solve
> by setting up some print variables and mounting a print device inside the
> container, have not addressed this yet.
>
>
> So we currently use the regular CGI for printing checks and sending
> invoices, but have migrated most other activities into Docker, with the
> much faster Starman configuration (2 instances sharing the same database).
> And for new users, this is a really fast way to get a working install up
> and running.
>

Interesting.

>
>
>
>
> On a technical side, I would like to switch to external tooling for
>>> administration and management.  This means one gets the admin tools, and
>>> can install and manage LedgerSMB instances from there.  It also means these
>>> tools need to play really well together.
>>>
>>
>> I'm affraid this sounds pretty vague from just this sentence. I'm affraid
>> people will have a hard time judging how this will impact them. Will we
>> still deliver these tools in such a way that installation will be easy?
>> Will these be part of the source distro so packagers for distributions
>> automatically get the right sources to package and distribute? Could you
>> elaborate a bit more on your plans in that respect?
>>
>>
> Other than that, if the tools come with the same source tarball and are a
>> huge improvement over what we have now, I think we only can cheer to that
>> :-)
>>
>
> I was actually thinking of a couple different approaches we could take
> here.
>
> 1.  Install the admin and installation tools.  Run them to check your
> system, download, and install latest versions of whatever major branch you
> want (starting with 1.6).  Such could also update a system and could be run
> with different permissions than the main web server.  In other words, you:
>a. download a tarball.
>b.  Run a program.  It checks your system.  If it can start a web
> server on a higher port it does so.  Otherewise it gives clear feedback
> about what you are missing.
>c.  Log into the web interface for more system diagnostics,
> installation and management help.
>
> Cool idea. But is this the best use of our time?
>
> With the rise of Docker, I think a lot of this is becoming largely
> unnecessary. We can create a single supported environment that can run
> anywhere, even on Windows, and not have to support all the possible
> variations of environments -- we should be focusing our efforts on the
> financial 

Re: [Ledger-smb-devel] Initial steps for 1.6, and some longer-term questions

[Chris Travers]

On Sun, Nov 1, 2015 at 12:06 AM, Erik Huelsmann  wrote:

>
>
>> As folks know here I have made several major attempts to rewrite the
>> financial logic but it is a very big task and I never get as far as I would
>> like.  While 1.6 is already expected to have great improvements in
>> multicurrency support, I am not expecting to do a lot on financial code at
>> present.
>>
>
> While this may seem like a weird move given the mantra of the development
> team that the "old code" needs to be removed and the recent confirmation
> that the old code is still very fragile, the existing code has shown to be
> "mostly working" (for most people). So, I agree that we need to do some
> maintenance on other aspects of the code which may have degenerated since
> the project started, due to the focus of replacing code: in the areas you
> point out below, there's a lot of room for improvement. With the desire to
> replace old code, it seems like the replacement code wasn't always required
> to either implement replacements for *all* aspects of the original
> functionality. While that may be acceptable short term, I don't think
> that's a good idea longer term. With the release of LedgerSMB 1.3.0 now
> more than 5 years ago (see:
> http://ledgersmb.org/release/ledgersmb-130-released -- dated
> 2011-10-10!), I concur with Chris that time for consolidation has come,
> even for newer code. Time to iron out some long standing inconveniences and
> problems which may have been caused by a heavy thrust forward over the past
> 10 years of LedgerSMB and the past 5 years of 1.3.x and 1.4.x.
>
>
>> There are a few things architecturally however that I would hope to
>> accomplish with 1.6:
>>
>> 1.  From 1.2 through 1.4, ease of installation has not been a priority
>> and has degraded  We need to reverse that.  Ease of installation and
>> management needs to be a top priority.  I think we have the back-end stuff
>> down, so the question is front-end and administrator tooling.
>>
>
> I would say this is particularly true: we don't have a web tool which
> points out which libraries or other dependencies are missing from an
> installation. Tools like Zabbix have that, allowing an admin to quickly
> address the issue(s) and assess the level of support for any of the
> optional features/dependencies. This could seriously improve our "first
> user experience".
>
>
>> 2.  Contact and user management has become more feature-rich but harder
>> to use.  The code here has also become  bit of a mess even though it is
>> rewritten code.  I would like to fix these problems too.
>>
>
> Right. Recent consulting support shows that our contact management --
> especially the difference between the "entity" and the "eca" -- and the
> relation between contacts ("entities" and "ecas") and orders/invoices is
> hard to understand at first.
>
> Personally, I actually completely understand this for multiple reasons
> (listing them here, because I think these should be on the list to be
> addressed as part of addressing any issues in "contact management"):
>
> * The system hinges on two thoughts regarding entities and ecas: when
> selecting a customer an invoice, one has to enter the name of the *entity*
> * The system clearly hinges on two thoughts regarding entities and ecas:
> when there are two ecas (both customers) within 1 entity, every "update"
> leads to the counterparty selection screen
> * The fact that there's a "contact type" (customer/vendor/...) on the
> entity definition (company/person) screen, leads people to believe that the
> customer already has been created when they save the company/person (being
> presented with the ECA screen which lets them define and save the customer
> for real)
> * The way there currently is no extremely obvious indication that
> addresses/bank accounts/notes/etc are being added to either the entity or
> the selected ECA, make the use of these items very hard to use
>

First, I agree with these points.

Secondly I think there are two major architectural problems also which need
to be sorted out.

On top of the UI issues we have the fact that right now we are generating a
large form in HTML which then gets dojoized on the client.  This leads to a
number of other problems.  So in addition to the points you make above
regarding customer confusion, there are some other problems that come out
of this.

1.  I start entering a note, switch to the address tab, enter an address,
and come back relatively gracefully. The whole page is submitted and
recreated and that is less than ideal.
2.  Because of this, the Perl code that generates the page, the controller
logic, is ugly, complicated, and difficult to maintain.

If the UI was in dojo, with a clear front/back-end separation, these
problems would go away.  This by itself would provide better usability, but
it also gives an ability to, as you point out, to clear up the user
confusion issues.


>
>
>> 3  Workflow and usability issues need to be the top 

Re: [Ledger-smb-devel] Initial steps for 1.6, and some longer-term questions

[John Locke]

Hi,

Love to hear the directions and priorities. Overall, I'm completely 
onboard, and support these. Couple comments:


On 10/31/2015 11:50 PM, Chris Travers wrote:

   There are a few things architecturally however that I would
   hope to accomplish with 1.6:

   1.  From 1.2 through 1.4, ease of installation has not been
   a priority and has degraded  We need to reverse that.  Ease
   of installation and management needs to be a top priority. 
   I think we have the back-end stuff down, so the question is

   front-end and administrator tooling.


   I would say this is particularly true: we don't have a web tool
   which points out which libraries or other dependencies are
   missing from an installation. Tools like Zabbix have that,
   allowing an admin to quickly address the issue(s) and assess the
   level of support for any of the optional features/dependencies.
   This could seriously improve our "first user experience".



Well, I do think that using Docker can alleviate a lot of this pain, in 
2 ways:


1. Provide a "ready-to-run" installation provisioned in two steps (run a 
Postgres container, run an LSMB container).


2. Provide a supported installation recipe for anyone who wants to run 
it themselves (the Dockerfile). We could support multiple flavors of 
this: install from git (what the current dockerfile does), install from 
tarball, install from package. And of course, some steps for creating a 
development environment (e.g. use Docker to provide all dependencies but 
mount a git tree into the container that you can develop on directly).


There are some small blockers/disadvantages here, but this is close to 
being ready for use, and I've been using a Dockerized install in 
production for many tasks. The current downsides/blockers:


- Size. Current images are > 1.5GB, largely due to LaTeX and other 
dependencies.
- Issues #854, #917 -- templates cannot be easily managed in the 
filesystem in Docker, so until all templates can be loaded into the 
database, this is impractical for casual users.
- Server-side printing -- this might be something relatively easy to 
solve by setting up some print variables and mounting a print device 
inside the container, have not addressed this yet.



So we currently use the regular CGI for printing checks and sending 
invoices, but have migrated most other activities into Docker, with the 
much faster Starman configuration (2 instances sharing the same 
database). And for new users, this is a really fast way to get a working 
install up and running.



   On a technical side, I would like to switch to external
   tooling for administration and management.  This means one
   gets the admin tools, and can install and manage LedgerSMB
   instances from there. It also means these tools need to play
   really well together.


   I'm affraid this sounds pretty vague from just this sentence.
   I'm affraid people will have a hard time judging how this will
   impact them. Will we still deliver these tools in such a way
   that installation will be easy? Will these be part of the source
   distro so packagers for distributions automatically get the
   right sources to package and distribute? Could you elaborate a
   bit more on your plans in that respect?

   Other than that, if the tools come with the same source tarball
   and are a huge improvement over what we have now, I think we
   only can cheer to that :-)


   I was actually thinking of a couple different approaches we could
   take here.

   1.  Install the admin and installation tools.  Run them to check
   your system, download, and install latest versions of whatever major
   branch you want (starting with 1.6).  Such could also update a
   system and could be run with different permissions than the main web
   server.  In other words, you:
   a. download a tarball.
   b.  Run a program.  It checks your system.  If it can start a
   web server on a higher port it does so. Otherewise it gives clear
   feedback about what you are missing.
   c.  Log into the web interface for more system diagnostics,
   installation and management help.

Cool idea. But is this the best use of our time?

With the rise of Docker, I think a lot of this is becoming largely 
unnecessary. We can create a single supported environment that can run 
anywhere, even on Windows, and not have to support all the possible 
variations of environments -- we should be focusing our efforts on the 
financial system rewrite, not admin tools to fix a bunch of different 
environments.


If somebody really wants a bare-bones install in their specific 
environment and not run in a container, that sounds like a perfect 
opportunity for paid consultants to step in -- for the core team, this 
seems like a huge distraction.



   2.  We could bundle all our tools together.  

Re: [Ledger-smb-devel] Initial steps for 1.6, and some longer-term questions

[Erik Huelsmann]

> As folks know here I have made several major attempts to rewrite the
> financial logic but it is a very big task and I never get as far as I would
> like.  While 1.6 is already expected to have great improvements in
> multicurrency support, I am not expecting to do a lot on financial code at
> present.
>

While this may seem like a weird move given the mantra of the development
team that the "old code" needs to be removed and the recent confirmation
that the old code is still very fragile, the existing code has shown to be
"mostly working" (for most people). So, I agree that we need to do some
maintenance on other aspects of the code which may have degenerated since
the project started, due to the focus of replacing code: in the areas you
point out below, there's a lot of room for improvement. With the desire to
replace old code, it seems like the replacement code wasn't always required
to either implement replacements for *all* aspects of the original
functionality. While that may be acceptable short term, I don't think
that's a good idea longer term. With the release of LedgerSMB 1.3.0 now
more than 5 years ago (see:
http://ledgersmb.org/release/ledgersmb-130-released -- dated 2011-10-10!),
I concur with Chris that time for consolidation has come, even for newer
code. Time to iron out some long standing inconveniences and problems which
may have been caused by a heavy thrust forward over the past 10 years of
LedgerSMB and the past 5 years of 1.3.x and 1.4.x.


> There are a few things architecturally however that I would hope to
> accomplish with 1.6:
>
> 1.  From 1.2 through 1.4, ease of installation has not been a priority and
> has degraded  We need to reverse that.  Ease of installation and management
> needs to be a top priority.  I think we have the back-end stuff down, so
> the question is front-end and administrator tooling.
>

I would say this is particularly true: we don't have a web tool which
points out which libraries or other dependencies are missing from an
installation. Tools like Zabbix have that, allowing an admin to quickly
address the issue(s) and assess the level of support for any of the
optional features/dependencies. This could seriously improve our "first
user experience".


> 2.  Contact and user management has become more feature-rich but harder to
> use.  The code here has also become  bit of a mess even though it is
> rewritten code.  I would like to fix these problems too.
>

Right. Recent consulting support shows that our contact management --
especially the difference between the "entity" and the "eca" -- and the
relation between contacts ("entities" and "ecas") and orders/invoices is
hard to understand at first.

Personally, I actually completely understand this for multiple reasons
(listing them here, because I think these should be on the list to be
addressed as part of addressing any issues in "contact management"):

* The system hinges on two thoughts regarding entities and ecas: when
selecting a customer an invoice, one has to enter the name of the *entity*
* The system clearly hinges on two thoughts regarding entities and ecas:
when there are two ecas (both customers) within 1 entity, every "update"
leads to the counterparty selection screen
* The fact that there's a "contact type" (customer/vendor/...) on the
entity definition (company/person) screen, leads people to believe that the
customer already has been created when they save the company/person (being
presented with the ECA screen which lets them define and save the customer
for real)
* The way there currently is no extremely obvious indication that
addresses/bank accounts/notes/etc are being added to either the entity or
the selected ECA, make the use of these items very hard to use


> 3  Workflow and usability issues need to be the top priority.
>

Agreed. I think this starts with defining what the various workflows should
look like and how we can create screens which support those workflows. I'm
thinking of presenting un-editable screens for documents (transactions,
orders, ...) which have been saved and/or posted; an "Edit"(for saved
documents) or "Copy to new"(for posted documents) button at the bottom
should move the document into a state where it can be edited, presenting
the user with a fully editable document. This will also help users
understand the idea of "Save Info", because the only editable field in an
invoice *after posting* will be the "internal notes" field. Completely
obvious to what the "Save Info" button will apply, right?

Another one of my gripes is with usability and scheduling of transactions.
I think we should present a listing of all scheduled transaction postings
when scheduling a transaction. That way the user gets to validate his/her
expectations regarding the planned postings. I for one *always* get this
wrong...


> On a technical side, I would like to switch to external tooling for
> administration and management.  This means one gets the admin tools, and
> can install and