On 7/17/08, Alex Balashov <[EMAIL PROTECTED]> wrote: > Matt Florell wrote: > > > No apologies necessary, I think a lot of what you said is mostly true. > > > Well, thank you. I really appreciate that you're willing to entertain > what I am saying without construing it as some sort of attack; it is > not in the least bit intended that way. > > > > The PHP and Perl code is not the prettiest around, and a lot of it is > > not commented or formatted as well as it should be. However, I would > > disagree that there is absolutely zero regard for maintanability or > > readability. As with many other GPL projects out there VICIDIAL is > > free to use and modify, and there are many people outside of our > > company who have worked with the code to provide patches and added > > functionality. > > > Indeed, and I acknowledge that this is a challenge with an open-source > project - a problem that is probably best solved by means that are not > necessarily received well politically, and is probably seen as > orthogonal to the spirit and philosophy of open-source. It requires > some degree of centralisation of the patch management process and high > standards for acceptance, testing, and coding conventions. This leads > to a process that gives the perception of being more "closed," a la the > Linux kernel, and, as some would have it, perhaps the Asterisk source > tree. (I can't really say, as I've never attempted to contribute any of > my modifications of the Asterisk source to the project. *hangs head in > shame*)
Bad programmer, no beer for you :) But seriously, Asterisk is a better example of doing things right more recently, a couple of years ago all sorts of stuff went into the "stable" releases of Asterisk without enough testing resulting in some pretty big bugs (like in asterisk 1.2.10-12) And even further back the code conventions of Asterisk 6 years ago were in many places about as good as VICIDIAL is today. I think it's the evolution of a project from being in its early years as opposed to its more mature years like Asterisk has started to enter. If I were to start writing VICIDIAL today I would have done many things a lot differently from how they are, but then again if I started from scratch today it wouldn't be a functioning product for many months. Right now, we do basic testing of everything before it is committed to SVN. We are planning on starting an official Q&A process when we are able to that would test a full range of functions before every commit. > But, maintainability and extensibility are probably the biggest > challenges to the adoption of an open-source project by a commercial > organisation, although those challenges are even more formidable for > proprietary, closed-source products. The stark, naked economic > realities of adopting something are still there. There are a few large companies that have adopted VICIDIAL and internalized the maintenance of the code for their systems, so we know that the state of the code has not been a road-block for all companies. > The integration paths, APIs, transparency, modularity and extensibility > are the most important central concerns. For instance, many likely use > cases of ViciDIAL entail at least some degree of integration with > existing business systems, rules, and logic; after all, the data that > goes into the hopper must come from somewhere. :-) Most companies will do some degree of basic lead loading and lead exporting integration on the back-end along with integration with web-based CRM systems on the agent front-end. And this has proven to usually be not too difficult for many companies to accomplish. > ... Which leads me to my point: contrary to what is often zealously > claimed by purveyors and advocates of open-source solutions, the simple > fact that the code is open **does not, ipso facto, offer the necessary > level of integration and extensibility.** Version changes, feature > changes, bug fixes, and other revisions cannot be readily or easily > applied if an organisation chooses to essentially "fork" an internal > revision of stock code, so invariably the preoccupation of good > engineering project management becomes whether the custom code can be > kept "outboard" in modules entirely separate from the main code tree, so > that the latter can remain more or less pristine across upgrades, > updates, add-ons, etc. This is a big reason that we use SVN and we started maintaining a "stable" branch that we apply only minor changes and bug fixes to and a "development" branch that all new development takes place on. > This is where the importance of data import/export, APIs, and other > integration paths comes in. It's the same reason why Asterisk worked > out so well for you in creating ViciDIAL; you can do pretty much > everything in AGI, instead of having to hack the source or even do a > whole lot in the configs. > > And, of course, in closed-source situations, these integration paths > become the lifeline - the only possible path for integration and > customisation. So, these access points to the "plumbing" of the > program, where tweaks can be made to the nature and content of its flow, > are extremely vital. > > ViciDIAL doesn't really have these, as far as I can tell. We have actually do now have a agent-side API that allows outside processes to have very limited control over agent interface functions. As it's features grow we will also work on more back-end API functionality not directly related to the agent. > > > The 50,000+ lines of code were mostly written by me over the course of > > 5 years, and as we are now having more people working on the code and > > we are going through the scripts we are working to make the code > > easier to understand. It is important to mention that VICIDIAL is > > quite complex and offers a lot of features that add to the complexity > > of the code. Many of these features were not even conceived when the > > project was started so they were added in in the most efficient manner > > that was available. At this point there are over 1000 database fields > > across 60+ tables that control how VICIDIAL works. > > > Yeah, I understand. It's just how the project evolved; I'm sure when > it was started it was intended to perform a relatively narrow subset of > what it does now, and things just got added on. Nothing wrong with that > intrinsically; it's how many open source projects got their beginnings, > and hardly to their detriment. > > Still, the ViciDIAL code is an example of something that appears to have > been written, but not really designed, engineered, or orchestrated. As > the complexity increases in the manner you describe above, it is > critically important to be able to decompose new elements out of the > core and into modules, libraries, packages, etc. And good developer > documentation for the interfaces to these things must be crafted. The > thinking must become a lot less "linear" in order for the > maintainability and economic feasibility of the code to scale > correspondingly to the features on the functional veneer of the application. Yes, I agree, modularization is a future goal of ours that unfortunately we do not have the time to work on at the moment, but it is something that we have talked about a great deal. > As a somewhat related note, a lot of the constructs used in the code are > simply naive, from a programmatic perspective. At this level of > complexity, the code should feel like it belongs to an application, > instead of a very, very complicated script. I must say my programming skills have grown a great deal since starting this project, and it shows if you compare the early code to the more recent code. Many newer major features are now starting with a planning phase before any programming is done, this has helped the development and documentation of new features greatly, and as we have the resources we will go back and reengineer things in a more cohearant manner. > But worst of all is the organisation and readability. I've worked to > evaluate the adoption of ViciDIAL with three different organisations > now, suggesting it with professional impartiality - apart from the > advocacy inherent in simply recommending it. Each time, I would say a > genuine, sincere, diligent effort was made by the developers to get to > understand the code and explore how to go about customising it, and in > each case the code was ultimately judged too spaghetti to be workable. > The general complaints were - (1) impossible to read (2) strangely and > inconsistently formatted (3) lack of modularisation, abstraction, > centralisation, and other vital architectural considerations. It really depends on the degree to which you need to alter the code as to how difficult it is. Simple integrations with customer data are fairly easy to do through data connecitons or scripts running and converting data between systems. On the other hand, if you want to significantly alter the schema or layout of the agent interface, then you are in for a lot of work. > > As for scalability, VICIDIAL scales to hundreds of seats across > > multiple Asterisk servers. It can do this because of it's reliance on > > the MySQL database that acts as the core of a VICIDIAL system. We > > chose to use MySQL instead of a dedicated communications protocol so > > that the data could be accessed and used by almost any programming or > > user interface, and still remain extremely fault-tolerant and > > resistant to issues on any individual system. > > > This is advantageous and portable, but a database really should not be > used for a high-speed, high-volume IPC mechanism inherently. That's not > what databases are for. Having a centralized system with universal data access has really helped us in scalability and integration with other systems. Initially I did not start out with MySQL handling everything, but as I ran tests and worked more with it, it proved to be extremely reliable and very fast, so I built everything around it and it has proven that it can handle the job. It might not be the best or most efficient tool for the job, and we may look into other methods of handling IPC in the future, but for now it has proven to be one of the most stable elements of a VICIDIAL system, with some databases running for over a year without restart while still handling 5000+ queries per second. > Secondly, I think that a lot of the need for such a mechanism arises > from the fact that ViciDIAL is driven by so many concurrent processes > and cron jobs; it ends up being a surrogate for locking and IPC. > > Of course, this is just my opinion, but I think the dialer component of > ViciDIAL (i.e. the AGI part) could benefit from a much more monolithic > approach involving multiple threads and a great deal more synchronous > I/O and event multiplexing. This would eliminate a lot of the need to > pump so much data in and out of the database for the purpose of > maintaining synchronisation in the first place. When VICIDIAL started 5 years ago there were some serious reliability issues with Asterisk under high loads, which is why some of the things are done the way they are in VICIDIAL. I had to spend a lot of time and effort finding ways to validate the data coming from Asterisk and still be tolerant of Asterisk system freezes on multi-server systems. Most of those issues have gone away, but they still do happen and the fault tolerance built in to VICIDIAL has made it a better system in the end. > Also, you don't necessarily have to develop a custom IPC protocol for > communication amongst components; there are plenty of workable > distributed service architectures out there. Is there a specific IPC protocol that you would recommend I take a look at? > > > As for efficiency, yes there are a lot of small inefficiencies in the > > code, but most of the major bottlenecks were removed after changing > > many AGI scripts to FastAGI and rewriting several other scripts, which > > resulted in a 70-80% reduction in non-Asterisk load on a VICIDIAL > > system. > > > Ah, well, glad to hear it! Yes, we were pretty amazed(and happy) by the difference it made, and we are in the process of doing more consolidation in development now. > It's been a while now, but I saw a main issue to be a lot of redundant > and unnecessary database calls in dozens of different places. > > In a lot of cases, what ViciDIAL's AGI scripts appear to do is: > > 1. Get data out of database; > > 2. Somehow programmatically transform the data (i.e. do something with > it in a Perl %hash, or sort it in an array); > > 3. Feed the data out of the programmatic construct back into the database. > > 4. Do more things with it there. > > 5. Get it back out of the database and do some more stuff with it in the > program. > > 6. Rinse, repeat. > > Sometimes this is unavoidable, and also, MySQL's abysmal lack of decent > features prior to 5.x might have been another constraint. For instance, > without supporting sub-queries, one doesn't have a lot of choice in many > cases except to get the data out, and use parts of it as identifiers for > other queries. > > All the same, a lot of this stuff could be done with stored procedures, > more joins, and other good, judicious use of in-database facilities. > There is also a dire need for centralisation of a lot of the database > calls. On more than one occasion while monkeying around with ViciDIAL I > discovered that to make some minor adjustments to the schema and data > treatment, I had to make changes in dozens of places across 15-20 files. After the next full release we will do away with MySQL 4.0 and 4.1 support and require 5.0+ so we can do a lot of what you suggest. We have hundreds of systems out there with 4.X MySQL and we have always tried to stay backwards compatible, but we have started to put notes into the documentation and in the forums that we will be dropping support for these older versions so that we can add new functions and increase efficiency in future versions. As for the schema, we are cleaning up the code to better handle modifications to the schema, but it is a big job. We are updating code as we work on newer features or go into files for bug fixes. After the next major release we will be doing code cleanup to finally remove any use of fixed data structures in the programming. > > Javascript and XMLHTTPRequest are wonderful in some > > ways, but are very hard to work with in a complex application like > > VICIDIAL and they are not the most efficient solution despite all of > > their advantages of client portability and > > zero-installation/configuration. > > > Ain't that the truth. :-) > > > > We would love to hear more about the specific issues that you found > > working with VICIDIAL, if you could email me back or just post them to > > the VICIDIAL forum or Issue Tracker. > > > I would, but a lot of my issues are philosophical and relate to broad > project management / business considerations rather than extremely > specific, well-defined lackings. > > > > Could you tell me what the last version of VICIDIAL that you worked with > was? > > > I do not recall. It was whatever was current and stable in January 2008. That would probably be the 2.0.4 release. We have done 3 updates to this branch in SVN and released new code snapshots with all sorts of bug fixes and enhancements in the last 7 months that do not require any database changes to upgrade. > > -- Alex Thanks again for your well though-out feedback. MATT--- > -- > Alex Balashov > Evariste Systems > Web : http://www.evaristesys.com/ > Tel : (+1) (678) 954-0670 > Direct : (+1) (678) 954-0671 > Mobile : (+1) (706) 338-8599 > > _______________________________________________ > -- Bandwidth and Colocation Provided by http://www.api-digital.com -- > > AstriCon 2008 - September 22 - 25 Phoenix, Arizona > Register Now: http://www.astricon.net > > asterisk-users mailing list > To UNSUBSCRIBE or update options visit: > http://lists.digium.com/mailman/listinfo/asterisk-users > _______________________________________________ -- Bandwidth and Colocation Provided by http://www.api-digital.com -- AstriCon 2008 - September 22 - 25 Phoenix, Arizona Register Now: http://www.astricon.net asterisk-users mailing list To UNSUBSCRIBE or update options visit: http://lists.digium.com/mailman/listinfo/asterisk-users
