Matt Florell wrote: > No apologies necessary, I think a lot of what you said is mostly true.
Well, thank you. I really appreciate that you're willing to entertain what I am saying without construing it as some sort of attack; it is not in the least bit intended that way. > The PHP and Perl code is not the prettiest around, and a lot of it is > not commented or formatted as well as it should be. However, I would > disagree that there is absolutely zero regard for maintanability or > readability. As with many other GPL projects out there VICIDIAL is > free to use and modify, and there are many people outside of our > company who have worked with the code to provide patches and added > functionality. Indeed, and I acknowledge that this is a challenge with an open-source project - a problem that is probably best solved by means that are not necessarily received well politically, and is probably seen as orthogonal to the spirit and philosophy of open-source. It requires some degree of centralisation of the patch management process and high standards for acceptance, testing, and coding conventions. This leads to a process that gives the perception of being more "closed," a la the Linux kernel, and, as some would have it, perhaps the Asterisk source tree. (I can't really say, as I've never attempted to contribute any of my modifications of the Asterisk source to the project. *hangs head in shame*) But, maintainability and extensibility are probably the biggest challenges to the adoption of an open-source project by a commercial organisation, although those challenges are even more formidable for proprietary, closed-source products. The stark, naked economic realities of adopting something are still there. The integration paths, APIs, transparency, modularity and extensibility are the most important central concerns. For instance, many likely use cases of ViciDIAL entail at least some degree of integration with existing business systems, rules, and logic; after all, the data that goes into the hopper must come from somewhere. :-) ... Which leads me to my point: contrary to what is often zealously claimed by purveyors and advocates of open-source solutions, the simple fact that the code is open **does not, ipso facto, offer the necessary level of integration and extensibility.** Version changes, feature changes, bug fixes, and other revisions cannot be readily or easily applied if an organisation chooses to essentially "fork" an internal revision of stock code, so invariably the preoccupation of good engineering project management becomes whether the custom code can be kept "outboard" in modules entirely separate from the main code tree, so that the latter can remain more or less pristine across upgrades, updates, add-ons, etc. This is where the importance of data import/export, APIs, and other integration paths comes in. It's the same reason why Asterisk worked out so well for you in creating ViciDIAL; you can do pretty much everything in AGI, instead of having to hack the source or even do a whole lot in the configs. And, of course, in closed-source situations, these integration paths become the lifeline - the only possible path for integration and customisation. So, these access points to the "plumbing" of the program, where tweaks can be made to the nature and content of its flow, are extremely vital. ViciDIAL doesn't really have these, as far as I can tell. > The 50,000+ lines of code were mostly written by me over the course of > 5 years, and as we are now having more people working on the code and > we are going through the scripts we are working to make the code > easier to understand. It is important to mention that VICIDIAL is > quite complex and offers a lot of features that add to the complexity > of the code. Many of these features were not even conceived when the > project was started so they were added in in the most efficient manner > that was available. At this point there are over 1000 database fields > across 60+ tables that control how VICIDIAL works. Yeah, I understand. It's just how the project evolved; I'm sure when it was started it was intended to perform a relatively narrow subset of what it does now, and things just got added on. Nothing wrong with that intrinsically; it's how many open source projects got their beginnings, and hardly to their detriment. Still, the ViciDIAL code is an example of something that appears to have been written, but not really designed, engineered, or orchestrated. As the complexity increases in the manner you describe above, it is critically important to be able to decompose new elements out of the core and into modules, libraries, packages, etc. And good developer documentation for the interfaces to these things must be crafted. The thinking must become a lot less "linear" in order for the maintainability and economic feasibility of the code to scale correspondingly to the features on the functional veneer of the application. As a somewhat related note, a lot of the constructs used in the code are simply naive, from a programmatic perspective. At this level of complexity, the code should feel like it belongs to an application, instead of a very, very complicated script. But worst of all is the organisation and readability. I've worked to evaluate the adoption of ViciDIAL with three different organisations now, suggesting it with professional impartiality - apart from the advocacy inherent in simply recommending it. Each time, I would say a genuine, sincere, diligent effort was made by the developers to get to understand the code and explore how to go about customising it, and in each case the code was ultimately judged too spaghetti to be workable. The general complaints were - (1) impossible to read (2) strangely and inconsistently formatted (3) lack of modularisation, abstraction, centralisation, and other vital architectural considerations. > As for scalability, VICIDIAL scales to hundreds of seats across > multiple Asterisk servers. It can do this because of it's reliance on > the MySQL database that acts as the core of a VICIDIAL system. We > chose to use MySQL instead of a dedicated communications protocol so > that the data could be accessed and used by almost any programming or > user interface, and still remain extremely fault-tolerant and > resistant to issues on any individual system. This is advantageous and portable, but a database really should not be used for a high-speed, high-volume IPC mechanism inherently. That's not what databases are for. Secondly, I think that a lot of the need for such a mechanism arises from the fact that ViciDIAL is driven by so many concurrent processes and cron jobs; it ends up being a surrogate for locking and IPC. Of course, this is just my opinion, but I think the dialer component of ViciDIAL (i.e. the AGI part) could benefit from a much more monolithic approach involving multiple threads and a great deal more synchronous I/O and event multiplexing. This would eliminate a lot of the need to pump so much data in and out of the database for the purpose of maintaining synchronisation in the first place. Also, you don't necessarily have to develop a custom IPC protocol for communication amongst components; there are plenty of workable distributed service architectures out there. > As for efficiency, yes there are a lot of small inefficiencies in the > code, but most of the major bottlenecks were removed after changing > many AGI scripts to FastAGI and rewriting several other scripts, which > resulted in a 70-80% reduction in non-Asterisk load on a VICIDIAL > system. Ah, well, glad to hear it! It's been a while now, but I saw a main issue to be a lot of redundant and unnecessary database calls in dozens of different places. In a lot of cases, what ViciDIAL's AGI scripts appear to do is: 1. Get data out of database; 2. Somehow programmatically transform the data (i.e. do something with it in a Perl %hash, or sort it in an array); 3. Feed the data out of the programmatic construct back into the database. 4. Do more things with it there. 5. Get it back out of the database and do some more stuff with it in the program. 6. Rinse, repeat. Sometimes this is unavoidable, and also, MySQL's abysmal lack of decent features prior to 5.x might have been another constraint. For instance, without supporting sub-queries, one doesn't have a lot of choice in many cases except to get the data out, and use parts of it as identifiers for other queries. All the same, a lot of this stuff could be done with stored procedures, more joins, and other good, judicious use of in-database facilities. There is also a dire need for centralisation of a lot of the database calls. On more than one occasion while monkeying around with ViciDIAL I discovered that to make some minor adjustments to the schema and data treatment, I had to make changes in dozens of places across 15-20 files. > Javascript and XMLHTTPRequest are wonderful in some > ways, but are very hard to work with in a complex application like > VICIDIAL and they are not the most efficient solution despite all of > their advantages of client portability and > zero-installation/configuration. Ain't that the truth. :-) > We would love to hear more about the specific issues that you found > working with VICIDIAL, if you could email me back or just post them to > the VICIDIAL forum or Issue Tracker. I would, but a lot of my issues are philosophical and relate to broad project management / business considerations rather than extremely specific, well-defined lackings. > Could you tell me what the last version of VICIDIAL that you worked with was? I do not recall. It was whatever was current and stable in January 2008. -- Alex -- Alex Balashov Evariste Systems Web : http://www.evaristesys.com/ Tel : (+1) (678) 954-0670 Direct : (+1) (678) 954-0671 Mobile : (+1) (706) 338-8599 _______________________________________________ -- Bandwidth and Colocation Provided by http://www.api-digital.com -- AstriCon 2008 - September 22 - 25 Phoenix, Arizona Register Now: http://www.astricon.net asterisk-users mailing list To UNSUBSCRIBE or update options visit: http://lists.digium.com/mailman/listinfo/asterisk-users
