http://www.salon.com/tech/col/garf/2000/09/08/patrick_ball/print.html Mining data on mutilations, beatings, murders A computer programmer digs up the truth behind atrocities in El Salvador, Kosovo and other troubled locales. - - - - - - - - - - - - By Simson Garfinkel Sep. 08, 2000 | "This would be a good time to leave." That's what Patrick Ball heard in 1992 when he was working for the Salvadoran Human Rights Commission. Ball, a peace activist with expertise in data mining, had spent two years in El Salvador building a large-scale database that tracked atrocities and human rights violations perpetrated by both the Salvadoran government and militias during the 1970s and 1980s. It was a digital record of this most troubled period in that country's history. The Human Rights Commission had actually created two databases. The first was a detailed account of threats, thefts, beatings, mutilations, murders and massacres. This database was largely created from eyewitness testimony -- more than 9,000 reports in all. The second was a database that tracked the careers of El Salvador's police and military, built largely from official records, newspaper accounts and some personal recollections. "What we were doing was tracking them by job, rank and unit from when they graduated the military academy as young lieutenants until they retired as senior colonels or generals," recalls Ball. "And then we crossed these two databases, by unit and time." The technique allowed the commission to develop "statistical human rights profiles" of individual officers and units. It showed how units became more violent when certain officers took control, and cataloged the crimes that had been committed under the watch of specific individuals. Essentially, the commission had created a Who's Who of the nastiest criminals of the country's 20-year civil war. "And then we published them in the newspaper!" It was a bold move for a Yankee living so far south of the border. But the move was calculated. El Salvador was in the middle of a closely watched transition from military to civilian rule. "Because it was 1992, and not 1982, they didn't blow up our office," says Ball. Instead, the people who had been named in the files -- most of whom by then were high-ranking officials -- attacked the commission in the courts. And as for Ball, he left the country. It certainly wasn't what Ball had expected when he signed up to work as a peace activist in El Salvador after graduating from Columbia University. His first job in Central America was as a so-called nonviolent accompaniment. "You hang around with people who were likely targets of political violence, on the premise that your witness would prevent people who wanted to do political violence from doing it," he remembers. "It's interesting work, but it's actually boring when you do it. They go to meetings, but you sit around out front" and talk to the secretaries. It was these secretaries who gave Ball his first big break. To hear him tell it, the universal experience of secretaries in offices around the world is losing files on their computers. "If you can do anything to recover their files you become a computer expert." As it turns out, Ball is a computer expert. He paid for his undergraduate education by working part time as a database and statistics programmer. Soon after moving to El Salvador he took a job doing computer work for the human rights office of the Lutheran Church. From there he moved to the Human Rights Commission, where he designed the databases to track El Salvador's bloody history. At first glance, it seems odd that the Human Rights Commission would have massive data-processing needs. Perhaps it might need a few dozen paid researchers to interview the victims and then a small team of writers to assemble the findings in a big report -- but is there really a need for SQL database programmers, forms designers and multivariable analysis? Sadly, the answer is yes. In recent years the scale of atrocities in places like Guatemala, El Salvador, Rwanda, South Africa and Kosovo has been so massive that it defies comprehension by a single person. Each of these countries has seen hundreds of thousands of victims, jointly suffering millions of individual actions and crimes. While such overwhelming brutality can simply be written up -- like the volumes of testimony and records cataloged by Argentina's Commission on Disappeared People in the Nunca Mas report -- such vast amounts of data cannot be easily made sense of in a Microsoft Word document. In El Salvador, researchers figured that if they could somehow capture these events, systematize them and put them in a data bank, they could produce summary reports showing trends, propose underlying theories and motives consistent with the data and ultimately draw a comprehensive portrait of the guilty. "I remember when I lived in El Salvador in the early '90s, I used to go to this Saturday afternoon drinking club at which there were these human rights lawyers, some journalists, and they would tell war stories to each other," recalls Ball. "The collective knowledge among these guys was incredible -- extremely detailed knowledge of who the intelligence service worked for and who was involved -- but it wasn't systematized. When we started putting it into databases, it became incredibly useful. It could be generalized to all kinds of purposes." Since then, Ball has worked around the world developing software that finds hidden patterns in large databases of people's actions. He has worked for the Truth and Reconciliation Commission in South Africa, as well as in Ethiopia, Haiti, Guatemala and the former Yugoslavia. Ironically, he uses many of the same database-mining techniques used by marketing firms to manipulate consumer opinion or by intelligence agencies to track the movements of dissidents. But in Ball's hands, these techniques instead become tools for justice and equity. "I think that Patrick is doing very important work for human rights by essentially professionalizing human rights, by making it more of a social science," says Fred Abrams, a senior researcher at Human Rights Watch. "Traditionally, human rights work has been through anecdotal case studies and narrative reporting based on field research. Patrick is addressing these issues in a more scientific manner. That's crucial. It complements the narrative reporting." In 1996, Ball got a full-time job at the Science and Human Rights Program at the American Association for the Advancement of Science. At the time, the association's main foray into the field of human rights was a project that used genetic fingerprinting to match up children who had been kidnapped during Argentina's "Dirty War" with their grandparents. Another group of scientists was in El Salvador using genetic techniques to identify remains dug up from unmarked graves. Ball met some of the association's scientists, who quickly realized that what he was doing fit their charter of applying science to the advancement of human rights. "Technology has leveled the playing field between human rights organizations and intelligence services," says Ball. "Back in the '70s, intelligence services all over the world were getting pretty impressive computer hardware. This gave them the ability to track activities, peaceful civilian activists as well as violent [individuals], in pretty precise ways, to infer patterns and to use the data analysis as the basis for oppression." Today the same tools can be used to build an irrefutable record that documents a history of oppression. Ball's work is "incredibly important," says Harvey Weinstein, associate director of the Human Rights Center at the University of California at Berkeley. "Patrick has the capacity with this statistical knowledge to develop hard, incontrovertible statistical data to provide the kind of evidence that people need to get a good sense of the kind of human rights violations that occur in these difficult situations. He is one of the leaders in the field of trying to develop and use statistics to provide substantiation for human rights abuses." Over the past decade, most of the large-scale human rights databases have been built either by Ball himself or by people he personally trained. Earlier this year, Ball co-edited a book called "Making the Case," which discusses the technical decisions that were made, and the problems that were encountered, in building these databases in El Salvador, Haiti, South Africa and Guatemala. "After I train people, I lose them to the private sector and to government," says Ball. "By May '99 those projects had all ended and pretty much wrapped up," he says. "I wanted to preserve the technology memory of how these projects happened." More recently, Ball has applied his statistical techniques to analyze interviews with refugees from Kosovo. In 1999, in the midst of the NATO bombing campaign, several hundred thousand people fled their homes. Although the refugees said they were fleeing Serbian militias and Serbian government forces, many on the American left claimed that the refugees were actually fleeing NATO bombs. Ball, who analyzed data from border crossing surveys of 275,000 individuals, doesn't believe this is true. "The core finding was that there were three phases of exodus: March 24 to April 6, when there was a huge wave. Those people almost exclusively came from the south and west. Then it starts creeping up again and peaks on April 17, and those people are all coming from north central. Then it goes down to another low point on April 24; then it comes up again in late April and early May, and those people are coming from the south. "My conclusion was that there is a pattern here, and that pattern does not match bombing at all. There has to be some centrally coordinating cause other than bombing causing migration" -- presumably, armed paramilitary groups that were traveling from village to village. "The findings were not a surprise to me," says Abrams of Human Rights Watch. "The Yugoslav government was claiming that people were fleeing NATO bombs, which we knew was not the case because we interviewed hundreds and hundreds of people and that is what they said. But to have the numbers reach the same conclusion was very powerful and irrefutable." And that's the reason Ball has been using the tools of data mining to bolster human rights causes for so many years. The statistical techniques turn individual accounts into hard data -- and data can be used to argue a cause in a public forum or in a court of law. "The notion 'Never forget' is an overriding principle" of his work, says Ball. "There has been a lot of psychiatric research that shows that individual victims have a much better outcome when the truth is acknowledged. The first level of goal is truth. The second level of goal is justice; if we know what happened, maybe there is some way that the perpetrators can be punished. The third is reconciliation. The fourth level is deterrence: 'Never again.'" Indeed, the statistical evidence can have a lasting impact on a nation that has been through the worst of times. In Croatia, Ball says, "some of the guys who held positions in the fascist government in the '40s now hold positions in Parliament." But things are different in El Salvador, at least for some of the worst offenders from the country's troubled past. Back in 1992, after Ball and his co-workers published their list of names in the newspaper and Ball had to leave the country, military officers whose names had appeared in print sued the Human Rights Commission for defamation. So the commission went to court with computer printouts of the 9,000 testimonies and presented them as depositions. It showed the court the statistics that it had used and its methodology, and asked to subpoena the army's own records to confirm its allegations. "The officers withdrew," says Ball. "They didn't think that our methods were good enough, because they thought, 'There is no way these guys can know these things.'" But the officers were wrong. After the court hearing, the Human Rights Commission turned over its records to the so-called Ad Hoc Commission that was overseeing the country's transition to civilian rule. One of the jobs of the Ad Hoc Commission was to come up with a list of people who would be barred from holding public office. The people on the Ad Hoc Commission's list matched the list that had been supplied by the HRC. Call it the triumph of the database. - - - - - - - - - - - - About the writer Simson Garfinkel is a columnist for Salon Technology and the part owner of Vineyard.NET, an ISP on Martha's Vineyard, Mass. Sound Off Send us a Letter to the Editor <A HREF="http://www.ctrl.org/">www.ctrl.org</A> DECLARATION & DISCLAIMER ========== CTRL is a discussion & informational exchange list. Proselytizing propagandic screeds are unwelcomed. Substance�not soap-boxing�please! These are sordid matters and 'conspiracy theory'�with its many half-truths, mis- directions and outright frauds�is used politically by different groups with major and minor effects spread throughout the spectrum of time and thought. That being said, CTRLgives no endorsement to the validity of posts, and always suggests to readers; be wary of what you read. CTRL gives no credence to Holocaust denial and nazi's need not apply. Let us please be civil and as always, Caveat Lector. ======================================================================== Archives Available at: http://peach.ease.lsoft.com/archives/ctrl.html <A HREF="http://peach.ease.lsoft.com/archives/ctrl.html">Archives of [EMAIL PROTECTED]</A> http:[EMAIL PROTECTED]/ <A HREF="http:[EMAIL PROTECTED]/">ctrl</A> ======================================================================== To subscribe to Conspiracy Theory Research List[CTRL] send email: SUBSCRIBE CTRL [to:] [EMAIL PROTECTED] To UNsubscribe to Conspiracy Theory Research List[CTRL] send email: SIGNOFF CTRL [to:] [EMAIL PROTECTED] Om
