http://www.sciencemag.org/content/335/6067/394.full
Science 27 January 2012: Vol. 335 no. 6067 pp. 394-396 DOI: 10.1126/science.335.6067.394 Computer Science What It'll Take to Go Exascale Robert F. Service Scientists hope the next generation of supercomputers will carry out a million trillion operations per second. But first they must change the way the machines are built and run. On fire. More powerful supercomputers now in the design stage should make modeling turbulent gas flames more accurate and revolutionize engine designs. "CREDIT: J. CHEN/CENTER FOR EXASCALE SIMULATION OF COMBUSTION IN TURBULENCE, SANDIA NATIONAL LABORATORIES" Using real climate data, scientists at Lawrence Berkeley National Laboratory (LBNL) in California recently ran a simulation on one of the world's most powerful supercomputers that replicated the number of tropical storms and hurricanes that had occurred over the past 30 years. Its accuracy was a landmark for computer modeling of global climate. But Michael Wehner and his LBNL colleagues have their eyes on a much bigger prize: understanding whether an increase in cloud cover from rising temperatures would retard climate change by reflecting more light back into space, or accelerate it by trapping additional heat close to Earth. To succeed, Wehner must be able to model individual cloud systems on a global scale. To do that, he will need supercomputers more powerful than any yet designed. These so-called exascale computers would be capable of carrying out 1018 floating point operations per second, or an exaflop. That's nearly 100 times more powerful than today's biggest supercomputer, Japan's “K Computer,” which achieves 11.3 petaflops (1015 flops) (see graph), and 1000 times faster than the Hopper supercomputer used by Wehner and his colleagues. The United States now appears poised to reach for the exascale, as do China, Japan, Russia, India, and the European Union. It won't be easy. Advances in supercomputers have come at a steady pace over the past 20 years, enabled by the continual improvement in computer chip manufacturing. But this evolutionary approach won't cut it in getting to the exascale. Instead, computer scientists must first figure out ways to make future machines far more energy efficient and tolerant of errors, and find novel ways to program them. “The step we are about to take to exascale computing will be very, very difficult,” says Robert Rosner, a physicist at the University of Chicago in Illinois, who chaired a recent Department of Energy (DOE) committee charged with exploring whether exascale computers would be achievable. Charles Shank, a former director of LBNL who recently headed a separate panel collecting widespread views on what it would take to build an exascale machine, agrees. “Nobody said it would be impossible,” Shank says. “But there are significant unknowns.” Gaining support The next generation of powerful supercomputers will be used to design high-efficiency engines tailored to burn biofuels, reveal the causes of supernova explosions, track the atomic workings of catalysts in real time, and study how persistent radiation damage might affect the metal casing surrounding nuclear weapons. “It's a technology that has become critically important for many scientific disciplines,” says Horst Simon, LBNL's deputy director. That versatility has made supercomputing an easy sell to politicians. The massive 2012 spending bill approved last month by Congress contained $1.06 billion for DOE's program in advanced computing, which includes a down payment to bring online the world's first exascale computer. Congress didn't specify exactly how much money should be spent on the exascale initiative, for which DOE had requested $126 million. But it asked for a detailed plan, due next month, with multiyear budget breakdowns listing who is expected to do what, when. Those familiar with the ways of Washington say that the request reflects an unusual bipartisan consensus on the importance of the initiative. “In today's political atmosphere, this is very unusual,” says Jack Dongarra, a computer scientist at the University of Tennessee, Knoxville, who closely follows national and international high-performance computing trends. “It shows how critical it really is and the threat perceived of the U.S. losing its dominance in the field.” The threat is real: Japan and China have built and operate the three most powerful supercomputers in the world. The rest of the world also hopes that their efforts will make them less dependent on U.S. technology. Of today's top 500 supercomputers, the vast majority were built using processors from Intel, Advanced Micro Devices (AMD), and NVIDIA, all U.S.-based companies. But that's beginning to change, at least at the top. Japan's K machine is built using specially designed processors from Fujitsu, a Japanese company. China, which had no supercomputers in the Top500 List in 2000, now has five petascale machines and is building another with processors made by a Chinese company. And an E.U. research effort plans to use ARM processing chips made by a U.K. company. Getting over the bumps Although bigger and faster, supercomputers aren't fundamentally different from our desktops and laptops, all of which rely on the same sorts of specialized components. Computer processors serve as the brains that carry out logical functions, such as adding two numbers together or sending a bit of data to a location where it is needed. Memory chips, by contrast, hold data for safekeeping for later use. A network of wires connects processors and memory and allows data to flow where and when they are needed. For decades, the primary way of improving computers was creating chips with ever smaller and faster circuitry. This increased the processor's frequency, allowing it to churn through tasks at a faster clip. Through the 1990s, chipmakers steadily boosted the frequency of chips. But the improvements came at a price: The power demanded by a processor is proportional to its frequency cubed. So doubling a processor's frequency requires an eightfold increase in power. New king. Japan has the fastest machine (bar), although the United States still has the most petascale computers (number in parentheses). "CREDIT: ADAPTED FROM JACK DONGARRA/TOP 500 LIST/UNIVERSITY OF TENNESSEE" On the rise. The gap in available supercomputing capacity between the United States and the rest of the world has narrowed, with China gaining the most ground. "CREDIT: ADAPTED FROM JACK DONGARRA/TOP 500 LIST/UNIVERSITY OF TENNESSEE" With the rise of mobile computing, chipmakers couldn't raise power demands beyond what batteries could store. So about 10 years ago, chip manufacturers began placing multiple processing “cores” side by side on single chips. This arrangement meant that only twice the power was needed to double a chip's performance. This trend swept through the world of supercomputers. Those with single souped-up processors gave way to today's “parallel” machines that couple vast numbers of off-the-shelf commercial processors together. This move to parallel computing “was a huge, disruptive change,” says Robert Lucas, an electrical engineer at the University of Southern California's Information Sciences Institute in Los Angeles. Hardware makers and software designers had to learn how to split problems apart, send individual pieces to different processors, synchronize the results, and synthesize the final ensemble. Today's top machine—Japan's “K Computer”—has 705,000 cores. If the trend continues, an exascale computer would have between 100 million and 1 billion processors. But simply scaling up today's models won't work. “Business as usual will not get us to the exascale,” Simon says. “These computers are becoming so complicated that a number of issues have come up that were not there before,” Rosner agrees. The biggest issue relates to a supercomputer's overall power use. The largest supercomputers today use about 10 megawatts (MW) of power, enough to power 10,000 homes. If the current trend of power use continues, an exascale supercomputer would require 200 MW. “It would take a nuclear power reactor to run it,” Shank says. Even if that much power were available, the cost would be prohibitive. At $1 million per megawatt per year, the electricity to run an exascale machine would cost $200 million annually. “That's a non-starter,” Shank says. So the current target is a machine that draws 20 MW at most. Even that goal will require a 300-fold improvement in flops per watt over today's technology. Ideas for getting to these low-power chips are already circulating. One would make use of different types of specialized cores. Today's top-of-the-line supercomputers already combine conventional processor chips, known as CPUs, with an alternative version called graphical processing units (GPUs), which are very fast at certain types of calculations. Chip manufacturers are now looking at going from “multicore” chips with four or eight cores to “many-core” chips, each containing potentially hundreds of CPU and GPU cores, allowing them to assign different calculations to specialized processors. That change is expected to make the overall chips more energy efficient. Intel, AMD, and other chip manufacturers have already announced plans to make hybrid many-core chips. Another stumbling block is memory. As the number of processors in a supercomputer skyrockets, so, too, does the need to add memory to feed bits of data to the processors. Yet, over the next few years, memory manufacturers are not projected to increase the storage density of their chips fast enough to keep up with the performance gains of processors. Supercomputer makers can get around this by adding additional memory modules. But that's threatening to drive costs too high, Simon says. Even if researchers could afford to add more memory modules, that still won't solve matters. Moving ever-growing streams of data back and forth to processors is already creating a backup for processors that can dramatically slow a computer's performance. Today's supercomputers use 70% of their power to move bits of data around from one place to another. One potential solution would stack memory chips on top of one another and run communication and power lines vertically through the stack. This more-compact architecture would require fewer steps to route data. Another approach would stack memory chips atop processors to minimize the distance bits need to travel. A third issue is errors. Modern processors compute with stunning accuracy, but they aren't perfect. The average processor will produce one error per year, as a thermal fluctuation or a random electrical spike flips a bit of data from one value to another. Such errors are relatively easy to ferret out when the number of processors is low. But it gets much harder when 100 million to 1 billion processors are involved. And increasing complexity produces additional software errors as well. One possible solution is to have the supercomputer crunch different problems multiple times and “vote” for the most common solution. But that creates a new problem. “How can I do this without wasting double or triple the resources?” Lucas asks. “Solving this problem will probably require new circuit designs and algorithms.” Finally, there is the challenge of redesigning the software applications themselves, such as a novel climate model or a simulation of a chemical reaction. “Even if we can produce a machine with 1 billion processors, it's not clear that we can write software to use it efficiently,” Lucas says. Current parallel computing machines use a strategy, known as message passing interface, that divides computational problems and parses out the pieces to individual processors, then collects the results. But coordinating all this traffic for millions of processors is becoming a programming nightmare. “There's a huge concern that the programming paradigm will have to change,” Rosner says. DOE has already begun laying the groundwork to tackle these and other challenges. Last year it began funding three “co-design” centers, multi-institution cooperatives led by researchers at Los Alamos, Argonne, and Sandia national laboratories. The centers bring together scientific users who write the software code and hardware makers to design complex software and computer architectures that work in the fastest and most energy-efficient manner. It poses a potential clash between scientists who favor openness and hardware companies that normally keep their activities secret for proprietary reasons. “But it's a worthy goal,” agrees Wilfred Pinfold, Intel's director of extreme-scale programming in Hillsboro, Oregon. Not so fast. Researchers have some ideas on how to overcome barriers to building exascale machines. Coming up with the cash Solving these challenges will take money, and lots of it. Two years ago, Simon says, DOE officials estimated that creating an exascale computer would cost $3 billion to $4 billion over 10 years. That amount would pay for one exascale computer for classified defense work, one for nonclassified work, and two 100-petaflops machines to work out some of the technology along the way. Those projections assumed that Congress would deliver a promised 10-year doubling of the budget of DOE's Office of Science. But those assumptions are “out of the window,” Simon says, replaced by the more likely scenario of budget cuts as Congress tries to reduce overall federal spending. Given that bleak fiscal picture, DOE officials must decide how aggressively they want to pursue an exascale computer. “What's the right balance of being aggressive to maintain a leadership position and having the plan sent back to the drawing board by [the Office of Management and Budget]?” Simon asks. “I'm curious to see.” DOE's strategic plan, due out next month, should provide some answers. The rest of the world faces a similar juggling act. China, Japan, the European Union, Russia, and India all have given indications that they hope to build an exascale computer within the next decade. Although none has released detailed plans, each will need to find the necessary resources despite these tight fiscal times. The victor will reap more than scientific glory. Companies use 57% of the computing time on the machines on the Top500 List, looking to speed product design and gain other competitive advantages, Dongarra says. So government officials see exascale computing as giving their industries a leg up. That's particularly true for chip companies that plan to use exascale designs to improve future commodity electronics. “It will have dividends all the way down to the laptop,” says Peter Beckman, who directs the Exascale Technology and Computing Initiative at Argonne National Laboratory in Illinois. The race to provide the hardware needed for exascale computing “will be extremely competitive,” Beckman predicts, and developing software and networking technology will be equally important, according to Dongarra. Even so, many observers think that the U.S. track record and the current alignment of its political and scientific forces makes it America's race to lose. Whatever happens, U.S. scientists are unlikely to be blindsided. The task of building the world's first exascale computer is so complex, Simon says, that it will be nearly impossible for a potential winner to hide in the shadows and come out of nowhere to claim the prize. _______________________________________________ Beowulf mailing list, [email protected] sponsored by Penguin Computing To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
