liangsibin created CASSANDRA-13266:
--------------------------------------
Summary: Bulk loading sometimes is very slow?
Key: CASSANDRA-13266
URL: https://issues.apache.org/jira/browse/CASSANDRA-13266
Project: Cassandra
Issue Type: Improvement
Reporter: liangsibin
When I bulkload sstable created with CQLSSTableWriter, it's sometimes very
slow.
use 2 nodes write SSTable and bulkload
1、Use CQLSSTableWriter create SSTable （60 threads）
2、When the directory over 100000 rows，bulkload the directory （20 threads）
the normal bulkload speed is about 70M/s per node，and bulkload 141G SStables
cost 90 minutes but sometimes is very slow，the same data cost 4 hours why?
here is the code bulkload sstable
public class JmxBulkLoader {
static final Logger LOGGER = LoggerFactory.getLogger(JmxBulkLoader.class);
private JMXConnector connector;
private StorageServiceMBean storageBean;
private Timer timer = new Timer();
public JmxBulkLoader(String host, int port) throws Exception {
connect(host, port);
}
private void connect(String host, int port) throws IOException,
MalformedObjectNameException {
JMXServiceURL jmxUrl = new JMXServiceURL(
String.format("service:jmx:rmi:///jndi/rmi://%s:%d/jmxrmi", host, port));
Map<String, Object> env = new HashMap<String, Object>();
connector = JMXConnectorFactory.connect(jmxUrl, env);
MBeanServerConnection mbeanServerConn =
connector.getMBeanServerConnection();
ObjectName name = new
ObjectName("org.apache.cassandra.db:type=StorageService");
storageBean = JMX.newMBeanProxy(mbeanServerConn, name,
StorageServiceMBean.class);
}
public void close() throws IOException {
connector.close();
}
public void bulkLoad(String path) {
LOGGER.info("begin load data to cassandra " + new
Path(path).getName());
timer.start();
storageBean.bulkLoad(path);
timer.end();
LOGGER.info("bulk load took " + timer.getTimeTakenMillis() +
"ms, path: " + new Path(path).getName());
}
}
bulkload thread
public class BulkThread implements Runnable {
private String path;
private String jmxHost;
private int jmxPort;
public BulkThread(String path, String jmxHost, int jmxPort) {
super();
this.path = path;
this.jmxHost = jmxHost;
this.jmxPort = jmxPort;
}
@Override
public void run() {
JmxBulkLoader bulkLoader = null;
try {
bulkLoader = new JmxBulkLoader(jmxHost, jmxPort);
bulkLoader.bulkLoad(path);
} catch (Exception e) {
e.printStackTrace();
} finally {
if (bulkLoader != null)
try {
bulkLoader.close();
bulkLoader = null;
} catch (IOException e) {
e.printStackTrace();
}
}
}
}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
