FYI
From: [email protected]
To: [email protected]
Subject: RE: [FastBit-users] Segmentation fault with multiple threads
Date: Fri, 26 Oct 2012 22:16:12 +0530
JohnPFA for my java test code, depending on the system RAM play with the NO# of
threads(THREADS=150), and NO# of iterations (ITERATIONS=1000).you will get
following errors after few queries executed.
Warning -- column[fastbit.c]::softWriteLock --
pthread_rwlock_trywrlock(0x1001dae60) for unloadIndex returned 16 (Resource
busy)
SunilKalva
> Date: Fri, 26 Oct 2012 07:58:34 -0700
> From: [email protected]
> To: [email protected]
> Subject: Re: [FastBit-users] Segmentation fault with multiple threads
>
> Thanks for the information, but I need more information to do anything
> about your problems. Please provide a test case with java code,
> instructions, and test data.
>
> John
>
>
> On 10/26/12 7:49 AM, sunil kalva wrote:
> >
> > Iam using "https://github.com/gauravagarwal/fastbijava"; as my java
> > interface,
> > i still getting following errors with multithread calls, iam closing
> > fastbit object after every query.
> >
> >
> > ===================================================================================================
> >
> > Invalid memory access of location 0x0 rip=0x11273f86c
> >
> >
> > Warning -- fileManager::unload yields to another thread already
> > waiting for memory ...
> >
> > Warning -- fileManager::storage::ctor unable to find 121,689,400 bytes
> > of space in memory
> >
> > Warning -- column[fastbit.gender]::softWriteLock --
> > pthread_rwlock_trywrlock(0x1001dae60) for unloadIndex returned 16
> > (Resource busy)
> >
> >
> > Warning -- fileManager::softWriteLock failed to acquire the write lock
> > (0x10dd47fc0) for fileManager::clear, error code = 16
> >
> >
> > Warning -- part[fastbit]::gainWriteAccess --
> > pthread_rwlock_trywrlock(0x102260020) for freeRIDs returned 16
> > (Resource busy)
> >
> > Warning -- part[fastbit]::gainWriteAccess --
> > pthread_rwlock_trywrlock(0x102260020) for freeRIDs returned 16
> > (Resource busy)
> >
> > ===================================================================================================
> >
> >
> > Please suggest me how should i fix these problems.
> >
> >
> > SunilKalva
> >
> >
> >
> >> Date: Thu, 25 Oct 2012 20:36:07 -0700
> >> From: [email protected]
> >> To: [email protected]
> >> CC: [email protected]
> >> Subject: Re: [FastBit-users] Segmentation fault with multiple threads
> >>
> >> Hi, Sunil,
> >>
> >> Looks like the memory is not freed somewhere. The warning messages
> >> you included all point to this problem. This could happen in a a few
> >> different ways,
> >>
> >> - neglecting to destroy the query objects (by calling destroy_query)
> >> - memory leak in the JNI code
> >> - memory leak in the C API
> >> - memory leak in the C++ code (the core of FastBit)
> >>
> >> I would encourage you to destroy the query objects as soon as possible
> >> in your program. The C API layer and the JNI code are not exercised
> >> nearly as much as the C++ code, so it is very possible that they
> >> contain some problems. I am willing to look into this, but would need
> >> some help from you. More specifically, I would need a small test case
> >> in java along with some test data. You can email them to me or put
> >> them in a place for me to pick them up.
> >>
> >> Thanks.
> >>
> >> John
> >>
> >>
> >>
> >> On 10/25/12 3:11 PM, sunil kalva wrote:
> >> > Hi
> >> > i get segmentation fault error, if i run queries from multiple threads
> >> > using java API. and if restart it works for me for few queries and
> >> > again i face same problem.
> >> >
> >> > and i also following errors very frequently ...!
> >> >
> >> > Warning -- fileManager::storage::ctor unable to find 28 bytes of space
> >> > in memory
> >> >
> >> > Warning -- part::init failed to read mask file
> >> > "/data/test/fastbit/2012-10-16-00/fastbit/-part.msk", assume all rows
> >> > (32394899) are active
> >> >
> >> > com.globalpark.fastbit.exception.FastbitPartitionException:
> >> > storage::ctor(memory) failed
> >> >
> >> >
> >> > Warning -- part[fastbit]::gainWriteAccess --
> >> > pthread_rwlock_trywrlock(0x102260020) for freeRIDs returned 16
> >> > (Resource busy)
> >> >
> >> > Warning -- part[fastbit]::gainWriteAccess --
> >> > pthread_rwlock_trywrlock(0x102260020) for freeRIDs returned 16
> >> > (Resource busy)
> >> >
> >> > Warning -- fileManager::softWriteLock failed to acquire the write lock
> >> > (0x10dd47fc0) for fileManager::clear, error code = 16
> >> >
> >> >
> >> > t
> >> > SunilKalva
> >> >
> >> >
> >> > _______________________________________________
> >> > FastBit-users mailing list
> >> > [email protected]
> >> > https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users
> >> >
import com.globalpark.fastbit.FastbitPartition;
import com.globalpark.fastbit.FastbitTable;
import com.globalpark.fastbit.FastbitTablex;
import java.io.File;
import java.io.FileWriter;
import java.util.ArrayList;
import java.util.List;
import java.util.Random;
/**
* User: sunilkalva
* Date: 26/10/12
*/
public class FBDataPopulator {
protected static long[] salaries = new long[]{25000, 50000, 75000, 100000};
public static List<String[]> queries = new ArrayList<String[]>();
private static String csvFile = "/data/monetization/tfs/sunil/load.csv";
private static String destFolder = "/data/monetization/tfs/sunil/";
private static String schema =
"a:int,b:int,c:int,d:int,e:int,f:int,g:int,h:int,i:int,j:int";
private static int THREADS = 250;
private static int ITERATIONS = 1000;
static {
queries.add(new String[]{"sum(d) salary", "a > %d and b = %d"});
queries.add(new String[]{"sum(d) salary", "f < %d and g > %d"});
queries.add(new String[]{"sum(d) salary", "c in (%d,%d)"});
}
public static void generateLoad(int rows) throws Exception {
long b = System.currentTimeMillis();
FileWriter writer = new FileWriter(new File(csvFile));
for (int i = 1; i <= rows; i++) {
writer.write(String.format("%d,%d,%d,%d,%d,%d,%d,%d,%d,%d\n",
random(5000), random(5000), random(5000),
salaries[random(salaries.length - 1)], salaries[random(salaries.length - 1)],
random(5000), random(5000), random(5000),
salaries[random(salaries.length - 1)], salaries[random(salaries.length - 1)]));
}
writer.close();
System.out.println("File created in " + (System.currentTimeMillis() -
b));
}
public static long importTable(String csvFile, String destFolder, String
schema, int size) {
long b = System.currentTimeMillis();
FastbitTablex tablex = new FastbitTablex();
tablex.importCSV(csvFile, destFolder, schema, size);
System.out.println("Table created in " + (System.currentTimeMillis() -
b));
b = System.currentTimeMillis();
FastbitPartition partition = new FastbitPartition(destFolder);
FastbitTable table = new FastbitTable(partition);
table.buildIndexes("<binning none/><encoding range/>");
System.out.println("Indexes created in " + (System.currentTimeMillis()
- b));
return table.nrows();
}
public static void runQuery(String select, String where) {
long b = System.currentTimeMillis();
FastbitPartition partition = new FastbitPartition(destFolder);
FastbitTable table = new FastbitTable(partition);
table.select(select, where);
System.out.println("Time for query (" + select + ", " + where + ") in
" + (System.currentTimeMillis() - b));
}
protected static int random(int max) {
Random random = new Random();
return random.nextInt(max);
}
public static void runLoadTest() throws Exception {
Thread[] workers = new Thread[THREADS];
for (int i = 0; i < THREADS; i++) {
workers[i] = new FBThread(ITERATIONS);
}
long b = System.currentTimeMillis();
for (int i = 0; i < THREADS; i++) {
workers[i].start();
}
for (int i = 0; i < THREADS; i++) {
workers[i].join();
}
System.out.println("==================Total Time = " +
(System.currentTimeMillis() - b));
}
public static boolean deleteFiles(String folderPath) {
boolean retValue = true;
File file = new File(folderPath);
File[] files = file.listFiles();
for (File idxFile : files) {
retValue = retValue && idxFile.delete();
}
return retValue;
}
private static class FBThread extends Thread {
private int cont;
private FBThread(int cont) {
this.cont = cont;
}
public void run() {
for (int i = 1; i <= cont; i++) {
String[] strings = nextQuery();
runQuery(strings[0], String.format(strings[1], random(5000),
random(5000)));
}
}
public String[] nextQuery() {
Random random = new Random();
return queries.get(random.nextInt(queries.size()));
}
}
public static void main(String[] args) throws Exception {
int buffer = 10000;
deleteFiles(destFolder);
generateLoad(buffer * 35);
importTable(csvFile, destFolder, schema, buffer);
runLoadTest();
}
}_______________________________________________
FastBit-users mailing list
[email protected]
https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users
