Hi Angie,
We were able to get our queries to work after making changes to
getSnpSeqFileOffset
function in hgc.c. It appears that the original code does not work in a 32 bit
environment. Here are the changes we needed to do to have it work in our
setting:
1) Line 23137 change declaration of offset
=== was
long offset = 0;
=== to
long long offset = 0;
2) Line 23135 change call of conversion of returned sql offset:
==== was
offset = sqlUnsignedLong(row[0])
=== to
offset = sqlLongLong(row[0])
I don't know if there are other 32/64 issues but it may be helpful to others
working in a 32 bit environment to have these adjustments made in the code.
Thanks for your help in locating this problem.
johnh...
________________________________
From: John Hayward
Sent: Tuesday, June 08, 2010 9:51 AM
To: Angie Hinrichs; Kyle Tretina
Cc: UCSC
Subject: RE: [Genome] Fwd: Fwd: Problem with Mirror
The problem seems to be a 64/32 bit issue.
I rebuilt the binaries and observed the flags for building (here is a snippet)
....
gcc -O -g -Wall -Werror -Wformat -Wimplicit -Wreturn-type -Wuninitialized
-D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -D_GNU_SOURCE -DMACHTYPE_i486
-Wall -Werror -Wformat -Wimplicit -Wreturn-type -Wuninitialized -I../inc
-I../../inc -I../../../inc -I../../../../inc -I../../../../../inc -o
hgLoadWiggle.o -c hgLoadWiggle.c
....
The error on the web page we see is:
=====
Expected FASTA header, got this line:
ATAATACATT TTAATTTTCT AGAAAGTAAT GGTATGAAGT TCCAACTTTT TTTAAAAAAG ACTTTTTTTC
at offset 344476137 in file /gbdb/hg18/snp/snp130.fa
======
The offset you are seeing from the database is 17524345321
Here is a bc script:
=====
bc 1.06.94
Copyright 1991-1994, 1997, 1998, 2000, 2004, 2006 Free Software Foundation, Inc.
This is free software with ABSOLUTELY NO WARRANTY.
For details type `warranty'.
obase=16
15
F
344476137 // this is the offset reported in the error
148849E9
17524345321 // this is the offset according to the database
4148849E9 // if you truncate this value to 32 bits (last 8 hex digits) then
you get the value
// above
======
This shows that the offset in the database is more than 32 bits and when
truncated to 32 bits is the same value reported in the error message.
The operating system is:
Linux ubuntu 2.6.31-302-ec2 #7-Ubuntu SMP Tue Oct 13 19:06:04 UTC 2009 i686
GNU/Linux
Ubuntu 10.04 (lucid)
Now we need to find out where the offset is being truncated - it seems that the
flags for file offsets are in place but I wonder if there are some other 32 bit
issues in the source.
I recall when originally building there were some warnings which required some
changes to compile. I think the next step is to repeat those steps and make
sure there are no 32/64 bit issues.
johnh...
________________________________
From: Angie Hinrichs [[email protected]]
Sent: Monday, June 07, 2010 5:26 PM
To: Kyle Tretina
Cc: UCSC; John Hayward
Subject: Re: [Genome] Fwd: Fwd: Problem with Mirror
Hi Kyle,
I think I know what it is now: snp130.fa is an extremely large file, and either
your operating system can't handle its size or perhaps the flags
"-D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE" were omitted from the build? (our
makefiles all include kent/src/inc/common.mk, which defines the variable
HG_DEFS to include those, and our compile rule uses $HG_DEFS)
snp130.fa is a whopping 18,668,682,162 bytes. (18GB or 17GiB depending on how
you count) If your filesystem can handle only 32-bit file offsets (or even
36), that is just too big.
rs72477211's offset ("select file_offset from snp130Seq where acc =
'rs72477211'") is 17524345321, which is (16 * 1024 * 1024 * 1024) + 344476137,
and 344476137 is the file offset that you see in the error message because the
top bit of the offset is lost.
What operating system and compiler are you using?
Angie
----- "Kyle Tretina" <[email protected]> wrote:
> From: "Kyle Tretina" <[email protected]>
> To: "Angie Hinrichs" <[email protected]>, "UCSC" <[email protected]>,
> "John Hayward" <[email protected]>
> Sent: Sunday, June 6, 2010 10:59:57 AM GMT -08:00 US/Canada Pacific
> Subject: Re: [Genome] Fwd: Fwd: Problem with Mirror
>
> Hello,
>
>
Note: this was in database hg19 using the command: select settings from trackDb
where tableName = "snp130";
=======================================================================
>
+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| settings
|
+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| chimpDb panTro2
chimpOrangMacOrthoTable snp130OrthoPt2Pa2Rm2
codingAnnoLabel_snp130CodingDbSnp dbSNP
codingAnnotations snp130CodingDbSnp,
defaultGeneTracks knownGene
group varRep
hapmapPhase III
longLabel Simple Nucleotide Polymorphisms (dbSNP build 130)
macaqueDb rheMac2
maxWindowToDraw 10000000
orangDb ponAbe2
priority 100.094
shortLabel SNPs (130)
snpSeqFile /gbdb/hg18/snp/snp130.fa
track snp130
type bed 6 +
url http://www.ncbi.nlm.nih.gov/SNP/snp_ref.cgi?type=rs&rs=$$
urlLabel dbSNP:
visibility dense
|
+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
1 row in set (0.07 sec)
>
>
=======================================================================
(This is the error message for rs72477211)
>
Error(s):
* Expected FASTA header, got this line:
> ATAATACATT TTAATTTTCT AGAAAGTAAT GGTATGAAGT TCCAACTTTT TTTAAAAAAG ACTTTTTTTC
> at offset 344476137 in file /gbdb/hg18/snp/snp130.fa
=======================================================================
>
So rs242 does work. When I enter it into my mirror, I am getting the same
output as your website. I was shocked when I saw it, because this was the first
one I had entered that did not return an error message.
>
>
Kyle Tretina
Wheaton College
>
>
>
_______________________________________________
Genome maillist - [email protected]
https://lists.soe.ucsc.edu/mailman/listinfo/genome
