On Fri, Oct 7, 2011 at 1:35 PM, bob gailer <[email protected]
<mailto:[email protected]>> wrote:
On 10/7/2011 2:19 PM, Guess?!? wrote:
Hello all,
I am trying to create some test data for a search module that I
am building. Since I dont want to hit performance related issues
late in the game, I decided to create half a million records in
the DB.
My approach is create a csv file with valid UUIDs which I already
have. Read each record from csv and generate random data and do
DB insert as long the csv file has records in them. This crashes
near 100000 record insertions.
The problem is that my program eventually fails with this error.
Any ideas what I can improve to make it go all the way?
Runtime Error!
Program: C:\Python27\pythonw.exe
This app has requested the Runtime to terminate it in an unusual
way. Please contact application support team
Here is the code that I wrote to do this.
=======================================================================
import pyodbc, random, datetime, uuid
#INT
conn = pyodbc.connect('DRIVER={SQL
Server};SERVER=SERVERNAME\INT_FOUNDATIONS;DATABASE=membership_service;UID=int_usr;PWD=blah')
c = conn.cursor()
# "enterprise_person_id" TEXT NOT NULL,
# read file
f = open('C:\\Users\\garora\\Desktop\\INTPPLHALFMIL.csv','r')
# Hardcoded valid seed values
# INT
programId = 50801
packageId = 46101
pkg_optionId = 40301
seed_data = "TEST_DATA"
numberPrePend = "AAAA-"
create_dt = datetime.datetime.now();
modified_dt = datetime.datetime.now();
i = 0
for each in f:
if (i == 0):
i = i + 1
continue;
(id, entrpid,) = each.split(',')
print id
id = id.replace('\n', '')
print id
id = uuid.UUID(id)
#enterprise_person_id
print entrpid
entrpid = entrpid.replace('\n', '')
print entrpid
entrpid = uuid.UUID(entrpid)
#random values for few columns
statusRandom = random.randint(1,4)
sourceRandom = random.randint(1,7)
inputChannelRandom = random.randint(1,5)
marketRandom= random.randint(1,10)
trialRandom = random.randint(0,1)
monthRandom = random.randint(3,12) # prevent month of feb
dateRandom = random.randint(1,30) # prevent 31st on invalid month
yearRandom = random.randint(2007,2010)
member_start_dt = datetime.datetime(yearRandom, monthRandom,
dateRandom,0,0,0)
current_term_start_dt = datetime.datetime(yearRandom, monthRandom,
dateRandom,0,0,0)
print "id: " + str(i)
print " programId: " + str(programId)
print " packageId: " + str(packageId)
print " pkgoptionId: " + str(pkg_optionId)
print " status id: " + str(statusRandom)
print " source id: " + str(sourceRandom)
print " input channel id: " + str(inputChannelRandom)
print " market id: " + str(marketRandom)
print " person id: " + str(id)
print "membership id: " + numberPrePend + str(i)
print " member_start_dt " + str(member_start_dt)
print " current_term_start_dt " + str(current_term_start_dt)
print " started_as_trial " + str(trialRandom)
print " enterprise_person_id " + str(entrpid)
record = (i, programId, packageId, pkg_optionId, statusRandom,
sourceRandom, inputChannelRandom,
marketRandom, str(id), numberPrePend + str(i), member_start_dt,
current_term_start_dt, trialRandom
,seed_data, seed_data, create_dt, modified_dt, str(entrpid))
print record
c.execute("""insert into Members(id, membership_program_id,
membership_package_id,
membership_pkg_option_id, membership_status_id,
membership_source_id, src_input_channel_id,
src_market_id, person_id_ref, membership_id, member_start_dt,
current_term_start_dt, started_as_trial,
created_by, modified_by, create_dt, modified_dt,
enterprise_person_id)
values (?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?)""", record)
conn.commit()
i = i + 1
c.close()
_______________________________________________
Tutor maillist [email protected] <mailto:[email protected]>
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor
--
Bob Gailer
919-636-4239 <tel:919-636-4239>
Chapel Hill NC
