[issue8552] msilib can't create large CAB files

2013-03-24 Thread R. David Murray 

R. David Murray added the comment:

It looks like it turned out that there is nothing specific in this issue that 
isn't covered by issue 2399.

--
nosy: +r.david.murray
resolution:  - duplicate
stage:  - committed/rejected
status: open - closed
superseder:  - Patches for Tools/msi

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue8552
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue8552] msilib can't create large CAB files

2010-04-29 Thread Bill Janssen 

Bill Janssen bill.jans...@gmail.com added the comment:

Another bit of info.  It's the frequent commits that seem to fix the problem; 
when I comment those out of the __del__ method, it fails as before.  I also 
notice that the finished installer is about twice the size of the two data 
files in it (the CAB file and my zip file).  Seems like a lot of overhead.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue8552
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue8552] msilib can't create large CAB files

2010-04-29 Thread Bill Janssen 

Bill Janssen bill.jans...@gmail.com added the comment:

Ah, found the size problem -- I was measuring something in 512 blocks not 1KB 
blocks.  Never mind.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue8552
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue8552] msilib can't create large CAB files

2010-04-28 Thread Martin v . Löwis 

Martin v. Löwis mar...@v.loewis.de added the comment:

You could also try to commit the MSI file in-between, which may release memory.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue8552
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue8552] msilib can't create large CAB files

2010-04-28 Thread Bill Janssen 

Bill Janssen bill.jans...@gmail.com added the comment:

Yes, I've tried that.  No joy.  Right now I'm trying an approach which
packages each top-level directory as a separate cab.

What I'm finding is that if I get up around 4200 files, it breaks,
regardless of the file sizes.  Out of curiosity, how many files are in the
Python MSI file?

Bill

On Tue, Apr 27, 2010 at 11:37 PM, Martin v. Löwis 
rep...@bugs.python.orgwrote:


 Martin v. Löwis mar...@v.loewis.de added the comment:

 You could also try to commit the MSI file in-between, which may release
 memory.

 --

 ___
 Python tracker rep...@bugs.python.org
 http://bugs.python.org/issue8552
 ___


--
Added file: http://bugs.python.org/file17118/unnamed

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue8552
___Yes, I#39;ve tried that.  No joy.  Right now I#39;m trying an approach 
which packages each top-level directory as a separate 
cab.divbr/divdivWhat I#39;m finding is that if I get up around 4200 
files, it breaks, regardless of the file sizes.  Out of curiosity, how many 
files are in the Python MSI file?br
divbr/divdivBillbrbrdiv class=gmail_quoteOn Tue, Apr 27, 2010 
at 11:37 PM, Martin v. Löwis span dir=ltrlt;a 
href=mailto:rep...@bugs.python.org;rep...@bugs.python.org/agt;/span 
wrote:brblockquote class=gmail_quote style=margin:0 0 0 
.8ex;border-left:1px #ccc solid;padding-left:1ex;
div class=imbr
Martin v. Löwis lt;a 
href=mailto:mar...@v.loewis.de;mar...@v.loewis.de/agt; added the 
comment:br
br
/divYou could also try to commit the MSI file in-between, which may release 
memory.br
br
--br
divdiv/divdiv class=h5br
___br
Python tracker lt;a 
href=mailto:rep...@bugs.python.org;rep...@bugs.python.org/agt;br
lt;a href=http://bugs.python.org/issue8552; 
target=_blankhttp://bugs.python.org/issue8552/agt;br
___br
/div/div/blockquote/divbr/div/div
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue8552] msilib can't create large CAB files

2010-04-28 Thread Bill Janssen 

Changes by Bill Janssen bill.jans...@gmail.com:


Removed file: http://bugs.python.org/file17118/unnamed

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue8552
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue8552] msilib can't create large CAB files

2010-04-28 Thread Bill Janssen 

Bill Janssen bill.jans...@gmail.com added the comment:

I've now been able to build my installer.

I applied Travis Oliphant's patch from http://bugs.python.org/issue2399 to 
Lib/msilib/__init__.py, then added a __del__ method to the Directory class:

def __del__(self):
if self._numfiles_wo_commit  0:
self.db.Commit()
self.db = None
self.keyfiles = None
self.cab = None
self.ids = None

This seems to release enough memory that I can deal with a larger number of 
files.

I don't think there's a good way to write a unit test for this, though.  
Success or failure will depend on the machine you're running it on, I think.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue8552
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue8552] msilib can't create large CAB files

2010-04-27 Thread Bill Janssen 

New submission from Bill Janssen bill.jans...@gmail.com:

I'm trying to create a CAB file containing about 69MB of data, in 4555 files.  
msilib fails in CAB.commit():

$ python build-msi-installer.py /c/UpLib/1.7.9 ~/uplib 1.7.9 ./uplib-1.7.9.msi
c:\Documents and Settings\wjanssen\uplib\win32\uplib-1.7.9.msi
install_location c:/UpLib/1.7.9
c:\docume~1\wjanssen\locals~1\temp\tmpu5dwz6 68943045
gc: collecting generation 0...
gc: objects in each generation: 670 702 8255
gc: done, 0.s elapsed.
gc: collecting generation 0...
gc: objects in each generation: 707 1364 8255
gc: done, 0.s elapsed.
gc: collecting generation 0...
gc: objects in each generation: 699 2031 8255
gc: done, 0.s elapsed.
gc: collecting generation 0...
gc: objects in each generation: 794 2680 8255
gc: done, 0.s elapsed.
gc: collecting generation 0...
gc: objects in each generation: 730 3373 8255
gc: done, 0.s elapsed.
gc: collecting generation 0...
gc: objects in each generation: 765 4018 8255
gc: done, 0.s elapsed.
gc: collecting generation 0...
gc: objects in each generation: 741 4697 8255
gc: done, 0.s elapsed.
(1, 4555, None, '#prereqs', None, None)
Traceback (most recent call last):
  File build-msi-installer.py, line 780, in module
p.run()
  File build-msi-installer.py, line 243, in run
self.add_files()
  File build-msi-installer.py, line 312, in add_files
cab.commit(self.db)
  File c:\Python26\lib\msilib\__init__.py, line 223, in commit
[(1, self.index, None, #+self.name, None, None)])
  File c:\Python26\lib\msilib\__init__.py, line 97, in add_data
v = db.OpenView(SELECT * FROM `%s` % table)
_msi.MSIError: 1: 2229 2: c:\Documents and 
Settings\wjanssen\uplib\win32\uplib-1.7.9.msi 3: Media 4: SELECT * FROM `Media` 
gc: collecting generation 2...
gc: objects in each generation: 416 5351 8254
gc: done, 0.s elapsed.
$

(I added a bit of code to print out the Media table record, and the size of the 
file created with FCICreate.)

This works fine if I choose a slightly smaller subset, such as 
/c/UpLib/1.7.9/lib, for instance.  So I'm guessing this is a memory problem; at 
some point there's just not enough memory to open the MSI file and load the 
Media table.

I opened the MSI file with orca and sure enough, there is no Media table in it. 
 Not sure how to take that, though;  perhaps the table only appears after a 
record has been written to it.

Another reason to allow multiple CAB files in a single installer?

--
components: Library (Lib), Windows
messages: 104379
nosy: janssen, loewis
priority: normal
severity: normal
status: open
title: msilib can't create large CAB files
type: resource usage
versions: Python 2.6

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue8552
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue8552] msilib can't create large CAB files

2010-04-27 Thread Martin v . Löwis 

Martin v. Löwis mar...@v.loewis.de added the comment:

 Another reason to allow multiple CAB files in a single installer?

I'm still curious as to what the first reason is.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue8552
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue8552] msilib can't create large CAB files

2010-04-27 Thread Bill Janssen 

Bill Janssen bill.jans...@gmail.com added the comment:

Ummm, just in case the packager wanted to.

In my case, I was putting the files which were registered as part of the 
installation in one CAB file, and another set of temporary files which were 
used by some of the installation scripts, but not part of the installation file 
set, in a second CAB file.  I've since changed that second package to a zip 
file.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue8552
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue8552] msilib can't create large CAB files

2010-04-27 Thread Bill Janssen 

Bill Janssen bill.jans...@gmail.com added the comment:

So, I subclassed msilib.CAB, and re-wrote commit() so that the CAB file is 
created in a different process, a la Tools/msi/msilib.py.  Still have the same 
problem, though.  So now I'm thinking it's not a memory problem, but I'm 
wondering if there's something about the length of the path names in the CAB 
file, or some such.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue8552
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

CAB files

2008-08-08 Thread Virgil Stokes 

I would appreciate python code for creating *.cab files.

--V. Stokes
--
http://mail.python.org/mailman/listinfo/python-list

Re: CAB files

2008-08-08 Thread Tim Golden 

Virgil Stokes wrote:

I would appreciate python code for creating *.cab files.


I'm not aware of any existing modules for creating
CABs. I thought there were some MS API calls for
doing that kind of thing but a quick search hasn't
shown anything up.

You may have to roll your own:

http://support.microsoft.com/kb/310618

TJG
--
http://mail.python.org/mailman/listinfo/python-list

Re: CAB files

2008-08-08 Thread Thomas Heller 
Virgil Stokes schrieb:
 I would appreciate python code for creating *.cab files.
 
 --V. Stokes

Here is some code that I have still laying around.  It has never been
used in production and I do not know what you can do with the cab files
it creates, but I have been able to create a cab and open it with winzip.

Thomas

cab.py
from ctypes import *
import sys, os, tempfile, glob

BOOL = c_int
ULONG = c_ulong
UINT = c_uint
USHORT = c_ushort

class ERF(Structure):
_fields_ = [(erfOper, c_int),
(erfType, c_int),
(fError, BOOL)]

CB_MAX_CHUNK =   32768
CB_MAX_DISK =0x7ff
CB_MAX_FILENAME =   256
CB_MAX_CABINET_NAME =   256
CB_MAX_CAB_PATH =   256
CB_MAX_DISK_NAME =  256

class CCAB(Structure):
_fields_ = [
(cb, ULONG),  # size available for cabinet on this 
media
(cbFolderThresh, ULONG),  # Thresshold for forcing a new Folder
(cbReserveCFHeader, UINT),   # Space to reserve in CFHEADER
(cbReserveCFFolder, UINT),   # Space to reserve in CFFOLDER
(cbReserveCFData, UINT), # Space to reserve in CFDATA
(iCab, c_int),# sequential numbers for cabinets
(iDisk, c_int),   # Disk number
(fFailOnIncompressible, c_int), # TRUE = Fail if a block is 
incompressible
(setID, USHORT),   # Cabinet set ID
(szDisk, c_char * CB_MAX_DISK_NAME),# current disk name
(szCab, c_char * CB_MAX_CABINET_NAME),  # current cabinet name
(szCabPath, c_char * CB_MAX_CAB_PATH),  # path for creating cabinet
]

cab = cdll.cabinet

class HFCI(object):
_handle = 0
_as_parameter_ = property(lambda self: self._handle)

def __init__(self, fnm, verbose=1):
self.verbose = verbose
self.erf = ERF()
ccab = self.ccab = CCAB()
ccab.cb = 1
ccab.cbFolderThresh = 10
ccab.szCab = os.path.basename(fnm)
dirname = os.path.dirname(fnm)
if not dirname:
dirname = .
ccab.szCabPath = dirname + \\
self._init_callbacks()
self._handle = cab.FCICreate(byref(self.erf),
 self.pfn_filedest,
 cdll.msvcrt.malloc,
 cdll.msvcrt.free,
 cdll.msvcrt._open,
 cdll.msvcrt._read,
 cdll.msvcrt._write,
 cdll.msvcrt._close,
 cdll.msvcrt._lseek,
 cdll.msvcrt._unlink,
 self.pfn_gettempfnm,
 byref(ccab),
 None)

def _init_callbacks(self):
self.pfn_gettempfnm = CFUNCTYPE(c_int, c_void_p, c_int, 
c_void_p)(self._gettempfnm)
self.pfn_filedest = CFUNCTYPE(c_int)(self._filedest)
self.pfn_status = CFUNCTYPE(c_int, c_int, c_uint, c_uint, 
c_void_p)(self._status)
self.pfn_getnextcab = CFUNCTYPE(c_int)(self._getnextcab)
self.pfn_getopeninfo = CFUNCTYPE(c_int, c_char_p)(self._getopeninfo)

def _getopeninfo(self, fnm):
if self.verbose:
print File, fnm
return cdll.msvcrt._open(fnm, os.O_BINARY | os.O_RDONLY)

def _status(self, typeStatus, cb1, cb2, pv):
return 0

def _filedest(self):
return 0

def _getnextcab(self):
return 0

def _gettempfnm(self, pszTempName, cbTempName, pv):
# same as tempfile.mktemp(), but this is deprecated
fh, fnm = tempfile.mkstemp()
os.close(fh)
os.remove(fnm)
cdll.msvcrt.strcpy(pszTempName, fnm)
return 1

def AddFile(self, src, dst=None, compressed=0):
if dst is None:
dst = os.path.basename(src)
cab.FCIAddFile(self,
   src,
   dst,
   0, # fExecute
   self.pfn_getnextcab,
   self.pfn_status,
   self.pfn_getopeninfo,
   compressed)

def Close(self):
if self._handle != 0:
cab.FCIFlushCabinet(self,
0, # fGetNextCab
self.pfn_getnextcab,
self.pfn_status)
cab.FCIDestroy(self)
self._handle = 0

if __name__ == __main__:
import os, glob

hfci = HFCI(my-first.cab, verbose=1)

files = glob.glob(r.\cab\*.*)

for fnm in files:
hfci.AddFile(fnm)

hfci.Close()
eof
--
http://mail.python.org/mailman/listinfo/python-list

Re: CAB files

2008-08-08 Thread oyster 
there is 2 files: text2pdf.py and cab.py
but I get a cab, in which there is a file text2pdf.py in it, but
cab.py is created as a directory!
[your cab.py starts]
blahblah

if __name__ == __main__:
   import os, glob

   hfci = HFCI(my-first.cab, verbose=1)

   files = glob.glob(r*.py)

   for fnm in files:
   hfci.AddFile(fnm)

   hfci.Close()
[/your cab.py ends]


 From: Thomas Heller [EMAIL PROTECTED]
 To: python-list@python.org
 Date: Fri, 08 Aug 2008 16:29:10 +0200
 Subject: Re: CAB files
 Virgil Stokes schrieb:
  I would appreciate python code for creating *.cab files.
 
  --V. Stokes

 Here is some code that I have still laying around.  It has never been
 used in production and I do not know what you can do with the cab files
 it creates, but I have been able to create a cab and open it with winzip.

--
http://mail.python.org/mailman/listinfo/python-list

CAB files manipulation API (again).

2005-06-28 Thread Isaac Rodriguez 
Hi,

I am sorry to post this question again, but when I did it the other day, my 
news reader got stucked downloading new messages, and it has been that way 
for a few days. It still gets stucked if I try to download old messages.

Anyway, does anyone know of a Python module, API, etc. that allows to 
manipulate CAB files?

Thanks,

-- 
Isaac Rodriguez
SWE Autodesk.

There are 10 types of people.
Those who undertand binary, and those who don't 


-- 
http://mail.python.org/mailman/listinfo/python-list

Re: CAB files manipulation API (again).

2005-06-28 Thread Robert Kern 
Isaac Rodriguez wrote:
 Hi,
 
 I am sorry to post this question again, but when I did it the other day, my 
 news reader got stucked downloading new messages, and it has been that way 
 for a few days. It still gets stucked if I try to download old messages.

Google is your friend.

http://groups-beta.google.com/group/comp.lang.python/browse_thread/thread/e9a2237d820a4964

-- 
Robert Kern
[EMAIL PROTECTED]

In the fields of hell where the grass grows high
  Are the graves of dreams allowed to die.
   -- Richard Harter

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Python API to manipulate CAB files.

2005-06-23 Thread Thomas Heller 
Konstantin Veretennicov [EMAIL PROTECTED] writes:

 On 6/22/05, Peter Maas [EMAIL PROTECTED] wrote:
 Isaac Rodriguez schrieb:
  Does anyone know of a Python API to manipulate CAB files?

 I guess you'll have to interface with setupapi.dll
 (SetupIterateCabinet) via ctypes, or with Microsoft Cabinet utilities
 via subprocess module. Neither is what people call fun...

 - kv
Probably not what you mean with 'manipulate CAB files', but it may give
a start with ctypes and cab:

http://starship.python.net/crew/theller/moin.cgi/CreateCab

Thomas
-- 
http://mail.python.org/mailman/listinfo/python-list

Python API to manipulate CAB files.

2005-06-22 Thread Isaac Rodriguez 
Does anyone know of a Python API to manipulate CAB files?

Thanks,

-- 
Isaac Rodriguez
SWE Autodesk.

There are 10 types of people.
Those who undertand binary, and those who don't 


-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Python API to manipulate CAB files.

2005-06-22 Thread Peter Maas 
Isaac Rodriguez schrieb:
 Does anyone know of a Python API to manipulate CAB files?

If there is a Windows API you can probybly call it from Python
using Mark Hammond's Win32 extensions.

-- 
---
Peter Maas,  M+R Infosysteme,  D-52070 Aachen,  Tel +49-241-93878-0
E-mail 'cGV0ZXIubWFhc0BtcGx1c3IuZGU=\n'.decode('base64')
---
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Python API to manipulate CAB files.

2005-06-22 Thread Konstantin Veretennicov 
On 6/22/05, Peter Maas [EMAIL PROTECTED] wrote:
 Isaac Rodriguez schrieb:
  Does anyone know of a Python API to manipulate CAB files?

I guess you'll have to interface with setupapi.dll
(SetupIterateCabinet) via ctypes, or with Microsoft Cabinet utilities
via subprocess module. Neither is what people call fun...

- kv
-- 
http://mail.python.org/mailman/listinfo/python-list