[issue37241] Item Count Error in Shelf

[SilentGhost]

Change by SilentGhost :


--
stage:  -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37241] Item Count Error in Shelf

[Jesse Bacon]

Change by Jesse Bacon :


--
resolution:  -> third party

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37241] Item Count Error in Shelf

[Jesse Bacon]

Jesse Bacon  added the comment:

I was using anaconda distribution on OSX.  It failed for 3.6 and 3.7.  I pulled 
off anaconda and compiled from source and the script executed correctly 
regardless of whether or not "--enable-optimizations" was set.  Anaconda claims 
to be geared towards scientists so this is alarming.  Thank you for your time.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37241] Item Count Error in Shelf

[Eric V. Smith]

Eric V. Smith  added the comment:

After fixing a missing import (import urllib.request), this is what I get:

$ /usr/local/bin/python3.6 pbr37241_Jesse_Bacon.py 
Fetching nvdcve-1.0-2019.json.gz
Storing Gzipped File
Loading JSON Content
4275 records
4275 unique records
Creating Shelve: cve_2019.shelf
Assembling Big Dictionary of 2019 Data in shelve
shelve reports 4275 unique records
Extracting data by keys from shelve
4275 extracted records
Number of missing records 0
data match

Are you seeing failures?

This is on a python3.6 that I compiled from source on an old Fedora box.

What OS are you using?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37241] Item Count Error in Shelf

[Jesse Bacon]

Jesse Bacon  added the comment:

Eric,

The interpreter said something about passing a negative value when I converted 
the db.keys to a list.  I have attached a script in txt format and a Jupyter 
notebook for further analysis.  I apologize for posting images,  I just saw 
your note.  I'll go ahead and look at the shelve source while you determine if 
this information is sufficient. Thank you for your time.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37241] Item Count Error in Shelf

[Jesse Bacon]

Change by Jesse Bacon :


Added file: https://bugs.python.org/file48415/pbr37241_Jesse_Bacon.py.txt

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37241] Item Count Error in Shelf

[Jesse Bacon]

Change by Jesse Bacon :


Added file: https://bugs.python.org/file48414/Python Proof.ipynb

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37241] Item Count Error in Shelf

[Eric V. Smith]

Eric V. Smith  added the comment:

This still isn't an example we can copy and paste to reproduce, so I'm going to 
be unable to help you. Sorry.

Again: please don't post images, for the reasons I previously stated.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37241] Item Count Error in Shelf

[Jesse Bacon]

Change by Jesse Bacon :


Added file: https://bugs.python.org/file48413/ShelfKeys.png

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37241] Item Count Error in Shelf

[Jesse Bacon]

Jesse Bacon  added the comment:

I am missing keys, when extracting the data back out with todays NVD pull.
---
KeyError  Traceback (most recent call last)
~/anaconda3/lib/python3.6/shelve.py in __getitem__(self, key)
110 try:
--> 111 value = self.cache[key]
112 except KeyError:

KeyError: 'CVE-2019-1842'

During handling of the above exception, another exception occurred:

KeyError  Traceback (most recent call last)
 in 
  1 results = []
  2 for x in raw_cves:
> 3 results.append(db[x])

~/anaconda3/lib/python3.6/shelve.py in __getitem__(self, key)
111 value = self.cache[key]
112 except KeyError:
--> 113 f = BytesIO(self.dict[key.encode(self.keyencoding)])
114 value = Unpickler(f).load()
115 if self.writeback:

KeyError: b'CVE-2019-1842'

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37241] Item Count Error in Shelf

[Eric V. Smith]

Eric V. Smith  added the comment:

Please do not post images: we can't copy and paste from them, and they're 
unfriendly to visually impaired users.

Can you create code that reproduces this? A small example, with no external 
dependencies would be best. Please attach the reproducer as a text file.

--
nosy: +eric.smith

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37241] Item Count Error in Shelf

[Jesse Bacon]

New submission from Jesse Bacon :

I have loaded the National Vulnerability Database from NIST for 2019 and it 
includes 3989 JSON Documents.  This data I have placed in a shelf.  when I run 
len(db.keys()) I get 3658.  len(set(cves)) == 3989 : True

When I extract the data from the shelf I have the right amount of records, 
3989.  I tested on python 3.7.3 and Python 3.6.5.  I am concerned this is going 
to ruin a metric in a security report.  For example, A risk exposure report may 
use the number of keys in a yearly vulnerability db as the baseline for a risk 
calculation which contrasts the number of patched CVE's.  

nvdcve-1.0-2019.json

--
components: Library (Lib)
files: KeyCount.png
messages: 345290
nosy: jessembacon
priority: normal
severity: normal
status: open
title: Item Count Error in Shelf
type: behavior
versions: Python 3.6, Python 3.7
Added file: https://bugs.python.org/file48411/KeyCount.png

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com