Re: Dataframe to postgresql - Saving the dataframe to memory using StringIO
On Fri, Oct 23, 2020 at 3:35 AM Grant Edwards wrote: > Moving from 2.x to 3.x isn't too bad, but trying to maintain > compatiblity with both is painful. At this point, I would probably > just abandon 2.x. > Definitely. No point trying to support both when you're starting with code from a Py3 example. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: Dataframe to postgresql - Saving the dataframe to memory using StringIO
On 2020-10-22, Chris Angelico wrote:
> On Fri, Oct 23, 2020 at 12:15 AM Shaozhong SHI wrote:
>> What should I know or watch out if I decide to move from Python 2.7
>> to Python 3?
>
> Key issues? Well, for starters, you don't have to worry about whether
> your strings are Unicode or not. They just are, unless you explicitly
> need them to be bytes.
The 'bytes' thing is important. If you use serial ports, sockets, or
anything else that's raw bytes, that code will need to be examined
carefully.
The usage of the 'bytes' type in 3.x isn't at all like 2.x:
Python2:
>>> bytes('abcd')
'abcd'
>>> bytes(3)
'3'
>>> bytes([0,1,2])
'[0, 1, 2]'
Python3:
>>> bytes('abcd')
Traceback (most recent call last):
File "", line 1, in
TypeError: string argument without an encoding
>>> bytes(3)
b'\x00\x00\x00'
>>> bytes([0,1,2])
b'\x00\x01\x02
Python2:
>>> b'abcd'[2]
'c'
Python3:
>>> b'abcd'[2]
99
Moving from 2.x to 3.x isn't too bad, but trying to maintain
compatiblity with both is painful. At this point, I would probably
just abandon 2.x.
--
Grant
--
https://mail.python.org/mailman/listinfo/python-list
Re: Dataframe to postgresql - Saving the dataframe to memory using StringIO
On Fri, Oct 23, 2020 at 12:15 AM Shaozhong SHI wrote: > > Thanks, Chris. > > What should I know or watch out if I decide to move from Python 2.7 to Python > 3? > > What are the key issues? Syntax? > Keep it on-list please :) Key issues? Well, for starters, you don't have to worry about whether your strings are Unicode or not. They just are, unless you explicitly need them to be bytes. Syntax is probably going to be fine, since you're taking examples that were written for Python 3 anyway. There's a LOT more features in Python 3, since it's had another decade of development than Python 2 has had. Give it a try. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: Dataframe to postgresql - Saving the dataframe to memory using StringIO
Never worked with ZFS, it sounds interesting. Anyway, IMHO it's much more simple to save to disk also for debugging: you have not to extract the data from the db if you need to inspect them. On Thu, 22 Oct 2020 at 14:39, D'Arcy Cain wrote: > On 10/22/20 7:23 AM, Marco Sulla wrote: > > I would add that usually I do not recommend saving files on databases. I > > usually save the file on the disk and the path and mime on a dedicated > > table. > > I used to do that because backing up the database became huge. Now I use > ZFS snapshots with send/receive so the backup only copies the changed > blocks > so I keep it in the database. That way the data is available from any > client with database access without the need to give access to the server. > > PostgreSQL can handle it. > > -- > D'Arcy J.M. Cain > Vybe Networks Inc. > A unit of Excelsior Solutions Corporation - Propelling Business Forward > http://www.VybeNetworks.com/ > IM:da...@vybenetworks.com VoIP: sip:da...@vybenetworks.com > -- > https://mail.python.org/mailman/listinfo/python-list > -- https://mail.python.org/mailman/listinfo/python-list
Re: Dataframe to postgresql - Saving the dataframe to memory using StringIO
On 10/22/20 7:23 AM, Marco Sulla wrote: I would add that usually I do not recommend saving files on databases. I usually save the file on the disk and the path and mime on a dedicated table. I used to do that because backing up the database became huge. Now I use ZFS snapshots with send/receive so the backup only copies the changed blocks so I keep it in the database. That way the data is available from any client with database access without the need to give access to the server. PostgreSQL can handle it. -- D'Arcy J.M. Cain Vybe Networks Inc. A unit of Excelsior Solutions Corporation - Propelling Business Forward http://www.VybeNetworks.com/ IM:da...@vybenetworks.com VoIP: sip:da...@vybenetworks.com OpenPGP_signature Description: OpenPGP digital signature -- https://mail.python.org/mailman/listinfo/python-list
Re: Dataframe to postgresql - Saving the dataframe to memory using StringIO
On Thu, Oct 22, 2020 at 8:28 PM Shaozhong SHI wrote: > > I found this last option is very interesting. > > Saving the dataframe to memory using StringIO > > https://naysan.ca/2020/06/21/pandas-to-postgresql-using-psycopg2-copy_from/ > > But, testing shows > unicode argument expected, got 'str' > > Any working example for getting DataFrame into a PostgreSQL table directly? > That error suggests that you're using a legacy version of Python. The page you're linking to has been written on the assumption that you're using Python 3. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: Dataframe to postgresql - Saving the dataframe to memory using StringIO
I would add that usually I do not recommend saving files on databases. I usually save the file on the disk and the path and mime on a dedicated table. -- https://mail.python.org/mailman/listinfo/python-list
Re: Dataframe to postgresql - Saving the dataframe to memory using StringIO
Try to save it in a binary field on PG using hdf5: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.to_hdf.html On Thu, 22 Oct 2020 at 11:29, Shaozhong SHI wrote: > I found this last option is very interesting. > > Saving the dataframe to memory using StringIO > > https://naysan.ca/2020/06/21/pandas-to-postgresql-using-psycopg2-copy_from/ > > But, testing shows > unicode argument expected, got 'str' > > Any working example for getting DataFrame into a PostgreSQL table directly? > > Regards, > > David > -- > https://mail.python.org/mailman/listinfo/python-list > -- https://mail.python.org/mailman/listinfo/python-list
Dataframe to postgresql - Saving the dataframe to memory using StringIO
I found this last option is very interesting. Saving the dataframe to memory using StringIO https://naysan.ca/2020/06/21/pandas-to-postgresql-using-psycopg2-copy_from/ But, testing shows unicode argument expected, got 'str' Any working example for getting DataFrame into a PostgreSQL table directly? Regards, David -- https://mail.python.org/mailman/listinfo/python-list
