Can anyone share experience about python backend developer?

[lampahome]

I have 3+years developer experience in python, but I always develop
about peer-to-peer service. Have no backend experience in python.

But now I want to change to backend engineer, somebody shares their
job is to do like
1. Develop customize API to receive data from global
2. Develop tools with k8s
3. Experience about NoSQL

After I heard above, it seems wide range and hard to start from scratch.

Without comparing performance with other languages, how to dig into
backend development?(Or some framework could help me?)

thx
-- 
https://mail.python.org/mailman/listinfo/python-list

How to improve epoll speed when recv from kernel via netlink?

[lampahome]

I tried to receive msg from kernel via netlink of socket.

And I use epoll to receive netlink events whenever it comes from kernel to
user space.

But I found the performance is poor e.g. epoll costs 90% time of execution
time after I profile it by cProfile module.

Are there any tips to improve this?
-- 
https://mail.python.org/mailman/listinfo/python-list

How to specific multiple dtypes in numpy.ndarray?

[lampahome]

I meet performance is low when I use struct.unpack to unpack binary data.

So I tried to use numpy.ndarray
But meet error when I want to unpack multiple dtypes

Can anyone teach me~

Code like below:
# python3
import struct
import numpy as np
s1 = struct.Struct("@QIQ")
ss1 = s1.pack(1,11,111)
np.ndarray((3,), [('Q','I','Q')], ss1)
# ValueError: mismatch in size of old and new data-descriptor.
-- 
https://mail.python.org/mailman/listinfo/python-list

What does the blue color section in background mean?

[lampahome]

I tried to plot graph about a time-series with library statsmodel.

I decide to plot autocorrelation function, but I don't know the blue
section in the example graph mean...

Can anyone tell me?

The plot_acf example link:
https://www.statsmodels.org/dev/generated/statsmodels.graphics.tsaplots.plot_acf.html#statsmodels.graphics.tsaplots.plot_acf


The  graph in example:
https://www.statsmodels.org/dev/plots/graphics_tsa_plot_acf.png

 thx
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Does module socketserver using epoll in python3?

[lampahome]

>
> The source code is here:
> https://github.com/python/cpython/blob/master/Lib/socketserver.py .  You
> should find all the technical details you are looking for in it.
>
>
# poll/select have the advantage of not requiring any extra file
> descriptor,# contrarily to epoll/kqueue (also, they require a single
> syscall).
> if hasattr(selectors, 'PollSelector'):
>  _ServerSelector = selectors.PollSelector
> else:
>  _ServerSelector = selectors.SelectSelector

 Oh..no It uses poll or select ranther than epoll. Maybe it suffer some
performance degrading.

thx lot.
-- 
https://mail.python.org/mailman/listinfo/python-list

Does module socketserver using epoll in python3?

[lampahome]

As title,

I want to use socketserver to replace my own server code to
maintain ealsier.

But I don't found any info about tech. detail of socketserver, epoll is
important.

Can anyone tell me?
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Use epoll but still lose packet

[lampahome]

Dennis Lee Bieber  於 2019年11月21日 週四 上午2:17寫道：

> On Wed, 20 Nov 2019 18:51:31 +0800, lampahome 
> declaimed the following:
>
> >
> >I only use a while loop to catch events like below:
> >import select, socket
> >sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
> >sock.bind()
>
> Documentation indicates that one must provide an address (ip,
> port) for
> the bind -- there is no default described.
>
> sorry, I just write a brief code so I don't write whole detail ex: (ip,
port). But it can receive any connection and bind successfully.


> >sock.listen(10)
>
> You've specified that 10 connections can be held in the backlog,
> any
> beyond that will be refused.
>
> But there's one client and it send 128 messages with different length.


> for fd,event in events:
> >  if event & select.EPOLLIN:
> >sock.recv(1024)
> >...
>
> Documentation is a bit week -- there is no description of what
> .poll()
> returns in the help file.
>
> Point #1: you are using STREAM => implies TCP. TCP is not a
> "message"
> related protocol. Multiple "sends" can be one "receive" packet. One "send"
> can also be multiple "receive" packets. You truncated anything that would
> show if your processing is parsing out some high-level protocol showing the
> handling of such combined or split packets.
>
> Maybe this is the point! Multiple sends which doesn't full a 1024-length
buffer will make me receive one packet.


> Also, the help system links to
> https://linux.die.net/man/4/epoll
> which has
> """
> Q7
> If more than one event comes in between epoll_wait(2) calls, are they
> combined or reported separately?
> A7
> They will be combined.
> """
>
-- 
https://mail.python.org/mailman/listinfo/python-list

Use epoll but still lose packet

[lampahome]

I use epoll to listen events to receive packet from remote client via tcp
socket

And I found it still lose packet from remote client when client sends 128
messages to me.

Is there any tips to avoid this? thx

I only use a while loop to catch events like below:
import select, socket
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.bind()
sock.listen(10)
epoll = select.epoll()
epoll.register(sock.fileno(), select.EPOLLIN)
while True:
 events = epoll.poll(10) # timeout=10 sec
 if not events:
  continue

 for fd,event in events:
  if event & select.EPOLLIN:
sock.recv(1024)
...
-- 
https://mail.python.org/mailman/listinfo/python-list

Pickle failed __getstate__ on my customized class inherited dict

[lampahome]

I make a class Wrapper inherited from dict and met problem when I want to
pickle it.

Is there anyway to make __getstate__ of Wrapper to output a normal
dict?(Output a dict will help pickleing easily)


=== code ===
import pickle
class Wrapper(dict):
def __getattr__(self, attr):
return self[attr]
def __getstate__(self):
# blablabla

d=Wrapper(zip((1,2), (3,4)))
pickle.dumps(d)
=== code ===

When I tried overwrite __getstate__ to below:
def __getstate__(self):
return self.__repr__()
pickle it will shows:
b'\x80...__main__...Wrapper...' <- I don't know why it shows the class name.
That makes me confused.
-- 
https://mail.python.org/mailman/listinfo/python-list

Any socket library to communicate with kernel via netlink?

[lampahome]

As title, I tried to communicate with kernel via netlink. But I failed when
I receive msg from kernel.

The weird point is sending successfully from user to kernel, failed when
receiving from kernel.

So I want to check code in 3rd library and dig in, but always found library
called netlinkg but it actually does something like modify network address
or check network card...

Any idea is welcome
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: python3 subprocess run sudo cmd in remote failed

[lampahome]

>
> Well, there's a Python library called "paramiko" which implements ssh.
> That might help.
>
> Later I will try lol.


> Note also that since stdin and stdout are pipes and not the terminal
> then ssh will not be interactive, and will not allocate a tty at the far
> end either. You can get ssh to open a remote tty with the -t option.
>
> But I suspect you don't want stdin=PIPE or stdout=PIPE at all. Why are
> they there?
>
> I thought I can use ps.stdin.write(password), so I make stdin and stdout
be pipe as input and output.

Here are I tried:
>from subprocess import Popen, PIPE
>ps = Popen('ssh -o \'StrictHostKeyChecking no\' hello@192.168.80.11 \'sudo
sysctl -w vm.drop_caches=3\', shell=True)
>  hello@192.168.80.11's password:

>from subprocess import Popen, PIPE
>ps = Popen(['ssh',  '-o \'StrictHostKeyChecking no\'',  '
hello@192.168.80.11', '\'sudo sysctl -w vm.drop_caches=3\''])
>  hello@192.168.80.11's password:

It always prompt immediately, that make me hard to enter password.
 Maybe I should try  paramiko...
-- 
https://mail.python.org/mailman/listinfo/python-list

python3 subprocess run sudo cmd in remote failed

[lampahome]

Hello, I use python3.5 and found no way to solve this problem

>from subprocess import Popen, PIPE
>ps = Popen('ssh -o \'StrictHostKeyChecking no\' hello@192.168.80.11 \'sudo
sysctl -w vm.drop_caches=3\', stdin=PIPE, stdout=PIPE, stderr=PIPE,
bufsize=0, shell=True)
>  hello@192.168.80.11's password:

what I tried many times like enter password, but it failed.
I just want to use ps.stdin.write(password) to send password, but it always
jump password prompt immediately.

How to solve this
-- 
https://mail.python.org/mailman/listinfo/python-list

How to use regex to search string between {}?

[lampahome]

I want to parse a path string with multiple files and try to figure out a
one-line way.

If I have path: /home/admin/hello/yo/{h1,h2,h3,h4}

What I thought is use regex.search, but the pattern always failed.

I use below:
import re
path = /home/admin/hello/yo/{h1,h2,h3,h4}
r = re.search('{.}', path)
# r should be ['h1,h2,h3,h4'] but I fail

Why always search nothing?
-- 
https://mail.python.org/mailman/listinfo/python-list

FDs will be closed after exception automatically in python2.7?

[lampahome]

as title,

I confused will fd will be close after exception automatically?

Like:
try:
fd=open("file","w+")
fd.get() //any useless function of fd
except Exception:
print 'hi'
-- 
https://mail.python.org/mailman/listinfo/python-list

Better way to recontruct a continuous and repeated array with low time complexity?

[lampahome]

I write program to do experiment about time series(weekly) with machine
learning.
I record changes of everyday of each ID and Count.
I read the csv as dataset like below:
ID, Count
1,30 // First Day
2,33
3,45
4,11
5,66
7,88
1,32 // 2nd Day
2,35
3,55
4,21
5,36
7,48

I have two array X, y. I want to put ID and Count in X, and put Count of
2nd Day in y.
The element of X and y is corresponding with the ID.
ex: X[0] and y[0] is the value where ID == 1
X[1] and y[1] is the value where ID == 2...etc

So X is like below:
array([
[1,30],
[2,33],
[3,45],
[4,11],
[5,66],
[7,88]
])

y is like below:
array([
[32],
[35],
[55],
[21],
[36],
[48]
])

Program what I write always cost O(n^2) complexity.
Code:

dataframe = pandas.read_csv(path, header=0, engine='python')
dataset = dataframe.dropna().values.astype('float64')
create_data(dataset)

def create_data(dataset):

uni = np.unique(dataset[:,0])
X = np.zeros((7, 2))
y = np.zeros((7, 1))
offset = 0
for i in uni:

index = dataset[:,0] == i

data = dataset[index]

for j in xrange(len(data)-2+1):

X[offset] = data[j]

y[offset] = data[j, 1]

offset += 1

return X, y


I use two for loop and estimate the complexity is O(n^2).

Is there any better way to re-write the code and reduct the time complexity?
-- 
https://mail.python.org/mailman/listinfo/python-list

how can I solve this problem simply and clearly

[lampahome]

I have two categories A,B, and A has 2 items A1,A2, and B have 2 items B1,
B2.

I have two class A and B, and A will handle A1,A2, B handle B1,B2.

I want to parse one of A1,A2,B1,B2 to script and generate the corresponding
class(object).

Ex: Both in class A and B, all have func1(), func2().
What I thought to design is below:
-
class Data(object):
def __Init__():
...
 def func1(self):
pass

def func2(self):
 pass

class A(Data):
def __Init__():
...
 def func1(self):
A_do()

def func2(self):
 A_does()

class B(Data):
def __Init__():
...
 def func1(self):
B_do()

def func2(self):
 B_does()

def get_class(obj):
if obj == 'A1' or obj == 'A2':
return A(obj)
 else:
return B(obj)

# A = get_class(A1)
# B = get_class(B2)

-

The function *get_class() *is the question that can I solve this problem
clearly? or other else code snippets ?

thx
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Does this behavior have a better design pattern?

[lampahome]

>
>
> The rest is just cruft ;)
> >
> > *Is there better design pattern for me?*
>
> If A does B to C, is that a crime?
>
No


> Your problem description suffers from overgeneralisation.
>
> Generally speaking you get better solutions when you ask yourself
> "How can I solve this problem efficiently?"
>
>
In another hand, how can I solve this problem efficiently?
or any pythonic way to solve this?
-- 
https://mail.python.org/mailman/listinfo/python-list

Does this behavior have a better design pattern?

[lampahome]

I have two categories A,B, and A has 2 items A1,A2
B have 2 items B1, B2.

I have two class A and B, and A will handle A1,A2, B handle B1,B2.

I want to parse one of A1,A2,B1,B2 to script and generate the corresponding
class(object).

Ex: Both in class A and B, all have func1(), func2().
What I thought to design is below:
-
class Data(object):
def __Init__():
...
 def func1(self):
pass

def func2(self):
 pass

class A(Data):
def __Init__():
...
 def func1(self):
A_do()

def func2(self):
 A_does()

class B(Data):
def __Init__():
...
 def func1(self):
B_do()

def func2(self):
 B_does()

def get_class(obj):
if obj == 'A1' or obj == 'A2':
return A(obj)
 else:
return B(obj)

# A = get_class(A1)
# B = get_class(B2)

-

Above is I thought to make code clear, and this pattern is called simple
factory?

*Is there better design pattern for me?*

thanks
-- 
https://mail.python.org/mailman/listinfo/python-list

matplotlib.plot.show always blocks the execution of python script

[lampahome]

I want to plot a graph and *still run following code without closing the
graph automatically like Matlab does*.

I try plt.show(block=False) , it failed and appear in a small moment then
close itself.

I also try plt.draw() or interactive mode , it failed, too.
plt.draw() will block until I close it.
interactive mode and plot.show() will appear nothing

My version is below:
user@ya:~/$ sudo pip freeze | grep matplotlib
matplotlib==2.2.3
user@ya:~/$ sudo pip -V
pip 18.1 from /usr/local/lib/python2.7/dist-packages/pip (python 2.7)

Environments:
I only execute script in Ubuntu ex: user@ya: python xxx.py Distributor ID:
Ubuntu
Description: Ubuntu 16.04.5 LTS
Release: 16.04
Codename: xenial

Can anyone help me? I just want to do like Matlab which won't close the
plotted graph even if the script finishes.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: abspath returns different results when py_compile input path is different in Python?

[lampahome]

so py_compile.compile() doesn't normalize the filename, but it will pass
the input filename to co_filename?

2018-02-12 19:27 GMT+08:00 eryk sun <eryk...@gmail.com>:

> On Mon, Feb 12, 2018 at 9:40 AM, lampahome <pahome.c...@mirlab.org> wrote:
> > I want to know abspath of python script followed by steps below.
> >
> >1. *built it to byte code by py_compile.*
> >2. *execute it to check abspath.*
> >
> > But I got *2 results* when I execute it.I found the *results based on the
> > path of script* followed by py_compile.
> >
> > Here is my script test.py :
> >
> > import os
> > import inspect
> > print os.path.dirname(os.path.abspath(inspect.getfile(
> inspect.currentframe(
> >
> > Build it with py_compile, then got 2 results when I enter *different path
> > of test.py*:
> >
> > *enter the folder and compile with only script name.*
> >
> > [~] cd /usr/local/bin/
> > [/usr/local/bin/] python -m py_compile test.py
> > [/usr/local/bin/] cd ~
> > [~] python /usr/local/bin/test.pyc
> > /home/UserXX
> >
> > *In other folder and compile with absolute script name.*
> >
> > [~] python -m py_compile /usr/local/bin/test.py
> > [~] python /usr/local/bin/test.pyc
> > /usr/local/bin
>
> A code object has a co_filename attribute for use in creating
> tracebacks. This path isn't necessarily fully qualified. Here are a
> couple of examples using the built-in compile() function:
>
> >>> compile('42', 'test.py', 'exec').co_filename
> 'test.py'
> >>> compile('42', '/usr/local/bin/test.py', 'exec').co_filename
> '/usr/local/bin/test.py'
>
> py_compile.compile() does not normalize the filename.
>
-- 
https://mail.python.org/mailman/listinfo/python-list

abspath returns different results when py_compile input path is different in Python?

[lampahome]

I want to know abspath of python script followed by steps below.

   1. *built it to byte code by py_compile.*
   2. *execute it to check abspath.*

But I got *2 results* when I execute it.I found the *results based on the
path of script* followed by py_compile.

Here is my script test.py :

import osimport inspect
print os.path.dirname(os.path.abspath(inspect.getfile(inspect.currentframe(

Build it with py_compile, then got 2 results when I enter *different path
of test.py*:

*enter the folder and compile with only script name.*

[~]cd /usr/local/bin/[/usr/local/bin/]python -m py_compile
test.py[/usr/local/bin/]cd ~[~]python
/usr/local/bin/test.pyc/home/UserXX

*In other folder and compile with absolute script name.*

[~]python -m py_compile /usr/local/bin/test.py[~]python
/usr/local/bin/test.pyc/usr/local/bin

how come got 2 different results?
-- 
https://mail.python.org/mailman/listinfo/python-list