Abacn commented on issue #23904:
URL: https://github.com/apache/beam/issues/23904#issuecomment-2027669667
THe memory profile (pipeline option `--profile_memory --profile_cpu`) wasn't
quite helpful. The generated memory profiles look like the following
```
Partition of a set of 22679 objects. Total size = 2069184 bytes.
Index Count % Size % Cumulative % Kind (class / dict of class)
0 9796 43 912749 44 912749 44 str
1 4565 20 324624 16 1237373 60 tuple
2 962 4 269480 13 1506853 73 dict (no owner)
3 194 1 180983 9 1687836 82 bytes
4 5185 23 124440 6 1812276 88 float
5 39 0 91312 4 1903588 92 list
6 1180 5 33192 2 1936780 94 int
7 50 0 18992 1 1955772 95 frozenset
8 92 0 16200 1 1971972 95 types.CodeType
9 110 0 14960 1 1986932 96 function
<125 more rows. Type e.g. '_.more' to view.>
Index Count % Size % Cumulative % Kind (class / dict of class)
10 9 0 9984 0 1996916 97 abc.ABCMeta
11 9 0 8976 0 2005892 97 dict of module
12 72 0 5760 0 2011652 97
org.apache.beam.model.pipeline.v1.metrics_pb2.Monitorin
gInfo
13 9 0 4896 0 2016548 97 dict of abc.ABCMeta
14 4 0 4256 0 2020804 98 type
15 6 0 3476 0 2024280 98 re.Pattern
16 5 0 3200 0 2027480 98 dict of
apache_beam.testing.synthetic_pipeline.SyntheticSource
17 1 0 2552 0 2030032 98 random.Random
18 2 0 2400 0 2032432 98 urllib.parse.Quoter
19 3 0 1920 0 2034352 98 dict of threading.Thread
<115 more rows. Type e.g. '_.more' to view.>
```
or
```
Partition of a set of 16762 objects. Total size = 3649165 bytes.
Index Count % Size % Cumulative % Kind (class / dict of class)
0 9 0 2189046 60 2189046 60 bytes
1 9966 59 901079 25 3090125 85 str
2 2322 14 162448 4 3252573 89 tuple
3 528 3 141944 4 3394517 93 dict (no owner)
4 51 0 97704 3 3492221 96 list
5 2671 16 64104 2 3556325 97 float
6 703 4 19800 1 3576125 98 int
7 19 0 11856 0 3587981 98 collections.deque
8 32 0 6912 0 3594893 99 frozenset
9 8 0 6048 0 3600941 99 re.Pattern
<73 more rows. Type e.g. '_.more' to view.>
Index Count % Size % Cumulative % Kind (class / dict of class)
10 68 0 5440 0 3606381 99
org.apache.beam.model.pipeline.v1.metrics_pb2.Monitorin
gInfo
11 19 0 4408 0 3610789 99 dict of threading.Condition
12 6 0 3840 0 3614629 99 dict of
apache_beam.testing.synthetic_pipeline.SyntheticSource
13 21 0 2856 0 3617485 99 function
14 39 0 2808 0 3620293 99 types.BuiltinMethodType
15 4 0 2560 0 3622853 99 dict of threading.Thread
16 2 0 2128 0 3624981 99 type
17 5 0 2000 0 3626981 99 dict of
urllib3.connection.HTTPSConnection
18 3 0 1368 0 3628349 99 types.FrameType
19 34 0 1360 0 3629709 99 types.CellType
<63 more rows. Type e.g. '_.more' to view.>
```
or
```
Partition of a set of 3959 objects. Total size = 271112 bytes.
Index Count % Size % Cumulative % Kind (class / dict of class)
0 1419 36 100336 37 100336 37 tuple
1 344 9 98784 36 199120 73 dict (no owner)
2 1696 43 40704 15 239824 88 float
3 130 3 11315 4 251139 93 str
4 289 7 8104 3 259243 96 int
5 4 0 3024 1 262267 97 re.Pattern
6 3 0 1368 1 263635 97 types.FrameType
7 17 0 1360 1 264995 98
org.apache.beam.model.pipeline.v1.metrics_pb2.Monitorin
gInfo
8 2 0 781 0 265776 98 bytes
9 5 0 680 0 266456 98 function
<24 more rows. Type e.g. '_.more' to view.>
Index Count % Size % Cumulative % Kind (class / dict of class)
10 1 0 640 0 267096 99 dict of threading.Thread
11 1 0 624 0 267720 99 collections.deque
12 7 0 608 0 268328 99 list
13 9 0 360 0 268688 99 types.CellType
14 8 0 320 0 269008 99 _thread.lock
15 2 0 256 0 269264 99
grpc._cython.cygrpc.SendMessageOperation
16 1 0 232 0 269496 99 dict of threading.Condition
17 2 0 160 0 269656 99
org.apache.beam.model.fn_execution.v1.beam_fn_api_pb2.D
ata
18 2 0 160 0 269816 100 urllib.parse.SplitResult
19 2 0 144 0 269960 100 builtins.weakref
<14 more rows. Type e.g. '_.more' to view.>
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
