[GitHub] [incubator-heron] huijunwu edited a comment on pull request #3588: Deep clean heron-explorer
huijunwu edited a comment on pull request #3588: URL: https://github.com/apache/incubator-heron/pull/3588#issuecomment-667168233 tested on macos, several issues: 1. need to toggle verbose, see below ``` [tw-mbp-huijunw heron-github (explorer-cli)]$ ~/.heron/bin/heron-explorer logical-plan local hello-world-topology [2020-07-31 07:32:21 -0700] [DEBUG]: fetching url http://127.0.0.1:/topologies/logicalplan?cluster=local=default=hello-world-topology=huijunw [2020-07-31 07:32:21 -0700] [DEBUG]: TIME: url fetch took 0.39 ms server time http://127.0.0.1:/topologies/logicalplan?cluster=local=default=hello-world-topology=huijunw [2020-07-31 07:32:21 -0700] [DEBUG]: TIME: url fetch took 29.01 ms round trip http://127.0.0.1:/topologies/logicalplan?cluster=local=default=hello-world-topology=huijunw [2020-07-31 07:32:21 -0700] [DEBUG]: fetching url http://127.0.0.1:/topologies/info?cluster=local=default=hello-world-topology=huijunw [2020-07-31 07:32:21 -0700] [DEBUG]: TIME: url fetch took 0.11 ms server time http://127.0.0.1:/topologies/info?cluster=local=default=hello-world-topology=huijunw [2020-07-31 07:32:21 -0700] [DEBUG]: TIME: url fetch took 4.45 ms round trip http://127.0.0.1:/topologies/info?cluster=local=default=hello-world-topology=huijunw typenameparallelism inputoutput -- - --- spout word 2 -exclaim1 boltexclaim1 4 word - [tw-mbp-huijunw heron-github (explorer-cli)]$ ~/.heron/bin/heron-explorer --verbose logical-plan local hello-world-topology typenameparallelism inputoutput -- - --- spout word 2 -exclaim1 boltexclaim1 4 word - ``` 2. some options order in the help-info are wrong? ``` [tw-mbp-huijunw heron-github (explorer-cli)]$ ~/.heron/bin/heron-explorer topologies --help Usage: heron-explorer topologies [OPTIONS] CLUSTER[/ROLE[/ENV]] Options: --tracker-url TEXT --help Show this message and exit. [tw-mbp-huijunw heron-github (explorer-cli)]$ ~/.heron/bin/heron-explorer topologies local --tracker-url http://127.0.0.1: [2020-07-31 07:43:20 -0700] [DEBUG]: fetching url http://127.0.0.1:/topologies?cluster=local [2020-07-31 07:43:20 -0700] [DEBUG]: TIME: url fetch took 0.10 ms server time http://127.0.0.1:/topologies?cluster=local [2020-07-31 07:43:20 -0700] [DEBUG]: TIME: url fetch took 22.43 ms round trip http://127.0.0.1:/topologies?cluster=local Topologies in local: role env topology --- --- huijunw default hello-world-topology [tw-mbp-huijunw heron-github (explorer-cli)]$ ~/.heron/bin/heron-explorer --tracker-url http://127.0.0.1: topologies local Usage: heron-explorer [OPTIONS] COMMAND [ARGS]... Try 'heron-explorer --help' for help. Error: no such option: --tracker-url ``` Should be `Usage: heron-explorer topologies CLUSTER[/ROLE[/ENV]] [OPTIONS]`rather than `Usage: heron-explorer topologies [OPTIONS] CLUSTER[/ROLE[/ENV]]` `--tracker-url TEXT`, `TEXT` could be more descriptive These issues exist as well for other sub-commands 3. looks like no difference of --component-type bolts/spouts/all ``` [tw-mbp-huijunw heron-github (explorer-cli)]$ ~/.heron/bin/heron-explorer --verbose logical-plan local hello-world-topology --component-type bolts typenameparallelism inputoutput -- - --- spout word 2 -exclaim1 boltexclaim1 4 word - [tw-mbp-huijunw heron-github (explorer-cli)]$ ~/.heron/bin/heron-explorer --verbose logical-plan local hello-world-topology --component-type spouts typenameparallelism inputoutput -- - --- spout word 2 -exclaim1 boltexclaim1 4 word - [tw-mbp-huijunw heron-github (explorer-cli)]$ ~/.heron/bin/heron-explorer --verbose logical-plan local hello-world-topology --component-type all typenameparallelism inputoutput -- - --- spout word 2 -exclaim1 boltexclaim1 4 word - [tw-mbp-huijunw heron-github (explorer-cli)]$ ~/.heron/bin/heron-explorer --verbose logical-plan local hello-world-topology --component-type all --help Usage: heron-explorer logical-plan [OPTIONS] CLUSTER[/ROLE[/ENV]] TOPOLOGY Options: --config-path TEXT --tracker-url TEXT --component-type [all|spouts|bolts] --help Show this message and exit. ``` ==
[GitHub] [incubator-heron] huijunwu edited a comment on pull request #3588: Deep clean heron-explorer
huijunwu edited a comment on pull request #3588: URL: https://github.com/apache/incubator-heron/pull/3588#issuecomment-667168233 tested on macos, several issues: 1. need to toggle verbose, see below ``` [tw-mbp-huijunw heron-github (explorer-cli)]$ ~/.heron/bin/heron-explorer logical-plan local hello-world-topology [2020-07-31 07:32:21 -0700] [DEBUG]: fetching url http://127.0.0.1:/topologies/logicalplan?cluster=local=default=hello-world-topology=huijunw [2020-07-31 07:32:21 -0700] [DEBUG]: TIME: url fetch took 0.39 ms server time http://127.0.0.1:/topologies/logicalplan?cluster=local=default=hello-world-topology=huijunw [2020-07-31 07:32:21 -0700] [DEBUG]: TIME: url fetch took 29.01 ms round trip http://127.0.0.1:/topologies/logicalplan?cluster=local=default=hello-world-topology=huijunw [2020-07-31 07:32:21 -0700] [DEBUG]: fetching url http://127.0.0.1:/topologies/info?cluster=local=default=hello-world-topology=huijunw [2020-07-31 07:32:21 -0700] [DEBUG]: TIME: url fetch took 0.11 ms server time http://127.0.0.1:/topologies/info?cluster=local=default=hello-world-topology=huijunw [2020-07-31 07:32:21 -0700] [DEBUG]: TIME: url fetch took 4.45 ms round trip http://127.0.0.1:/topologies/info?cluster=local=default=hello-world-topology=huijunw typenameparallelism inputoutput -- - --- spout word 2 -exclaim1 boltexclaim1 4 word - [tw-mbp-huijunw heron-github (explorer-cli)]$ ~/.heron/bin/heron-explorer --verbose logical-plan local hello-world-topology typenameparallelism inputoutput -- - --- spout word 2 -exclaim1 boltexclaim1 4 word - ``` 2. some options order in the help-info are wrong? ``` [tw-mbp-huijunw heron-github (explorer-cli)]$ ~/.heron/bin/heron-explorer topologies --help Usage: heron-explorer topologies [OPTIONS] CLUSTER[/ROLE[/ENV]] Options: --tracker-url TEXT --help Show this message and exit. [tw-mbp-huijunw heron-github (explorer-cli)]$ ~/.heron/bin/heron-explorer topologies local --tracker-url http://127.0.0.1: [2020-07-31 07:43:20 -0700] [DEBUG]: fetching url http://127.0.0.1:/topologies?cluster=local [2020-07-31 07:43:20 -0700] [DEBUG]: TIME: url fetch took 0.10 ms server time http://127.0.0.1:/topologies?cluster=local [2020-07-31 07:43:20 -0700] [DEBUG]: TIME: url fetch took 22.43 ms round trip http://127.0.0.1:/topologies?cluster=local Topologies in local: role env topology --- --- huijunw default hello-world-topology [tw-mbp-huijunw heron-github (explorer-cli)]$ ~/.heron/bin/heron-explorer --tracker-url http://127.0.0.1: topologies local Usage: heron-explorer [OPTIONS] COMMAND [ARGS]... Try 'heron-explorer --help' for help. Error: no such option: --tracker-url ``` Should be `Usage: heron-explorer topologies CLUSTER[/ROLE[/ENV]] [OPTIONS]`rather than `Usage: heron-explorer topologies [OPTIONS] CLUSTER[/ROLE[/ENV]]` `--tracker-url TEXT`, `TEXT` could be more descriptive These issues exist as well for other sub-commands 3. looks like no difference of --component-type bolts/spouts/all ``` [tw-mbp-huijunw heron-github (explorer-cli)]$ ~/.heron/bin/heron-explorer --verbose logical-plan local hello-world-topology --component-type bolts typenameparallelism inputoutput -- - --- spout word 2 -exclaim1 boltexclaim1 4 word - [tw-mbp-huijunw heron-github (explorer-cli)]$ ~/.heron/bin/heron-explorer --verbose logical-plan local hello-world-topology --component-type spouts typenameparallelism inputoutput -- - --- spout word 2 -exclaim1 boltexclaim1 4 word - [tw-mbp-huijunw heron-github (explorer-cli)]$ ~/.heron/bin/heron-explorer --verbose logical-plan local hello-world-topology --component-type all typenameparallelism inputoutput -- - --- spout word 2 -exclaim1 boltexclaim1 4 word - [tw-mbp-huijunw heron-github (explorer-cli)]$ ~/.heron/bin/heron-explorer --verbose logical-plan local hello-world-topology --component-type all --help Usage: heron-explorer logical-plan [OPTIONS] CLUSTER[/ROLE[/ENV]] TOPOLOGY Options: --config-path TEXT --tracker-url TEXT --component-type [all|spouts|bolts] --help Show this message and exit. ```
[GitHub] [incubator-heron] huijunwu commented on pull request #3588: Deep clean heron-explorer
huijunwu commented on pull request #3588: URL: https://github.com/apache/incubator-heron/pull/3588#issuecomment-667168233 tested on macos, several issues: 1. need to toggle verbose, see below ``` [tw-mbp-huijunw heron-github (explorer-cli)]$ ~/.heron/bin/heron-explorer logical-plan local hello-world-topology [2020-07-31 07:32:21 -0700] [DEBUG]: fetching url http://127.0.0.1:/topologies/logicalplan?cluster=local=default=hello-world-topology=huijunw [2020-07-31 07:32:21 -0700] [DEBUG]: TIME: url fetch took 0.39 ms server time http://127.0.0.1:/topologies/logicalplan?cluster=local=default=hello-world-topology=huijunw [2020-07-31 07:32:21 -0700] [DEBUG]: TIME: url fetch took 29.01 ms round trip http://127.0.0.1:/topologies/logicalplan?cluster=local=default=hello-world-topology=huijunw [2020-07-31 07:32:21 -0700] [DEBUG]: fetching url http://127.0.0.1:/topologies/info?cluster=local=default=hello-world-topology=huijunw [2020-07-31 07:32:21 -0700] [DEBUG]: TIME: url fetch took 0.11 ms server time http://127.0.0.1:/topologies/info?cluster=local=default=hello-world-topology=huijunw [2020-07-31 07:32:21 -0700] [DEBUG]: TIME: url fetch took 4.45 ms round trip http://127.0.0.1:/topologies/info?cluster=local=default=hello-world-topology=huijunw typenameparallelism inputoutput -- - --- spout word 2 -exclaim1 boltexclaim1 4 word - [tw-mbp-huijunw heron-github (explorer-cli)]$ ~/.heron/bin/heron-explorer --verbose logical-plan local hello-world-topology typenameparallelism inputoutput -- - --- spout word 2 -exclaim1 boltexclaim1 4 word - ``` 2. some options order in the help-info are wrong? ``` [tw-mbp-huijunw heron-github (explorer-cli)]$ ~/.heron/bin/heron-explorer topologies --help Usage: heron-explorer topologies [OPTIONS] CLUSTER[/ROLE[/ENV]] Options: --tracker-url TEXT --help Show this message and exit. [tw-mbp-huijunw heron-github (explorer-cli)]$ ~/.heron/bin/heron-explorer topologies local --tracker-url http://127.0.0.1: [2020-07-31 07:43:20 -0700] [DEBUG]: fetching url http://127.0.0.1:/topologies?cluster=local [2020-07-31 07:43:20 -0700] [DEBUG]: TIME: url fetch took 0.10 ms server time http://127.0.0.1:/topologies?cluster=local [2020-07-31 07:43:20 -0700] [DEBUG]: TIME: url fetch took 22.43 ms round trip http://127.0.0.1:/topologies?cluster=local Topologies in local: role env topology --- --- huijunw default hello-world-topology [tw-mbp-huijunw heron-github (explorer-cli)]$ ~/.heron/bin/heron-explorer --tracker-url http://127.0.0.1: topologies local Usage: heron-explorer [OPTIONS] COMMAND [ARGS]... Try 'heron-explorer --help' for help. Error: no such option: --tracker-url ``` Should be `Usage: heron-explorer topologies CLUSTER[/ROLE[/ENV]] [OPTIONS]`rather than `Usage: heron-explorer topologies [OPTIONS] CLUSTER[/ROLE[/ENV]]` `--tracker-url TEXT`, `TEXT` could be more descriptive These issues exist as well for other sub-commands 3. looks like no difference of --component-type bolts/spouts/all ``` [tw-mbp-huijunw heron-github (explorer-cli)]$ ~/.heron/bin/heron-explorer --verbose logical-plan local hello-world-topology --component-type bolts typenameparallelism inputoutput -- - --- spout word 2 -exclaim1 boltexclaim1 4 word - [tw-mbp-huijunw heron-github (explorer-cli)]$ ~/.heron/bin/heron-explorer --verbose logical-plan local hello-world-topology --component-type spouts typenameparallelism inputoutput -- - --- spout word 2 -exclaim1 boltexclaim1 4 word - [tw-mbp-huijunw heron-github (explorer-cli)]$ ~/.heron/bin/heron-explorer --verbose logical-plan local hello-world-topology --component-type all typenameparallelism inputoutput -- - --- spout word 2 -exclaim1 boltexclaim1 4 word - [tw-mbp-huijunw heron-github (explorer-cli)]$ ~/.heron/bin/heron-explorer --verbose logical-plan local hello-world-topology --component-type all --help Usage: heron-explorer logical-plan [OPTIONS] CLUSTER[/ROLE[/ENV]] TOPOLOGY Options: --config-path TEXT --tracker-url TEXT --component-type [all|spouts|bolts] --help Show this message and exit. ``` 'consolidating the explorer
[GitHub] [incubator-heron] windhamwong commented on pull request #3595: Switch to pigz to support multi-core tar command
windhamwong commented on pull request #3595: URL: https://github.com/apache/incubator-heron/pull/3595#issuecomment-667061592 Docker build testing result: ubuntu18.04 - success ubuntu20.04 - success ubuntu16.04 - success debian9 - fail - python3.6 not supported (unrelated to pigz) debian10 - fail - protobuf issue (unrelated to pigz) centos7 - success This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-heron] nicknezis merged pull request #3595: Switch to pigz to support multi-core tar command
nicknezis merged pull request #3595: URL: https://github.com/apache/incubator-heron/pull/3595 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-heron] nicknezis merged pull request #3596: Updating docker build for fixing python library click issue
nicknezis merged pull request #3596: URL: https://github.com/apache/incubator-heron/pull/3596 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-heron] nicknezis commented on pull request #3596: Updating docker build for fixing python library click issue
nicknezis commented on pull request #3596: URL: https://github.com/apache/incubator-heron/pull/3596#issuecomment-666901866 @huijunw @windhamwong Is the underlying issue something that needs to be documented for those not using Docker images? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-heron] windhamwong commented on pull request #3595: Switch to pigz to support multi-core tar command
windhamwong commented on pull request #3595: URL: https://github.com/apache/incubator-heron/pull/3595#issuecomment-666502369 let me try out all docker builds tomorrow. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-heron] joshfischer1108 commented on pull request #3589: Reorder the init containers in the Helm chart
joshfischer1108 commented on pull request #3589: URL: https://github.com/apache/incubator-heron/pull/3589#issuecomment-66637 > @joshfischer1108 The order that init containers run is sequentially. The check for Zookeeper to be up and healthy was happening after the Bookkeeper format call. Bookkeeper needs Zookeeper to be healthy and therefore startup issues would occur when installing using Helm (everything at once). Gotcha. Thanks for explaining. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-heron] windhamwong commented on pull request #3595: Switch to pigz to support multi-core tar command
windhamwong commented on pull request #3595: URL: https://github.com/apache/incubator-heron/pull/3595#issuecomment-666062598 https://zlib.net/pigz/ It is gzip format but with the parallel multi-core feature. The traditional gzip command uses only one core but modern CPU/vCPU has more than one. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-heron] windhamwong opened a new pull request #3596: Updating docker build for fixing python library click issue
windhamwong opened a new pull request #3596: URL: https://github.com/apache/incubator-heron/pull/3596 https://github.com/apache/incubator-heron/commit/39cf3a76b143b5c491de88cfe40cd83f29d2c0be#commitcomment-41026206 This PR adds 2 lines of ENV for applying the fix of click under Py36 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-heron] nicknezis commented on pull request #3595: Switch to pigz to support multi-core tar command
nicknezis commented on pull request #3595: URL: https://github.com/apache/incubator-heron/pull/3595#issuecomment-665910932 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-heron] Code0x58 commented on pull request #3595: Switch to pigz to support multi-core tar command
Code0x58 commented on pull request #3595: URL: https://github.com/apache/incubator-heron/pull/3595#issuecomment-666023113 A couple of tests to show equivalence for consumers: With gzip: ``` $ cat txt oink $ ls txt $ pigz txt $ ls txt.gz $ gunzip txt.gz $ ls txt $ cat txt oink ``` With tar: ``` $ tar --remove-files --use-compress-program=pigz -cf out.tar.gz txt $ ls out.tar.gz $ tar -zxf out.tar.gz $ cat txt oink ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-heron] windhamwong opened a new pull request #3595: Switch to pigz to support multi-core tar command
windhamwong opened a new pull request #3595: URL: https://github.com/apache/incubator-heron/pull/3595 pigz is a multi-core version of tar, which makes the docker build process much faster. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-heron] windhamwong closed pull request #3594: Switch to pigz from gzip to enable multi-thread processing
windhamwong closed pull request #3594: URL: https://github.com/apache/incubator-heron/pull/3594 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-heron] huijunwu merged pull request #3591: Remove unused heron.tools.common.src.python.utils.heronparser
huijunwu merged pull request #3591: URL: https://github.com/apache/incubator-heron/pull/3591 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-heron] huijunwu commented on pull request #3591: Remove unused heron.tools.common.src.python.utils.heronparser
huijunwu commented on pull request #3591: URL: https://github.com/apache/incubator-heron/pull/3591#issuecomment-665564256 reference #1767 #1595 #1119 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-heron] windhamwong commented on pull request #3594: Switch to pigz from gzip to enable multi-thread processing
windhamwong commented on pull request #3594: URL: https://github.com/apache/incubator-heron/pull/3594#issuecomment-665796887 Messed up with the code, i will create another PR This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-heron] windhamwong opened a new pull request #3594: Switch to pigz from gzip to enable multi-thread processing
windhamwong opened a new pull request #3594: URL: https://github.com/apache/incubator-heron/pull/3594 This helps to make the docker build faster with using multi-core/thread. Ubuntu18 has pigz installed by default, but still need test on other distro. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-heron] huijunwu merged pull request #3592: Fix for misisng JDK tools
huijunwu merged pull request #3592: URL: https://github.com/apache/incubator-heron/pull/3592 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-heron] nicknezis merged pull request #3590: Fix licenses in 2020/7/20 master
nicknezis merged pull request #3590: URL: https://github.com/apache/incubator-heron/pull/3590 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-heron] nicknezis closed issue #3516: The build does not work with Java version greater than 11
nicknezis closed issue #3516: URL: https://github.com/apache/incubator-heron/issues/3516 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-heron] nicknezis opened a new pull request #3592: Fix for misisng JDK tools
nicknezis opened a new pull request #3592: URL: https://github.com/apache/incubator-heron/pull/3592 When we updated the Distribution images to use JRE instead of JDK, we lost the ability to run commands like `jmap` and `jstack`. This Pull Request updates the images to use JDK as they did before. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-heron] nicknezis opened a new pull request #3593: Updating Kryo
nicknezis opened a new pull request #3593: URL: https://github.com/apache/incubator-heron/pull/3593 Updating Kryo to a newer version. Currently 5.0.0-RC7 is the most recent release. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-heron] nwangtw opened a new pull request #3590: Fix licenses in 2020/7/20 master
nwangtw opened a new pull request #3590: URL: https://github.com/apache/incubator-heron/pull/3590 License is missing in a few files. And the license of a new font needs to be listed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-heron] Code0x58 opened a new pull request #3588: Deep clean heron-explorer
Code0x58 opened a new pull request #3588: URL: https://github.com/apache/incubator-heron/pull/3588 * use click instead of argparse pattern * list all topologies instead of just 20 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-heron] nicknezis merged pull request #3589: Reorder the init containers in the Helm chart
nicknezis merged pull request #3589: URL: https://github.com/apache/incubator-heron/pull/3589 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-heron] nicknezis opened a new pull request #3589: Reorder the init containers in the Helm chart
nicknezis opened a new pull request #3589: URL: https://github.com/apache/incubator-heron/pull/3589 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-heron] Code0x58 opened a new pull request #3591: Remove unused heron.tools.common.src.python.utils.heronparser
Code0x58 opened a new pull request #3591: URL: https://github.com/apache/incubator-heron/pull/3591 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-heron] nicknezis commented on pull request #3589: Reorder the init containers in the Helm chart
nicknezis commented on pull request #3589: URL: https://github.com/apache/incubator-heron/pull/3589#issuecomment-665448238 @joshfischer1108 The order that init containers run is sequentially. The check for Zookeeper to be up and healthy was happening after the Bookkeeper format call. Bookkeeper needs Zookeeper to be healthy and therefore startup issues would occur when installing using Helm (everything at once). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-heron] nicknezis commented on issue #3516: The build does not work with Java version greater than 11
nicknezis commented on issue #3516: URL: https://github.com/apache/incubator-heron/issues/3516#issuecomment-665480260 I believe the Javadoc issue was resolved. Closing this issue. Please let us know if the issue persists. If so we will re-open this issue. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-heron] huijunwu commented on pull request #3587: Correct heron-executor CLI usage
huijunwu commented on pull request #3587: URL: https://github.com/apache/incubator-heron/pull/3587#issuecomment-664290210 instance may need similar refactoring https://github.com/apache/incubator-heron/blob/master/heron/instance/src/python/instance/st_heron_instance.py#L328 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-heron] Code0x58 opened a new pull request #3587: Correct heron-executor CLI usage
Code0x58 opened a new pull request #3587: URL: https://github.com/apache/incubator-heron/pull/3587 The usage was inaccurate/out of date for `heron-executor`, so this change leaves the parsing and documentation to [click](https://click.palletsprojects.com/) in a change that tried to be minimally invasive. The click options were made by automated substitution of the argparse arguments, then slightly organised (mostly alphabetical). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-heron] huijunwu commented on issue #3545: Help, error zlib
huijunwu commented on issue #3545: URL: https://github.com/apache/incubator-heron/issues/3545#issuecomment-664004338 I tried ubuntu20.04+python3, looks good to me. Do you still see this issue? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-heron] huijunwu commented on issue #3582: Not able to compile on Darwin platform
huijunwu commented on issue #3582:
URL:
https://github.com/apache/incubator-heron/issues/3582#issuecomment-664003979
My MacOs has several PEX environments, I used the following to force PEX on
py3
```
$ PY3=$(which python3)
$ echo "PEX_PYTHON_PATH=$(dirname ${PY3})" > ~/.pexrc
$ echo "PEX_PYTHON=${PY3}" >> ~/.pexrc
$ cat ~/.pexrc
PEX_PYTHON_PATH=/usr/local/bin
PEX_PYTHON=/usr/local/bin/python3
```
not sure if this helps..
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [incubator-heron] huijunwu merged pull request #3580: Fix log-reader for Python3
huijunwu merged pull request #3580: URL: https://github.com/apache/incubator-heron/pull/3580 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-heron] huijunwu closed pull request #3586: fix shell download
huijunwu closed pull request #3586: URL: https://github.com/apache/incubator-heron/pull/3586 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-heron] huijunwu commented on pull request #3586: fix shell download
huijunwu commented on pull request #3586: URL: https://github.com/apache/incubator-heron/pull/3586#issuecomment-663934276 close it, duplicate #3580 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-heron] huijunwu merged pull request #3579: Replace graph code in tracker
huijunwu merged pull request #3579: URL: https://github.com/apache/incubator-heron/pull/3579 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-heron] huijunwu merged pull request #3581: Support lifecycle for Mesos/Aurora
huijunwu merged pull request #3581: URL: https://github.com/apache/incubator-heron/pull/3581 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-heron] huijunwu edited a comment on pull request #3585: Purge heron-admin
huijunwu edited a comment on pull request #3585: URL: https://github.com/apache/incubator-heron/pull/3585#issuecomment-663923208 > * remove heron-admin binary from release > I am all for it. Hasn't had a chance to use it in the past 3 years. @nlu90 @huijunw Is there any case heron-admin is needed? Even if there is a unique function in the tool, given how rare it is used, we may still want to consider maybe there is a better place for the unique function. > > remove nomad binary from release > It could be useful for some people. I would suggest moving it into a new "extra" folder just in case some people use it. > > remove "standalone" cluster config from release and documentation > Overall I think it is ok. I am wondering about the use cases. Does it mean running on a Baremetal machine? Nomad seems for running a set of machines ranther than a single machine. call @jerrypeng for comments This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-heron] huijunwu commented on pull request #3585: Purge heron-admin
huijunwu commented on pull request #3585: URL: https://github.com/apache/incubator-heron/pull/3585#issuecomment-663923208 > * remove heron-admin binary from release > I am all for it. Hasn't had a chance to use it in the past 3 years. @nlu90 @huijunw Is there any case heron-admin is needed? Even if there is a unique function in the tool, given how rare it is used, we may still want to consider maybe there is a better place for the unique function. > > remove nomad binary from release > It could be useful for some people. I would suggest moving it into a new "extra" folder just in case some people use it. > > remove "standalone" cluster config from release and documentation > Overall I think it is ok. I am wondering about the use cases. Does it mean running on a Baremetal machine? Nomad seems for running a set of machines ranther than single machine. call @jerrypeng for comments This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-heron] huijunwu opened a new pull request #3586: fix shell download
huijunwu opened a new pull request #3586: URL: https://github.com/apache/incubator-heron/pull/3586 fix https://heronstreaming.slack.com/archives/C0VKQLKV0/p1595687004234500 The shell exception at str.decode https://github.com/apache/incubator-heron/blob/master/heron/shell/src/python/utils.py#L152 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-heron] nwangtw commented on pull request #3585: Purge heron-admin
nwangtw commented on pull request #3585: URL: https://github.com/apache/incubator-heron/pull/3585#issuecomment-663905226 - remove heron-admin binary from release I am all for it. Hasn't had a chance to use it in the past 3 years. @nlu90 @huijunw Is there any case heron-admin is needed? Even if there is a unique function in the tool, given how rare it is used, we may still want to consider maybe there is a better place for the unique function. remove nomad binary from release It could be useful for some people. I would suggest moving it into a new "extra" folder just in case some people use it. remove "standalone" cluster config from release and documentation Overall I think it is ok. I am wondering about the use cases. Does it mean running on a Baremetal machine? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-heron] Code0x58 opened a new pull request #3585: Purge heron-admin
Code0x58 opened a new pull request #3585: URL: https://github.com/apache/incubator-heron/pull/3585 * remove heron-admin binary from release * remove nomad binary from release * remove "standalone" cluster config from release and documentation This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-heron] Code0x58 opened a new pull request #3584: Deep clean heron-admin - probably better to remove entirely
Code0x58 opened a new pull request #3584: URL: https://github.com/apache/incubator-heron/pull/3584 I started cleaning up `heron-admin`'s code to make it a bit clearer, but along the way found that it was broken (at least because it tried to find nomad in the wrong place. Given all it does is try to manage one of many schedulers, when you can already use the local "cluster" or deploy your own cluster (including minikube or kind which seem to have support within the community), I'm thinking it would be better to strip out the `heron-admin` binary all together to simplify the ecosystem and reduce debt/maintenance cost as well. That said, the changes here are: * use [click](http://click.palletsprojects.com/) instead of argparse to make things more readable - parameters are explicitly passed instead of aways passing a mysterious `cl_args` dict everywhere - the interface is still the same as before, although docstrings may be different but also easier to change. I would be tempted to suggest reworking the interface to something more succinct if I didn't think it would be better to drop the whole thing * use `$VISUAL`, `$EDITOR`, or `vi` when editing the `inventory.yaml` in `heron-admin standalone set` * the code now at least passes a manual smoke test This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-heron] Code0x58 opened a new pull request #3583: Upgrade virtualenv
Code0x58 opened a new pull request #3583: URL: https://github.com/apache/incubator-heron/pull/3583 replace the 4 year old version with the latest version (released today) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-heron] nicknezis opened a new issue #3582: Not able to compile on Darwin platform
nicknezis opened a new issue #3582: URL: https://github.com/apache/incubator-heron/issues/3582 **Describe the bug** I can not run `bazel build --config=darwin_nostyle heron/...` without errors. **To Reproduce** Steps to reproduce the behavior: 1. `/.bazel_configure.py` ```zsh ❯ ./bazel_configure.py Platform Darwin Using C compiler : /usr/bin/gcc (4.2.1) Using C++ compiler: /usr/bin/g++ (4.2.1) Using C preprocessor : /usr/bin/cpp (11.0.0) Using C++ preprocessor: /usr/bin/cpp (11.0.0) Using linker : /usr/bin/ld Using JDK : /System/Library/Frameworks/JavaVM.framework/Versions/A Using Automake: /usr/local/Cellar/automake/1.16.2/bin/automake (1.16.2) Using Autoconf: /usr/local/Cellar/autoconf/2.69/bin/autoconf (2.69) Using Make: /usr/bin/make (3.81) Using Python3 : /usr/bin/python3 (3.7.3) Using Libtool : /usr/local/Cellar/libtool/2.4.6_2/bin/glibtool (2.4.6) Using archiver: /usr/bin/ar Using coverage tool : /usr/bin/gcov Using ant : /usr/local/Cellar/ant/1.10.8_1/bin/ant dwp : not found, but ok Using nm : /usr/bin/nm objcopy : not found, but ok Using objdump : /usr/bin/objdump Using strip : /usr/bin/strip ``` 2. `bazel build --config=darwin_nostyle heron/...` ```zsh ERROR: /Users/nnezis/workspace/incubator-heron-tracker/tools/rules/pex/BUILD:54:8: Bootstrapping pex //tools/rules/pex:pex_wrapper failed (Exit 1) /private/var/tmp/_bazel_nnezis/dcf39782d6886b57ad31493941aeb019/execroot/org_apache_heron/bazel-out/host/bin/external/virtualenv/virtualenv.runfiles/virtualenv/virtualenv.py:1039: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses import imp Traceback (most recent call last): File "/private/var/tmp/_bazel_nnezis/dcf39782d6886b57ad31493941aeb019/execroot/org_apache_heron/bazel-out/host/bin/external/virtualenv/virtualenv.runfiles/virtualenv/virtualenv.py", line 2328, in main() File "/private/var/tmp/_bazel_nnezis/dcf39782d6886b57ad31493941aeb019/execroot/org_apache_heron/bazel-out/host/bin/external/virtualenv/virtualenv.runfiles/virtualenv/virtualenv.py", line 713, in main symlink=options.symlink) File "/private/var/tmp/_bazel_nnezis/dcf39782d6886b57ad31493941aeb019/execroot/org_apache_heron/bazel-out/host/bin/external/virtualenv/virtualenv.runfiles/virtualenv/virtualenv.py", line 925, in create_environment site_packages=site_packages, clear=clear, symlink=symlink)) File "/private/var/tmp/_bazel_nnezis/dcf39782d6886b57ad31493941aeb019/execroot/org_apache_heron/bazel-out/host/bin/external/virtualenv/virtualenv.runfiles/virtualenv/virtualenv.py", line 1317, in install_python shutil.copy(original_python, py_executable) UnboundLocalError: local variable 'original_python' referenced before assignment ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-heron] nicknezis merged pull request #3576: Upgrade to Bazel 3.4.1
nicknezis merged pull request #3576: URL: https://github.com/apache/incubator-heron/pull/3576 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-heron] Code0x58 commented on a change in pull request #3580: Fix log-reader for Python3
Code0x58 commented on a change in pull request #3580:
URL: https://github.com/apache/incubator-heron/pull/3580#discussion_r459404375
##
File path: heron/shell/src/python/utils.py
##
@@ -135,21 +135,22 @@ def read_chunk(filename, offset=-1, length=-1,
escape_data=False):
if length == -1:
length = fstat.st_size - offset
- with open(filename, "r") as fp:
+ with open(filename, "rb") as fp:
fp.seek(offset)
try:
data = fp.read(length)
except IOError:
return {}
if data:
-data = _escape_data(data) if escape_data else data
+# use permissive decoding and escaping if escape_data is set, otherwise
use strict decoding
+data = _escape_data(data) if escape_data else data.decode()
return dict(offset=offset, length=len(data), data=data)
return dict(offset=offset, length=0)
def _escape_data(data):
- return escape(data.decode('utf8', 'replace'))
Review comment:
yep, although I think that decode will still want `errors='replace'` to
be compatible with the python2 behaviour
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [incubator-heron] thinker0 commented on a change in pull request #3580: Fix log-reader for Python3
thinker0 commented on a change in pull request #3580:
URL: https://github.com/apache/incubator-heron/pull/3580#discussion_r459189749
##
File path: heron/shell/src/python/utils.py
##
@@ -135,21 +135,22 @@ def read_chunk(filename, offset=-1, length=-1,
escape_data=False):
if length == -1:
length = fstat.st_size - offset
- with open(filename, "r") as fp:
+ with open(filename, "rb") as fp:
fp.seek(offset)
try:
data = fp.read(length)
except IOError:
return {}
if data:
-data = _escape_data(data) if escape_data else data
+# use permissive decoding and escaping if escape_data is set, otherwise
use strict decoding
+data = _escape_data(data) if escape_data else data.decode()
return dict(offset=offset, length=len(data), data=data)
return dict(offset=offset, length=0)
def _escape_data(data):
- return escape(data.decode('utf8', 'replace'))
Review comment:
```
escape(data.decode())
```
It works because I modified it.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [incubator-heron] thinker0 commented on a change in pull request #3580: Fix log-reader for Python3
thinker0 commented on a change in pull request #3580:
URL: https://github.com/apache/incubator-heron/pull/3580#discussion_r459174401
##
File path: heron/shell/src/python/utils.py
##
@@ -135,21 +135,22 @@ def read_chunk(filename, offset=-1, length=-1,
escape_data=False):
if length == -1:
length = fstat.st_size - offset
- with open(filename, "r") as fp:
+ with open(filename, "rb") as fp:
fp.seek(offset)
try:
data = fp.read(length)
except IOError:
return {}
if data:
-data = _escape_data(data) if escape_data else data
+# use permissive decoding and escaping if escape_data is set, otherwise
use strict decoding
+data = _escape_data(data) if escape_data else data.decode()
return dict(offset=offset, length=len(data), data=data)
return dict(offset=offset, length=0)
def _escape_data(data):
- return escape(data.decode('utf8', 'replace'))
Review comment:
```
with open(filename, "rb") as fp:
```
If you read it as binary, escape doesn't work.
```
[E 200723 10:40:39 web:1407] Uncaught exception GET
/filedata/./log-files/container_46_pulsar-prod-4_96.log.0?offset=4191685=5
(10.128.139.55)
HTTPServerRequest(protocol='http',
host='shared-aurora-044-ladp-jp2p-prod:31516', method='GET',
uri='/filedata/./log-files/container_46_pulsar-prod-4_96.log.0?offset=4191685=5',
version='HTTP/1.1', remote_ip='10.128.139.55', headers={'Connection': 'close',
'Host': 'shared-aurora-044-ladp-jp2p-prod:31516', 'Accept-Encoding': 'gzip'})
Traceback (most recent call last):
File
"/var/lib/mesos/slaves/9a7b96a9-a670-48e3-bedf-051a91ac5a9b-S251/frameworks/c663397e-a472-43bd-92dd-d97027fcf6ce-/executors/thermos-www-release-heron-system-access-es-others-46-02c80d1d-6257-4b42-829c-6fafbed25b8d/runs/8597c5be-55d4-4b8b-9e2e-2b8faddc3372/sandbox/.pex/installed_wheels/ba642ca162d5bf8bbf2fad77d02edcf2a3188eb8/tornado-4.0.2-cp36-cp36m-linux_x86_64.whl/tornado/web.py",
line 1288, in _stack_context_handle_exception
raise_exc_info((type, value, traceback))
File "", line 3, in raise_exc_info
File
"/var/lib/mesos/slaves/9a7b96a9-a670-48e3-bedf-051a91ac5a9b-S251/frameworks/c663397e-a472-43bd-92dd-d97027fcf6ce-/executors/thermos-www-release-heron-system-access-es-others-46-02c80d1d-6257-4b42-829c-6fafbed25b8d/runs/8597c5be-55d4-4b8b-9e2e-2b8faddc3372/sandbox/.pex/installed_wheels/ba642ca162d5bf8bbf2fad77d02edcf2a3188eb8/tornado-4.0.2-cp36-cp36m-linux_x86_64.whl/tornado/web.py",
line 1475, in wrapper
result = method(self, *args, **kwargs)
File
"/var/lib/mesos/slaves/9a7b96a9-a670-48e3-bedf-051a91ac5a9b-S251/frameworks/c663397e-a472-43bd-92dd-d97027fcf6ce-/executors/thermos-www-release-heron-system-access-es-others-46-02c80d1d-6257-4b42-829c-6fafbed25b8d/runs/8597c5be-55d4-4b8b-9e2e-2b8faddc3372/sandbox/.pex/code/ca989480a7444b1c46f32b02147ba41568922357/heron/shell/src/python/handlers/filedatahandler.py",
line 50, in get
data = utils.read_chunk(path, offset=offset, length=length,
escape_data=True)
File
"/var/lib/mesos/slaves/9a7b96a9-a670-48e3-bedf-051a91ac5a9b-S251/frameworks/c663397e-a472-43bd-92dd-d97027fcf6ce-/executors/thermos-www-release-heron-system-access-es-others-46-02c80d1d-6257-4b42-829c-6fafbed25b8d/runs/8597c5be-55d4-4b8b-9e2e-2b8faddc3372/sandbox/.pex/code/ca989480a7444b1c46f32b02147ba41568922357/heron/shell/src/python/utils.py",
line 147, in read_chunk
data = _escape_data(data) if escape_data else data.decode()
File
"/var/lib/mesos/slaves/9a7b96a9-a670-48e3-bedf-051a91ac5a9b-S251/frameworks/c663397e-a472-43bd-92dd-d97027fcf6ce-/executors/thermos-www-release-heron-system-access-es-others-46-02c80d1d-6257-4b42-829c-6fafbed25b8d/runs/8597c5be-55d4-4b8b-9e2e-2b8faddc3372/sandbox/.pex/code/ca989480a7444b1c46f32b02147ba41568922357/heron/shell/src/python/utils.py",
line 153, in _escape_data
return escape(data)
File "/usr/lib64/python3.6/xml/sax/saxutils.py", line 27, in escape
data = data.replace("&", "")
TypeError: a bytes-like object is required, not 'str'
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [incubator-heron] thinker0 commented on a change in pull request #3580: Fix log-reader for Python3
thinker0 commented on a change in pull request #3580:
URL: https://github.com/apache/incubator-heron/pull/3580#discussion_r459174401
##
File path: heron/shell/src/python/utils.py
##
@@ -135,21 +135,22 @@ def read_chunk(filename, offset=-1, length=-1,
escape_data=False):
if length == -1:
length = fstat.st_size - offset
- with open(filename, "r") as fp:
+ with open(filename, "rb") as fp:
fp.seek(offset)
try:
data = fp.read(length)
except IOError:
return {}
if data:
-data = _escape_data(data) if escape_data else data
+# use permissive decoding and escaping if escape_data is set, otherwise
use strict decoding
+data = _escape_data(data) if escape_data else data.decode()
return dict(offset=offset, length=len(data), data=data)
return dict(offset=offset, length=0)
def _escape_data(data):
- return escape(data.decode('utf8', 'replace'))
Review comment:
```
with open(filename, "r") as fp:
```
If you read it as binary, escape doesn't work.
```
[E 200723 10:40:39 web:1407] Uncaught exception GET
/filedata/./log-files/container_46_pulsar-prod-4_96.log.0?offset=4191685=5
(10.128.139.55)
HTTPServerRequest(protocol='http',
host='shared-aurora-044-ladp-jp2p-prod:31516', method='GET',
uri='/filedata/./log-files/container_46_pulsar-prod-4_96.log.0?offset=4191685=5',
version='HTTP/1.1', remote_ip='10.128.139.55', headers={'Connection': 'close',
'Host': 'shared-aurora-044-ladp-jp2p-prod:31516', 'Accept-Encoding': 'gzip'})
Traceback (most recent call last):
File
"/var/lib/mesos/slaves/9a7b96a9-a670-48e3-bedf-051a91ac5a9b-S251/frameworks/c663397e-a472-43bd-92dd-d97027fcf6ce-/executors/thermos-www-release-heron-system-access-es-others-46-02c80d1d-6257-4b42-829c-6fafbed25b8d/runs/8597c5be-55d4-4b8b-9e2e-2b8faddc3372/sandbox/.pex/installed_wheels/ba642ca162d5bf8bbf2fad77d02edcf2a3188eb8/tornado-4.0.2-cp36-cp36m-linux_x86_64.whl/tornado/web.py",
line 1288, in _stack_context_handle_exception
raise_exc_info((type, value, traceback))
File "", line 3, in raise_exc_info
File
"/var/lib/mesos/slaves/9a7b96a9-a670-48e3-bedf-051a91ac5a9b-S251/frameworks/c663397e-a472-43bd-92dd-d97027fcf6ce-/executors/thermos-www-release-heron-system-access-es-others-46-02c80d1d-6257-4b42-829c-6fafbed25b8d/runs/8597c5be-55d4-4b8b-9e2e-2b8faddc3372/sandbox/.pex/installed_wheels/ba642ca162d5bf8bbf2fad77d02edcf2a3188eb8/tornado-4.0.2-cp36-cp36m-linux_x86_64.whl/tornado/web.py",
line 1475, in wrapper
result = method(self, *args, **kwargs)
File
"/var/lib/mesos/slaves/9a7b96a9-a670-48e3-bedf-051a91ac5a9b-S251/frameworks/c663397e-a472-43bd-92dd-d97027fcf6ce-/executors/thermos-www-release-heron-system-access-es-others-46-02c80d1d-6257-4b42-829c-6fafbed25b8d/runs/8597c5be-55d4-4b8b-9e2e-2b8faddc3372/sandbox/.pex/code/ca989480a7444b1c46f32b02147ba41568922357/heron/shell/src/python/handlers/filedatahandler.py",
line 50, in get
data = utils.read_chunk(path, offset=offset, length=length,
escape_data=True)
File
"/var/lib/mesos/slaves/9a7b96a9-a670-48e3-bedf-051a91ac5a9b-S251/frameworks/c663397e-a472-43bd-92dd-d97027fcf6ce-/executors/thermos-www-release-heron-system-access-es-others-46-02c80d1d-6257-4b42-829c-6fafbed25b8d/runs/8597c5be-55d4-4b8b-9e2e-2b8faddc3372/sandbox/.pex/code/ca989480a7444b1c46f32b02147ba41568922357/heron/shell/src/python/utils.py",
line 147, in read_chunk
data = _escape_data(data) if escape_data else data.decode()
File
"/var/lib/mesos/slaves/9a7b96a9-a670-48e3-bedf-051a91ac5a9b-S251/frameworks/c663397e-a472-43bd-92dd-d97027fcf6ce-/executors/thermos-www-release-heron-system-access-es-others-46-02c80d1d-6257-4b42-829c-6fafbed25b8d/runs/8597c5be-55d4-4b8b-9e2e-2b8faddc3372/sandbox/.pex/code/ca989480a7444b1c46f32b02147ba41568922357/heron/shell/src/python/utils.py",
line 153, in _escape_data
return escape(data)
File "/usr/lib64/python3.6/xml/sax/saxutils.py", line 27, in escape
data = data.replace("&", "")
TypeError: a bytes-like object is required, not 'str'
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [incubator-heron] thinker0 commented on a change in pull request #3580: Fix log-reader for Python3
thinker0 commented on a change in pull request #3580:
URL: https://github.com/apache/incubator-heron/pull/3580#discussion_r459171310
##
File path: heron/shell/src/python/utils.py
##
@@ -135,21 +135,22 @@ def read_chunk(filename, offset=-1, length=-1,
escape_data=False):
if length == -1:
length = fstat.st_size - offset
- with open(filename, "r") as fp:
+ with open(filename, "rb") as fp:
fp.seek(offset)
try:
data = fp.read(length)
except IOError:
return {}
if data:
-data = _escape_data(data) if escape_data else data
+# use permissive decoding and escaping if escape_data is set, otherwise
use strict decoding
+data = _escape_data(data) if escape_data else data.decode()
return dict(offset=offset, length=len(data), data=data)
return dict(offset=offset, length=0)
def _escape_data(data):
- return escape(data.decode('utf8', 'replace'))
Review comment:
The code is not working in python3 rather than python compatible. I am
trying to modify it because the above error occurs.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [incubator-heron] Code0x58 commented on a change in pull request #3580: Fix log-reader for Python3
Code0x58 commented on a change in pull request #3580:
URL: https://github.com/apache/incubator-heron/pull/3580#discussion_r459163401
##
File path: heron/shell/src/python/utils.py
##
@@ -135,21 +135,22 @@ def read_chunk(filename, offset=-1, length=-1,
escape_data=False):
if length == -1:
length = fstat.st_size - offset
- with open(filename, "r") as fp:
+ with open(filename, "rb") as fp:
fp.seek(offset)
try:
data = fp.read(length)
except IOError:
return {}
if data:
-data = _escape_data(data) if escape_data else data
+# use permissive decoding and escaping if escape_data is set, otherwise
use strict decoding
+data = _escape_data(data) if escape_data else data.decode()
return dict(offset=offset, length=len(data), data=data)
return dict(offset=offset, length=0)
def _escape_data(data):
- return escape(data.decode('utf8', 'replace'))
Review comment:
once the change to this is reverted, the PR will be compatible with the
python2 behaviour - alternativley you could do the decoding in the `data = ...`
line above
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [incubator-heron] Code0x58 commented on a change in pull request #3580: Fix log-reader for Python3
Code0x58 commented on a change in pull request #3580:
URL: https://github.com/apache/incubator-heron/pull/3580#discussion_r459163401
##
File path: heron/shell/src/python/utils.py
##
@@ -135,21 +135,22 @@ def read_chunk(filename, offset=-1, length=-1,
escape_data=False):
if length == -1:
length = fstat.st_size - offset
- with open(filename, "r") as fp:
+ with open(filename, "rb") as fp:
fp.seek(offset)
try:
data = fp.read(length)
except IOError:
return {}
if data:
-data = _escape_data(data) if escape_data else data
+# use permissive decoding and escaping if escape_data is set, otherwise
use strict decoding
+data = _escape_data(data) if escape_data else data.decode()
return dict(offset=offset, length=len(data), data=data)
return dict(offset=offset, length=0)
def _escape_data(data):
- return escape(data.decode('utf8', 'replace'))
Review comment:
once the change to this is reverted, the PR will be compatible with the
python2 behaviour
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [incubator-heron] Code0x58 closed issue #3559: Use builtin python3 modules instead of externals
Code0x58 closed issue #3559: URL: https://github.com/apache/incubator-heron/issues/3559 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-heron] Code0x58 closed issue #3556: Use cloudpickle via pypi rather than vendoring
Code0x58 closed issue #3556: URL: https://github.com/apache/incubator-heron/issues/3556 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-heron] Code0x58 closed issue #3575: Unvendor javaobj
Code0x58 closed issue #3575: URL: https://github.com/apache/incubator-heron/issues/3575 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-heron] Code0x58 commented on a change in pull request #3580: Fix log-reader for Python3
Code0x58 commented on a change in pull request #3580: URL: https://github.com/apache/incubator-heron/pull/3580#discussion_r459152556 ## File path: heron/shell/src/python/utils.py ## @@ -135,7 +135,7 @@ def read_chunk(filename, offset=-1, length=-1, escape_data=False): if length == -1: length = fstat.st_size - offset - with open(filename, "r") as fp: Review comment: I think the `encoding='utf-8'` is implied on normal builds, including WIndows as of python3.6, so the change on this line could be dropped This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-heron] thinker0 opened a new pull request #3581: Support lifecycle for Mesos/Aurora
thinker0 opened a new pull request #3581: URL: https://github.com/apache/incubator-heron/pull/3581 Support lifecycle for Mesos/Aurora http://aurora.apache.org/documentation/latest/reference/configuration/#lifecycleconfig-objects This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-heron] Code0x58 commented on a change in pull request #3580: Fix log-reader for Python3
Code0x58 commented on a change in pull request #3580: URL: https://github.com/apache/incubator-heron/pull/3580#discussion_r459154368 ## File path: heron/shell/src/python/utils.py ## @@ -135,7 +135,7 @@ def read_chunk(filename, offset=-1, length=-1, escape_data=False): if length == -1: length = fstat.st_size - offset - with open(filename, "r") as fp: Review comment: to be consistent with the previous behaviour (not sure it still makes sense), I think the whole PR change would be something like: ```python with open(filename, "rb") as fp: ... if data: # use permissive decoding and escaping if escape_data is set, otherwise use strict decoding data = _escape_data(data) if escape_data else data.decode() ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-heron] Code0x58 commented on a change in pull request #3580: Fix log-reader for Python3
Code0x58 commented on a change in pull request #3580: URL: https://github.com/apache/incubator-heron/pull/3580#discussion_r459154368 ## File path: heron/shell/src/python/utils.py ## @@ -135,7 +135,7 @@ def read_chunk(filename, offset=-1, length=-1, escape_data=False): if length == -1: length = fstat.st_size - offset - with open(filename, "r") as fp: Review comment: to be consistent with the previous behaviour (not sure it still makes sense), I think the whole PR change would be something like: ```python with open(filename, "rb") as fp: ... if data: # use permissive decoding and escaping if escape_data is set, otherwise use strict decoding data = _escape_data(data) if escape_data else data.decode() This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-heron] Code0x58 commented on a change in pull request #3580: Fix log-reader for Python3
Code0x58 commented on a change in pull request #3580: URL: https://github.com/apache/incubator-heron/pull/3580#discussion_r459154368 ## File path: heron/shell/src/python/utils.py ## @@ -135,7 +135,7 @@ def read_chunk(filename, offset=-1, length=-1, escape_data=False): if length == -1: length = fstat.st_size - offset - with open(filename, "r") as fp: Review comment: to be consistent with the previous behaviour (not sure it still makes sense), I think this would be something like: ```python with open(filename, "rb") as fp: ... if data: # use permissive decoding and escaping if escape_data is set, otherwise use strict decoding data = _escape_data(data) if escape_data else data.decode() This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-heron] Code0x58 commented on a change in pull request #3580: Fix log-reader for Python3
Code0x58 commented on a change in pull request #3580: URL: https://github.com/apache/incubator-heron/pull/3580#discussion_r459152556 ## File path: heron/shell/src/python/utils.py ## @@ -135,7 +135,7 @@ def read_chunk(filename, offset=-1, length=-1, escape_data=False): if length == -1: length = fstat.st_size - offset - with open(filename, "r") as fp: Review comment: I think the `encoding='utf-8'` is implied on normal builds, including WIndows as of python3.6, so the change on this line could be dropped This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-heron] thinker0 opened a new pull request #3580: Fix log-reader for Python3
thinker0 opened a new pull request #3580: URL: https://github.com/apache/incubator-heron/pull/3580 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-heron] Code0x58 opened a new pull request #3579: Replace graph code in tracker
Code0x58 opened a new pull request #3579: URL: https://github.com/apache/incubator-heron/pull/3579 :broom: :dash: This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-heron] nicknezis merged pull request #3574: Unvendor pyobj + fix py2->py3 compatability fixes.
nicknezis merged pull request #3574: URL: https://github.com/apache/incubator-heron/pull/3574 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-heron] thinker0 edited a comment on pull request #3574: Unvendor pyobj + fix py2->py3 compatability fixes.
thinker0 edited a comment on pull request #3574: URL: https://github.com/apache/incubator-heron/pull/3574#issuecomment-662158049 +1 really good work. Approved This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-heron] thinker0 commented on pull request #3574: Unvendor pyobj + fix py2->py3 compatability fixes.
thinker0 commented on pull request #3574: URL: https://github.com/apache/incubator-heron/pull/3574#issuecomment-662158049 +1 Approved This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-heron] Code0x58 commented on pull request #3574: Unvendor pyobj + fix py2->py3 compatability fixes.
Code0x58 commented on pull request #3574: URL: https://github.com/apache/incubator-heron/pull/3574#issuecomment-662133945 an initial look into `convert_pb_kvs(...)` makes it me think it's a pretty hairy thing, so I didn't try to do anything smart and pushed what might get things running. I'll keep having a look through the file to see if I can work out how to clean it up, but I'm not optimistic This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-heron] nicknezis merged pull request #3578: update readme
nicknezis merged pull request #3578: URL: https://github.com/apache/incubator-heron/pull/3578 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-heron] nicknezis closed pull request #3512: Change url of javadoc
nicknezis closed pull request #3512: URL: https://github.com/apache/incubator-heron/pull/3512 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-heron] joshfischer1108 opened a new pull request #3578: update readme
joshfischer1108 opened a new pull request #3578: URL: https://github.com/apache/incubator-heron/pull/3578 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-heron] joshfischer1108 edited a comment on pull request #3512: Change url of javadoc
joshfischer1108 edited a comment on pull request #3512: URL: https://github.com/apache/incubator-heron/pull/3512#issuecomment-661785928 I think this should be closed or modified. We are on java 11 now. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-heron] joshfischer1108 edited a comment on pull request #3512: Change url of javadoc
joshfischer1108 edited a comment on pull request #3512: URL: https://github.com/apache/incubator-heron/pull/3512#issuecomment-661785928 I think this should be closed. We are on java 11 now. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-heron] joshfischer1108 commented on pull request #3512: Change url of javadoc
joshfischer1108 commented on pull request #3512: URL: https://github.com/apache/incubator-heron/pull/3512#issuecomment-661785928 I think this should be closed. We are java 11 now. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-heron] nicknezis commented on pull request #3512: Change url of javadoc
nicknezis commented on pull request #3512: URL: https://github.com/apache/incubator-heron/pull/3512#issuecomment-661605988 Do we want this merged in or closed? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-heron] thinker0 edited a comment on pull request #3574: Unvendor pyobj + fix py2->py3 compatability fixes.
thinker0 edited a comment on pull request #3574: URL: https://github.com/apache/incubator-heron/pull/3574#issuecomment-661433596 @Code0x58 sorry. The last one goes like this. Other features seem to work fine. ``` [2020-07-21 08:21:52 +0900] [ERROR]: Failed to parse data as java object Traceback (most recent call last): File "/Users/thinker0/.pex/code/33e1a410e8efde46fe6074757a0631f8eb7c7d3b/heron/tools/tracker/src/python/tracker.py", line 65, in _convert_java_value if pobj.is_primitive(): AttributeError: 'JavaMap' object has no attribute 'is_primitive' [2020-07-21 08:21:52 +0900] [ERROR]: Failed to parse data as java object Traceback (most recent call last): File "/Users/thinker0/.pex/code/33e1a410e8efde46fe6074757a0631f8eb7c7d3b/heron/tools/tracker/src/python/tracker.py", line 65, in _convert_java_value if pobj.is_primitive(): AttributeError: 'JavaMap' object has no attribute 'is_primitive' [2020-07-21 08:21:52 +0900] [ERROR]: Failed to parse data as java object Traceback (most recent call last): File "/Users/thinker0/.pex/code/33e1a410e8efde46fe6074757a0631f8eb7c7d3b/heron/tools/tracker/src/python/tracker.py", line 65, in _convert_java_value if pobj.is_primitive(): AttributeError: 'JavaObject' object has no attribute 'is_primitive' [2020-07-21 09:29:25 +0900] [ERROR]: Failed to parse data as java object Traceback (most recent call last): File "/Users/thinker0/.pex/code/33e1a410e8efde46fe6074757a0631f8eb7c7d3b/heron/tools/tracker/src/python/tracker.py", line 65, in _convert_java_value if pobj.is_primitive(): AttributeError: 'JavaInt' object has no attribute 'is_primitive' [2020-07-21 09:29:25 +0900] [ERROR]: Failed to parse data as java object Traceback (most recent call last): File "/Users/thinker0/.pex/code/33e1a410e8efde46fe6074757a0631f8eb7c7d3b/heron/tools/tracker/src/python/tracker.py", line 65, in _convert_java_value if pobj.is_primitive(): AttributeError: 'JavaBool' object has no attribute 'is_primitive' [2020-07-21 09:29:25 +0900] [ERROR]: Failed to parse data as java object Traceback (most recent call last): File "/Users/thinker0/.pex/code/33e1a410e8efde46fe6074757a0631f8eb7c7d3b/heron/tools/tracker/src/python/tracker.py", line 65, in _convert_java_value if pobj.is_primitive(): AttributeError: 'JavaInt' object has no attribute 'is_primitive' [2020-07-21 09:29:25 +0900] [ERROR]: Failed to parse data as java object Traceback (most recent call last): File "/Users/thinker0/.pex/code/33e1a410e8efde46fe6074757a0631f8eb7c7d3b/heron/tools/tracker/src/python/tracker.py", line 65, in _convert_java_value if pobj.is_primitive(): AttributeError: 'JavaInt' object has no attribute 'is_primitive' [2020-07-21 09:29:25 +0900] [ERROR]: Failed to parse data as java object Traceback (most recent call last): File "/Users/thinker0/.pex/code/33e1a410e8efde46fe6074757a0631f8eb7c7d3b/heron/tools/tracker/src/python/tracker.py", line 65, in _convert_java_value if pobj.is_primitive(): AttributeError: 'JavaBool' object has no attribute 'is_primitive' [2020-07-21 09:29:25 +0900] [ERROR]: Failed to parse data as java object Traceback (most recent call last): File "/Users/thinker0/.pex/code/33e1a410e8efde46fe6074757a0631f8eb7c7d3b/heron/tools/tracker/src/python/tracker.py", line 65, in _convert_java_value if pobj.is_primitive(): AttributeError: 'JavaInt' object has no attribute 'is_primitive' [2020-07-21 09:29:25 +0900] [ERROR]: Failed to parse data as java object Traceback (most recent call last): File "/Users/thinker0/.pex/code/33e1a410e8efde46fe6074757a0631f8eb7c7d3b/heron/tools/tracker/src/python/tracker.py", line 65, in _convert_java_value if pobj.is_primitive(): AttributeError: 'JavaInt' object has no attribute 'is_primitive' [2020-07-21 09:29:25 +0900] [ERROR]: Failed to parse data as java object Traceback (most recent call last): File "/Users/thinker0/.pex/code/33e1a410e8efde46fe6074757a0631f8eb7c7d3b/heron/tools/tracker/src/python/tracker.py", line 65, in _convert_java_value if pobj.is_primitive(): AttributeError: 'JavaBool' object has no attribute 'is_primitive' [2020-07-21 09:29:25 +0900] [ERROR]: Failed to parse data as java object Traceback (most recent call last): File "/Users/thinker0/.pex/code/33e1a410e8efde46fe6074757a0631f8eb7c7d3b/heron/tools/tracker/src/python/tracker.py", line 65, in _convert_java_value if pobj.is_primitive(): AttributeError: 'JavaInt' object has no attribute 'is_primitive' [2020-07-21 09:29:25 +0900] [ERROR]: Failed to parse data as java object Traceback (most recent call last): File "/Users/thinker0/.pex/code/33e1a410e8efde46fe6074757a0631f8eb7c7d3b/heron/tools/tracker/src/python/tracker.py", line 65, in _convert_java_value if pobj.is_primitive(): AttributeError:
[GitHub] [incubator-heron] thinker0 edited a comment on pull request #3574: Unvendor pyobj + fix py2->py3 compatability fixes.
thinker0 edited a comment on pull request #3574: URL: https://github.com/apache/incubator-heron/pull/3574#issuecomment-661433596 @Code0x58 sorry. The last one goes like this. ``` [2020-07-21 08:21:52 +0900] [ERROR]: Failed to parse data as java object Traceback (most recent call last): File "/Users/thinker0/.pex/code/33e1a410e8efde46fe6074757a0631f8eb7c7d3b/heron/tools/tracker/src/python/tracker.py", line 65, in _convert_java_value if pobj.is_primitive(): AttributeError: 'JavaMap' object has no attribute 'is_primitive' [2020-07-21 08:21:52 +0900] [ERROR]: Failed to parse data as java object Traceback (most recent call last): File "/Users/thinker0/.pex/code/33e1a410e8efde46fe6074757a0631f8eb7c7d3b/heron/tools/tracker/src/python/tracker.py", line 65, in _convert_java_value if pobj.is_primitive(): AttributeError: 'JavaMap' object has no attribute 'is_primitive' [2020-07-21 08:21:52 +0900] [ERROR]: Failed to parse data as java object Traceback (most recent call last): File "/Users/thinker0/.pex/code/33e1a410e8efde46fe6074757a0631f8eb7c7d3b/heron/tools/tracker/src/python/tracker.py", line 65, in _convert_java_value if pobj.is_primitive(): AttributeError: 'JavaObject' object has no attribute 'is_primitive' [2020-07-21 09:29:25 +0900] [ERROR]: Failed to parse data as java object Traceback (most recent call last): File "/Users/thinker0/.pex/code/33e1a410e8efde46fe6074757a0631f8eb7c7d3b/heron/tools/tracker/src/python/tracker.py", line 65, in _convert_java_value if pobj.is_primitive(): AttributeError: 'JavaInt' object has no attribute 'is_primitive' [2020-07-21 09:29:25 +0900] [ERROR]: Failed to parse data as java object Traceback (most recent call last): File "/Users/thinker0/.pex/code/33e1a410e8efde46fe6074757a0631f8eb7c7d3b/heron/tools/tracker/src/python/tracker.py", line 65, in _convert_java_value if pobj.is_primitive(): AttributeError: 'JavaBool' object has no attribute 'is_primitive' [2020-07-21 09:29:25 +0900] [ERROR]: Failed to parse data as java object Traceback (most recent call last): File "/Users/thinker0/.pex/code/33e1a410e8efde46fe6074757a0631f8eb7c7d3b/heron/tools/tracker/src/python/tracker.py", line 65, in _convert_java_value if pobj.is_primitive(): AttributeError: 'JavaInt' object has no attribute 'is_primitive' [2020-07-21 09:29:25 +0900] [ERROR]: Failed to parse data as java object Traceback (most recent call last): File "/Users/thinker0/.pex/code/33e1a410e8efde46fe6074757a0631f8eb7c7d3b/heron/tools/tracker/src/python/tracker.py", line 65, in _convert_java_value if pobj.is_primitive(): AttributeError: 'JavaInt' object has no attribute 'is_primitive' [2020-07-21 09:29:25 +0900] [ERROR]: Failed to parse data as java object Traceback (most recent call last): File "/Users/thinker0/.pex/code/33e1a410e8efde46fe6074757a0631f8eb7c7d3b/heron/tools/tracker/src/python/tracker.py", line 65, in _convert_java_value if pobj.is_primitive(): AttributeError: 'JavaBool' object has no attribute 'is_primitive' [2020-07-21 09:29:25 +0900] [ERROR]: Failed to parse data as java object Traceback (most recent call last): File "/Users/thinker0/.pex/code/33e1a410e8efde46fe6074757a0631f8eb7c7d3b/heron/tools/tracker/src/python/tracker.py", line 65, in _convert_java_value if pobj.is_primitive(): AttributeError: 'JavaInt' object has no attribute 'is_primitive' [2020-07-21 09:29:25 +0900] [ERROR]: Failed to parse data as java object Traceback (most recent call last): File "/Users/thinker0/.pex/code/33e1a410e8efde46fe6074757a0631f8eb7c7d3b/heron/tools/tracker/src/python/tracker.py", line 65, in _convert_java_value if pobj.is_primitive(): AttributeError: 'JavaInt' object has no attribute 'is_primitive' [2020-07-21 09:29:25 +0900] [ERROR]: Failed to parse data as java object Traceback (most recent call last): File "/Users/thinker0/.pex/code/33e1a410e8efde46fe6074757a0631f8eb7c7d3b/heron/tools/tracker/src/python/tracker.py", line 65, in _convert_java_value if pobj.is_primitive(): AttributeError: 'JavaBool' object has no attribute 'is_primitive' [2020-07-21 09:29:25 +0900] [ERROR]: Failed to parse data as java object Traceback (most recent call last): File "/Users/thinker0/.pex/code/33e1a410e8efde46fe6074757a0631f8eb7c7d3b/heron/tools/tracker/src/python/tracker.py", line 65, in _convert_java_value if pobj.is_primitive(): AttributeError: 'JavaInt' object has no attribute 'is_primitive' [2020-07-21 09:29:25 +0900] [ERROR]: Failed to parse data as java object Traceback (most recent call last): File "/Users/thinker0/.pex/code/33e1a410e8efde46fe6074757a0631f8eb7c7d3b/heron/tools/tracker/src/python/tracker.py", line 65, in _convert_java_value if pobj.is_primitive(): AttributeError: 'JavaInt' object has no attribute
[GitHub] [incubator-heron] Code0x58 commented on pull request #3574: Unvendor pyobj + fix py2->py3 compatability fixes.
Code0x58 commented on pull request #3574: URL: https://github.com/apache/incubator-heron/pull/3574#issuecomment-661468879 Thanks again @thinker0! So much for hopes of a dumb upgrade. I'll have to spend some time getting to understand the old and new `javaobj` modules to do a proper migration for `convert_pb_kvs(...)`, which I'll aim to get on tomorrow evening This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-heron] thinker0 commented on pull request #3574: Unvendor pyobj + fix py2->py3 compatability fixes.
thinker0 commented on pull request #3574: URL: https://github.com/apache/incubator-heron/pull/3574#issuecomment-661433596 @Code0x58 sorry. The last one goes like this. ``` [2020-07-21 08:21:52 +0900] [ERROR]: Failed to parse data as java object Traceback (most recent call last): File "/Users/thinker0/.pex/code/33e1a410e8efde46fe6074757a0631f8eb7c7d3b/heron/tools/tracker/src/python/tracker.py", line 65, in _convert_java_value if pobj.is_primitive(): AttributeError: 'JavaMap' object has no attribute 'is_primitive' [2020-07-21 08:21:52 +0900] [ERROR]: Failed to parse data as java object Traceback (most recent call last): File "/Users/thinker0/.pex/code/33e1a410e8efde46fe6074757a0631f8eb7c7d3b/heron/tools/tracker/src/python/tracker.py", line 65, in _convert_java_value if pobj.is_primitive(): AttributeError: 'JavaMap' object has no attribute 'is_primitive' [2020-07-21 08:21:52 +0900] [ERROR]: Failed to parse data as java object Traceback (most recent call last): File "/Users/thinker0/.pex/code/33e1a410e8efde46fe6074757a0631f8eb7c7d3b/heron/tools/tracker/src/python/tracker.py", line 65, in _convert_java_value if pobj.is_primitive(): AttributeError: 'JavaObject' object has no attribute 'is_primitive' ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-heron] Code0x58 commented on pull request #3574: Unvendor pyobj + fix py2->py3 compatability fixes.
Code0x58 commented on pull request #3574: URL: https://github.com/apache/incubator-heron/pull/3574#issuecomment-661324466 CI passed on my machine, so hopefully the switch to the maintained version of `pyobj` sorts things out! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-heron] nicknezis merged pull request #3568: Unvendor cloudpickle
nicknezis merged pull request #3568: URL: https://github.com/apache/incubator-heron/pull/3568 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-heron] joshfischer1108 commented on a change in pull request #3574: Decode string when deserializing Java object in python
joshfischer1108 commented on a change in pull request #3574: URL: https://github.com/apache/incubator-heron/pull/3574#discussion_r457585129 ## File path: heron/statemgrs/src/python/zkstatemanager.py ## @@ -170,7 +170,7 @@ def _get_topology_with_watch(self, topologyName, callback, isWatching): # pylint: disable=unused-variable, unused-argument @self.client.DataWatch(path) -def watch_topology(data, stats): +def watch_topology(data, stats, event): """ watch topology """ if data: topology = Topology() Review comment: Ok. Thanks for clarifying. Makes sense to me. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-heron] Code0x58 commented on pull request #3574: Decode string when deserializing Java object in python
Code0x58 commented on pull request #3574: URL: https://github.com/apache/incubator-heron/pull/3574#issuecomment-661050567 humm, that error looks like it could be a deeper `pyobj` issue, so I should probably stop trying to patch it, and just upgrade `pyobj` for https://github.com/apache/incubator-heron/issues/3575 - I'll be able to give it more attention tonight This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-heron] thinker0 commented on pull request #3574: Decode string when deserializing Java object in python
thinker0 commented on pull request #3574:
URL: https://github.com/apache/incubator-heron/pull/3574#issuecomment-661046491
@Code0x58 Another
```
[2020-07-20 22:40:03 +0900] [INFO]: Setting topology info for topology:
heron-ladp_oa_message_id_report-to-ladp-log-es
[2020-07-20 22:40:03 +0900] [ERROR]: ==Oops state dump=
[2020-07-20 22:40:03 +0900] [ERROR]: References:
[[com.typesafe.config.impl.SerializedConfigValue:0x0],
]
[2020-07-20 22:40:03 +0900] [ERROR]: Stream seeking back at -16 byte (2nd
line is an actual position!):
[2020-07-20 22:40:03 +0900] [ERROR]: Failed to parse data as java object
Traceback (most recent call last):
File
"/Users/thinker0/.pex/code/89ddc931e94debf0336dd56177721e9f35b441e3/heron/tools/tracker/src/python/javaobj.py",
line 221, in readObject
_, res = self._read_and_exec_opcode(ident=0)
File
"/Users/thinker0/.pex/code/89ddc931e94debf0336dd56177721e9f35b441e3/heron/tools/tracker/src/python/javaobj.py",
line 256, in _read_and_exec_opcode
return (opid, handler(ident=ident))
File
"/Users/thinker0/.pex/code/89ddc931e94debf0336dd56177721e9f35b441e3/heron/tools/tracker/src/python/javaobj.py",
line 420, in do_object
opcode, obj = self._read_and_exec_opcode(ident=ident+1)
File
"/Users/thinker0/.pex/code/89ddc931e94debf0336dd56177721e9f35b441e3/heron/tools/tracker/src/python/javaobj.py",
line 255, in _read_and_exec_opcode
raise RuntimeError("Unknown OpCode in the stream: 0x%x" % opid)
RuntimeError: Unknown OpCode in the stream: 0x7a
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File
"/Users/thinker0/.pex/code/89ddc931e94debf0336dd56177721e9f35b441e3/heron/tools/tracker/src/python/tracker.py",
line 62, in _convert_java_value
pobj = javaobj.loads(kv.serialized_value)
File
"/Users/thinker0/.pex/code/89ddc931e94debf0336dd56177721e9f35b441e3/heron/tools/tracker/src/python/javaobj.py",
line 65, in loads
return marshaller.readObject()
File
"/Users/thinker0/.pex/code/89ddc931e94debf0336dd56177721e9f35b441e3/heron/tools/tracker/src/python/javaobj.py",
line 235, in readObject
self._oops_dump_state()
File
"/Users/thinker0/.pex/code/89ddc931e94debf0336dd56177721e9f35b441e3/heron/tools/tracker/src/python/javaobj.py",
line 545, in _oops_dump_state
self.object_stream.seek(-16, mode=1)
TypeError: seek() takes no keyword arguments
[2020-07-20 22:40:03 +0900] [INFO]: Adding new topology:
heron-system-access-es-a-line-me, state_manager: lad
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [incubator-heron] Code0x58 commented on pull request #3574: Decode string when deserializing Java object in python
Code0x58 commented on pull request #3574: URL: https://github.com/apache/incubator-heron/pull/3574#issuecomment-661031266 thanks again @thinker0, I pushed another amendment. The amendment isn't 1-to-1 with the python2 implementation, but it's probably saner. I don't know if it will have a knock on effect down the line 爛 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-heron] thinker0 commented on pull request #3574: Decode string when deserializing Java object in python
thinker0 commented on pull request #3574:
URL: https://github.com/apache/incubator-heron/pull/3574#issuecomment-661021922
@Code0x58 another
```
[2020-07-20 21:33:24 +0900] [INFO]: Setting topology info for topology:
heron-ladp_ladm_perf_adgroup_breakdown_report-kafka-to-ladp-log-es-eco
[2020-07-20 21:33:24 +0900] [ERROR]: Failed to parse data as java object
Traceback (most recent call last):
File
"/Users/thinker0/.pex/code/20c8103cc4fd30b436b4787554e90a405988f63f/heron/tools/tracker/src/python/tracker.py",
line 77, in _convert_java_value
'raw' : utils.hex_escape(kv.serialized_value)}
File
"/Users/thinker0/.pex/code/20c8103cc4fd30b436b4787554e90a405988f63f/heron/tools/tracker/src/python/utils.py",
line 44, in hex_escape
return ''.join(ch if ch in printable else r'0x{0:02x}'.format(ord(ch))
for ch in bin_str)
File
"/Users/thinker0/.pex/code/20c8103cc4fd30b436b4787554e90a405988f63f/heron/tools/tracker/src/python/utils.py",
line 44, in
return ''.join(ch if ch in printable else r'0x{0:02x}'.format(ord(ch))
for ch in bin_str)
TypeError: 'in ' requires string as left operand, not int
[2020-07-20 21:33:24 +0900] [ERROR]: Caught exception while triggering
callback: 'in ' requires string as left operand, not int
[2020-07-20 21:33:24 +0900] [INFO]: Unregister a watch with uid:
e56c8661-0da9-4793-a2a8-60d733e74528
[2020-07-20 21:33:24 +0900] [INFO]: Adding data watch for path:
/heron/tmasters/heron-ladp_ladm_perf_adgroup_breakdown_report-kafka-to-ladp-log-es-eco
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [incubator-heron] Code0x58 edited a comment on pull request #3574: Decode string when deserializing Java object in python
Code0x58 edited a comment on pull request #3574: URL: https://github.com/apache/incubator-heron/pull/3574#issuecomment-661013477 thanks @thinker0, I just pushed a quick amendment that should resolve the `/`→`//` issue. There is still be the separate seek issue which I think is because the underlying (C) stream type doesn't implement seek, but that hopefully only shows itself when when things blow up. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-heron] Code0x58 edited a comment on pull request #3574: Decode string when deserializing Java object in python
Code0x58 edited a comment on pull request #3574: URL: https://github.com/apache/incubator-heron/pull/3574#issuecomment-661013477 thanks @thinker0, I just pushed a quick amendment that should resolve the `/`→`//` issue, but there'd still be the separate seek issue which I think is because the underlying (C) stream type doesn't implement seek This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-heron] Code0x58 commented on a change in pull request #3574: Decode string when deserializing Java object in python
Code0x58 commented on a change in pull request #3574: URL: https://github.com/apache/incubator-heron/pull/3574#discussion_r457352071 ## File path: heron/statemgrs/src/python/zkstatemanager.py ## @@ -170,7 +170,7 @@ def _get_topology_with_watch(self, topologyName, callback, isWatching): # pylint: disable=unused-variable, unused-argument @self.client.DataWatch(path) -def watch_topology(data, stats): +def watch_topology(data, stats, event): """ watch topology """ if data: topology = Topology() Review comment: I think it helps by avoiding the questionable pattern [here](https://github.com/python-zk/kazoo/blob/master/kazoo/recipe/watchers.py#L161) which can obscure the exceptions when they occur within the handler - at least we certainly won't see the TypeError now even if the parameter is junk (no more "exception occurred while handling exception", so that much easier to read) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-heron] Code0x58 commented on a change in pull request #3574: Decode string when deserializing Java object in python
Code0x58 commented on a change in pull request #3574: URL: https://github.com/apache/incubator-heron/pull/3574#discussion_r457352071 ## File path: heron/statemgrs/src/python/zkstatemanager.py ## @@ -170,7 +170,7 @@ def _get_topology_with_watch(self, topologyName, callback, isWatching): # pylint: disable=unused-variable, unused-argument @self.client.DataWatch(path) -def watch_topology(data, stats): +def watch_topology(data, stats, event): """ watch topology """ if data: topology = Topology() Review comment: I think it helps by avoiding the questionable pattern [here](https://github.com/python-zk/kazoo/blob/master/kazoo/recipe/watchers.py#L161) which can obscure the exceptions when they occur within the handler - at least we certainly won't see the TypeError now even if the parameter is junk This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-heron] Code0x58 commented on a change in pull request #3574: Decode string when deserializing Java object in python
Code0x58 commented on a change in pull request #3574: URL: https://github.com/apache/incubator-heron/pull/3574#discussion_r457352071 ## File path: heron/statemgrs/src/python/zkstatemanager.py ## @@ -170,7 +170,7 @@ def _get_topology_with_watch(self, topologyName, callback, isWatching): # pylint: disable=unused-variable, unused-argument @self.client.DataWatch(path) -def watch_topology(data, stats): +def watch_topology(data, stats, event): """ watch topology """ if data: topology = Topology() Review comment: I think it helps by avoiding the questionable pattern [here](https://github.com/python-zk/kazoo/blob/master/kazoo/recipe/watchers.py#L161) which might be having some weird interaction with the rest of the codebase - at least we certainly won't see the TypeError now even if the parameter is junk This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-heron] Code0x58 commented on a change in pull request #3574: Decode string when deserializing Java object in python
Code0x58 commented on a change in pull request #3574: URL: https://github.com/apache/incubator-heron/pull/3574#discussion_r457352071 ## File path: heron/statemgrs/src/python/zkstatemanager.py ## @@ -170,7 +170,7 @@ def _get_topology_with_watch(self, topologyName, callback, isWatching): # pylint: disable=unused-variable, unused-argument @self.client.DataWatch(path) -def watch_topology(data, stats): +def watch_topology(data, stats, event): """ watch topology """ if data: topology = Topology() Review comment: I think it helps by avoiding the questionable pattern [here](https://github.com/python-zk/kazoo/blob/master/kazoo/recipe/watchers.py#L155) which might be having some weird interaction with the rest of the codebase - at least we certainly won't see the TypeError now even if the parameter is junk This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-heron] Code0x58 commented on pull request #3574: Decode string when deserializing Java object in python
Code0x58 commented on pull request #3574: URL: https://github.com/apache/incubator-heron/pull/3574#issuecomment-661013477 thanks @thinker0, I just pushed a quick amendment that should resolve that issue (`/`→`//`) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-heron] thinker0 commented on pull request #3574: Decode string when deserializing Java object in python
thinker0 commented on pull request #3574:
URL: https://github.com/apache/incubator-heron/pull/3574#issuecomment-661002357
@Code0x58
```
TypeError: 'float' object cannot be interpreted as an integer
```
```
[2020-07-20 21:33:24 +0900] [INFO]: Setting topology info for topology:
heron-lass_win_loss_notice_log-to-ladp-log-es
[2020-07-20 21:33:24 +0900] [ERROR]: ==Oops state dump=
[2020-07-20 21:33:24 +0900] [ERROR]: References:
[[java.util.HashMap:0x507DAC1], ,
'NJ_INSTANCE_PHASE', 'BETA', 'LD_LIBRARY_PATH',
'${LD_LIBRARY_PATH}:./lib64:/usr/lib/hadoop/lib/native', 'SENTRY_DSN',
'https://55d2bf90467e4814908d94c10d154...@lad-sentry.line-apps.com/3?sample.rate=0.10',
'SERVER_ENV', 'beta', 'NJ_INSTANCE_ID',
'heron-lass_win_loss_notice_log-to-ladp-log-es', 'SENTRY_ENVIRONMENT']
[2020-07-20 21:33:24 +0900] [ERROR]: Stream seeking back at -16 byte (2nd
line is an actual position!):
[2020-07-20 21:33:24 +0900] [ERROR]: Failed to parse data as java object
Traceback (most recent call last):
File
"/Users/thinker0/.pex/code/20c8103cc4fd30b436b4787554e90a405988f63f/heron/tools/tracker/src/python/javaobj.py",
line 221, in readObject
_, res = self._read_and_exec_opcode(ident=0)
File
"/Users/thinker0/.pex/code/20c8103cc4fd30b436b4787554e90a405988f63f/heron/tools/tracker/src/python/javaobj.py",
line 256, in _read_and_exec_opcode
return (opid, handler(ident=ident))
File
"/Users/thinker0/.pex/code/20c8103cc4fd30b436b4787554e90a405988f63f/heron/tools/tracker/src/python/javaobj.py",
line 428, in do_object
tmp_object = transformer.transform(java_object)
File
"/Users/thinker0/.pex/code/20c8103cc4fd30b436b4787554e90a405988f63f/heron/tools/tracker/src/python/javaobj.py",
line 625, in transform
for i in range((len(obj.annotations)-1)/2):
TypeError: 'float' object cannot be interpreted as an integer
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File
"/Users/thinker0/.pex/code/20c8103cc4fd30b436b4787554e90a405988f63f/heron/tools/tracker/src/python/tracker.py",
line 62, in _convert_java_value
pobj = javaobj.loads(kv.serialized_value)
File
"/Users/thinker0/.pex/code/20c8103cc4fd30b436b4787554e90a405988f63f/heron/tools/tracker/src/python/javaobj.py",
line 65, in loads
return marshaller.readObject()
File
"/Users/thinker0/.pex/code/20c8103cc4fd30b436b4787554e90a405988f63f/heron/tools/tracker/src/python/javaobj.py",
line 235, in readObject
self._oops_dump_state()
File
"/Users/thinker0/.pex/code/20c8103cc4fd30b436b4787554e90a405988f63f/heron/tools/tracker/src/python/javaobj.py",
line 545, in _oops_dump_state
self.object_stream.seek(-16, mode=1)
TypeError: seek() takes no keyword arguments
[2020-07-20 21:33:24 +0900] [ERROR]: Caught exception while triggering
callback: 'in ' requires string as left operand, not int
[2020-07-20 21:33:24 +0900] [INFO]: Unregister a watch with uid:
0ce59fcd-9c28-473e-94d7-f09aebc29085
[2020-07-20 21:33:24 +0900] [INFO]: Adding data watch for path:
/heron/tmasters/heron-lass_win_loss_notice_log-to-ladp-log-es
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [incubator-heron] nwangtw commented on a change in pull request #3576: Upgrade to Bazel 3.4.1
nwangtw commented on a change in pull request #3576: URL: https://github.com/apache/incubator-heron/pull/3576#discussion_r457146473 ## File path: WORKSPACE ## @@ -363,25 +363,27 @@ http_archive( # end helm # for docker image building -DOCKER_RULES_VERSION = "0.14.1" Review comment: could be cleaner to keep this variable? not a blocker. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-heron] huijunwu merged pull request #3577: try to fix setup-intellj.sh
huijunwu merged pull request #3577: URL: https://github.com/apache/incubator-heron/pull/3577 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-heron] joshfischer1108 removed a comment on pull request #3568: Unvendor cloudpickle
joshfischer1108 removed a comment on pull request #3568: URL: https://github.com/apache/incubator-heron/pull/3568#issuecomment-660779925 @Code0x58 Could you rebase this branch from master and resolve the conflict please? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
