My PC: My PC is a 8 Core Xeon E5 with 16 GB of RAM, when the test starts, I only have 8 GB of memory occupied. I increased the memory of the Java VM to 4 GB and it only uses 1 GB when the test runs.
The Topology: On my PC, I have three Spouts in mono, and one Bolt in mono. The topology is described in Flux – so I have basically zero code in Java, all in Flux .yaml + .Net with mono. All the messages use SHUFFLE and there is one worker only (my PC) I run in local mode and I also have a Docker container where I deployed this. Topology details: The Spouts read from an internal service, I collect about 60/70,000 records each minute. The Bolt reads from the three Spouts and makes aggregation in memory using SqlLite, the records are added to SqlLite as they arrive, then every 30 seconds SqlLite runs an aggregation and emits the data to an instance of Redis cache (via another Bolt hop). To test with Java, I replaced the Bolt with a simple Java Bolt that was only logging every 10,000 records. To compare with Mono, I created an empty .net Bolt and did the same. My Tests: The Flux topology is attached. The Java class I used to test and the .Net Bolt are as well. Again, the Spouts are .Net classes that emits 65K rows per minute. The log files are attached, you can see how much time it takes for the Bolt to consume 10,000 records – Inter-Language.txt is on my PC using the mono debug bolt, each 10,000 records takes around 4.5 seconds. The Java.txt is on my PC using Java (TransformEchoBolt.Java), each 10,000 records takes around 0.7 seconds. The Linux.txt is on the Docker container (still on my PC but using Docker for Windows in Linux Dockers mode), using mono but on Linux this time - the results are compatible with Mono on Windows (4.5 seconds per 10.000 records). I also tried calling directly the Windows exe on Windows in local mode, bypassing mono – the results were not pretty: 15 seconds per 10,000 records (NetExe.txt) Results: I know I can scale out and partition the data, but the amount of processing did not seem to require that – Maybe one issue is that the object I am moving has 11 fields? I can try to create a mini-repro if the dev team is interested – hopefully this might find what the bottleneck is - Thanks for your attention - Mauro. From: P. Taylor Goetz [mailto:[email protected]] Sent: Friday, May 12, 2017 4:55 PM To: [email protected]; [email protected] Subject: Re: Performance of Multi-Lang protocol Adding dev@ mailing list... There is definitely a performance hit. But it shouldn't be as drastic as you describe. Can you share some of your environment characteristics? I've been looking at the Apache Arrow project (full disclosure: I'm a PMC member) as a means for improved performance (it essentially would remove the performance hit for serialize/deserialize operations). This is particularly relevant to multi-lang, but could also apply to same-machine inter-worker communication. At this point I don't feel Arrow is at a production level maturity, but is getting close. I definitely feel it's worth exploring at PoC level. -Taylor On May 12, 2017, at 6:56 PM, Mauro Giusti <[email protected]<mailto:[email protected]>> wrote: Hi – We are using multi-lang to pass data between storm and mono – We observe a 6x time increase when messages go from spout to bolt if the bolt is in mono vs. being in Java – Java can process 10,000 records in 0.7 seconds, while mono requires 4.5 seconds. The mono bolt was an empty one created with Storm.Net.Adapter<https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fziyunhx%2Fstorm-net-adapter&data=02%7C01%7Cmaurgi%40microsoft.com%7Cc1d9c2b13bab4297b2b508d499924f9d%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636302300991869578&sdata=kaE4OjEttJv0KuGcwdUoJA%2BBDXIO1qvyv65S%2BBpMM%2F0%3D&reserved=0> library This is on a single machine topology – we are still in dev phase and using this solution for now - Is this expected? Should we try to minimize multi-lang and inter-process or is this a problem with my specific scenario (mono and/or single machine) ? Thank you – Mauro.
topology.yaml
Description: topology.yaml
TransformEchoBolt.java
Description: TransformEchoBolt.java
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
using Storm;
namespace DebugBolt
{
public class DebugBolt : IBasicBolt
{
private int _numProcessedMessages = 0;
public static DebugBolt Get(Context ctx)
{
return new DebugBolt(ctx);
}
public DebugBolt(Context ctx)
{
Dictionary<string, List<Type>> inputSchema = new Dictionary<string,
List<Type>>
{
{ "default", new List<Type>() { typeof(string), typeof(string),
typeof(string), typeof(string), typeof(string), typeof(string), typeof(string),
typeof(string), typeof(string), typeof(string), typeof(double) } },
{ "__tick", new List<Type>() { typeof(string) } }
};
Dictionary<string, List<Type>> outputSchema = new
Dictionary<string, List<Type>> {{"default", new List<Type>() {typeof(string),
typeof(int)}}};
ctx.DeclareComponentSchema(new ComponentStreamSchema(inputSchema,
outputSchema));
}
public void Execute(StormTuple tuple)
{
if ((++_numProcessedMessages % 10000) == 0)
Context.Logger.Info($"Processed {_numProcessedMessages}");
}
public void Prepare(Config stormConf, TopologyContext context)
{
}
}
}
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
using Storm;
namespace DebugBolt
{
class Program
{
static void Main(string[] args)
{
ApacheStorm.LaunchPlugin(new newPlugin(DebugBolt.Get));
}
}
}
Storage ci ver 1.3 Killing old instances of Mono and Storm...SUCCESS: The process "mono.exe" with PID 80324 has been terminated. SUCCESS: The process "mono.exe" with PID 30776 has been terminated. SUCCESS: The process "mono.exe" with PID 28092 has been terminated. SUCCESS: The process "mono.exe" with PID 21540 has been terminated. Done. Setting Visual Studio 2017 environment ...Done. "Restoring NuGet Packages..." MSBuild auto-detection: using msbuild version '14.0' from 'C:\Program Files (x86)\MSBuild\14.0\bin'. All packages listed in packages.config are already installed. Done. Cleaninig up folder multilang\resources...Done. Microsoft (R) Build Engine version 14.0.25420.1 Copyright (C) Microsoft Corporation. All rights reserved. Common -> C:\OneBranch\SRE\DataInsightsHub\src\StormDataHub\Topologies\Storage\multilang\resources\mono\storage\extractspout\AzureSre.DataHub.Common.dll KeyVault -> C:\OneBranch\SRE\DataInsightsHub\src\StormDataHub\Topologies\Storage\multilang\resources\mono\storage\extractspout\AzureSre.DataHub.KeyVault.dll HubConfig -> C:\OneBranch\SRE\DataInsightsHub\src\StormDataHub\Topologies\Storage\multilang\resources\mono\storage\extractspout\AzureSre.DataHub.HubConfig.dll HubLog -> C:\OneBranch\SRE\DataInsightsHub\src\StormDataHub\Topologies\Storage\multilang\resources\mono\storage\extractspout\AzureSre.DataHub.HubLog.dll Storm -> C:\OneBranch\SRE\DataInsightsHub\src\StormDataHub\Topologies\Storage\multilang\resources\mono\storage\extractspout\AzureSre.DataHub.Storm.dll Mdm -> C:\OneBranch\SRE\DataInsightsHub\src\StormDataHub\Topologies\Storage\multilang\resources\mono\storage\extractspout\AzureSre.DataHub.Mdm.dll SqlLite -> C:\OneBranch\SRE\DataInsightsHub\src\StormDataHub\Topologies\Storage\multilang\resources\mono\storage\extractspout\AzureSre.DataHub.SqlLite.dll StormMessages -> C:\OneBranch\SRE\DataInsightsHub\src\StormDataHub\Topologies\Storage\multilang\resources\mono\storage\extractspout\AzureSre.DataHub.Topologies.Common.StormMessages.dll Storage.ExtractSpout -> C:\OneBranch\SRE\DataInsightsHub\src\StormDataHub\Topologies\Storage\multilang\resources\mono\storage\extractspout\Storage.ExtractSpout.exe Microsoft (R) Build Engine version 14.0.25420.1 Copyright (C) Microsoft Corporation. All rights reserved. Common -> C:\OneBranch\SRE\DataInsightsHub\src\StormDataHub\Topologies\Storage\multilang\resources\mono\storage\regionbolt\AzureSre.DataHub.Common.dll HubLog -> C:\OneBranch\SRE\DataInsightsHub\src\StormDataHub\Topologies\Storage\multilang\resources\mono\storage\regionbolt\AzureSre.DataHub.HubLog.dll SqlLite -> C:\OneBranch\SRE\DataInsightsHub\src\StormDataHub\Topologies\Storage\multilang\resources\mono\storage\regionbolt\AzureSre.DataHub.SqlLite.dll KeyVault -> C:\OneBranch\SRE\DataInsightsHub\src\StormDataHub\Topologies\Storage\multilang\resources\mono\storage\regionbolt\AzureSre.DataHub.KeyVault.dll HubConfig -> C:\OneBranch\SRE\DataInsightsHub\src\StormDataHub\Topologies\Storage\multilang\resources\mono\storage\regionbolt\AzureSre.DataHub.HubConfig.dll Storm -> C:\OneBranch\SRE\DataInsightsHub\src\StormDataHub\Topologies\Storage\multilang\resources\mono\storage\regionbolt\AzureSre.DataHub.Storm.dll StormMessages -> C:\OneBranch\SRE\DataInsightsHub\src\StormDataHub\Topologies\Storage\multilang\resources\mono\storage\regionbolt\AzureSre.DataHub.Topologies.Common.StormMessages.dll Storage.RegionBolt -> C:\OneBranch\SRE\DataInsightsHub\src\StormDataHub\Topologies\Storage\multilang\resources\mono\storage\regionbolt\Storage.RegionBolt.exe Microsoft (R) Build Engine version 14.0.25420.1 Copyright (C) Microsoft Corporation. All rights reserved. Common -> C:\OneBranch\SRE\DataInsightsHub\src\StormDataHub\Topologies\Storage\multilang\resources\common\redisbolt\AzureSre.DataHub.Common.dll KeyVault -> C:\OneBranch\SRE\DataInsightsHub\src\StormDataHub\Topologies\Storage\multilang\resources\common\redisbolt\AzureSre.DataHub.KeyVault.dll HubConfig -> C:\OneBranch\SRE\DataInsightsHub\src\StormDataHub\Topologies\Storage\multilang\resources\common\redisbolt\AzureSre.DataHub.HubConfig.dll RedisLibrary -> C:\OneBranch\SRE\DataInsightsHub\src\StormDataHub\Topologies\Storage\multilang\resources\common\redisbolt\RedisLibrary.dll HubLog -> C:\OneBranch\SRE\DataInsightsHub\src\StormDataHub\Topologies\Storage\multilang\resources\common\redisbolt\AzureSre.DataHub.HubLog.dll Storm -> C:\OneBranch\SRE\DataInsightsHub\src\StormDataHub\Topologies\Storage\multilang\resources\common\redisbolt\AzureSre.DataHub.Storm.dll SqlLite -> C:\OneBranch\SRE\DataInsightsHub\src\StormDataHub\Topologies\Storage\multilang\resources\common\redisbolt\AzureSre.DataHub.SqlLite.dll StormMessages -> C:\OneBranch\SRE\DataInsightsHub\src\StormDataHub\Topologies\Storage\multilang\resources\common\redisbolt\AzureSre.DataHub.Topologies.Common.StormMessages.dll RedisBolt -> C:\OneBranch\SRE\DataInsightsHub\src\StormDataHub\Topologies\Storage\multilang\resources\common\redisbolt\RedisBolt.exe Microsoft (R) Build Engine version 14.0.25420.1 Copyright (C) Microsoft Corporation. All rights reserved. DebugBolt -> C:\OneBranch\SRE\DataInsightsHub\src\StormDataHub\Topologies\Storage\multilang\resources\common\debugbolt\DebugBolt.exe [INFO] Scanning for projects... [INFO] [INFO] ------------------------------------------------------------------------ [INFO] Building StorageTopology 1.0-SNAPSHOT [INFO] ------------------------------------------------------------------------ [INFO] [INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ StorageTopology --- [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] Copying 1 resource [INFO] Copying 1 resource [INFO] Copying 144 resources [INFO] [INFO] --- maven-compiler-plugin:3.3:compile (default-compile) @ StorageTopology --- [INFO] Nothing to compile - all classes are up to date [INFO] [INFO] --- exec-maven-plugin:1.4.0:java (default-cli) @ StorageTopology --- ??????????? ??? ?????? ??? ??????????? ??? ??????????? ?????? ??? ??? ??? ?????? ?????? ??? ??? ??? ?????? ??? ????????????????????? ??? ??? ???????? ??????? ??? ??? +- Apache Storm -+ +- data FLow User eXperience -+ Version: 1.1.0 Parsing classpath resource: /topology.yaml ---------- TOPOLOGY DETAILS ---------- Topology Name: storage-topology --------------- SPOUTS --------------- storage-throughput-sla-spout [1] (org.apache.storm.flux.wrappers.spouts.FluxShellSpout) storage-throughput-billable-spout [1] (org.apache.storm.flux.wrappers.spouts.FluxShellSpout) storage-throughput-spout [1] (org.apache.storm.flux.wrappers.spouts.FluxShellSpout) ---------------- BOLTS --------------- debug-bolt [1] (org.apache.storm.flux.wrappers.bolts.FluxShellBolt) --------------- STREAMS --------------- storage-throughput-sla-spout --SHUFFLE--> debug-bolt storage-throughput-billable-spout --SHUFFLE--> debug-bolt storage-throughput-spout --SHUFFLE--> debug-bolt -------------------------------------- 13:33:27,603 [r[2 2]] INFO task.ShellBolt - Launched subprocess with pid 50888 13:33:27,609 [r[2 2]] INFO task.ShellBolt - Start checking heartbeat... 13:33:27,629 [ead-31] INFO task.ShellBolt - ShellLog pid:50888, name:debug-bolt LaunchPlugin, pluginType: BASICBOLT 13:33:27,630 [ead-31] INFO task.ShellBolt - ShellLog pid:50888, name:debug-bolt [BasicBolt] Launch ... 13:33:28,055 [r[5 5]] INFO spout.ShellSpout - Launched subprocess with pid 13896 13:33:28,057 [r[5 5]] INFO spout.ShellSpout - Start checking heartbeat... 13:33:28,063 [r[3 3]] INFO spout.ShellSpout - Launched subprocess with pid 9716 13:33:28,064 [r[3 3]] INFO spout.ShellSpout - Start checking heartbeat... 13:33:28,086 [r[3 3]] INFO spout.ShellSpout - ShellLog pid:9716, name:storage-throughput-billable-spout LaunchPlugin, pluginType: SPOUT 13:33:28,086 [r[5 5]] INFO spout.ShellSpout - ShellLog pid:13896, name:storage-throughput-spout LaunchPlugin, pluginType: SPOUT 13:33:28,086 [r[3 3]] INFO spout.ShellSpout - ShellLog pid:9716, name:storage-throughput-billable-spout [Spout] Launch ... 13:33:28,087 [r[5 5]] INFO spout.ShellSpout - ShellLog pid:13896, name:storage-throughput-spout [Spout] Launch ... 13:33:28,095 [r[3 3]] ERROR spout.ShellSpout - ShellLog pid:9716, name:storage-throughput-billable-spout [Spout] unexpected message. 13:33:28,097 [r[5 5]] ERROR spout.ShellSpout - ShellLog pid:13896, name:storage-throughput-spout [Spout] unexpected message. 13:33:28,107 [r[3 3]] INFO spout.ShellSpout - ShellLog pid:9716, name:storage-throughput-billable-spout Extracting ThroughputBillable 13:33:28,110 [r[5 5]] INFO spout.ShellSpout - ShellLog pid:13896, name:storage-throughput-spout Extracting Throughput 13:33:28,169 [r[4 4]] INFO spout.ShellSpout - Launched subprocess with pid 110684 13:33:28,170 [r[4 4]] INFO spout.ShellSpout - Start checking heartbeat... 13:33:28,208 [r[4 4]] INFO spout.ShellSpout - ShellLog pid:110684, name:storage-throughput-sla-spout LaunchPlugin, pluginType: SPOUT 13:33:28,210 [r[4 4]] INFO spout.ShellSpout - ShellLog pid:110684, name:storage-throughput-sla-spout [Spout] Launch ... 13:33:28,231 [r[4 4]] ERROR spout.ShellSpout - ShellLog pid:110684, name:storage-throughput-sla-spout [Spout] unexpected message. 13:33:28,244 [r[4 4]] INFO spout.ShellSpout - ShellLog pid:110684, name:storage-throughput-sla-spout Extracting ThroughputSla 13:33:44,544 [r[3 3]] INFO spout.ShellSpout - ShellLog pid:9716, name:storage-throughput-billable-spout Emitting 68390 records. 13:33:44,549 [r[4 4]] INFO spout.ShellSpout - ShellLog pid:110684, name:storage-throughput-sla-spout Emitting 67569 records. 13:33:51,240 [ead-31] INFO task.ShellBolt - ShellLog pid:50888, name:debug-bolt Processed 10000 13:33:52,392 [r[5 5]] INFO spout.ShellSpout - ShellLog pid:13896, name:storage-throughput-spout Emitting 68720 records. 13:33:58,375 [ead-31] INFO task.ShellBolt - ShellLog pid:50888, name:debug-bolt Processed 20000 13:34:01,028 [r[4 4]] INFO spout.ShellSpout - ShellLog pid:110684, name:storage-throughput-sla-spout Emitting 67569 records.DONE 13:34:01,199 [r[3 3]] INFO spout.ShellSpout - ShellLog pid:9716, name:storage-throughput-billable-spout Emitting 68390 records.DONE 13:34:04,417 [ead-31] INFO task.ShellBolt - ShellLog pid:50888, name:debug-bolt Processed 30000 13:34:08,494 [r[5 5]] INFO spout.ShellSpout - ShellLog pid:13896, name:storage-throughput-spout Emitting 68720 records.DONE 13:34:09,232 [ead-31] INFO task.ShellBolt - ShellLog pid:50888, name:debug-bolt Processed 40000 13:34:14,152 [ead-31] INFO task.ShellBolt - ShellLog pid:50888, name:debug-bolt Processed 50000 13:34:18,687 [ead-31] INFO task.ShellBolt - ShellLog pid:50888, name:debug-bolt Processed 60000 13:34:23,089 [ead-31] INFO task.ShellBolt - ShellLog pid:50888, name:debug-bolt Processed 70000 13:34:27,790 [ead-31] INFO task.ShellBolt - ShellLog pid:50888, name:debug-bolt Processed 80000 13:34:32,754 [ead-31] INFO task.ShellBolt - ShellLog pid:50888, name:debug-bolt Processed 90000 13:34:37,794 [ead-31] INFO task.ShellBolt - ShellLog pid:50888, name:debug-bolt Processed 100000 13:34:44,135 [ead-31] INFO task.ShellBolt - ShellLog pid:50888, name:debug-bolt Processed 110000 13:34:48,772 [ead-31] INFO task.ShellBolt - ShellLog pid:50888, name:debug-bolt Processed 120000 13:34:53,253 [ead-31] INFO task.ShellBolt - ShellLog pid:50888, name:debug-bolt Processed 130000 13:34:57,642 [ead-31] INFO task.ShellBolt - ShellLog pid:50888, name:debug-bolt Processed 140000 13:35:01,889 [ead-31] INFO task.ShellBolt - ShellLog pid:50888, name:debug-bolt Processed 150000 13:35:06,154 [ead-31] INFO task.ShellBolt - ShellLog pid:50888, name:debug-bolt Processed 160000 13:35:10,396 [ead-31] INFO task.ShellBolt - ShellLog pid:50888, name:debug-bolt Processed 170000 13:35:14,713 [ead-31] INFO task.ShellBolt - ShellLog pid:50888, name:debug-bolt Processed 180000 13:35:18,977 [ead-31] INFO task.ShellBolt - ShellLog pid:50888, name:debug-bolt Processed 190000 13:35:23,354 [ead-31] INFO task.ShellBolt - ShellLog pid:50888, name:debug-bolt Processed 200000 Terminate batch job (Y/N)? Terminate batch job (Y/N)?
Storage ci ver 1.3 Killing old instances of Mono and Storm...Done. Setting Visual Studio 2017 environment ...Done. "Restoring NuGet Packages..." MSBuild auto-detection: using msbuild version '14.0' from 'C:\Program Files (x86)\MSBuild\14.0\bin'. All packages listed in packages.config are already installed. Done. Cleaninig up folder multilang\resources...Done. Microsoft (R) Build Engine version 14.0.25420.1 Copyright (C) Microsoft Corporation. All rights reserved. Common -> C:\OneBranch\SRE\DataInsightsHub\src\StormDataHub\Topologies\Storage\multilang\resources\mono\storage\extractspout\AzureSre.DataHub.Common.dll KeyVault -> C:\OneBranch\SRE\DataInsightsHub\src\StormDataHub\Topologies\Storage\multilang\resources\mono\storage\extractspout\AzureSre.DataHub.KeyVault.dll HubConfig -> C:\OneBranch\SRE\DataInsightsHub\src\StormDataHub\Topologies\Storage\multilang\resources\mono\storage\extractspout\AzureSre.DataHub.HubConfig.dll HubLog -> C:\OneBranch\SRE\DataInsightsHub\src\StormDataHub\Topologies\Storage\multilang\resources\mono\storage\extractspout\AzureSre.DataHub.HubLog.dll Storm -> C:\OneBranch\SRE\DataInsightsHub\src\StormDataHub\Topologies\Storage\multilang\resources\mono\storage\extractspout\AzureSre.DataHub.Storm.dll Mdm -> C:\OneBranch\SRE\DataInsightsHub\src\StormDataHub\Topologies\Storage\multilang\resources\mono\storage\extractspout\AzureSre.DataHub.Mdm.dll SqlLite -> C:\OneBranch\SRE\DataInsightsHub\src\StormDataHub\Topologies\Storage\multilang\resources\mono\storage\extractspout\AzureSre.DataHub.SqlLite.dll StormMessages -> C:\OneBranch\SRE\DataInsightsHub\src\StormDataHub\Topologies\Storage\multilang\resources\mono\storage\extractspout\AzureSre.DataHub.Topologies.Common.StormMessages.dll Storage.ExtractSpout -> C:\OneBranch\SRE\DataInsightsHub\src\StormDataHub\Topologies\Storage\multilang\resources\mono\storage\extractspout\Storage.ExtractSpout.exe Microsoft (R) Build Engine version 14.0.25420.1 Copyright (C) Microsoft Corporation. All rights reserved. Common -> C:\OneBranch\SRE\DataInsightsHub\src\StormDataHub\Topologies\Storage\multilang\resources\mono\storage\regionbolt\AzureSre.DataHub.Common.dll HubLog -> C:\OneBranch\SRE\DataInsightsHub\src\StormDataHub\Topologies\Storage\multilang\resources\mono\storage\regionbolt\AzureSre.DataHub.HubLog.dll SqlLite -> C:\OneBranch\SRE\DataInsightsHub\src\StormDataHub\Topologies\Storage\multilang\resources\mono\storage\regionbolt\AzureSre.DataHub.SqlLite.dll KeyVault -> C:\OneBranch\SRE\DataInsightsHub\src\StormDataHub\Topologies\Storage\multilang\resources\mono\storage\regionbolt\AzureSre.DataHub.KeyVault.dll HubConfig -> C:\OneBranch\SRE\DataInsightsHub\src\StormDataHub\Topologies\Storage\multilang\resources\mono\storage\regionbolt\AzureSre.DataHub.HubConfig.dll Storm -> C:\OneBranch\SRE\DataInsightsHub\src\StormDataHub\Topologies\Storage\multilang\resources\mono\storage\regionbolt\AzureSre.DataHub.Storm.dll StormMessages -> C:\OneBranch\SRE\DataInsightsHub\src\StormDataHub\Topologies\Storage\multilang\resources\mono\storage\regionbolt\AzureSre.DataHub.Topologies.Common.StormMessages.dll Storage.RegionBolt -> C:\OneBranch\SRE\DataInsightsHub\src\StormDataHub\Topologies\Storage\multilang\resources\mono\storage\regionbolt\Storage.RegionBolt.exe Microsoft (R) Build Engine version 14.0.25420.1 Copyright (C) Microsoft Corporation. All rights reserved. Common -> C:\OneBranch\SRE\DataInsightsHub\src\StormDataHub\Topologies\Storage\multilang\resources\common\redisbolt\AzureSre.DataHub.Common.dll KeyVault -> C:\OneBranch\SRE\DataInsightsHub\src\StormDataHub\Topologies\Storage\multilang\resources\common\redisbolt\AzureSre.DataHub.KeyVault.dll HubConfig -> C:\OneBranch\SRE\DataInsightsHub\src\StormDataHub\Topologies\Storage\multilang\resources\common\redisbolt\AzureSre.DataHub.HubConfig.dll RedisLibrary -> C:\OneBranch\SRE\DataInsightsHub\src\StormDataHub\Topologies\Storage\multilang\resources\common\redisbolt\RedisLibrary.dll HubLog -> C:\OneBranch\SRE\DataInsightsHub\src\StormDataHub\Topologies\Storage\multilang\resources\common\redisbolt\AzureSre.DataHub.HubLog.dll Storm -> C:\OneBranch\SRE\DataInsightsHub\src\StormDataHub\Topologies\Storage\multilang\resources\common\redisbolt\AzureSre.DataHub.Storm.dll SqlLite -> C:\OneBranch\SRE\DataInsightsHub\src\StormDataHub\Topologies\Storage\multilang\resources\common\redisbolt\AzureSre.DataHub.SqlLite.dll StormMessages -> C:\OneBranch\SRE\DataInsightsHub\src\StormDataHub\Topologies\Storage\multilang\resources\common\redisbolt\AzureSre.DataHub.Topologies.Common.StormMessages.dll RedisBolt -> C:\OneBranch\SRE\DataInsightsHub\src\StormDataHub\Topologies\Storage\multilang\resources\common\redisbolt\RedisBolt.exe Microsoft (R) Build Engine version 14.0.25420.1 Copyright (C) Microsoft Corporation. All rights reserved. DebugBolt -> C:\OneBranch\SRE\DataInsightsHub\src\StormDataHub\Topologies\Storage\multilang\resources\common\debugbolt\DebugBolt.exe [INFO] Scanning for projects... [INFO] [INFO] ------------------------------------------------------------------------ [INFO] Building StorageTopology 1.0-SNAPSHOT [INFO] ------------------------------------------------------------------------ [INFO] [INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ StorageTopology --- [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] Copying 1 resource [INFO] Copying 1 resource [INFO] Copying 144 resources [INFO] [INFO] --- maven-compiler-plugin:3.3:compile (default-compile) @ StorageTopology --- [INFO] Nothing to compile - all classes are up to date [INFO] [INFO] --- exec-maven-plugin:1.4.0:java (default-cli) @ StorageTopology --- ??????????? ??? ?????? ??? ??????????? ??? ??????????? ?????? ??? ??? ??? ?????? ?????? ??? ??? ??? ?????? ??? ????????????????????? ??? ??? ???????? ??????? ??? ??? +- Apache Storm -+ +- data FLow User eXperience -+ Version: 1.1.0 Parsing classpath resource: /topology.yaml ---------- TOPOLOGY DETAILS ---------- Topology Name: storage-topology --------------- SPOUTS --------------- storage-throughput-sla-spout [1] (org.apache.storm.flux.wrappers.spouts.FluxShellSpout) storage-throughput-billable-spout [1] (org.apache.storm.flux.wrappers.spouts.FluxShellSpout) storage-throughput-spout [1] (org.apache.storm.flux.wrappers.spouts.FluxShellSpout) ---------------- BOLTS --------------- debug-bolt [1] (org.apache.storm.flux.wrappers.bolts.FluxShellBolt) java-debug-bolt [1] (com.microsoft.azure.sre.datahub.TransformEchoBolt) --------------- STREAMS --------------- storage-throughput-sla-spout --SHUFFLE--> java-debug-bolt storage-throughput-billable-spout --SHUFFLE--> java-debug-bolt storage-throughput-spout --SHUFFLE--> java-debug-bolt -------------------------------------- 13:51:25,469 [r[3 3]] INFO spout.ShellSpout - Launched subprocess with pid 54688 13:51:25,480 [r[3 3]] INFO spout.ShellSpout - Start checking heartbeat... 13:51:25,493 [r[3 3]] INFO spout.ShellSpout - ShellLog pid:54688, name:storage-throughput-billable-spout LaunchPlugin, pluginType: SPOUT 13:51:25,493 [r[3 3]] INFO spout.ShellSpout - ShellLog pid:54688, name:storage-throughput-billable-spout [Spout] Launch ... 13:51:25,503 [r[3 3]] ERROR spout.ShellSpout - ShellLog pid:54688, name:storage-throughput-billable-spout [Spout] unexpected message. 13:51:25,515 [r[3 3]] INFO spout.ShellSpout - ShellLog pid:54688, name:storage-throughput-billable-spout Extracting ThroughputBillable 13:51:25,520 [r[5 5]] INFO spout.ShellSpout - Launched subprocess with pid 73396 13:51:25,520 [r[5 5]] INFO spout.ShellSpout - Start checking heartbeat... 13:51:25,553 [r[5 5]] INFO spout.ShellSpout - ShellLog pid:73396, name:storage-throughput-spout LaunchPlugin, pluginType: SPOUT 13:51:25,554 [r[5 5]] INFO spout.ShellSpout - ShellLog pid:73396, name:storage-throughput-spout [Spout] Launch ... 13:51:25,568 [r[5 5]] ERROR spout.ShellSpout - ShellLog pid:73396, name:storage-throughput-spout [Spout] unexpected message. 13:51:25,575 [r[4 4]] INFO spout.ShellSpout - Launched subprocess with pid 87196 13:51:25,576 [r[4 4]] INFO spout.ShellSpout - Start checking heartbeat... 13:51:25,580 [r[5 5]] INFO spout.ShellSpout - ShellLog pid:73396, name:storage-throughput-spout Extracting Throughput 13:51:25,622 [r[4 4]] INFO spout.ShellSpout - ShellLog pid:87196, name:storage-throughput-sla-spout LaunchPlugin, pluginType: SPOUT 13:51:25,624 [r[4 4]] INFO spout.ShellSpout - ShellLog pid:87196, name:storage-throughput-sla-spout [Spout] Launch ... 13:51:25,633 [r[4 4]] ERROR spout.ShellSpout - ShellLog pid:87196, name:storage-throughput-sla-spout [Spout] unexpected message. 13:51:25,646 [r[4 4]] INFO spout.ShellSpout - ShellLog pid:87196, name:storage-throughput-sla-spout Extracting ThroughputSla 13:51:43,274 [r[4 4]] INFO spout.ShellSpout - ShellLog pid:87196, name:storage-throughput-sla-spout Emitting 66955 records. 13:51:44,407 [r[3 3]] INFO spout.ShellSpout - ShellLog pid:54688, name:storage-throughput-billable-spout Emitting 67597 records. 13:51:45,197 [r[2 2]] INFO ransformEchoBolt - Received record # 10000 13:51:45,754 [r[5 5]] INFO spout.ShellSpout - ShellLog pid:73396, name:storage-throughput-spout Emitting 67728 records. 13:51:46,114 [r[2 2]] INFO ransformEchoBolt - Received record # 20000 13:51:46,862 [r[2 2]] INFO ransformEchoBolt - Received record # 30000 13:51:47,583 [r[2 2]] INFO ransformEchoBolt - Received record # 40000 13:51:48,288 [r[2 2]] INFO ransformEchoBolt - Received record # 50000 13:51:48,979 [r[2 2]] INFO ransformEchoBolt - Received record # 60000 13:51:49,667 [r[2 2]] INFO ransformEchoBolt - Received record # 70000 13:51:50,387 [r[2 2]] INFO ransformEchoBolt - Received record # 80000 13:51:51,087 [r[2 2]] INFO ransformEchoBolt - Received record # 90000 13:51:51,888 [r[2 2]] INFO ransformEchoBolt - Received record # 100000 13:51:52,584 [r[2 2]] INFO ransformEchoBolt - Received record # 110000 13:51:53,265 [r[2 2]] INFO ransformEchoBolt - Received record # 120000 13:51:53,966 [r[2 2]] INFO ransformEchoBolt - Received record # 130000 13:51:54,682 [r[2 2]] INFO ransformEchoBolt - Received record # 140000 13:51:55,339 [r[2 2]] INFO ransformEchoBolt - Received record # 150000 13:51:56,046 [r[2 2]] INFO ransformEchoBolt - Received record # 160000 13:51:56,726 [r[2 2]] INFO ransformEchoBolt - Received record # 170000 13:51:57,389 [r[2 2]] INFO ransformEchoBolt - Received record # 180000 13:51:57,825 [r[4 4]] INFO spout.ShellSpout - ShellLog pid:87196, name:storage-throughput-sla-spout Emitting 66955 records.DONE 13:51:58,207 [r[2 2]] INFO ransformEchoBolt - Received record # 190000 13:51:58,799 [r[3 3]] INFO spout.ShellSpout - ShellLog pid:54688, name:storage-throughput-billable-spout Emitting 67597 records.DONE 13:51:59,777 [r[2 2]] INFO ransformEchoBolt - Received record # 200000 13:52:00,250 [r[5 5]] INFO spout.ShellSpout - ShellLog pid:73396, name:storage-throughput-spout Emitting 67728 records.DONE
2017-05-12 22:03:53.413 o.a.s.d.executor Thread-4-debug-bolt-executor[2 2] [INFO] Preparing bolt debug-bolt:(2) 2017-05-12 22:03:53.418 o.a.s.u.ShellProcess Thread-4-debug-bolt-executor[2 2] [INFO] Storm multilang serializer: org.apache.storm.multilang.JsonSerializer 2017-05-12 22:03:53.418 o.a.s.u.ShellProcess Thread-12-storage-throughput-spout-executor[5 5] [INFO] Storm multilang serializer: org.apache.storm.multilang.JsonSerializer 2017-05-12 22:03:53.450 o.a.s.d.executor Thread-14-storage-throughput-sla-spout-executor[4 4] [INFO] Opening spout storage-throughput-sla-spout:(4) 2017-05-12 22:03:53.456 o.a.s.d.executor Thread-10-__system-executor[-1 -1] [INFO] Preparing bolt __system:(-1) 2017-05-12 22:03:53.461 o.a.s.d.executor Thread-10-__system-executor[-1 -1] [INFO] Prepared bolt __system:(-1) 2017-05-12 22:03:53.463 o.a.s.u.ShellProcess Thread-14-storage-throughput-sla-spout-executor[4 4] [INFO] Storm multilang serializer: org.apache.storm.multilang.JsonSerializer 2017-05-12 22:03:53.469 o.a.s.d.executor Thread-6-storage-throughput-billable-spout-executor[3 3] [INFO] Opening spout storage-throughput-billable-spout:(3) 2017-05-12 22:03:53.470 o.a.s.u.ShellProcess Thread-6-storage-throughput-billable-spout-executor[3 3] [INFO] Storm multilang serializer: org.apache.storm.multilang.JsonSerializer 2017-05-12 22:03:54.615 o.a.s.t.ShellBolt Thread-4-debug-bolt-executor[2 2] [INFO] Launched subprocess with pid 786 2017-05-12 22:03:54.625 o.a.s.t.ShellBolt Thread-4-debug-bolt-executor[2 2] [INFO] Start checking heartbeat... 2017-05-12 22:03:54.642 o.a.s.d.executor Thread-4-debug-bolt-executor[2 2] [INFO] Prepared bolt debug-bolt:(2) 2017-05-12 22:03:54.772 o.a.s.t.ShellBolt Thread-19 [INFO] ShellLog pid:786, name:debug-bolt LaunchPlugin, pluginType: BASICBOLT 2017-05-12 22:03:54.773 o.a.s.t.ShellBolt Thread-19 [INFO] ShellLog pid:786, name:debug-bolt [BasicBolt] Launch ... 2017-05-12 22:03:55.403 o.a.s.s.ShellSpout Thread-14-storage-throughput-sla-spout-executor[4 4] [INFO] Launched subprocess with pid 789 2017-05-12 22:03:55.404 o.a.s.d.executor Thread-14-storage-throughput-sla-spout-executor[4 4] [INFO] Opened spout storage-throughput-sla-spout:(4) 2017-05-12 22:03:55.406 o.a.s.d.executor Thread-14-storage-throughput-sla-spout-executor[4 4] [INFO] Activating spout storage-throughput-sla-spout:(4) 2017-05-12 22:03:55.407 o.a.s.s.ShellSpout Thread-14-storage-throughput-sla-spout-executor[4 4] [INFO] Start checking heartbeat... 2017-05-12 22:03:55.432 o.a.s.s.ShellSpout Thread-14-storage-throughput-sla-spout-executor[4 4] [INFO] ShellLog pid:789, name:storage-throughput-sla-spout LaunchPlugin, pluginType: SPOUT 2017-05-12 22:03:55.433 o.a.s.s.ShellSpout Thread-14-storage-throughput-sla-spout-executor[4 4] [INFO] ShellLog pid:789, name:storage-throughput-sla-spout [Spout] Launch ... 2017-05-12 22:03:55.439 o.a.s.s.ShellSpout Thread-14-storage-throughput-sla-spout-executor[4 4] [ERROR] ShellLog pid:789, name:storage-throughput-sla-spout [Spout] unexpected message. 2017-05-12 22:03:55.455 o.a.s.s.ShellSpout Thread-14-storage-throughput-sla-spout-executor[4 4] [INFO] ShellLog pid:789, name:storage-throughput-sla-spout Extracting ThroughputSla 2017-05-12 22:03:55.497 o.a.s.s.ShellSpout Thread-6-storage-throughput-billable-spout-executor[3 3] [INFO] Launched subprocess with pid 788 2017-05-12 22:03:55.498 o.a.s.d.executor Thread-6-storage-throughput-billable-spout-executor[3 3] [INFO] Opened spout storage-throughput-billable-spout:(3) 2017-05-12 22:03:55.498 o.a.s.d.executor Thread-6-storage-throughput-billable-spout-executor[3 3] [INFO] Activating spout storage-throughput-billable-spout:(3) 2017-05-12 22:03:55.499 o.a.s.s.ShellSpout Thread-6-storage-throughput-billable-spout-executor[3 3] [INFO] Start checking heartbeat... 2017-05-12 22:03:55.506 o.a.s.s.ShellSpout Thread-12-storage-throughput-spout-executor[5 5] [INFO] Launched subprocess with pid 783 2017-05-12 22:03:55.526 o.a.s.s.ShellSpout Thread-6-storage-throughput-billable-spout-executor[3 3] [INFO] ShellLog pid:788, name:storage-throughput-billable-spout LaunchPlugin, pluginType: SPOUT 2017-05-12 22:03:55.526 o.a.s.s.ShellSpout Thread-6-storage-throughput-billable-spout-executor[3 3] [INFO] ShellLog pid:788, name:storage-throughput-billable-spout [Spout] Launch ... 2017-05-12 22:03:55.526 o.a.s.d.executor Thread-12-storage-throughput-spout-executor[5 5] [INFO] Opened spout storage-throughput-spout:(5) 2017-05-12 22:03:55.527 o.a.s.d.executor Thread-12-storage-throughput-spout-executor[5 5] [INFO] Activating spout storage-throughput-spout:(5) 2017-05-12 22:03:55.527 o.a.s.s.ShellSpout Thread-12-storage-throughput-spout-executor[5 5] [INFO] Start checking heartbeat... 2017-05-12 22:03:55.538 o.a.s.s.ShellSpout Thread-6-storage-throughput-billable-spout-executor[3 3] [ERROR] ShellLog pid:788, name:storage-throughput-billable-spout [Spout] unexpected message. 2017-05-12 22:03:55.551 o.a.s.s.ShellSpout Thread-6-storage-throughput-billable-spout-executor[3 3] [INFO] ShellLog pid:788, name:storage-throughput-billable-spout Extracting ThroughputBillable 2017-05-12 22:03:55.591 o.a.s.s.ShellSpout Thread-12-storage-throughput-spout-executor[5 5] [INFO] ShellLog pid:783, name:storage-throughput-spout LaunchPlugin, pluginType: SPOUT 2017-05-12 22:03:55.603 o.a.s.s.ShellSpout Thread-12-storage-throughput-spout-executor[5 5] [INFO] ShellLog pid:783, name:storage-throughput-spout [Spout] Launch ... 2017-05-12 22:03:55.645 o.a.s.s.ShellSpout Thread-12-storage-throughput-spout-executor[5 5] [ERROR] ShellLog pid:783, name:storage-throughput-spout [Spout] unexpected message. 2017-05-12 22:03:55.667 o.a.s.s.ShellSpout Thread-12-storage-throughput-spout-executor[5 5] [INFO] ShellLog pid:783, name:storage-throughput-spout Extracting Throughput 2017-05-12 22:04:07.859 o.a.s.s.ShellSpout Thread-12-storage-throughput-spout-executor[5 5] [INFO] ShellLog pid:783, name:storage-throughput-spout Emitting 78232 records. 2017-05-12 22:04:22.740 o.a.s.t.ShellBolt Thread-19 [INFO] ShellLog pid:786, name:debug-bolt Processed 10000 2017-05-12 22:04:30.265 o.a.s.s.ShellSpout Thread-6-storage-throughput-billable-spout-executor[3 3] [INFO] ShellLog pid:788, name:storage-throughput-billable-spout Emitting 78230 records. 2017-05-12 22:04:33.055 o.a.s.s.ShellSpout Thread-14-storage-throughput-sla-spout-executor[4 4] [INFO] ShellLog pid:789, name:storage-throughput-sla-spout Emitting 78200 records. 2017-05-12 22:04:42.832 o.a.s.t.ShellBolt Thread-19 [INFO] ShellLog pid:786, name:debug-bolt Processed 20000 2017-05-12 22:05:01.854 o.a.s.t.ShellBolt Thread-19 [INFO] ShellLog pid:786, name:debug-bolt Processed 30000 2017-05-12 22:05:16.884 o.a.s.t.ShellBolt Thread-19 [INFO] ShellLog pid:786, name:debug-bolt Processed 40000 2017-05-12 22:05:21.239 o.a.s.s.ShellSpout Thread-12-storage-throughput-spout-executor[5 5] [INFO] ShellLog pid:783, name:storage-throughput-spout Emitting 78232 records.DONE 2017-05-12 22:05:36.531 o.a.s.t.ShellBolt Thread-19 [INFO] ShellLog pid:786, name:debug-bolt Processed 50000 2017-05-12 22:05:48.939 o.a.s.s.ShellSpout Thread-6-storage-throughput-billable-spout-executor[3 3] [INFO] ShellLog pid:788, name:storage-throughput-billable-spout Emitting 78230 records.DONE 2017-05-12 22:05:51.018 o.a.s.s.ShellSpout Thread-14-storage-throughput-sla-spout-executor[4 4] [INFO] ShellLog pid:789, name:storage-throughput-sla-spout Emitting 78200 records.DONE 2017-05-12 22:05:52.913 o.a.s.t.ShellBolt Thread-19 [INFO] ShellLog pid:786, name:debug-bolt Processed 60000 2017-05-12 22:06:09.027 o.a.s.t.ShellBolt Thread-19 [INFO] ShellLog pid:786, name:debug-bolt Processed 70000 2017-05-12 22:06:21.133 o.a.s.t.ShellBolt Thread-19 [INFO] ShellLog pid:786, name:debug-bolt Processed 80000 2017-05-12 22:06:27.627 o.a.s.t.ShellBolt Thread-19 [INFO] ShellLog pid:786, name:debug-bolt Processed 90000 2017-05-12 22:06:32.447 o.a.s.t.ShellBolt Thread-19 [INFO] ShellLog pid:786, name:debug-bolt Processed 100000 2017-05-12 22:06:37.227 o.a.s.t.ShellBolt Thread-19 [INFO] ShellLog pid:786, name:debug-bolt Processed 110000 2017-05-12 22:06:43.535 o.a.s.t.ShellBolt Thread-19 [INFO] ShellLog pid:786, name:debug-bolt Processed 120000 2017-05-12 22:06:48.275 o.a.s.t.ShellBolt Thread-19 [INFO] ShellLog pid:786, name:debug-bolt Processed 130000 2017-05-12 22:06:53.695 o.a.s.t.ShellBolt Thread-19 [INFO] ShellLog pid:786, name:debug-bolt Processed 140000 2017-05-12 22:06:58.974 o.a.s.t.ShellBolt Thread-19 [INFO] ShellLog pid:786, name:debug-bolt Processed 150000 2017-05-12 22:07:05.019 o.a.s.t.ShellBolt Thread-19 [INFO] ShellLog pid:786, name:debug-bolt Processed 160000 2017-05-12 22:07:10.720 o.a.s.t.ShellBolt Thread-19 [INFO] ShellLog pid:786, name:debug-bolt Processed 170000 2017-05-12 22:07:16.333 o.a.s.t.ShellBolt Thread-19 [INFO] ShellLog pid:786, name:debug-bolt Processed 180000 2017-05-12 22:07:21.625 o.a.s.t.ShellBolt Thread-19 [INFO] ShellLog pid:786, name:debug-bolt Processed 190000 2017-05-12 22:07:26.689 o.a.s.t.ShellBolt Thread-19 [INFO] ShellLog pid:786, name:debug-bolt Processed 200000 2017-05-12 22:07:31.232 o.a.s.t.ShellBolt Thread-19 [INFO] ShellLog pid:786, name:debug-bolt Processed 210000 2017-05-12 22:07:36.099 o.a.s.t.ShellBolt Thread-19 [INFO] ShellLog pid:786, name:debug-bolt Processed 220000 2017-05-12 22:07:40.509 o.a.s.t.ShellBolt Thread-19 [INFO] ShellLog pid:786, name:debug-bolt Processed 230000
Storage ci ver 1.3 Killing old instances of Mono and Storm...SUCCESS: The process "mono.exe" with PID 68716 has been terminated. SUCCESS: The process "mono.exe" with PID 72648 has been terminated. SUCCESS: The process "mono.exe" with PID 117436 has been terminated. Done. Setting Visual Studio 2017 environment ...Done. "Restoring NuGet Packages..." MSBuild auto-detection: using msbuild version '14.0' from 'C:\Program Files (x86)\MSBuild\14.0\bin'. All packages listed in packages.config are already installed. Done. Cleaninig up folder multilang\resources...Done. Microsoft (R) Build Engine version 14.0.25420.1 Copyright (C) Microsoft Corporation. All rights reserved. Common -> C:\OneBranch\SRE\DataInsightsHub\src\StormDataHub\Topologies\Storage\multilang\resources\mono\storage\extractspout\AzureSre.DataHub.Common.dll KeyVault -> C:\OneBranch\SRE\DataInsightsHub\src\StormDataHub\Topologies\Storage\multilang\resources\mono\storage\extractspout\AzureSre.DataHub.KeyVault.dll HubConfig -> C:\OneBranch\SRE\DataInsightsHub\src\StormDataHub\Topologies\Storage\multilang\resources\mono\storage\extractspout\AzureSre.DataHub.HubConfig.dll HubLog -> C:\OneBranch\SRE\DataInsightsHub\src\StormDataHub\Topologies\Storage\multilang\resources\mono\storage\extractspout\AzureSre.DataHub.HubLog.dll Storm -> C:\OneBranch\SRE\DataInsightsHub\src\StormDataHub\Topologies\Storage\multilang\resources\mono\storage\extractspout\AzureSre.DataHub.Storm.dll Mdm -> C:\OneBranch\SRE\DataInsightsHub\src\StormDataHub\Topologies\Storage\multilang\resources\mono\storage\extractspout\AzureSre.DataHub.Mdm.dll SqlLite -> C:\OneBranch\SRE\DataInsightsHub\src\StormDataHub\Topologies\Storage\multilang\resources\mono\storage\extractspout\AzureSre.DataHub.SqlLite.dll StormMessages -> C:\OneBranch\SRE\DataInsightsHub\src\StormDataHub\Topologies\Storage\multilang\resources\mono\storage\extractspout\AzureSre.DataHub.Topologies.Common.StormMessages.dll Storage.ExtractSpout -> C:\OneBranch\SRE\DataInsightsHub\src\StormDataHub\Topologies\Storage\multilang\resources\mono\storage\extractspout\Storage.ExtractSpout.exe Microsoft (R) Build Engine version 14.0.25420.1 Copyright (C) Microsoft Corporation. All rights reserved. Common -> C:\OneBranch\SRE\DataInsightsHub\src\StormDataHub\Topologies\Storage\multilang\resources\mono\storage\regionbolt\AzureSre.DataHub.Common.dll HubLog -> C:\OneBranch\SRE\DataInsightsHub\src\StormDataHub\Topologies\Storage\multilang\resources\mono\storage\regionbolt\AzureSre.DataHub.HubLog.dll SqlLite -> C:\OneBranch\SRE\DataInsightsHub\src\StormDataHub\Topologies\Storage\multilang\resources\mono\storage\regionbolt\AzureSre.DataHub.SqlLite.dll KeyVault -> C:\OneBranch\SRE\DataInsightsHub\src\StormDataHub\Topologies\Storage\multilang\resources\mono\storage\regionbolt\AzureSre.DataHub.KeyVault.dll HubConfig -> C:\OneBranch\SRE\DataInsightsHub\src\StormDataHub\Topologies\Storage\multilang\resources\mono\storage\regionbolt\AzureSre.DataHub.HubConfig.dll Storm -> C:\OneBranch\SRE\DataInsightsHub\src\StormDataHub\Topologies\Storage\multilang\resources\mono\storage\regionbolt\AzureSre.DataHub.Storm.dll StormMessages -> C:\OneBranch\SRE\DataInsightsHub\src\StormDataHub\Topologies\Storage\multilang\resources\mono\storage\regionbolt\AzureSre.DataHub.Topologies.Common.StormMessages.dll Storage.RegionBolt -> C:\OneBranch\SRE\DataInsightsHub\src\StormDataHub\Topologies\Storage\multilang\resources\mono\storage\regionbolt\Storage.RegionBolt.exe Microsoft (R) Build Engine version 14.0.25420.1 Copyright (C) Microsoft Corporation. All rights reserved. Common -> C:\OneBranch\SRE\DataInsightsHub\src\StormDataHub\Topologies\Storage\multilang\resources\common\redisbolt\AzureSre.DataHub.Common.dll KeyVault -> C:\OneBranch\SRE\DataInsightsHub\src\StormDataHub\Topologies\Storage\multilang\resources\common\redisbolt\AzureSre.DataHub.KeyVault.dll HubConfig -> C:\OneBranch\SRE\DataInsightsHub\src\StormDataHub\Topologies\Storage\multilang\resources\common\redisbolt\AzureSre.DataHub.HubConfig.dll RedisLibrary -> C:\OneBranch\SRE\DataInsightsHub\src\StormDataHub\Topologies\Storage\multilang\resources\common\redisbolt\RedisLibrary.dll HubLog -> C:\OneBranch\SRE\DataInsightsHub\src\StormDataHub\Topologies\Storage\multilang\resources\common\redisbolt\AzureSre.DataHub.HubLog.dll Storm -> C:\OneBranch\SRE\DataInsightsHub\src\StormDataHub\Topologies\Storage\multilang\resources\common\redisbolt\AzureSre.DataHub.Storm.dll SqlLite -> C:\OneBranch\SRE\DataInsightsHub\src\StormDataHub\Topologies\Storage\multilang\resources\common\redisbolt\AzureSre.DataHub.SqlLite.dll StormMessages -> C:\OneBranch\SRE\DataInsightsHub\src\StormDataHub\Topologies\Storage\multilang\resources\common\redisbolt\AzureSre.DataHub.Topologies.Common.StormMessages.dll RedisBolt -> C:\OneBranch\SRE\DataInsightsHub\src\StormDataHub\Topologies\Storage\multilang\resources\common\redisbolt\RedisBolt.exe Microsoft (R) Build Engine version 14.0.25420.1 Copyright (C) Microsoft Corporation. All rights reserved. DebugBolt -> C:\OneBranch\SRE\DataInsightsHub\src\StormDataHub\Topologies\Storage\multilang\resources\common\debugbolt\DebugBolt.exe [INFO] Scanning for projects... [INFO] [INFO] ------------------------------------------------------------------------ [INFO] Building StorageTopology 1.0-SNAPSHOT [INFO] ------------------------------------------------------------------------ [INFO] [INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ StorageTopology --- [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] Copying 1 resource [INFO] Copying 1 resource [INFO] Copying 144 resources [INFO] [INFO] --- maven-compiler-plugin:3.3:compile (default-compile) @ StorageTopology --- [INFO] Nothing to compile - all classes are up to date [INFO] [INFO] --- exec-maven-plugin:1.4.0:java (default-cli) @ StorageTopology --- ??????????? ??? ?????? ??? ??????????? ??? ??????????? ?????? ??? ??? ??? ?????? ?????? ??? ??? ??? ?????? ??? ????????????????????? ??? ??? ???????? ??????? ??? ??? +- Apache Storm -+ +- data FLow User eXperience -+ Version: 1.1.0 Parsing classpath resource: /topology.yaml ---------- TOPOLOGY DETAILS ---------- Topology Name: storage-topology --------------- SPOUTS --------------- storage-throughput-sla-spout [1] (org.apache.storm.flux.wrappers.spouts.FluxShellSpout) storage-throughput-billable-spout [1] (org.apache.storm.flux.wrappers.spouts.FluxShellSpout) storage-throughput-spout [1] (org.apache.storm.flux.wrappers.spouts.FluxShellSpout) ---------------- BOLTS --------------- debug-bolt [1] (org.apache.storm.flux.wrappers.bolts.FluxShellBolt) --------------- STREAMS --------------- storage-throughput-sla-spout --SHUFFLE--> debug-bolt storage-throughput-billable-spout --SHUFFLE--> debug-bolt storage-throughput-spout --SHUFFLE--> debug-bolt -------------------------------------- 14:02:47,781 [r[2 2]] INFO task.ShellBolt - Launched subprocess with pid 66376 14:02:47,787 [r[2 2]] INFO task.ShellBolt - Start checking heartbeat... 14:02:47,826 [ead-31] INFO task.ShellBolt - ShellLog pid:66376, name:debug-bolt LaunchPlugin, pluginType: BASICBOLT 14:02:47,831 [ead-31] INFO task.ShellBolt - ShellLog pid:66376, name:debug-bolt [BasicBolt] Launch ... 14:02:48,229 [r[4 4]] INFO spout.ShellSpout - Launched subprocess with pid 65580 14:02:48,232 [r[4 4]] INFO spout.ShellSpout - Start checking heartbeat... 14:02:48,257 [r[4 4]] INFO spout.ShellSpout - ShellLog pid:65580, name:storage-throughput-sla-spout LaunchPlugin, pluginType: SPOUT 14:02:48,259 [r[4 4]] INFO spout.ShellSpout - ShellLog pid:65580, name:storage-throughput-sla-spout [Spout] Launch ... 14:02:48,272 [r[4 4]] ERROR spout.ShellSpout - ShellLog pid:65580, name:storage-throughput-sla-spout [Spout] unexpected message. 14:02:48,284 [r[4 4]] INFO spout.ShellSpout - ShellLog pid:65580, name:storage-throughput-sla-spout Extracting ThroughputSla 14:02:48,403 [r[3 3]] INFO spout.ShellSpout - Launched subprocess with pid 112668 14:02:48,404 [r[3 3]] INFO spout.ShellSpout - Start checking heartbeat... 14:02:48,433 [r[3 3]] INFO spout.ShellSpout - ShellLog pid:112668, name:storage-throughput-billable-spout LaunchPlugin, pluginType: SPOUT 14:02:48,444 [r[3 3]] INFO spout.ShellSpout - ShellLog pid:112668, name:storage-throughput-billable-spout [Spout] Launch ... 14:02:48,454 [r[3 3]] ERROR spout.ShellSpout - ShellLog pid:112668, name:storage-throughput-billable-spout [Spout] unexpected message. 14:02:48,468 [r[3 3]] INFO spout.ShellSpout - ShellLog pid:112668, name:storage-throughput-billable-spout Extracting ThroughputBillable 14:02:48,532 [r[5 5]] INFO spout.ShellSpout - Launched subprocess with pid 58440 14:02:48,533 [r[5 5]] INFO spout.ShellSpout - Start checking heartbeat... 14:02:48,566 [r[5 5]] INFO spout.ShellSpout - ShellLog pid:58440, name:storage-throughput-spout LaunchPlugin, pluginType: SPOUT 14:02:48,567 [r[5 5]] INFO spout.ShellSpout - ShellLog pid:58440, name:storage-throughput-spout [Spout] Launch ... 14:02:48,583 [r[5 5]] ERROR spout.ShellSpout - ShellLog pid:58440, name:storage-throughput-spout [Spout] unexpected message. 14:02:48,596 [r[5 5]] INFO spout.ShellSpout - ShellLog pid:58440, name:storage-throughput-spout Extracting Throughput 14:03:05,388 [r[3 3]] INFO spout.ShellSpout - ShellLog pid:112668, name:storage-throughput-billable-spout Emitting 77754 records. 14:03:07,411 [r[4 4]] INFO spout.ShellSpout - ShellLog pid:65580, name:storage-throughput-sla-spout Emitting 77501 records. 14:03:14,829 [r[5 5]] INFO spout.ShellSpout - ShellLog pid:58440, name:storage-throughput-spout Emitting 77843 records. 14:03:24,234 [r[3 3]] INFO spout.ShellSpout - ShellLog pid:112668, name:storage-throughput-billable-spout Emitting 77754 records.DONE 14:03:25,968 [r[4 4]] INFO spout.ShellSpout - ShellLog pid:65580, name:storage-throughput-sla-spout Emitting 77501 records.DONE 14:03:28,074 [ead-31] INFO task.ShellBolt - ShellLog pid:66376, name:debug-bolt Processed 10000 14:03:32,705 [r[5 5]] INFO spout.ShellSpout - ShellLog pid:58440, name:storage-throughput-spout Emitting 77843 records.DONE 14:03:46,100 [ead-31] INFO task.ShellBolt - ShellLog pid:66376, name:debug-bolt Processed 20000 14:04:06,043 [ead-31] INFO task.ShellBolt - ShellLog pid:66376, name:debug-bolt Processed 30000 14:04:24,069 [ead-31] INFO task.ShellBolt - ShellLog pid:66376, name:debug-bolt Processed 40000 14:04:43,667 [ead-31] INFO task.ShellBolt - ShellLog pid:66376, name:debug-bolt Processed 50000 14:05:01,833 [ead-31] INFO task.ShellBolt - ShellLog pid:66376, name:debug-bolt Processed 60000 14:05:20,189 [ead-31] INFO task.ShellBolt - ShellLog pid:66376, name:debug-bolt Processed 70000 14:05:38,304 [ead-31] INFO task.ShellBolt - ShellLog pid:66376, name:debug-bolt Processed 80000 14:05:55,561 [ead-31] INFO task.ShellBolt - ShellLog pid:66376, name:debug-bolt Processed 90000 14:06:12,820 [ead-31] INFO task.ShellBolt - ShellLog pid:66376, name:debug-bolt Processed 100000 14:06:30,039 [ead-31] INFO task.ShellBolt - ShellLog pid:66376, name:debug-bolt Processed 110000 14:06:47,284 [ead-31] INFO task.ShellBolt - ShellLog pid:66376, name:debug-bolt Processed 120000 14:07:04,502 [ead-31] INFO task.ShellBolt - ShellLog pid:66376, name:debug-bolt Processed 130000 14:07:21,714 [ead-31] INFO task.ShellBolt - ShellLog pid:66376, name:debug-bolt Processed 140000 14:07:39,851 [ead-31] INFO task.ShellBolt - ShellLog pid:66376, name:debug-bolt Processed 150000 14:07:57,964 [ead-31] INFO task.ShellBolt - ShellLog pid:66376, name:debug-bolt Processed 160000 14:08:15,649 [ead-31] INFO task.ShellBolt - ShellLog pid:66376, name:debug-bolt Processed 170000 14:08:33,879 [ead-31] INFO task.ShellBolt - ShellLog pid:66376, name:debug-bolt Processed 180000 14:08:51,596 [ead-31] INFO task.ShellBolt - ShellLog pid:66376, name:debug-bolt Processed 190000 14:09:10,203 [ead-31] INFO task.ShellBolt - ShellLog pid:66376, name:debug-bolt Processed 200000 14:09:17,220 [r[3 3]] ERROR spout.ShellSpout - ShellLog pid:112668, name:storage-throughput-billable-spout [Spout] unexpected message. 14:09:17,224 [r[5 5]] ERROR spout.ShellSpout - ShellLog pid:58440, name:storage-throughput-spout [Spout] unexpected message. 14:09:17,228 [r[4 4]] ERROR spout.ShellSpout - ShellLog pid:65580, name:storage-throughput-sla-spout [Spout] unexpected message.
