Re: High-performance string reading and parsing
How does it compare to the c++? On Wed, Dec 11, 2019 at 4:32 PM Greg Keogh wrote: > Folks, I just went through an performance comparison exercise and I > thought a summary of the results might be of interest here. A colleague is > converting some C++ code to C# to see if it's possible to maintain the > legacy high performance while enjoying the benefits of the managed world. > The core code reads from 1 to 15 text files line-by-line and parses the > contents of the lines which may look like these samples: > > 83;61;58;18;42;96;24;15;42;39 > a1b1*0.333a2b1*0.333a3b1 > a3b1*826;2*93;3*101a19b1*526;2*557;3*518 > > The input files often contain up to 1 million lines. Each parsed number is > used to update a cell in a large matrix that is typically hundreds wide or > high, but might be tens of thousands wide. So you can see that this is > mainly a CPU and memory intensive task. We know that most of the time is > taken in the tight loop parsing of millions of numbers out of the input > lines. I wrote a test harness that simulated the processing in C# and > discovered the following: > >- Release or Debug build made little difference. >- Using compiled Regex slows by a factor of 5. >- Using string Split slows by a factor of about 3. >- Using Parallel.ForEach slows things slightly. >- Using an unmanaged buffer with unsafe unchecked pointers slows >things slightly. >- The fastest way to parse the lines is with an index loop over the >chars in the line string. > > In a normal business app you would of course use Regex or string methods > for parsing because it's clear and maintainable, but in this case where > every millisecond counts I found that any FCL usage would blow-out the time > and only a for-loop was viable. > > Parallelism is probably useless in this case because the processing on > each worker thread is just a blink, meaning the threading burden was > heavier than the processing it carried. > > So it turns out that an old-fashioned C-style for-loop to manually parse > the lines is the fastest by a long-shot. It's fragile of course, but my > colleague has translated the old well-tested C++ code directly over to C# > (it's rather ugly). This whole scenario is rather unusual and not very > applicable to LOB apps, but I thought it was worth posting anyway. > > Cheers, > *Greg Keogh* > > [image: image.png] > > Regex.Match(es) > Regex.Match(es) with Parallel Processing (PPL) > String Split > String Split with PPL > For-loop > For-loop with PPL > Plain file reads with no parsing (lowest baseline) >
Re: [OT] Sit/stand desk results
Actually, if you read /r/keto they'll tell you to "keep calm and keto on" - I lost 20kg on keto, and plateaued many times doing that. Just keep doing what you're doing, your body will adjust and adapt and it'll go down again. It's common. Drink a lot of water. I hopped off Keto for a bit for Christmas 18 months ago, and never hopped back on, and put it all back on and more. *shrug* I didn't learn anything I guess. Anyway, great diet, but you really have to change your lifestyle - and I'll have to get back on it. Standing desk? Co-worker wasn't using theirs, so I borrowed it, used it a bit for the first few weeks, went on holidays, came back and haven't used it in two months now. Apparently gimmicks can't overcome laziness :) On Tue, Jun 20, 2017 at 10:48 AM, Chris F wrote: > Stephen - > > Off topic too - but if you've found you've stalled on the keto diet have > you tried carbing up? > > Especially if you're lifting weights. One meal on a Saturday night once a > week or fortnight is great. > > Doesn't have to be clean either, I found I'd eat an entire large pizza and > within a day my weight would drop a significant amount once everything > flushed through. > > Depending on how long you've been on it, you'll kick back into ketosis > either Sunday afternoon or Monday and it'll give you a bit more energy for > lifting weights for the week to come. Also gives you a bit of an > psychological break too. > > > On 20 June 2017 at 02:22, Stephen Price wrote: > >> Totally agree on this point. I've been ketogenic for six months now (lost >> 6kg in the first month, have plateaued now but feel great). Some .net >> people may know Carl Franklin's been podcasting at 2ketodudes.com, and >> he's done an awesome job recording his progress. 6 months and he lost 80lb >> and is no longer type 2 diabetic. >> Got so much out of it, I backed his kickstarter project to turn his town >> keto for a weekend. Flying out with my wife in a couple of weeks. Will be >> seeing the sights in New York, then up to New London for ketofest. >> Btw, you don't have to be over weight to suffer the damaging effects of >> too much carbs/sugar. The inflammatory damage in your veins can't be seen >> from the outside. >> >> One of the strange side effects I have noticed is that some days I forget >> to eat. Today, I had accidentally turned off my alarm so was running a bit >> late. Went to work with no breakfast, had one coffee at work, and worked >> right through lunch as I hadn't taken anything and office is a bit of a >> drive from places to eat. Barely noticed. >> Don't miss sugar. Finding some awesome recipes along the way. Recently >> made deep fried chicken crumbed in pork rinds combined with Parmesan >> cheese. >> So good. Hmm... this might possibly be the first recipe shared on this >> elist. :) >> >> Anyway to keep on topic, had a standup desk and my last project, one of >> those motorised ones. Great for exercise and strengthening but not losing >> weight. What you put in your body has way more effect in that regard. You >> can lose weight with zero exercise, but exercise is important for other >> reasons. I.e. Preventing muscles wasting away. If you don't use it, you >> lose it. >> >> Cheers, >> Stephen >> -- >> *From:* ozdotnet-boun...@ozdotnet.com on >> behalf of Piers Williams >> *Sent:* Monday, June 19, 2017 8:46:35 PM >> *To:* ozDotNet >> *Subject:* Re: [OT] Sit/stand desk results >> >> There are quite a few people in my office now using sit-to-stand desks. I >> sent a scary article around about a year or so ago about the health issues, >> but I think it's mostly the availability of reasonable quality converters >> (Varidesk etc) that's really changed things. >> >> I'm between offices too much to have one myself (I'd need 3), so I just >> concentrate on having a regular walk around the office instead, and having >> all meetings as stand ups. And - on the insulin front - be sure to read up >> on (and cut down on) the sugar that's crammed into everything these days. >> >> On 19 Jun. 2017 14:33, "Tony Wright" wrote: >> >>> Hi Tom, >>> >>> No, not the back for me. They say that your body stops regulating >>> insulin properly after 4 hours of sitting, and that's about when I was >>> feeling unwell/lethargic from sitting. >>> >>> Regards, >>> Tony >>> >>> On Mon, Jun 19, 2017 at 4:00 PM, Tom Rutter wrote: >>> Did you switch to this for a specific reason (lower back problem for example)? If so did this help at all in a noticeable way? On Monday, 19 June 2017, Tony Wright wrote: > Hi Tom, > > It changes for me. Sometimes I stand, sometimes I sit. If I feel that > I've been sitting for too long, I hit the buttons and stand for a while. > I'm not regretful for one second that I have the option. > > Regards, > Tony > > On Mon, Jun 19, 2017 at 1:54 PM, Tom Rutter > wrote: > >> Hey folks >> >> I recall a while ag
[OT] Surface Studio
Hey all, What do we think of the Surface Studio? I've long said that it'll be great when we have that sort of touch interface directly in front of us to manipulate... My thinking is it'd be more powerful if instead of moving from the tilted position to vertical, if you had one or more screens vertically and had Kinect or similar to "drag" windows down to be manipulated directly in front of you, and then back up when just viewing. I can't imagine it'll be that far away.
Re: Entity Framework - the lay of the land
Document/Object DBs are kind of the solution for that, however, they still need to be managed. Grow a beard and become "full-stack". On Fri, Sep 16, 2016 at 11:56 AM, Greg Keogh wrote: > The people who think that ORMs are a good idea have a code-centric view of >> the world. >> > > Stored procs! > > Here's the black hole of the argument. To me it's a simple reductionist > problem... Clean-shaven code-centric people write real-world apps. Beardy > DBAs design databases. Coders need databases. Databases are unlikely to > have a structure that suits coders. What can bridge the "impedance" gap? > Something has to. > > It doesn't matter how much you like stored procs, you still have to get > stuff in and out of them across the gap to the coder's classes. How do > procs help? Are you proposing that more business logic be moved into procs? > If so, that way lies madness, as you can't easily integrate proc code into > source control, testing, versioning, builds, etc. I've seen whole apps > written in T-SQL, and it's quite frightening. > > Coders need ORMs, heavy or light. I've only used 2 this century, but they > simplify and reduce code by a staggering amount. The benefits far outweigh > the deficits. > > I was just discussing this with a colleague and we have made a casual pact > that in future apps we will try to meet more halfway across the "gap". He > will write procs that aggregate and deep load joined data for complex > queries, then my code on the ORM side will query what the proc assembles. > We haven't worked out the details of this plan, but I think it lets each > part do what they are best at. The procs will assemble and shred data out > of tables and the code will work with business objects. > > *Greg* >
Re: Entity Framework - the lay of the land
I haven't had significant problems that I haven't been able to fix. That said, I think they're marking the current as Stable and just working on features of EF Core now, which isn't as complete by far yet. On Fri, Sep 16, 2016 at 10:40 AM, Craig van Nieuwkerk wrote: > Using something like PetaPoco makes that pretty easy. > > > >> What about the classical problem of "impedance mismatch". You have to >> carefully maintain DataSets or similar and use DataAdapter to fill them, >> then writing data back is a circus trick with the ADO.NET classes. Then >> they invented ORMs, why did they do that!? -- *GK* >> > >
Re: DDD Melbourne
I think that's fairly cynical to think they were trying to exclude people - they were actually trying to stagger it so more people had a chance, if they were unable to even get to a computer on a single particular day, from what I understand. I hope to see you too - and keep an eye out - there's usually one or two folk who can't go at last minute who look to sell a ticket. At least it can work out for those in the area. On Mon, Jun 20, 2016 at 3:06 PM, David Burstin wrote: > Hi Andrew. > > Firstly, well done on securing tickets - I hope you have a fantastic day > (genuinely, not sarcastic). > > I hear what you are saying but I must respectfully disagree. Yes, each > room requires more management - but the jump from 2 rooms to 3 is bigger > than 3 to 4. If the aim is to disseminate knowledge and grow community then > these benefits should surely outweigh any management issues. Also, I would > suggest that the risk of attendees rushing one session is *reduced* with > more rooms. > > Clearly there was a conscious decision made to exclude a large number of > potential attendees, given the staggered approach to ticket sales. I have > yet to hear a good reason why that decision was made. Creating scarcity and > hype would be fine when selling an exclusive product, but is that really > what DDD is about? I have attended for many years and I never thought so. > > Anyway, enjoy the day and maybe I will see you there next year. > > > > On 20 June 2016 at 14:53, Andrew Tobin wrote: > >> It happens. We were lucky to get tickets this year, and it was only >> allowing 1 per login - but a few of my team got them. >> >> The thing is, I can understand the organisers not wanting to grow it >> bigger, and even if they had more rooms/more sessions - that would require >> more management, and the risk that 100% of the people try to rush one >> session that is full. >> >> The other issue is whether it would ruin the feel of the conference if it >> were to grow too big and unmanageable... and I get that view... >> >> It does say something though that each session sold out within the minute >> though and longtime attendees didn't even get a chance. >> >> On Mon, Jun 20, 2016 at 2:28 PM, Greg Keogh wrote: >> >>> Given how quickly tickets have gone each week, it is obvious that a >>>> sizeable proportion of the developer community in Melbourne has missed out. >>>> >>> >>> I also saw these posts and decided to not even bother to try and get a >>> ticket. I did plan to email the organisers and suggest they find a venue >>> proportional in size to the expected ticket demand to avoid this idiocy. If >>> by some fluke tickets do become available without the need for >>> embarrassed suffering, then I'd like to get one -- GK >>> >> >> >
Re: DDD Melbourne
It happens. We were lucky to get tickets this year, and it was only allowing 1 per login - but a few of my team got them. The thing is, I can understand the organisers not wanting to grow it bigger, and even if they had more rooms/more sessions - that would require more management, and the risk that 100% of the people try to rush one session that is full. The other issue is whether it would ruin the feel of the conference if it were to grow too big and unmanageable... and I get that view... It does say something though that each session sold out within the minute though and longtime attendees didn't even get a chance. On Mon, Jun 20, 2016 at 2:28 PM, Greg Keogh wrote: > Given how quickly tickets have gone each week, it is obvious that a >> sizeable proportion of the developer community in Melbourne has missed out. >> > > I also saw these posts and decided to not even bother to try and get a > ticket. I did plan to email the organisers and suggest they find a venue > proportional in size to the expected ticket demand to avoid this idiocy. If > by some fluke tickets do become available without the need for > embarrassed suffering, then I'd like to get one -- GK >
Re: [OT] Data Retention Bill
Obviously the solution is to make VPN technology illegal - can't have people circumventing it. On Fri, Mar 27, 2015 at 12:44 PM, Greg Keogh wrote: > So the bill has passed, opposed by everyone but the two major parties. Oh > well, it's not all bad, as at least it was nice to see Libs and Labs > finally cooperate on something, anything, and I'm sure they know what's > best for us! Now, I don't want my metadata, or anything tracked by the > government, and perhaps there are others in here who feel the same way. > Senator Ian McDonald said that if you have nothing to hide then you having > nothing to worry about, but I would urge him to go to Brandis' bookshelves > and look for a book on the 20th history of Europe to see if that changes > his mind. > > There are many overlapping questions here ... Exactly what is being > stored? Who stores the metadata where? How is the data protected? Who can > access it? How much will it cost the end users? > > I learned in a radio interview this week that metadata for offshore > webmail accounts like Gmail and Hotmail cannot be retained by Australian > ISPs (I hope this is true). I presume that Australian officials will have > to jump through unchanged hoops to request information about such accounts > and activity, and I hope these hoops are reasonably narrow. > > I'm not sure if they're planning to retain web browsing data, does anyone > know? If so, rumours have it that using a foreign VPN is the easiest way > around this. If so again, is it not too much of a burden to setup and run? > > I reckon it would be cheaper if the Australian government just paid for a > live data fed from Facebook and Google, as they already know more about us > than the NSA. > > TGIF, Greg >
Re: Azure and security trust
One alternative that I haven't looked into much at all, so take this with a grain of salt - is to have anything identifying on a local network, firewalled, and accessible via a site-to-site VPN connection to an Azure hosted server. Like I said, I haven't looked at what an implementation would take, but if you could create a firewalled, safe, tunnel to your data hosted on prem, and other data in the cloud - then it's an option? http://azure.microsoft.com/en-us/documentation/articles/virtual-networks-create-site-to-site-cross-premises-connectivity/ On Wed, Feb 25, 2015 at 2:28 PM, Greg Keogh wrote: > Folks, I have a demo SQL database in Azure and it's working nicely, but > now we have to consider how to get it into production use. My demo DB > doesn't contain any real names and addresses, but the live DB will have > information about hospital patients, and you can imagine how confidential > that is! I'm told they will demand the DB be stored on hospital managed > servers, which is a damn nuisance in reality as I'm sure many of you know > how tedious it can be trying to break through walls of bureaucracy around > IT departments in places like hospitals and the government. > > This opens up the whole issues of "trust and the cloud". Since the Snowden > revelations, I don't know how anyone with confidential data can trust cloud > storage. Even I don't trust it and all of my backups in Rackspace and Azure > blobs are pkzipc AES encrypted. So how on earth could a hospital be > convinced that cloud store is an attractive option? > > I just remembered that Amazon has a special area that is certified secure > so they can get government contracts. I haven't seen anything like that in > Azure. Despite that, it doesn't make me feel much better, as we now know > the NSA was intercepting hardware and bugging it, and coercing huge telcos > to put splitters in the backbones, and using secret FISA orders to threaten > other even huger companies to secretly hand over their records. So who the > hell can trust anyone in the cloud?! > > Is anyone dealing in this sort of cloud/trust business at the moment? > What's the state of play? is there any hope? Am I just paranoid? (who's > monitoring this email?) > > *Greg K* >
Re: DDD Melbourne Ticket
Hey Michael, I'd be happy to take those off your hands - contact me off list? On Wed, Jul 16, 2014 at 2:19 PM, Michael Ridland wrote: > Hi All > > My friend has decided to have his 30th birthday on the day of DDD. > > So I have two tickets for DDD if anyone wants them maybe 2 for $40? > > Thanks > > >
Recommendations for a ERP system
Hey all, We're currently looking at changing our Warehousing/Inventory/Financials package and I was just wondering if anyone had some experience with a good product they would recommend? We're looking at something that is current technology, would be not impossible to interface with, and has a local support presence. Anyone come across a good package they're happy to recommend to others? Andrew
Re: Access Database Replication
I've got an app I've got to stick with an Access front-end that has > 50 people on it. It depends on what you want to achieve, not that I'm saying it's good, and that replacing it with a .NET equivalent wouldn't be better, but it does what it needs to. > > Whenever you try scaling Access, it becomes ugly and fragile. But it's > nice for a single user app. > > >> Best of luck >> Martin >> >> -Original Message- >> From: ozdotnet-boun...@ozdotnet.com [mailto:ozdotnet-boun...@ozdotnet.com] >> On Behalf Of Les Hughes >> Sent: Tuesday, 7 September 2010 12:33 PM >> To: ozDotNet >> Subject: Access Database Replication >> >> >> Hi All, >> >> I've got a legacy MSAccess app in VBA which is been used at two separate >> office locations (Melbourne & Singapore), with two separate copies of >> the database. >> >> There is a table with 5,000 rows in it (about 30 columns) which has >> inserts/updates at both offices, and we were looking for some way to >> propagate the changes. >> >> Ideally, we would either migrate the app to .NET/SQL Server (which would >> be fairly expensive... management say 'grrr'), or use RDP and having the >> app on one computer. (which is also for some reason is also 'g') >> >> Based of my knowledge of access/mdb's, there are no triggers, meaning >> the only way we could really track updates is to modify the Access forms >> to update a flag column with a datetime or something similar, and then >> have a batch process move updates every now and then. >> >> This idea seems dirty to me because if someone changes with the tables >> directly, or there are updates around the same time it becomes quite >> messy in keeping a clean dataset/dealing with race conditions/etc. >> >> Long story short: >> >> Has anyone dealt with something similar to this before? Any ideas would >> be appreciated. >> >> Thanks heaps, >> -- >> Les Hughes >> l...@datarev.com.au >> No virus found in this incoming message. >> Checked by AVG - www.avg.com >> Version: 9.0.851 / Virus Database: 271.1.1/3117 - Release Date: 09/06/10 >> 16:35:00 >> >> > > > -- > Meski > > "Going to Starbucks for coffee is like going to prison for sex. Sure, > you'll get it, but it's going to be rough" - Adam Hills >
Re: Starting out with Entity Framework
You could map the string for the months to a string in your class, and on the setter, do the logic to put it into the list, and on the getter make it back into the correctly ordered string? On Mon, May 31, 2010 at 2:02 PM, James Chapman-Smith wrote: > Hi Clint, > > > > You can easily use partial classes to add the list of months member to the > generated classes from entity framework. Will that do what you need? > > > > Cheers. > > > > James. > > > > *From:* ozdotnet-boun...@ozdotnet.com [mailto: > ozdotnet-boun...@ozdotnet.com] *On Behalf Of *Clint Colefax > *Sent:* Monday, 31 May 2010 12:00 > > *To:* ozdotnet@ozdotnet.com > *Subject:* Starting out with Entity Framework > > > > Hi all, > > > > I’m starting out with Entity Framework (.NET 3.5) and am curious if it can > handle the following scenario and how to go about implementing it. > > > > Say I have single table like so > > > > TableName: Languages > > Columns: Name > > TranslatedMonths > > > > An example row would be > > Name=English > > TranslatedMonths =“Jan, Feb, Mar….” > > > > In the entities layer, I want this represented as to types of classes, > > > > ClassName:Languages > > Memebers:Name > > List(Of Months) > > > > Can I do this with EF? Where do I start. I’ve played a bit with this > scenario but can’t seem to get it to where I want. I’ve done a bit with > Partial Classes for this so far, but I’m just not happy with trying to keep > the String property created by ef in sync with my custom added list > property. For this I’d like to remove the property from the entity in the > designer, but not sure how then to have that database value loaded so I can > parse it out into my list. > > > > Thanks > > [image: Clint_C] > > >
